From owner-freebsd-hackers@freebsd.org Mon Nov 21 09:02:40 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 90C6FC4DDA3 for ; Mon, 21 Nov 2016 09:02:40 +0000 (UTC) (envelope-from Olivier.Nicole@cs.ait.ac.th) Received: from mail.cs.ait.ac.th (mail.cs.ait.ac.th [192.41.170.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2591A1804 for ; Mon, 21 Nov 2016 09:02:39 +0000 (UTC) (envelope-from Olivier.Nicole@cs.ait.ac.th) Received: from mail.cs.ait.ac.th (localhost [127.0.0.1]) by mail.cs.ait.ac.th (Postfix) with ESMTP id 5ED27D7885 for ; Mon, 21 Nov 2016 16:02:29 +0700 (ICT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cs.ait.ac.th; h= content-type:content-type:mime-version:message-id:date:date :subject:subject:from:from:received:received:received; s= selector1; t=1479718948; x=1481533349; bh=PFRlL181NPWGbIgr3EUHSr sIR5Px9jNEegiZlq0xGYg=; b=bYjfjaGEXnUY1o485YPPQf/RFJjL65QkHTVG/g fItYsU3pQrITwXOZrbIlHYJiv0GQgfiHoX0oszrBIA9D/Bhm5jDlNdIfonG6yFfa y63g2qvoOBpMpeQXRbkWFFunCDQ0aae4E637snOlM6ONeoFqDCQQBBXcRhZREmJo Zo8TE= X-Virus-Scanned: amavisd-new at cs.ait.ac.th Received: from mail.cs.ait.ac.th ([127.0.0.1]) by mail.cs.ait.ac.th (mail.cs.ait.ac.th [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id DvTL2gwGvJrj for ; Mon, 21 Nov 2016 16:02:28 +0700 (ICT) Received: from banyan.cs.ait.ac.th (banyan.cs.ait.ac.th [192.41.170.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.cs.ait.ac.th (Postfix) with ESMTPS id 45AA8D7884 for ; Mon, 21 Nov 2016 16:02:28 +0700 (ICT) Received: (from on@localhost) by banyan.cs.ait.ac.th (8.15.2/8.15.2/Submit) id uAL92RsZ038367; Mon, 21 Nov 2016 16:02:27 +0700 (ICT) (envelope-from on@banyan.cs.ait.ac.th) From: Olivier To: freebsd-hackers@freebsd.org Subject: New behavious in /dev/consolectl Date: Mon, 21 Nov 2016 16:02:27 +0700 Message-ID: MIME-Version: 1.0 Content-Type: text/plain X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 09:02:40 -0000 Hi, I am not sure this is the right list to ask the question, but I have the following problem that is bugging me. I have a simple program that read the middle button of the mouse at start-up and reply with (close/pressed or open/released) (see below). In previous versions of FreeBSD (up to 9 I think), if the middle button was pressed down when I launched the program, it was detected, and I could launch and relaunch the program, it was always replying with "close". Now (FreeBSD 10), the close is only detected on the first run. The second run will reply "open", even if I keep the button pressed all the time between the 2 runs; like if by reading the button, I was erasing the memory. Please consider the following examples: The first run is just to show what is happening to answer the mouse button action. Run 2 is started with the button pressed and the output is correct. After run 2 is aborted and run 3 is launched, the ouput is incorrect. If I cycle the mouse button between run 3 and 4, the output is then correct. Run # mouse button output release 1 start program open open press closed & hold closed hold closed release open open open CTRL-C press & hold 2 start program hold hold closed hold closed hold closed CTRL-C hold hold 3 start program hold open hold open hold open hold open CTRL-C hold release press & hold 4 start program hold closed hold closed hold closed hold closed CTRL-C hold I am pretty positive it was not an issue on previous versions of FreeBSD, as I have been relying on this for a long time because all my system is built around this feature (mouse button is wired to a switch that detects if a door if open or closed). What has been changed in the recent versions of FreeBSD? Thanks in advance, Olivier #include #include #include #include #include int main (int argc, char ** argv, char** env) { int fd_mouse; struct mouse_info mouse_info; fd_mouse=open("/dev/consolectl", O_RDWR); while (1) { mouse_info.operation=MOUSE_GETINFO; ioctl(fd_mouse, CONS_MOUSECTL, &mouse_info); if (((mouse_info.u.data.buttons&(int)2)==2)) { printf("closed\n"); } else { printf("open\n"); } sleep(1); } } From owner-freebsd-hackers@freebsd.org Mon Nov 21 09:11:53 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 465DBC4B035 for ; Mon, 21 Nov 2016 09:11:53 +0000 (UTC) (envelope-from athanasioskostopoulos@gmail.com) Received: from mail-wm0-x233.google.com (mail-wm0-x233.google.com [IPv6:2a00:1450:400c:c09::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CF9EF1B1F for ; Mon, 21 Nov 2016 09:11:52 +0000 (UTC) (envelope-from athanasioskostopoulos@gmail.com) Received: by mail-wm0-x233.google.com with SMTP id u144so19511665wmu.1 for ; Mon, 21 Nov 2016 01:11:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=5OkAU5mfOWbcyrB4b4QQe+N4LRlgkHMcqxOY/Bg78Nw=; b=vh0epJMLr22PDH337xX/t1Mz7Lgtt+mz9HnnVh2RAt/qYhprRoVMQ+TxK+S6BbdOd5 Ypyy9UVtFyejVHW2ImRktLVGFBAtfN/VUVei8oknEMOUW9f3G3Vdfp9hT0bQCzjRV/dB 6wzrp06J0li+vuQ0YJzZBZFESlPXrkOuiB0FzEGIRQf2qAXHNyL6BgW7/+V7PdwU6Yvg 0aoCun12JHxdiSHVbF6QCBEuoc/4i7ZivebQ/TvFmcwoBaETBCDZiqUGoZLMgehICNTy vEMRfO35kuNik/p9SMuFaQPRQdrIy5AkEp2xq9byjf1cv+bAq99pqLDRgtKoDqCr8d1e HZzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=5OkAU5mfOWbcyrB4b4QQe+N4LRlgkHMcqxOY/Bg78Nw=; b=Q+MMkPLYs+erRA3X+YMr2GESmkFp0754tFI1s5NeTZks5yxlgcqgwsg24P0R6F/RX+ cXHaSY1/XTBMJ5ZdGBh31bbkt/LhGQzGXnVQc9UFOpyzbo3BgVjQJQ8BMtWIWu/2cHwW Ig40twEDESlhC8LeAhUaj6aQn6EjmX8ep6gfAhtWTeNbljWVjDHHLCYTeGXp/EdhvJdQ SJkeNspeKDbiaJpyoqds1ugtBNGTyULRh+AYonoAAp1FqTJDO5pgIcVsXSZpqXjOEaiV L5sz6yH7E5rS+PfqI3Gdl1wz+i4rTa7x/TMd7pTv3jtRHZFNOyleJUCv4XSTCoIqPyRf qJNg== X-Gm-Message-State: AKaTC03dbPJ7TgTqiBkxXrcnargQnVSPxlK/5iPrR19rK7+BHe3SIFS62RRhPhcWauRCIqetsuqT9ZXGrtnb1A== X-Received: by 10.28.107.77 with SMTP id g74mr12205298wmc.109.1479719510301; Mon, 21 Nov 2016 01:11:50 -0800 (PST) MIME-Version: 1.0 Received: by 10.194.114.69 with HTTP; Mon, 21 Nov 2016 01:11:49 -0800 (PST) From: Athanasios Kostopoulos Date: Mon, 21 Nov 2016 10:11:49 +0100 Message-ID: Subject: Kernel fuzzing suite - any technical point of contact? To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 09:11:53 -0000 Hi all, "long time listener, first time caller". Is there a technical point of contact for the aforementioned project, as I want to discuss some ideas. Thanks and Best Regards, Athanasios From owner-freebsd-hackers@freebsd.org Mon Nov 21 13:45:11 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A1B1C4C223 for ; Mon, 21 Nov 2016 13:45:11 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from mail.SerNet.de (mail.SerNet.de [193.175.80.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44714274 for ; Mon, 21 Nov 2016 13:45:10 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from intern.SerNet.DE by mail.SerNet.DE with esmtps (Exim 4.84_2 #1) for freebsd-hackers@freebsd.org id 1c8oks-0004g8-Cs; Mon, 21 Nov 2016 14:35:30 +0100 Received: by intern.sernet.de id 1c8oks-0004Gl-AH; Mon, 21 Nov 2016 14:35:30 +0100 Date: Mon, 21 Nov 2016 14:35:28 +0100 From: Volker Lendecke To: freebsd-hackers@freebsd.org Subject: process shared mutexes? Message-ID: <20161121133528.GA30947@sernet.de> Reply-To: Volker.Lendecke@SerNet.DE MIME-Version: 1.0 User-Agent: Mutt/1.5.23 (2014-03-12) Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 13:45:11 -0000 Hello! For Samba's tdb I'm trying to get process shared robust mutexes to work. However, tdb has a usage pattern that seems to confuse FreeBSD 11 (32-bit x86 if that matters). The attached program fails in the final pthread_mutex_lock call. If I comment out the call to ptr = mmap(NULL, 0xb0, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x1000); it works. FWIW, tdb uses robust shared mutexes on Linux successfully for a while now. I haven't tried Solaris yet, the only other platform I know about that has them. What am I doing wrong? Thanks, Volker From owner-freebsd-hackers@freebsd.org Mon Nov 21 13:50:43 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 620DBC4C4A8 for ; Mon, 21 Nov 2016 13:50:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E78F5911 for ; Mon, 21 Nov 2016 13:50:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uALDoaNc062398 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 21 Nov 2016 15:50:37 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uALDoaNc062398 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uALDoaMY062397; Mon, 21 Nov 2016 15:50:36 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 21 Nov 2016 15:50:36 +0200 From: Konstantin Belousov To: Volker Lendecke Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121135036.GY54029@kib.kiev.ua> References: <20161121133528.GA30947@sernet.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121133528.GA30947@sernet.de> User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 13:50:43 -0000 On Mon, Nov 21, 2016 at 02:35:28PM +0100, Volker Lendecke wrote: > Hello! > > For Samba's tdb I'm trying to get process shared robust mutexes to > work. However, tdb has a usage pattern that seems to confuse FreeBSD > 11 (32-bit x86 if that matters). > > The attached program fails in the final pthread_mutex_lock call. If I > comment out the call to > > ptr = mmap(NULL, 0xb0, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x1000); > > it works. > > FWIW, tdb uses robust shared mutexes on Linux successfully for a while > now. I haven't tried Solaris yet, the only other platform I know about > that has them. > > What am I doing wrong? > > Thanks, > There is no attached program, please mail it either inline or put it somewhere on web. From owner-freebsd-hackers@freebsd.org Mon Nov 21 14:16:22 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C1FE9C4C115 for ; Mon, 21 Nov 2016 14:16:22 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from mail.SerNet.de (mail1.SerNet.de [193.175.80.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8C5B71DB7 for ; Mon, 21 Nov 2016 14:16:21 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from intern.SerNet.DE by mail.SerNet.DE with esmtps (Exim 4.84_2 #1) id 1c8pOM-0001KM-Mq; Mon, 21 Nov 2016 15:16:18 +0100 Received: by intern.sernet.de id 1c8pOM-0004gu-K9; Mon, 21 Nov 2016 15:16:18 +0100 Date: Mon, 21 Nov 2016 15:16:16 +0100 From: Volker Lendecke To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121141616.GB30947@sernet.de> Reply-To: Volker.Lendecke@SerNet.DE References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121135036.GY54029@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 14:16:22 -0000 On Mon, Nov 21, 2016 at 03:50:36PM +0200, Konstantin Belousov wrote: > On Mon, Nov 21, 2016 at 02:35:28PM +0100, Volker Lendecke wrote: > > Hello! > > > > For Samba's tdb I'm trying to get process shared robust mutexes to > > work. However, tdb has a usage pattern that seems to confuse FreeBSD > > 11 (32-bit x86 if that matters). > > > > The attached program fails in the final pthread_mutex_lock call. If I > > comment out the call to > > > > ptr = mmap(NULL, 0xb0, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x1000); > > > > it works. > > > > FWIW, tdb uses robust shared mutexes on Linux successfully for a while > > now. I haven't tried Solaris yet, the only other platform I know about > > that has them. > > > > What am I doing wrong? > > > > Thanks, > > > > There is no attached program, please mail it either inline or put it > somewhere on web. Hmm. Inline now. Volker #include #include #include #include #include #include #include #include #include int main(int argc, const char *argv[]) { int fd, ret; void *ptr; pthread_mutex_t *m; pthread_mutexattr_t attr; if (argc != 2) { fprintf(stderr, "usage: %s \n", argv[0]); return 1; } fd = open(argv[1], O_RDWR|O_CREAT, 0644); if (fd == -1) { perror("open failed"); return 1; } ret = ftruncate(fd, 0x1000+0xb0); if (ret == -1) { perror("ftruncate failed"); return 1; } m = mmap(NULL, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); if (m == MAP_FAILED) { perror("mmap failed"); return 1; } ret = pthread_mutexattr_init(&attr); if (ret != 0) { fprintf(stderr, "pthread_mutexattr_init failed: %s\n", strerror(ret)); return 1; } ret = pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED); if (ret != 0) { fprintf(stderr, "pthread_mutexattr_setpshared failed: %s\n", strerror(ret)); return 1; } ret = pthread_mutex_init(m, &attr); if (ret != 0) { fprintf(stderr, "pthread_mutex_init failed: %s\n", strerror(ret)); return 1; } ret = munmap(m, 0x1000); if (ret == -1) { perror("munmap failed"); return 1; } #if 1 ptr = mmap(NULL, 0xb0, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x1000); if (ptr == MAP_FAILED) { perror("mmap failed"); return 1; } #endif m = mmap(NULL, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); if (m == MAP_FAILED) { perror("mmap failed"); return 1; } ret = pthread_mutex_lock(m); if (ret != 0) { fprintf(stderr, "pthread_mutex_lock failed: %s\n", strerror(ret)); return 1; } ret = pthread_mutex_unlock(m); if (ret != 0) { fprintf(stderr, "pthread_mutex_lock failed: %s\n", strerror(ret)); return 1; } return 0; } From owner-freebsd-hackers@freebsd.org Mon Nov 21 15:10:48 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57C47C4D263 for ; Mon, 21 Nov 2016 15:10:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F2328F90 for ; Mon, 21 Nov 2016 15:10:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uALFAfTA082317 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 21 Nov 2016 17:10:41 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uALFAfTA082317 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uALFAff6082316; Mon, 21 Nov 2016 17:10:41 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 21 Nov 2016 17:10:40 +0200 From: Konstantin Belousov To: Volker Lendecke Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121151040.GA54029@kib.kiev.ua> References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121141616.GB30947@sernet.de> User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 15:10:48 -0000 On Mon, Nov 21, 2016 at 03:16:16PM +0100, Volker Lendecke wrote: > On Mon, Nov 21, 2016 at 03:50:36PM +0200, Konstantin Belousov wrote: > > On Mon, Nov 21, 2016 at 02:35:28PM +0100, Volker Lendecke wrote: > > > Hello! > > > > > > For Samba's tdb I'm trying to get process shared robust mutexes to > > > work. However, tdb has a usage pattern that seems to confuse FreeBSD > > > 11 (32-bit x86 if that matters). > > > > > > The attached program fails in the final pthread_mutex_lock call. If I > > > comment out the call to > > > > > > ptr = mmap(NULL, 0xb0, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x1000); > > > > > > it works. > > > > > > FWIW, tdb uses robust shared mutexes on Linux successfully for a while > > > now. I haven't tried Solaris yet, the only other platform I know about > > > that has them. > > > > > > What am I doing wrong? Please see the libthr(3) man page, in particular, read the RUN-TIME SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent sysctl. Does setting the sysctl to 1 allow your program to run ? > > > > > > Thanks, > > > > > > > There is no attached program, please mail it either inline or put it > > somewhere on web. > > Hmm. Inline now. > > Volker > > > #include > #include > #include > #include > #include > #include > #include > #include > #include > > int main(int argc, const char *argv[]) > { > int fd, ret; > void *ptr; > pthread_mutex_t *m; > pthread_mutexattr_t attr; > > if (argc != 2) { > fprintf(stderr, "usage: %s \n", argv[0]); > return 1; > } > > fd = open(argv[1], O_RDWR|O_CREAT, 0644); > if (fd == -1) { > perror("open failed"); > return 1; > } > > ret = ftruncate(fd, 0x1000+0xb0); > if (ret == -1) { > perror("ftruncate failed"); > return 1; > } > > m = mmap(NULL, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); > if (m == MAP_FAILED) { > perror("mmap failed"); > return 1; > } > > ret = pthread_mutexattr_init(&attr); > if (ret != 0) { > fprintf(stderr, "pthread_mutexattr_init failed: %s\n", > strerror(ret)); > return 1; > } > > ret = pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED); > if (ret != 0) { > fprintf(stderr, "pthread_mutexattr_setpshared failed: %s\n", > strerror(ret)); > return 1; > } > > ret = pthread_mutex_init(m, &attr); > if (ret != 0) { > fprintf(stderr, "pthread_mutex_init failed: %s\n", > strerror(ret)); > return 1; > } > > ret = munmap(m, 0x1000); > if (ret == -1) { > perror("munmap failed"); > return 1; > } > > #if 1 > ptr = mmap(NULL, 0xb0, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0x1000); > if (ptr == MAP_FAILED) { > perror("mmap failed"); > return 1; > } > #endif > > m = mmap(NULL, 0x1000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); > if (m == MAP_FAILED) { > perror("mmap failed"); > return 1; > } > > ret = pthread_mutex_lock(m); > if (ret != 0) { > fprintf(stderr, "pthread_mutex_lock failed: %s\n", > strerror(ret)); > return 1; > } > > ret = pthread_mutex_unlock(m); > if (ret != 0) { > fprintf(stderr, "pthread_mutex_lock failed: %s\n", > strerror(ret)); > return 1; > } > > return 0; > } From owner-freebsd-hackers@freebsd.org Mon Nov 21 15:26:10 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 54F78C4D83B for ; Mon, 21 Nov 2016 15:26:10 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from mail.SerNet.de (mail.SerNet.de [193.175.80.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 20B1419C5 for ; Mon, 21 Nov 2016 15:26:09 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from intern.SerNet.DE by mail.SerNet.DE with esmtps (Exim 4.84_2 #1) id 1c8qTu-00019X-Ij; Mon, 21 Nov 2016 16:26:06 +0100 Received: by intern.sernet.de id 1c8qTu-0005Qp-Fz; Mon, 21 Nov 2016 16:26:06 +0100 Date: Mon, 21 Nov 2016 16:25:42 +0100 From: Volker Lendecke To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121152542.GA31733@sernet.de> Reply-To: Volker.Lendecke@SerNet.DE References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> <20161121151040.GA54029@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121151040.GA54029@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 15:26:10 -0000 On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote: > Please see the libthr(3) man page, in particular, read the RUN-TIME > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent > sysctl. > > Does setting the sysctl to 1 allow your program to run ? Yes, that does make it work. The description says that the umtx vnode is dropped on the last munmap. If I #if 0 the middle mmap, it works, although there is no mmap around anymore. So the description is not 100% accurate I'd say. When does the recycling happen exactly? Can it break running applications? And -- how can I make sure for Samba that this is set properly at runtime? We already have a runtime mutex test for some ancient Linux kernels that were broken. We could add this as a subtest too. But -- what happens if the admin resets this while Samba is running? Does the kernel make sure that existing files still get the correct behaviour when the sysctl changes? Thanks! Volker From owner-freebsd-hackers@freebsd.org Mon Nov 21 15:39:51 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51FD6C4DB65 for ; Mon, 21 Nov 2016 15:39:51 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from mail.SerNet.de (mail.SerNet.de [193.175.80.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1BDE624D for ; Mon, 21 Nov 2016 15:39:50 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from intern.SerNet.DE by mail.SerNet.DE with esmtps (Exim 4.84_2 #1) id 1c8qh9-0002W1-FK; Mon, 21 Nov 2016 16:39:47 +0100 Received: by intern.sernet.de id 1c8qh9-0005Ze-Cc; Mon, 21 Nov 2016 16:39:47 +0100 Date: Mon, 21 Nov 2016 16:39:45 +0100 From: Volker Lendecke To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121153945.GB31733@sernet.de> Reply-To: Volker.Lendecke@SerNet.DE References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> <20161121151040.GA54029@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121151040.GA54029@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 15:39:51 -0000 On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote: > Please see the libthr(3) man page, in particular, read the RUN-TIME > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent > sysctl. One more question: man libtrh(3) says that this behaviour is allowed by Posix. Do you have a reference that is available online? Or do I have to buy the Posix standards documents? Thanks, Volker From owner-freebsd-hackers@freebsd.org Mon Nov 21 15:58:36 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01C74C4C4AE for ; Mon, 21 Nov 2016 15:58:36 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9DA8915F3 for ; Mon, 21 Nov 2016 15:58:35 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uALFwPpa094099 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 21 Nov 2016 17:58:25 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uALFwPpa094099 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uALFwNdP094097; Mon, 21 Nov 2016 17:58:23 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 21 Nov 2016 17:58:23 +0200 From: Konstantin Belousov To: Volker Lendecke Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121155823.GB54029@kib.kiev.ua> References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> <20161121151040.GA54029@kib.kiev.ua> <20161121152542.GA31733@sernet.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121152542.GA31733@sernet.de> User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 15:58:36 -0000 On Mon, Nov 21, 2016 at 04:25:42PM +0100, Volker Lendecke wrote: > On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote: > > Please see the libthr(3) man page, in particular, read the RUN-TIME > > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent > > sysctl. > > > > Does setting the sysctl to 1 allow your program to run ? > > Yes, that does make it work. The description says that the umtx vnode > is dropped on the last munmap. If I #if 0 the middle mmap, it works, > although there is no mmap around anymore. So the description is not > 100% accurate I'd say. No, not umtx vnode is dropped, but the shared umutexes attached to the file' page. What you observed is the consequence of an implementation detail. It is impossible to execute umtx cleanup while dereferencing vm object, due to locking issues. An asynchronous task is scheduled to perform the cleanup. But when the task is run, it is quite possible that your process is already executed second mmap() and pthread_mutex_lock(), creating another reference on the umtx data. The mmap(NULL, 0xb, ...) delays execution of that part of the program, so the task wins more reliably. This initially puzzled me as well, since I was not able to observe your reported behaviour on the amd64 host, either on 64 or 32bit binary. > > When does the recycling happen exactly? Can it break running > applications? See above. I cannot answer the second question. > > And -- how can I make sure for Samba that this is set properly at > runtime? We already have a runtime mutex test for some ancient Linux > kernels that were broken. We could add this as a subtest too. But -- > what happens if the admin resets this while Samba is running? Does > the kernel make sure that existing files still get the correct > behaviour when the sysctl changes? It is reasonable to expect that administrator set the knob very early during the boot, and not twiddle it at runtime, at least if explicit instructions to do so are provided. Samba may, additionally, check the sysctl value during the initialization and provide a hint to user if needed. On Mon, Nov 21, 2016 at 04:39:45PM +0100, Volker Lendecke wrote: > On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote: > > Please see the libthr(3) man page, in particular, read the RUN-TIME > > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent > > sysctl. > > One more question: man libtrh(3) says that this behaviour is allowed > by Posix. Do you have a reference that is available online? Or do I > have to buy the Posix standards documents? Look at the Single Unix Specification, particularly to the following paragraph in the mmap() description: The state of synchronization objects such as mutexes, semaphores, barriers, and conditional variables placed in shared memory mapped with MAP_SHARED becomes undefined when the last region in any process containing the synchronization object is unmapped. The reference is available at http://pubs.opengroup.org/onlinepubs/9699919799/functions/mmap.html but you might need to register before getting the online access or downloading the entire SUSv4tc2 in archive. From owner-freebsd-hackers@freebsd.org Mon Nov 21 16:15:00 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 782B7C4CE5E for ; Mon, 21 Nov 2016 16:15:00 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from mail.SerNet.de (mail1.SerNet.de [193.175.80.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 41873331 for ; Mon, 21 Nov 2016 16:14:59 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from intern.SerNet.DE by mail.SerNet.DE with esmtps (Exim 4.84_2 #1) id 1c8rFA-0006D1-KK; Mon, 21 Nov 2016 17:14:56 +0100 Received: by intern.sernet.de id 1c8rFA-0005ts-Hs; Mon, 21 Nov 2016 17:14:56 +0100 Date: Mon, 21 Nov 2016 17:14:54 +0100 From: Volker Lendecke To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121161454.GA32128@sernet.de> Reply-To: Volker.Lendecke@SerNet.DE References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> <20161121151040.GA54029@kib.kiev.ua> <20161121152542.GA31733@sernet.de> <20161121155823.GB54029@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121155823.GB54029@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 16:15:00 -0000 On Mon, Nov 21, 2016 at 05:58:23PM +0200, Konstantin Belousov wrote: > On Mon, Nov 21, 2016 at 04:25:42PM +0100, Volker Lendecke wrote: > > On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote: > > > Please see the libthr(3) man page, in particular, read the RUN-TIME > > > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent > > > sysctl. > > > > > > Does setting the sysctl to 1 allow your program to run ? > > > > Yes, that does make it work. The description says that the umtx vnode > > is dropped on the last munmap. If I #if 0 the middle mmap, it works, > > although there is no mmap around anymore. So the description is not > > 100% accurate I'd say. > No, not umtx vnode is dropped, but the shared umutexes attached to the > file' page. > > What you observed is the consequence of an implementation detail. It is > impossible to execute umtx cleanup while dereferencing vm object, due to > locking issues. An asynchronous task is scheduled to perform the cleanup. > But when the task is run, it is quite possible that your process is already > executed second mmap() and pthread_mutex_lock(), creating another reference > on the umtx data. Hmm. If I do a poll(NULL, 0, 60000) between the munmap and mmap without the intervening mmap, it still works. It's really the mmap(NULL,0xb0...) that kills it. > Look at the Single Unix Specification, particularly to the following > paragraph in the mmap() description: > > The state of synchronization objects such as mutexes, semaphores, > barriers, and conditional variables placed in shared memory mapped > with MAP_SHARED becomes undefined when the last region in any process > containing the synchronization object is unmapped. Thanks! Hidden deep in mmap(2)... No hint in any of the pthread calls. So -- all of the above discussion becomes irrelevant if I change tdb such that it keeps the mutex area mmappe'd at least once? Then no GC will kick in regardless of the sysctl? This would be possible, because we use mutexes on so-called CLEAR_IF_FIRST databases only. When the last process closes the db, it will be wiped on the next open. Volker From owner-freebsd-hackers@freebsd.org Mon Nov 21 16:41:17 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57774C4D894 for ; Mon, 21 Nov 2016 16:41:17 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F3453165D for ; Mon, 21 Nov 2016 16:41:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id uALGf9GT005122 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Mon, 21 Nov 2016 18:41:09 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua uALGf9GT005122 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id uALGf9XM005121; Mon, 21 Nov 2016 18:41:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 21 Nov 2016 18:41:09 +0200 From: Konstantin Belousov To: Volker Lendecke Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121164109.GC54029@kib.kiev.ua> References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> <20161121151040.GA54029@kib.kiev.ua> <20161121152542.GA31733@sernet.de> <20161121155823.GB54029@kib.kiev.ua> <20161121161454.GA32128@sernet.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121161454.GA32128@sernet.de> User-Agent: Mutt/1.7.1 (2016-10-04) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 16:41:17 -0000 On Mon, Nov 21, 2016 at 05:14:54PM +0100, Volker Lendecke wrote: > On Mon, Nov 21, 2016 at 05:58:23PM +0200, Konstantin Belousov wrote: > > On Mon, Nov 21, 2016 at 04:25:42PM +0100, Volker Lendecke wrote: > > > On Mon, Nov 21, 2016 at 05:10:40PM +0200, Konstantin Belousov wrote: > > > > Please see the libthr(3) man page, in particular, read the RUN-TIME > > > > SETTINGS section, the description of the kern.ipc.umtx_vnode_persistent > > > > sysctl. > > > > > > > > Does setting the sysctl to 1 allow your program to run ? > > > > > > Yes, that does make it work. The description says that the umtx vnode > > > is dropped on the last munmap. If I #if 0 the middle mmap, it works, > > > although there is no mmap around anymore. So the description is not > > > 100% accurate I'd say. > > No, not umtx vnode is dropped, but the shared umutexes attached to the > > file' page. > > > > What you observed is the consequence of an implementation detail. It is > > impossible to execute umtx cleanup while dereferencing vm object, due to > > locking issues. An asynchronous task is scheduled to perform the cleanup. > > But when the task is run, it is quite possible that your process is already > > executed second mmap() and pthread_mutex_lock(), creating another reference > > on the umtx data. > > Hmm. If I do a poll(NULL, 0, 60000) between the munmap and mmap > without the intervening mmap, it still works. It's really the > mmap(NULL,0xb0...) that kills it. Yes, because at that time, the cleanup task already completed. So the mutex is auto-reinited with default attributes, but as shared. When the cleanup is pending, the mutex off-page data is marked for pending removal, and lookup of that data in pthread_mutex_lock() returns an error. This is unfortunate consequence of the initially limiting ABI which we are trying to preserve still. > > > Look at the Single Unix Specification, particularly to the following > > paragraph in the mmap() description: > > > > The state of synchronization objects such as mutexes, semaphores, > > barriers, and conditional variables placed in shared memory mapped > > with MAP_SHARED becomes undefined when the last region in any process > > containing the synchronization object is unmapped. > > Thanks! Hidden deep in mmap(2)... No hint in any of the pthread calls. > > So -- all of the above discussion becomes irrelevant if I change tdb > such that it keeps the mutex area mmappe'd at least once? Then no GC > will kick in regardless of the sysctl? This would be possible, because > we use mutexes on so-called CLEAR_IF_FIRST databases only. When the last > process closes the db, it will be wiped on the next open. If the file is mmaped, then yes, the mutex must be not destroyed. If it is, then there is a bug in the current implementation. From owner-freebsd-hackers@freebsd.org Mon Nov 21 20:23:32 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6528AC4D86E for ; Mon, 21 Nov 2016 20:23:32 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from mail.SerNet.de (mail1.SerNet.de [193.175.80.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2E261D38 for ; Mon, 21 Nov 2016 20:23:31 +0000 (UTC) (envelope-from Volker.Lendecke@SerNet.DE) Received: from intern.SerNet.DE by mail.SerNet.DE with esmtps (Exim 4.84_2 #1) id 1c8v7g-0004oD-Ed; Mon, 21 Nov 2016 21:23:28 +0100 Received: by intern.sernet.de id 1c8v7g-0000d0-CH; Mon, 21 Nov 2016 21:23:28 +0100 Date: Mon, 21 Nov 2016 21:23:25 +0100 From: Volker Lendecke To: Konstantin Belousov Cc: freebsd-hackers@freebsd.org Subject: Re: process shared mutexes? Message-ID: <20161121202325.GA32463@sernet.de> Reply-To: Volker.Lendecke@SerNet.DE References: <20161121133528.GA30947@sernet.de> <20161121135036.GY54029@kib.kiev.ua> <20161121141616.GB30947@sernet.de> <20161121151040.GA54029@kib.kiev.ua> <20161121152542.GA31733@sernet.de> <20161121155823.GB54029@kib.kiev.ua> <20161121161454.GA32128@sernet.de> <20161121164109.GC54029@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20161121164109.GC54029@kib.kiev.ua> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Nov 2016 20:23:32 -0000 On Mon, Nov 21, 2016 at 06:41:09PM +0200, Konstantin Belousov wrote: > > So -- all of the above discussion becomes irrelevant if I change tdb > > such that it keeps the mutex area mmappe'd at least once? Then no GC > > will kick in regardless of the sysctl? This would be possible, because > > we use mutexes on so-called CLEAR_IF_FIRST databases only. When the last > > process closes the db, it will be wiped on the next open. > > If the file is mmaped, then yes, the mutex must be not destroyed. If it > is, then there is a bug in the current implementation. Just wanted to say thanks! With a pretty simple change to tdb our tdbtorture runs smoothly on FreeBSD 11! Volker From owner-freebsd-hackers@freebsd.org Tue Nov 22 02:25:00 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0EA1C4E79D for ; Tue, 22 Nov 2016 02:25:00 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x22c.google.com (mail-yw0-x22c.google.com [IPv6:2607:f8b0:4002:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9356E772 for ; Tue, 22 Nov 2016 02:25:00 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x22c.google.com with SMTP id t125so3592656ywc.1 for ; Mon, 21 Nov 2016 18:25:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=pgIWC34XDJEmfSXy74fU9budAMDOl9VsEYUKHG2q4lM=; b=zN0iJZkb6LGsWg2tWeOHNqGko2q57+dvgb8gzilv1gTVPtOGWxfP6JhB7oZCcchjC6 eIqpux7hlDQYa/6XL9SO0u3/WzYwZXyXUuNXwOLEMXBHE/AotbAzZJDB0wEiNfXuhwgg 115LRsN0IGOZ/vPgI9tbZ+a+QR1myWZg5shuCbMzscDJ2YEbKMdaJz0cU9OvlUsw8rEj ylA47EhYDpLm6assHtMD5FD1zyFGEdte0gcW8PY8bzMhxxXtO06+qZf9Vqh5COJHFBLp L6snSp5QeUbfec8/AkMIIGJUxjcvJCxfQz7oN8UulxcNYPAJgXhC6SKiS906ZjLJPDK6 yFxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=pgIWC34XDJEmfSXy74fU9budAMDOl9VsEYUKHG2q4lM=; b=PQeR+HY4Nf1wxOKzPEFb0csH4x6tHr4FeIJgTLSBVlD+KYjBkapG8046AVd1V7ahUe wKJcYn5BMTzEBf3lLlrrAc8YBVH5Ja6GId5LfLUMQ9BZsgtygyY5n4lcpG9UO+CA9nSU 6Z4LySMcXOhHRideualTsqszlWYcprqjx4D0G3HWfdynJrygiCxSGhkyz4SgfY+19YU3 Fc/6PI5NqyLyO9tH+aEJv5G+QbO2wF99fZZnAeL4pdULJKVSlUR2jiePh+rIwCQXEiuY 7Jmq8gBVwkm0DYpO2NUgY49o4CxlaBZ/nEW3XS6FR29+H03/afttCcBuHKQ/CYcHRnvA 8fZg== X-Gm-Message-State: AKaTC00lEFsXEpdV5VR6upZSDYvM+9OQo2w+kGcrcWXdVUM5ES08A7PXRz7dJmZfW6tme7cgGn6860OZguO6Nw== X-Received: by 10.129.113.84 with SMTP id m81mr18340222ywc.227.1479781499694; Mon, 21 Nov 2016 18:24:59 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.244.18 with HTTP; Mon, 21 Nov 2016 18:24:59 -0800 (PST) From: David Cross Date: Mon, 21 Nov 2016 21:24:59 -0500 Message-ID: Subject: FreeBSD 11 sparc64 instability To: freebsd-hackers@freebsd.org X-Mailman-Approved-At: Tue, 22 Nov 2016 02:37:32 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 02:25:00 -0000 After upgrading to FreeBSD 11 on some of my sparc64 machines (Sun Ultra 10), I noticed some truly disturbing behavior. 1) ipfw won't work, at all. kldload ipfw ipfw add 100 allow all from any to any ipfw -a list you will get errors about invalid arguments. *HOWEVER* if you use the ipfw that was from a 10.x install (10.3 in my case), it works perfectly. ??? 2) fsck will destroy your filesystems clean filesystem, "fsck -t ufs /dev/foo" you will see more errors than you knew were possible. *HOWEVER* if you take a version of fsck from 10.x (again, 10.3 in my case) ***AND*** libufs.so from that (LD_LIBRARY_PATH), everything is fine I suspected this may have been because I did a 11 buildworld on a 10.3 machine and "something leaked" with respect to headers, so I id a fresh buildworld, and when that too didn't fix it, I grabbed install images and used binaries and reproduced both of these errors. What is going on here? From owner-freebsd-hackers@freebsd.org Tue Nov 22 02:33:05 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F976C4EE2D for ; Tue, 22 Nov 2016 02:33:05 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: from mail-yw0-x232.google.com (mail-yw0-x232.google.com [IPv6:2607:f8b0:4002:c05::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 115E0D29 for ; Tue, 22 Nov 2016 02:33:05 +0000 (UTC) (envelope-from dcrosstech@gmail.com) Received: by mail-yw0-x232.google.com with SMTP id a10so3635403ywa.3 for ; Mon, 21 Nov 2016 18:33:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=mQh/X1BPL4ZyreHPMHPe4i4JQLQA2VNHQzltyO8Lwkk=; b=dL5DVPnSHKTii28VXi/9RHV2qjqrKvlJPmkkwuXVBZnF6HeF2BI5i/lIOHnhslmVDm qVy3Vde85ow9PT09fLfctarWv/Y1vmh9/NUoynoordS2vxFWQTFG2rQL3fWRxvU053wc ZNQEa3vCNNuMX61VbIeFHqePDfTVBbPjL9TKrqx6qNUE/fbPAWHVW+B77+uyyFHNVM/d ndRdlFO89u4q20ny7L6rCkRamzqlofJ4vHZmFQwpY2QnnZonoPExyx5Qe1taM+AeL5mq uizdxgOBHXoAE0uR1JxBdPlU19j2pkSIlonhUhDSqUgk9JpYaIwLtENP4MeUFjtBj/GR 28WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=mQh/X1BPL4ZyreHPMHPe4i4JQLQA2VNHQzltyO8Lwkk=; b=O5dZg+LbCDqdwYp9vMjre/B9fWANSw5kKLqrabx2vML7PBYJjePuRFKq9t02T+G86R aQu/tUVFyagS2pTE9s7w/ymsWIa9IOA5iEsD0YBe7xIsdwa76kwiIAn6YqdnZmr4ag+T Z4Lu+lwQJcz9fVOqswhKhZdsIqcjGXgDSC2pmWN8jfAYC7ZfPsQRdKZfr2C7HNP+Popy cuqlq+ycBhXV5sojCu+JgoTCmZTtCmk9TL5hfBoy9pXgp61n0LRP3RgWmx1yGP6Ca0FE US06gviDH+PYIXzB6UbwYdDWHUTb4nmUNzdxfS5ig0CBDTe2c3Sx35kgx8Zs+PGRmDlK yYDQ== X-Gm-Message-State: AKaTC00yrUyfkR9lwVi9nZs2xivfzWOf2AiPGWNgVnnW20vv6CekcWrj2VD5+112S/KxDMmapJN78XVNNjjtiA== X-Received: by 10.13.213.206 with SMTP id x197mr19307556ywd.299.1479781984278; Mon, 21 Nov 2016 18:33:04 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.244.18 with HTTP; Mon, 21 Nov 2016 18:33:04 -0800 (PST) From: David Cross Date: Mon, 21 Nov 2016 21:33:04 -0500 Message-ID: Subject: FreeBSD 11 i386 disk deadlock (I think) To: freebsd-hackers@freebsd.org X-Mailman-Approved-At: Tue, 22 Nov 2016 02:40:48 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 02:33:05 -0000 I noticed this situation on a Soekris machine I have (geode processor), It has happened twice, once in installworld ,once in buildworld. I am not sure how to trigger it more easily, but the symptoms are quite straightforward. Both times this happened in objcopy. And objcopy was stuck in the "vm_wait" wait-channel. The first time I was able to control-C it (after a few Control-Ts that showed it was not progressing) and everything came back, I was then able to re-run the installworld (that had been the source of the objcopy) without incident. That I was able to Control-C it, and re-run it without a disk-wait process, or blocking in the same spot is telling to me. The second time I had wandered away and by the time I got back I was no longer able to interact with the machine. It still passed traffic, but any attempt to open a new ssh connection was greeted with an opened TCP socket and IMMEDIATE reset/termination. I was forced to reset it. While this was happening, it was successfully routing 6 megabytes/second through its interfaces. I am going to try to run this again to see if I can get any more information, it is apparently fairly easy for me to trip. From owner-freebsd-hackers@freebsd.org Tue Nov 22 03:11:56 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8A923C4E754 for ; Tue, 22 Nov 2016 03:11:56 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 51FD9CEB for ; Tue, 22 Nov 2016 03:11:56 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 6A23013DE8 for ; Tue, 22 Nov 2016 03:11:49 +0000 (UTC) Subject: Re: FreeBSD 11 sparc64 instability To: freebsd-hackers@freebsd.org References: From: Allan Jude Message-ID: <83d26fd3-35d5-e416-a118-5f49ee6f2dc4@freebsd.org> Date: Mon, 21 Nov 2016 22:11:45 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="NaLlstmou7XJsF9HAjIUHrQBlDIpa4a6F" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 03:11:56 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --NaLlstmou7XJsF9HAjIUHrQBlDIpa4a6F Content-Type: multipart/mixed; boundary="e0B1McUSNclsPxCnhhoi0mJvpkfGeKNeX"; protected-headers="v1" From: Allan Jude To: freebsd-hackers@freebsd.org Message-ID: <83d26fd3-35d5-e416-a118-5f49ee6f2dc4@freebsd.org> Subject: Re: FreeBSD 11 sparc64 instability References: In-Reply-To: --e0B1McUSNclsPxCnhhoi0mJvpkfGeKNeX Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2016-11-21 21:24, David Cross wrote: > After upgrading to FreeBSD 11 on some of my sparc64 machines (Sun Ultra= > 10), I noticed some truly disturbing behavior. >=20 > 1) ipfw won't work, at all. >=20 > kldload ipfw > ipfw add 100 allow all from any to any >=20 > ipfw -a list >=20 > you will get errors about invalid arguments. >=20 > *HOWEVER* if you use the ipfw that was from a 10.x install (10.3 in my > case), it works perfectly. ??? >=20 > 2) fsck will destroy your filesystems > clean filesystem, "fsck -t ufs /dev/foo" you will see more errors than= you > knew were possible. >=20 > *HOWEVER* if you take a version of fsck from 10.x (again, 10.3 in my ca= se) > ***AND*** libufs.so from that (LD_LIBRARY_PATH), everything is fine >=20 > I suspected this may have been because I did a 11 buildworld on a 10.3 > machine and "something leaked" with respect to headers, so I id a fresh= > buildworld, and when that too didn't fix it, I grabbed install images a= nd > used binaries and reproduced both of these errors. >=20 > What is going on here? > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" >=20 Try running: uname -K and uname -U This will print the API versions of your kernel and userland. If they do not match, this is definitely the source of your issues. --=20 Allan Jude --e0B1McUSNclsPxCnhhoi0mJvpkfGeKNeX-- --NaLlstmou7XJsF9HAjIUHrQBlDIpa4a6F Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJYM7d1AAoJEBmVNT4SmAt+ptsP/AlXyH+z6KrRE5I18QP+6RD/ TobPJNpFgeL90+2LOsb3k0YdAju78cJwb7dN1H7leCQe87drmZjRyzTWIjsVsaY+ U/Poht80ZR3ia9CuNvbvlac/b9ZzefZXDtmfxJ31hBX4MdUMyT4E8JihNH3v7l/B Tx2tt+B/AhsigIThCudZaCj1QFPU1Ex9KQtfKt+JMFIZmqr3R5hlU0lmjhS/3SBX HTIy+JTG0H4evWSXtO/xn2MhXGxsoexg6EmI+5VXukiKbE3miEJpTgWRUkQoK4hZ 4Ruo5u99Pss33RuYeKhTwI19QjuEuXe86sAxNvqJIp24YHL+06zz3M0BCebar8YQ olU1x+NJJEmykQ8T0OnVdppVEaxv0adtVS9um0TOvJ0Mmek9UIPDx9vgKDkwD2MV g31yK7XPzWJsHGuhpwwwtBmugEZcsof7umuuoFksSGYdnzgw9TE+Lkpo4oVC0ZZO RSg4WGLboC4YmvQqY9ixKt/TbOHPTXfURM0d6E92qlXk1tjlncOfpHsIYGrjBNs3 RuZDW/qPQdzt+bevXCs8Tavgiv0BgMfwKlGnwBuTT5a0zzbwU2oMZlXcz0ZLqwlP vDGyvuX7PkIuWsQd4FqE1BwYJ2k/Blv86M24s6BChDDofIeyeBiBlPZ+Xy9HJuDY +KFqFNYxtYrbb5cj3QeW =UTV3 -----END PGP SIGNATURE----- --NaLlstmou7XJsF9HAjIUHrQBlDIpa4a6F-- From owner-freebsd-hackers@freebsd.org Tue Nov 22 10:32:58 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5BBBC4FFFF for ; Tue, 22 Nov 2016 10:32:58 +0000 (UTC) (envelope-from joel.bertrand@systella.fr) Received: from rayleigh.systella.fr (newton-ipv6.systella.fr [IPv6:2001:7a8:a8ed:253::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "rayleigh.systella.fr", Issuer "rayleigh.systella.fr" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7D7BF108E for ; Tue, 22 Nov 2016 10:32:58 +0000 (UTC) (envelope-from joel.bertrand@systella.fr) Received: from [192.168.2.3] (schroedinger.eikeo.com [192.168.2.3]) (authenticated bits=0) by rayleigh.systella.fr (8.15.2/8.15.2/Debian-6) with ESMTPSA id uAMAWhK3029611 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 22 Nov 2016 11:32:45 +0100 Subject: Re: RTL 8111G To: "freebsd-hackers@freebsd.org" References: <582AEEAE.2030702@systella.fr> From: =?UTF-8?Q?BERTRAND_Jo=c3=abl?= Message-ID: <58341ECB.8080605@systella.fr> Date: Tue, 22 Nov 2016 11:32:43 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40 MIME-Version: 1.0 In-Reply-To: <582AEEAE.2030702@systella.fr> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.99.2 at rayleigh X-Virus-Status: Clean X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 10:32:58 -0000 BERTRAND Joël a écrit : > Hello, > > I have installed some months ago a diskless terminal running > FreeBSD 10.x. This workstation contains a MSI H97I AC motherboard and > sadly a "Realtek® 8111G Gigabit LAN controller". > > With re driver provided by official FreeBSD 10.x kernel, network > adapter can stop and, as this system is diskless, kernel crashes. I want > to upgrade this system to FreeBSD 11.x as I have to use wifi adapter > (Intel Wireless 7260) that is not supported by 10.x. Anyone ? Does someone try to use this kind of ethernet controller ? Regards, JKB > From owner-freebsd-hackers@freebsd.org Tue Nov 22 11:44:51 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AA5E0C4FB8A for ; Tue, 22 Nov 2016 11:44:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7623D95B; Tue, 22 Nov 2016 11:44:51 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id D8EE02774E; Tue, 22 Nov 2016 12:44:46 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5ziDPz9SWaTw; Tue, 22 Nov 2016 12:44:45 +0100 (CET) Received: from [192.168.10.67] (opteron [192.168.10.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 9CF222774C; Tue, 22 Nov 2016 12:44:45 +0100 (CET) Subject: Re: Mising ENODATA To: John Baldwin , freebsd-hackers@freebsd.org References: <56E6C5EA.2080005@digiware.nl> <5030334.fMbvND8flt@ralph.baldwin.cx> From: Willem Jan Withagen Message-ID: Date: Tue, 22 Nov 2016 12:44:44 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.5.0 MIME-Version: 1.0 In-Reply-To: <5030334.fMbvND8flt@ralph.baldwin.cx> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 11:44:51 -0000 On 23-5-2016 22:47, John Baldwin wrote: > On Monday, March 14, 2016 03:08:42 PM Willem Jan Withagen wrote: >> Hi, >> >> According the standard is ENODATA an extention of errno.h defines... >> >> http://pubs.opengroup.org/onlinepubs/9699919799/ >> >> The Open Group Base Specifications Issue 7 >> IEEE Std 1003.1, 2013 Edition >> >> [ENODATA] >> [OB XSR] [Option Start] >> No message available. No message is available on the STREAM head >> read queue. [Option End] >> >> [XSR] [Option Start] XSI STREAMS [Option End] >> The functionality described is optional. The functionality described is >> also an extension to the ISO C standard. >> >> Where applicable, functions are marked with the XSR margin legend in the >> SYNOPSIS section. Where additional semantics apply to a function, the >> material is identified by use of the XSR margin legend. >> >> [OB] [Option Start] Obsolescent [Option End] >> The functionality described may be removed in a future version of this >> volume of POSIX.1-2008. Strictly Conforming POSIX Applications and >> Strictly Conforming XSI Applications shall not use obsolescent features. >> >> Where applicable, the material is identified by use of the OB margin legend. >> ---- >> >> The OB part makes a bit strange to ask for definition, but would it be >> possible to add ENODATA to our headers? >> The alternative question is: why would we not? > > Well, it's defined for STREAMS and FreeBSD (and BSDs in general) don't > implement STREAMS. OTOH, if Ceph has (ab)used it for their own internal > errors then we could perhaps add our own ENODATA. Do you want to make a > patch to do so? > Hi John, Rather old Email, but now it comes to the point that it is going to be used. Uptil now I just patched my onw errno.h, but once I'm going to build a port for Cep, it no long works. I do not think anubody will allow a port to modify /usr/include/errno.h 8-) For my/Ceph needs the path is rather simple. *** /usr/include/errno.h Mon Oct 3 02:05:43 2016 --- /usr/srcs/head/src/sys/sys/errno.h Sun Aug 21 18:25:05 2016 *************** *** 164,170 **** #define ECANCELED 85 /* Operation canceled */ #define EILSEQ 86 /* Illegal byte sequence */ #define ENOATTR 87 /* Attribute not found */ - #define ENODATA 87 /* Attribute not found */ #define EDOOFUS 88 /* Programming error */ #endif /* _POSIX_SOURCE */ --- 164,169 ---- I'll submit this as "bug" report and will see what comes of it. --WjW From owner-freebsd-hackers@freebsd.org Tue Nov 22 17:32:31 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 134D9C50E87 for ; Tue, 22 Nov 2016 17:32:31 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qk0-x242.google.com (mail-qk0-x242.google.com [IPv6:2607:f8b0:400d:c09::242]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C2F191ACC; Tue, 22 Nov 2016 17:32:30 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-qk0-x242.google.com with SMTP id x190so5516281qkb.0; Tue, 22 Nov 2016 09:32:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=mb4Nt+PqnJC88k7z7WXetXrBF/3E9iDzEYmV1tDU5NU=; b=Kh2m2QjTbbBZVnYVd8lCy+/HdgmTBRlN8ijW25DabqpWVcDJq6s+5rY5Va18OV/JBH qAzC1X9zaoNxLa0BLkmx3PO1XuciA1paRjD5V6i7XHGdgVyjvhFP0abuuLaU4PNG97mF /MBwx/UlTxqUEGC4GxyU0+5i90WByF9Fh9RdEVABCmZbhedcLaBVBKfA/NoaHIsvcUSS cSPMNet5XD4dk9qlfEfDzJbybX/+s7U/I+vA12pUzyIIjE+K9GR11AjD/ra8Fug5IoLj gDOFMtRPe5W2avR4a5mW/jMfI9yxjDTKzn9OWwMkA4KSz88F4RylBwBnn/UJlmOnTJti 1gAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=mb4Nt+PqnJC88k7z7WXetXrBF/3E9iDzEYmV1tDU5NU=; b=Qvk1C1g/XcybAsTQUq5Oi7sbg3ltKcLDcN647Xni1aCK0BLoMQrm7qgeQI5RPllNpq YqYEXUAO4j3TVN/Hdd13LPh4791pGshqsxA1C7qGiFL2eueGHfiXDEgOTGJ8nPdveuaZ KT6WdE0E2DJWwJmeodKJ3ZCpflHfhgWPzFfjFl7364GOLYCLm4tRKPT0BOKG/+dbkx+p BbCAXrIyMtai7BrhIgBIGUtsO90ryL87/5nBEf6hG/ALmE28N5j5F5Nm2vnDxbN29lPG hC//DMaO4sfbCGdEKpp9wzqKm/bnYRTOOecpqVY2XqDyVVNWCJVLN/n2wPxvnuJQTFOz xNAA== X-Gm-Message-State: AKaTC00E8tv4ewKQvQt9EArI1bSFKYxPxSVoz+6oLT+vjYyJLtTys6aHVEOFjJs9GftZrJrdntHkY8aCidOEqA== X-Received: by 10.55.104.68 with SMTP id d65mr22818188qkc.119.1479835949882; Tue, 22 Nov 2016 09:32:29 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.12.166.129 with HTTP; Tue, 22 Nov 2016 09:32:29 -0800 (PST) In-Reply-To: References: <56E6C5EA.2080005@digiware.nl> <5030334.fMbvND8flt@ralph.baldwin.cx> From: Alan Somers Date: Tue, 22 Nov 2016 10:32:29 -0700 X-Google-Sender-Auth: rDHU9HEEqGAxfcsjM3dFaTqcCjw Message-ID: Subject: Re: Mising ENODATA To: Willem Jan Withagen Cc: John Baldwin , "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 17:32:31 -0000 On Tue, Nov 22, 2016 at 4:44 AM, Willem Jan Withagen wrote: > On 23-5-2016 22:47, John Baldwin wrote: >> On Monday, March 14, 2016 03:08:42 PM Willem Jan Withagen wrote: >>> Hi, >>> >>> According the standard is ENODATA an extention of errno.h defines... >>> >>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>> >>> The Open Group Base Specifications Issue 7 >>> IEEE Std 1003.1, 2013 Edition >>> >>> [ENODATA] >>> [OB XSR] [Option Start] >>> No message available. No message is available on the STREAM head >>> read queue. [Option End] >>> >>> [XSR] [Option Start] XSI STREAMS [Option End] >>> The functionality described is optional. The functionality described is >>> also an extension to the ISO C standard. >>> >>> Where applicable, functions are marked with the XSR margin legend in the >>> SYNOPSIS section. Where additional semantics apply to a function, the >>> material is identified by use of the XSR margin legend. >>> >>> [OB] [Option Start] Obsolescent [Option End] >>> The functionality described may be removed in a future version of this >>> volume of POSIX.1-2008. Strictly Conforming POSIX Applications and >>> Strictly Conforming XSI Applications shall not use obsolescent features. >>> >>> Where applicable, the material is identified by use of the OB margin legend. >>> ---- >>> >>> The OB part makes a bit strange to ask for definition, but would it be >>> possible to add ENODATA to our headers? >>> The alternative question is: why would we not? >> >> Well, it's defined for STREAMS and FreeBSD (and BSDs in general) don't >> implement STREAMS. OTOH, if Ceph has (ab)used it for their own internal >> errors then we could perhaps add our own ENODATA. Do you want to make a >> patch to do so? >> > > Hi John, > > Rather old Email, but now it comes to the point that it is going to be > used. Uptil now I just patched my onw errno.h, but once I'm going to > build a port for Cep, it no long works. I do not think anubody will > allow a port to modify /usr/include/errno.h 8-) > > For my/Ceph needs the path is rather simple. > *** /usr/include/errno.h Mon Oct 3 02:05:43 2016 > --- /usr/srcs/head/src/sys/sys/errno.h Sun Aug 21 18:25:05 2016 > *************** > *** 164,170 **** > #define ECANCELED 85 /* Operation canceled */ > #define EILSEQ 86 /* Illegal byte sequence */ > #define ENOATTR 87 /* Attribute not found */ > - #define ENODATA 87 /* Attribute not found */ > > #define EDOOFUS 88 /* Programming error */ > #endif /* _POSIX_SOURCE */ > --- 164,169 ---- > > I'll submit this as "bug" report and will see what comes of it. > > --WjW I too ran into this problem a few years ago. My solution was to patch Ceph instead. Would these patches still work? --- src/include/compat.h.orig 2013-11-01 16:14:01.000000000 +0000 +++ src/include/compat.h 2013-11-04 18:21:43.000000000 +0000 @@ -13,7 +13,15 @@ #define CEPH_COMPAT_H #if defined(__FreeBSD__) -#define ENODATA 61 +/* + * FreeBSD does not have ENODATA. We must define it here. However, it MAY be + * defined by boost. We can't simply include boost/cerrno.hpp here, because + * that header is not includable by C code. So we must duplicate boost's + * definition :( + */ +#ifndef ENODATA +#define ENODATA 9919 +#endif #endif /* !__FreeBSD__ */ #if defined(__FreeBSD__) || defined(__APPLE__) --- src/pybind/rados.py.orig 2013-11-04 17:36:06.000000000 +0000 +++ src/pybind/rados.py 2013-11-04 17:37:02.000000000 +0000 @@ -89,10 +89,14 @@ errno.EIO : IOError, errno.ENOSPC : NoSpace, errno.EEXIST : ObjectExists, - errno.ENODATA : NoData, errno.EINTR : InterruptedOrTimeoutError, errno.ETIMEDOUT : TimedOut } + # errno.ENODATA is not implemented on all platforms + try: + errors[errno.ENODATA] = NoData + except AttributeError: + pass ret = abs(ret) if ret in errors: return errors[ret](msg) --- src/pybind/cephfs.py.orig 2013-10-10 16:14:07.000000000 +0000 +++ src/pybind/cephfs.py 2013-10-10 16:15:22.000000000 +0000 @@ -49,8 +49,12 @@ errno.EIO : IOError, errno.ENOSPC : NoSpace, errno.EEXIST : ObjectExists, - errno.ENODATA : NoData } + # errno.ENODATA is not implemented on all platforms + try: + errors[errno.ENODATA] = NoData + except AttributeError: + pass ret = abs(ret) if ret in errors: return errors[ret](msg) From owner-freebsd-hackers@freebsd.org Tue Nov 22 22:14:37 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15878C4F5C9 for ; Tue, 22 Nov 2016 22:14:37 +0000 (UTC) (envelope-from david.marec@davenulle.org) Received: from smtp.smtpout.orange.fr (smtp05.smtpout.orange.fr [80.12.242.127]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (Client CN "Bizanga Labs SMTP Client Certificate", Issuer "Bizanga Labs CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 7EF0D1DC2 for ; Tue, 22 Nov 2016 22:14:35 +0000 (UTC) (envelope-from david.marec@davenulle.org) Received: from dmarec.local ([86.199.68.25]) by mwinf5d61 with ME id BAER1u00k0YigXJ03AEStJ; Tue, 22 Nov 2016 23:14:27 +0100 X-ME-Helo: dmarec.local X-ME-Date: Tue, 22 Nov 2016 23:14:27 +0100 X-ME-IP: 86.199.68.25 Subject: Re: RTL 8111G To: freebsd-hackers@freebsd.org References: <582AEEAE.2030702@systella.fr> <58341ECB.8080605@systella.fr> From: David Marec Message-ID: <66187adb-2240-e63f-3800-b454cd4cbc0b@davenulle.org> Date: Tue, 22 Nov 2016 23:14:25 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <58341ECB.8080605@systella.fr> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 22:14:37 -0000 On 22.11.2016 11:32, BERTRAND Joël wrote: > > Anyone ? Does someone try to use this kind of ethernet controller ? I m currently running FreeBSD 11 ( after running 10.3 one season ), with this chipset embedded on the motherboard (MSI H81M-P33) . I didn't encounter any issue, but I mostly use this system as a desktop. While, at times, It runs a whole night long, to receive zfs backups from remote servers. Even then, I didn't notice anything that may have gone wrong. Just in case, to pass through the failure on 'M_DONTWAIT', you have to replace this flag with the /new/ (FreeBSD 9) mbuf allocation parameter: M_NOWAIT. But doing so, I guess you'll trigger a error related to another interface evolution. -- David Marec From owner-freebsd-hackers@freebsd.org Tue Nov 22 22:29:35 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DD497C4F999 for ; Tue, 22 Nov 2016 22:29:35 +0000 (UTC) (envelope-from joel.bertrand@systella.fr) Received: from rayleigh.systella.fr (newton-ipv6.systella.fr [IPv6:2001:7a8:a8ed:253::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "rayleigh.systella.fr", Issuer "rayleigh.systella.fr" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A25E676A for ; Tue, 22 Nov 2016 22:29:35 +0000 (UTC) (envelope-from joel.bertrand@systella.fr) Received: from [192.168.254.1] (bertrand@rayleigh.systella.fr [192.168.254.1]) (authenticated bits=0) by rayleigh.systella.fr (8.15.2/8.15.2/Debian-6) with ESMTPSA id uAMMTMMb029796 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 22 Nov 2016 23:29:22 +0100 Subject: Re: RTL 8111G To: freebsd-hackers@freebsd.org References: <582AEEAE.2030702@systella.fr> <58341ECB.8080605@systella.fr> <66187adb-2240-e63f-3800-b454cd4cbc0b@davenulle.org> From: =?UTF-8?Q?BERTRAND_Jo=c3=abl?= Message-ID: <5834C6C2.4070102@systella.fr> Date: Tue, 22 Nov 2016 23:29:22 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0 SeaMonkey/2.40 MIME-Version: 1.0 In-Reply-To: <66187adb-2240-e63f-3800-b454cd4cbc0b@davenulle.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.99.2 at rayleigh X-Virus-Status: Clean X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 22:29:36 -0000 David Marec a écrit : > On 22.11.2016 11:32, BERTRAND Joël wrote: >> >> Anyone ? Does someone try to use this kind of ethernet controller ? > > I m currently running FreeBSD 11 ( after running 10.3 one season ), > with this chipset embedded on the motherboard (MSI H81M-P33) . > > I didn't encounter any issue, but I mostly use this system as a desktop. > > While, at times, It runs a whole night long, to receive zfs backups > from remote servers. Even then, I didn't notice anything that may have > gone wrong. Thanks a lot for your answer. > Just in case, > to pass through the failure on 'M_DONTWAIT', you have to replace this > flag with the /new/ (FreeBSD 9) mbuf allocation parameter: M_NOWAIT. > > But doing so, I guess you'll trigger a error related to another > interface evolution. I don't understand your last sentences. What is the failure on 'M_DONTWAIT' ? My motherboard is also an MSI (mini-ITX, MSI H97I AC). With FreeBSD 10.x driver, ethernet controller randomly hangs (after a long uptime). It doesn't hang with Realtek driver. Regards, JKB From owner-freebsd-hackers@freebsd.org Tue Nov 22 22:46:16 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B8744C50068 for ; Tue, 22 Nov 2016 22:46:16 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [IPv6:2001:4cb8:90:ffff::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 527701345; Tue, 22 Nov 2016 22:46:16 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id BE1B227BBA; Tue, 22 Nov 2016 23:46:12 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9p8YRkM9U_pB; Tue, 22 Nov 2016 23:46:11 +0100 (CET) Received: from [192.168.10.120] (10G [192.168.10.120]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 12D1827BB9; Tue, 22 Nov 2016 23:46:11 +0100 (CET) References: <56E6C5EA.2080005@digiware.nl> <5030334.fMbvND8flt@ralph.baldwin.cx> In-Reply-To: Mime-Version: 1.0 (1.0) Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Message-Id: <1D7D7363-F503-4322-BBAC-556010E11AF1@digiware.nl> Cc: John Baldwin , "freebsd-hackers@freebsd.org" X-Mailer: iPad Mail (9B206) From: Willem Jan Withagen Subject: Re: Mising ENODATA Date: Tue, 22 Nov 2016 23:46:12 +0100 To: Alan Somers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 22:46:16 -0000 Op 22 nov. 2016 om 18:32 heeft Alan Somers het volgend= e geschreven: > On Tue, Nov 22, 2016 at 4:44 AM, Willem Jan Withagen wro= te: >> On 23-5-2016 22:47, John Baldwin wrote: >>> On Monday, March 14, 2016 03:08:42 PM Willem Jan Withagen wrote: >>>> Hi, >>>>=20 >>>> According the standard is ENODATA an extention of errno.h defines... >>>>=20 >>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>=20 >>>> The Open Group Base Specifications Issue 7 >>>> IEEE Std 1003.1, 2013 Edition >>>>=20 >>>> [ENODATA] >>>> [OB XSR] [Option Start] >>>> No message available. No message is available on the STREAM head >>>> read queue. [Option End] >>>>=20 >>>> [XSR] [Option Start] XSI STREAMS [Option End] >>>> The functionality described is optional. The functionality described is= >>>> also an extension to the ISO C standard. >>>>=20 >>>> Where applicable, functions are marked with the XSR margin legend in th= e >>>> SYNOPSIS section. Where additional semantics apply to a function, the >>>> material is identified by use of the XSR margin legend. >>>>=20 >>>> [OB] [Option Start] Obsolescent [Option End] >>>> The functionality described may be removed in a future version of this >>>> volume of POSIX.1-2008. Strictly Conforming POSIX Applications and >>>> Strictly Conforming XSI Applications shall not use obsolescent features= . >>>>=20 >>>> Where applicable, the material is identified by use of the OB margin le= gend. >>>> ---- >>>>=20 >>>> The OB part makes a bit strange to ask for definition, but would it be >>>> possible to add ENODATA to our headers? >>>> The alternative question is: why would we not? >>>=20 >>> Well, it's defined for STREAMS and FreeBSD (and BSDs in general) don't >>> implement STREAMS. OTOH, if Ceph has (ab)used it for their own internal= >>> errors then we could perhaps add our own ENODATA. Do you want to make a= >>> patch to do so? >>>=20 >>=20 >> Hi John, >>=20 >> Rather old Email, but now it comes to the point that it is going to be >> used. Uptil now I just patched my onw errno.h, but once I'm going to >> build a port for Cep, it no long works. I do not think anubody will >> allow a port to modify /usr/include/errno.h 8-) >>=20 >> For my/Ceph needs the path is rather simple. >> *** /usr/include/errno.h Mon Oct 3 02:05:43 2016 >> --- /usr/srcs/head/src/sys/sys/errno.h Sun Aug 21 18:25:05 2016 >> *************** >> *** 164,170 **** >> #define ECANCELED 85 /* Operation canceled */ >> #define EILSEQ 86 /* Illegal byte sequence *= / >> #define ENOATTR 87 /* Attribute not found */ >> - #define ENODATA 87 /* Attribute not found */= >>=20 >> #define EDOOFUS 88 /* Programming error */ >> #endif /* _POSIX_SOURCE */ >> --- 164,169 ---- >>=20 >> I'll submit this as "bug" report and will see what comes of it. >>=20 >> --WjW >=20 > I too ran into this problem a few years ago. My solution was to patch > Ceph instead. Would these patches still work? >=20 > --- src/include/compat.h.orig 2013-11-01 16:14:01.000000000 +0000 > +++ src/include/compat.h 2013-11-04 18:21:43.000000000 +0000 > @@ -13,7 +13,15 @@ > #define CEPH_COMPAT_H >=20 > #if defined(__FreeBSD__) > -#define ENODATA 61 > +/* > + * FreeBSD does not have ENODATA. We must define it here. However, it M= AY be > + * defined by boost. We can't simply include boost/cerrno.hpp here, beca= use > + * that header is not includable by C code. So we must duplicate boost's= > + * definition :( > + */ > +#ifndef ENODATA > +#define ENODATA 9919 > +#endif > #endif /* !__FreeBSD__ */ >=20 > #if defined(__FreeBSD__) || defined(__APPLE__) > --- src/pybind/rados.py.orig 2013-11-04 17:36:06.000000000 +0000 > +++ src/pybind/rados.py 2013-11-04 17:37:02.000000000 +0000 > @@ -89,10 +89,14 @@ > errno.EIO : IOError, > errno.ENOSPC : NoSpace, > errno.EEXIST : ObjectExists, > - errno.ENODATA : NoData, > errno.EINTR : InterruptedOrTimeoutError, > errno.ETIMEDOUT : TimedOut > } > + # errno.ENODATA is not implemented on all platforms > + try: > + errors[errno.ENODATA] =3D NoData > + except AttributeError: > + pass > ret =3D abs(ret) > if ret in errors: > return errors[ret](msg) > --- src/pybind/cephfs.py.orig 2013-10-10 16:14:07.000000000 +0000 > +++ src/pybind/cephfs.py 2013-10-10 16:15:22.000000000 +0000 > @@ -49,8 +49,12 @@ > errno.EIO : IOError, > errno.ENOSPC : NoSpace, > errno.EEXIST : ObjectExists, > - errno.ENODATA : NoData > } > + # errno.ENODATA is not implemented on all platforms > + try: > + errors[errno.ENODATA] =3D NoData > + except AttributeError: > + pass > ret =3D abs(ret) > if ret in errors: > return errors[ret](msg) But it works just the other way around... There are calls to freebsd xattr() which return ENOATTR on error, the Linux code however uses ENODATA (in value =3D=3D 87) to test for errors.= So if anything needs to be defined ENODATA =3D=3D ENOATTR. This I do in compat.h but defining it to 9919 would not help. Stronger, I test for it to not be 9919, or I Error compilation. And the trouble with all python code is that ignoring ENODATA is not always a= wise decision. Just having it in /usr/include is the right place where python/cython can fi= nd it. And it would help other porting efforts would be helped as well. --WjW From owner-freebsd-hackers@freebsd.org Tue Nov 22 23:05:15 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A99A7C5063A for ; Tue, 22 Nov 2016 23:05:15 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-qt0-x243.google.com (mail-qt0-x243.google.com [IPv6:2607:f8b0:400d:c0d::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 515271BCF; Tue, 22 Nov 2016 23:05:15 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-qt0-x243.google.com with SMTP id n6so4646856qtd.0; Tue, 22 Nov 2016 15:05:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=ZVc0toGG9ZDzsPGtr1s7O2antYd+T6Qx71CNxF/aOKI=; b=1IWg2eKqXQW+1K5cWY/ME1FoHxJcqoeOAj0ZpGqbmXremtYVot8gTzGb8Cw9UK6e4S hP0oBbWno6RpR8uhrVX5O2Rz9ks0X6c2J4hYJgFsasrrbQWj28afziVczZCE7O1iK/eg 7vjWceMpx4z7CItx9YyLye3CRdgWEGZ4pMd/Tj5r82YQkWpTqcNPvzpjBnhkLnAN5Cr4 0As6RS+80C7i68MwhzK2h/NFf/sluaATWa+R1hewTG6yMjTUa5rfGU/Zd8YtI4U7znhB CBnsZMWlsTsUUw4Kr6pbxhUsd/95OA6wHbgJ1X5AED3MVRdcN/jTZREylb4pmc7uMS0k sQeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=ZVc0toGG9ZDzsPGtr1s7O2antYd+T6Qx71CNxF/aOKI=; b=M56gxcvSH6/QUSGL1Wd9CxEwxlPECk1Sb2/wwEwpVOZwh/V338ibLyfxiOXzfLyeWz 9hE4Mw1UiKRg/YRX0+VZy9gA1EYDPFUVEgdWrFO7wBCiVwqlHMGA5y7t0PEvsUhIqH6/ Tcgo9SZ1BkoIPKQwW0jq3s8MiTz7r6U3vPBai/dPPh2roqEUQiryaGivXX6+z3jsLeSk bC3DtlTUp7dfmUf8tjRdSkdbQ1wBgxjXWncwNBFZdfHNbC5Yk3C+Fg6sl+1Xe2H3DjFN 71RyI8AGWFxDZofDZAV68wplEkNHTjCztQhCyC8u9jK/l3Bc+3CKTlf2aXCtoAo2wsen +7HQ== X-Gm-Message-State: AKaTC03VZQNAd1v1nLe1GWuvtyejQCtYmm7t/H8NvdX04RaMXYmz3S0pIK2N1CAVg21QjmcwKa4B+whYAK9Dpg== X-Received: by 10.200.48.28 with SMTP id f28mr52158qte.247.1479855914119; Tue, 22 Nov 2016 15:05:14 -0800 (PST) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 10.12.166.129 with HTTP; Tue, 22 Nov 2016 15:05:13 -0800 (PST) In-Reply-To: <1D7D7363-F503-4322-BBAC-556010E11AF1@digiware.nl> References: <56E6C5EA.2080005@digiware.nl> <5030334.fMbvND8flt@ralph.baldwin.cx> <1D7D7363-F503-4322-BBAC-556010E11AF1@digiware.nl> From: Alan Somers Date: Tue, 22 Nov 2016 16:05:13 -0700 X-Google-Sender-Auth: cImjeloMfAAPOEPWTot0ym8lfYc Message-ID: Subject: Re: Mising ENODATA To: Willem Jan Withagen Cc: John Baldwin , "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Nov 2016 23:05:15 -0000 On Tue, Nov 22, 2016 at 3:46 PM, Willem Jan Withagen wrote: > > > Op 22 nov. 2016 om 18:32 heeft Alan Somers het volgende geschreven: > >> On Tue, Nov 22, 2016 at 4:44 AM, Willem Jan Withagen wrote: >>> On 23-5-2016 22:47, John Baldwin wrote: >>>> On Monday, March 14, 2016 03:08:42 PM Willem Jan Withagen wrote: >>>>> Hi, >>>>> >>>>> According the standard is ENODATA an extention of errno.h defines... >>>>> >>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>> >>>>> The Open Group Base Specifications Issue 7 >>>>> IEEE Std 1003.1, 2013 Edition >>>>> >>>>> [ENODATA] >>>>> [OB XSR] [Option Start] >>>>> No message available. No message is available on the STREAM head >>>>> read queue. [Option End] >>>>> >>>>> [XSR] [Option Start] XSI STREAMS [Option End] >>>>> The functionality described is optional. The functionality described is >>>>> also an extension to the ISO C standard. >>>>> >>>>> Where applicable, functions are marked with the XSR margin legend in the >>>>> SYNOPSIS section. Where additional semantics apply to a function, the >>>>> material is identified by use of the XSR margin legend. >>>>> >>>>> [OB] [Option Start] Obsolescent [Option End] >>>>> The functionality described may be removed in a future version of this >>>>> volume of POSIX.1-2008. Strictly Conforming POSIX Applications and >>>>> Strictly Conforming XSI Applications shall not use obsolescent features. >>>>> >>>>> Where applicable, the material is identified by use of the OB margin legend. >>>>> ---- >>>>> >>>>> The OB part makes a bit strange to ask for definition, but would it be >>>>> possible to add ENODATA to our headers? >>>>> The alternative question is: why would we not? >>>> >>>> Well, it's defined for STREAMS and FreeBSD (and BSDs in general) don't >>>> implement STREAMS. OTOH, if Ceph has (ab)used it for their own internal >>>> errors then we could perhaps add our own ENODATA. Do you want to make a >>>> patch to do so? >>>> >>> >>> Hi John, >>> >>> Rather old Email, but now it comes to the point that it is going to be >>> used. Uptil now I just patched my onw errno.h, but once I'm going to >>> build a port for Cep, it no long works. I do not think anubody will >>> allow a port to modify /usr/include/errno.h 8-) >>> >>> For my/Ceph needs the path is rather simple. >>> *** /usr/include/errno.h Mon Oct 3 02:05:43 2016 >>> --- /usr/srcs/head/src/sys/sys/errno.h Sun Aug 21 18:25:05 2016 >>> *************** >>> *** 164,170 **** >>> #define ECANCELED 85 /* Operation canceled */ >>> #define EILSEQ 86 /* Illegal byte sequence */ >>> #define ENOATTR 87 /* Attribute not found */ >>> - #define ENODATA 87 /* Attribute not found */ >>> >>> #define EDOOFUS 88 /* Programming error */ >>> #endif /* _POSIX_SOURCE */ >>> --- 164,169 ---- >>> >>> I'll submit this as "bug" report and will see what comes of it. >>> >>> --WjW >> >> I too ran into this problem a few years ago. My solution was to patch >> Ceph instead. Would these patches still work? >> >> --- src/include/compat.h.orig 2013-11-01 16:14:01.000000000 +0000 >> +++ src/include/compat.h 2013-11-04 18:21:43.000000000 +0000 >> @@ -13,7 +13,15 @@ >> #define CEPH_COMPAT_H >> >> #if defined(__FreeBSD__) >> -#define ENODATA 61 >> +/* >> + * FreeBSD does not have ENODATA. We must define it here. However, it MAY be >> + * defined by boost. We can't simply include boost/cerrno.hpp here, because >> + * that header is not includable by C code. So we must duplicate boost's >> + * definition :( >> + */ >> +#ifndef ENODATA >> +#define ENODATA 9919 >> +#endif >> #endif /* !__FreeBSD__ */ >> >> #if defined(__FreeBSD__) || defined(__APPLE__) >> --- src/pybind/rados.py.orig 2013-11-04 17:36:06.000000000 +0000 >> +++ src/pybind/rados.py 2013-11-04 17:37:02.000000000 +0000 >> @@ -89,10 +89,14 @@ >> errno.EIO : IOError, >> errno.ENOSPC : NoSpace, >> errno.EEXIST : ObjectExists, >> - errno.ENODATA : NoData, >> errno.EINTR : InterruptedOrTimeoutError, >> errno.ETIMEDOUT : TimedOut >> } >> + # errno.ENODATA is not implemented on all platforms >> + try: >> + errors[errno.ENODATA] = NoData >> + except AttributeError: >> + pass >> ret = abs(ret) >> if ret in errors: >> return errors[ret](msg) >> --- src/pybind/cephfs.py.orig 2013-10-10 16:14:07.000000000 +0000 >> +++ src/pybind/cephfs.py 2013-10-10 16:15:22.000000000 +0000 >> @@ -49,8 +49,12 @@ >> errno.EIO : IOError, >> errno.ENOSPC : NoSpace, >> errno.EEXIST : ObjectExists, >> - errno.ENODATA : NoData >> } >> + # errno.ENODATA is not implemented on all platforms >> + try: >> + errors[errno.ENODATA] = NoData >> + except AttributeError: >> + pass >> ret = abs(ret) >> if ret in errors: >> return errors[ret](msg) > > But it works just the other way around... > > There are calls to freebsd xattr() which return ENOATTR on error, > the Linux code however uses ENODATA (in value == 87) to test for errors. > So if anything needs to be defined ENODATA == ENOATTR. > This I do in compat.h but defining it to 9919 would not help. > Stronger, I test for it to not be 9919, or I Error compilation. > > And the trouble with all python code is that ignoring ENODATA is not always a wise decision. > Just having it in /usr/include is the right place where python/cython can find it. > > And it would help other porting efforts would be helped as well. > > --WjW If the problem is that Linux's xattr() and FreeBSD's xattr() behave differently, then the solution should be in the code that uses xattr(), right? Something like this: err = extattr_get_fd(...) #if defined( freebsd ) if (err == ENOATTR) { #else if defined (linux) if (err == ENODATA) { #endif return CEPH_ENODATA } From owner-freebsd-hackers@freebsd.org Wed Nov 23 00:08:17 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2808C50A12 for ; Wed, 23 Nov 2016 00:08:17 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [176.74.240.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 89B6B188A; Wed, 23 Nov 2016 00:08:17 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from router.digiware.nl (localhost.digiware.nl [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 8C06427E82; Wed, 23 Nov 2016 01:08:09 +0100 (CET) X-Virus-Scanned: amavisd-new at digiware.com Received: from smtp.digiware.nl ([127.0.0.1]) by router.digiware.nl (router.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id x0luZxoBbm7D; Wed, 23 Nov 2016 01:08:06 +0100 (CET) Received: from [192.168.10.10] (asus [192.168.10.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.digiware.nl (Postfix) with ESMTPSA id 95E5227E81; Wed, 23 Nov 2016 01:08:06 +0100 (CET) Subject: Re: Mising ENODATA To: Alan Somers References: <56E6C5EA.2080005@digiware.nl> <5030334.fMbvND8flt@ralph.baldwin.cx> <1D7D7363-F503-4322-BBAC-556010E11AF1@digiware.nl> Cc: John Baldwin , "freebsd-hackers@freebsd.org" From: Willem Jan Withagen Message-ID: <4fad89b0-cf93-a4df-2164-b5393cb9eea5@digiware.nl> Date: Wed, 23 Nov 2016 01:08:07 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Nov 2016 00:08:17 -0000 On 23-11-2016 00:05, Alan Somers wrote: > On Tue, Nov 22, 2016 at 3:46 PM, Willem Jan Withagen wrote: >> >> >> Op 22 nov. 2016 om 18:32 heeft Alan Somers het volgende geschreven: >> >>> On Tue, Nov 22, 2016 at 4:44 AM, Willem Jan Withagen wrote: >>>> On 23-5-2016 22:47, John Baldwin wrote: >>>>> On Monday, March 14, 2016 03:08:42 PM Willem Jan Withagen wrote: >>>>>> Hi, >>>>>> >>>>>> According the standard is ENODATA an extention of errno.h defines... >>>>>> >>>>>> http://pubs.opengroup.org/onlinepubs/9699919799/ >>>>>> >>>>>> The Open Group Base Specifications Issue 7 >>>>>> IEEE Std 1003.1, 2013 Edition >>>>>> >>>>>> [ENODATA] >>>>>> [OB XSR] [Option Start] >>>>>> No message available. No message is available on the STREAM head >>>>>> read queue. [Option End] >>>>>> >>>>>> [XSR] [Option Start] XSI STREAMS [Option End] >>>>>> The functionality described is optional. The functionality described is >>>>>> also an extension to the ISO C standard. >>>>>> >>>>>> Where applicable, functions are marked with the XSR margin legend in the >>>>>> SYNOPSIS section. Where additional semantics apply to a function, the >>>>>> material is identified by use of the XSR margin legend. >>>>>> >>>>>> [OB] [Option Start] Obsolescent [Option End] >>>>>> The functionality described may be removed in a future version of this >>>>>> volume of POSIX.1-2008. Strictly Conforming POSIX Applications and >>>>>> Strictly Conforming XSI Applications shall not use obsolescent features. >>>>>> >>>>>> Where applicable, the material is identified by use of the OB margin legend. >>>>>> ---- >>>>>> >>>>>> The OB part makes a bit strange to ask for definition, but would it be >>>>>> possible to add ENODATA to our headers? >>>>>> The alternative question is: why would we not? >>>>> >>>>> Well, it's defined for STREAMS and FreeBSD (and BSDs in general) don't >>>>> implement STREAMS. OTOH, if Ceph has (ab)used it for their own internal >>>>> errors then we could perhaps add our own ENODATA. Do you want to make a >>>>> patch to do so? >>>>> >>>> >>>> Hi John, >>>> >>>> Rather old Email, but now it comes to the point that it is going to be >>>> used. Uptil now I just patched my onw errno.h, but once I'm going to >>>> build a port for Cep, it no long works. I do not think anubody will >>>> allow a port to modify /usr/include/errno.h 8-) >>>> >>>> For my/Ceph needs the path is rather simple. >>>> *** /usr/include/errno.h Mon Oct 3 02:05:43 2016 >>>> --- /usr/srcs/head/src/sys/sys/errno.h Sun Aug 21 18:25:05 2016 >>>> *************** >>>> *** 164,170 **** >>>> #define ECANCELED 85 /* Operation canceled */ >>>> #define EILSEQ 86 /* Illegal byte sequence */ >>>> #define ENOATTR 87 /* Attribute not found */ >>>> - #define ENODATA 87 /* Attribute not found */ >>>> >>>> #define EDOOFUS 88 /* Programming error */ >>>> #endif /* _POSIX_SOURCE */ >>>> --- 164,169 ---- >>>> >>>> I'll submit this as "bug" report and will see what comes of it. >>>> >>>> --WjW >>> >>> I too ran into this problem a few years ago. My solution was to patch >>> Ceph instead. Would these patches still work? >>> >>> --- src/include/compat.h.orig 2013-11-01 16:14:01.000000000 +0000 >>> +++ src/include/compat.h 2013-11-04 18:21:43.000000000 +0000 >>> @@ -13,7 +13,15 @@ >>> #define CEPH_COMPAT_H >>> >>> #if defined(__FreeBSD__) >>> -#define ENODATA 61 >>> +/* >>> + * FreeBSD does not have ENODATA. We must define it here. However, it MAY be >>> + * defined by boost. We can't simply include boost/cerrno.hpp here, because >>> + * that header is not includable by C code. So we must duplicate boost's >>> + * definition :( >>> + */ >>> +#ifndef ENODATA >>> +#define ENODATA 9919 >>> +#endif >>> #endif /* !__FreeBSD__ */ >>> >>> #if defined(__FreeBSD__) || defined(__APPLE__) >>> --- src/pybind/rados.py.orig 2013-11-04 17:36:06.000000000 +0000 >>> +++ src/pybind/rados.py 2013-11-04 17:37:02.000000000 +0000 >>> @@ -89,10 +89,14 @@ >>> errno.EIO : IOError, >>> errno.ENOSPC : NoSpace, >>> errno.EEXIST : ObjectExists, >>> - errno.ENODATA : NoData, >>> errno.EINTR : InterruptedOrTimeoutError, >>> errno.ETIMEDOUT : TimedOut >>> } >>> + # errno.ENODATA is not implemented on all platforms >>> + try: >>> + errors[errno.ENODATA] = NoData >>> + except AttributeError: >>> + pass >>> ret = abs(ret) >>> if ret in errors: >>> return errors[ret](msg) >>> --- src/pybind/cephfs.py.orig 2013-10-10 16:14:07.000000000 +0000 >>> +++ src/pybind/cephfs.py 2013-10-10 16:15:22.000000000 +0000 >>> @@ -49,8 +49,12 @@ >>> errno.EIO : IOError, >>> errno.ENOSPC : NoSpace, >>> errno.EEXIST : ObjectExists, >>> - errno.ENODATA : NoData >>> } >>> + # errno.ENODATA is not implemented on all platforms >>> + try: >>> + errors[errno.ENODATA] = NoData >>> + except AttributeError: >>> + pass >>> ret = abs(ret) >>> if ret in errors: >>> return errors[ret](msg) >> >> But it works just the other way around... >> >> There are calls to freebsd xattr() which return ENOATTR on error, >> the Linux code however uses ENODATA (in value == 87) to test for errors. >> So if anything needs to be defined ENODATA == ENOATTR. >> This I do in compat.h but defining it to 9919 would not help. >> Stronger, I test for it to not be 9919, or I Error compilation. >> >> And the trouble with all python code is that ignoring ENODATA is not always a wise decision. >> Just having it in /usr/include is the right place where python/cython can find it. >> >> And it would help other porting efforts would be helped as well. >> >> --WjW > > If the problem is that Linux's xattr() and FreeBSD's xattr() behave > differently, then the solution should be in the code that uses > xattr(), right? Something like this: > > err = extattr_get_fd(...) > #if defined( freebsd ) > if (err == ENOATTR) { > #else if defined (linux) > if (err == ENODATA) { > #endif > return CEPH_ENODATA > } > Sure that is a partial solution, but cython needs to be fixed as well. And that is a lot of place to fix it. After which there will be LUA problems, or any other interpreter that gets compiled. And code is again swamped with awkward If-endifs. ENODATA is currently not really defined, other than for C++ where it is only defined when it has not been defined in other headers. And then it gets something that is incompatible with what Linux thinks. So I think that it is "a good thing" (tm) if this could be part of errno.h. It might, will, help in other cases as well. Hence my patch, to me more compatible with Linux. And the standard gives room for that. But I understand that there a multiple views on the matter. --WjW From owner-freebsd-hackers@freebsd.org Wed Nov 23 03:22:14 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 268ECC4FB94 for ; Wed, 23 Nov 2016 03:22:14 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-yw0-x235.google.com (mail-yw0-x235.google.com [IPv6:2607:f8b0:4002:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D91E97C2 for ; Wed, 23 Nov 2016 03:22:13 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-yw0-x235.google.com with SMTP id a10so713740ywa.3 for ; Tue, 22 Nov 2016 19:22:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=N6VgoyoVm2BGwgGMO1QuwqJmPiu8ykvetr7NtKkYcsg=; b=j7vSwCGdxAssWF9s98IQ13J51DMgDkH3BICsN1ltVYYBnTNikT0OzpZcCi3ANkDlU+ vNGglSKFDgXfLtgX15cfCFfgM6wMkWeXSZplMxYlbrsuskdyrh0R47ftnTnctjjFSTcN CEfVbqqth5D2WlZtlqZ5MO1U2TZ2qUguvmID8j0NwU0oKU5ZMjqAQvVgZNAU1BRYn/GH ZhfVhE0Ya6gaflPs9TCfkP5b/2mBN7KWcNNLzf3Ih6RNGeniudxEteDq0kNHsnwk1soZ YmhE4yHbEf66w2kE6mozJWe2ygHZjCdbqFfvgGV5CJoW9uwvLeiyVt5x64iExpO8ySnJ /1jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=N6VgoyoVm2BGwgGMO1QuwqJmPiu8ykvetr7NtKkYcsg=; b=HQdo7zkk0imIvhqxDyBRncH+sCEFe3ncc3LlM4snj0XIheVEQ2ydMRzDVJ+o1f2Ozv ImQaUxlD4I+zGTTKe6fUggDKkHrd2furUJUESubzmaPGyl9riCJHWJmPAw2RLtfeDLJa 9JejmawGdIKeVGiRyOB/GkvI1DC2mNloJdTB4ntrxCV8RSaBPu32tvB2cbaZ36GbMnsO neS3ETdZ6QCnQ2rGCN7avWyrHA/B/lgvUUdNKRndAbGyG6ytp7cCVuTsZlVxOkndKTwa eQ41oyvnUVpbw20wxnQRvX6zkEg5QcnI8blude8dSIK3j7o8VgbCDudzgolxPsOWtDTW WYaA== X-Gm-Message-State: AKaTC01XqZKAOjR5fu4daKnEFhiPHA+BYyht6IHjjUiiJyFTjCoT1vyUDYfKL0XYMBWcuIV5lVvCys4S3JP/6w== X-Received: by 10.129.161.73 with SMTP id y70mr1097478ywg.190.1479871332188; Tue, 22 Nov 2016 19:22:12 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.171.239 with HTTP; Tue, 22 Nov 2016 19:22:11 -0800 (PST) From: Zaphod Beeblebrox Date: Tue, 22 Nov 2016 22:22:11 -0500 Message-ID: Subject: Two threads call vsyslog(), two threads in death embrace. To: FreeBSD Hackers Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Nov 2016 03:22:14 -0000 I'm just agitating... but bug 214482 seems to detail that two threads calling vsyslog() lead to a death embrace. I'm just posting to get some wider eyes or opinions on that. In the bug, I've kill -QUIT the mpd5 daemon and looked at it's corpse. I had to kill -QUIT because the daemon is completely unresponsive. Regular kills don't do anything. In the mpd5 binary, there are 3 threads. One is sitting in poll() and the other two are stuck asking for a lock inside the vsyslog() library function. From owner-freebsd-hackers@freebsd.org Wed Nov 23 13:12:59 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 95E9DC505C2 for ; Wed, 23 Nov 2016 13:12:59 +0000 (UTC) (envelope-from david@madTurtle.lapinbilly.eu) Received: from madTurtle.lapinbilly.eu (xvm-127-18.dc2.ghst.net [185.26.127.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "madTurtle", Issuer "madTurtle" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2FA02DEC for ; Wed, 23 Nov 2016 13:12:58 +0000 (UTC) (envelope-from david@madTurtle.lapinbilly.eu) Received: from madTurtle.lapinbilly.eu (localhost [127.0.0.1]) by madTurtle.lapinbilly.eu (8.15.2/8.15.2) with ESMTPS id uANCsDED092226 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 23 Nov 2016 12:54:13 GMT (envelope-from david@madTurtle.lapinbilly.eu) Received: (from david@localhost) by madTurtle.lapinbilly.eu (8.15.2/8.15.2/Submit) id uANCsDlS092225 for freebsd-hackers@freebsd.org; Wed, 23 Nov 2016 13:54:13 +0100 (CET) (envelope-from david) Date: Wed, 23 Nov 2016 13:54:13 +0100 From: David Marec To: freebsd-hackers@freebsd.org Subject: Re: RTL 8111G Message-ID: <20161123125412.GA92220@madTurtle.lapinbilly.eu> References: <582AEEAE.2030702@systella.fr> <58341ECB.8080605@systella.fr> <66187adb-2240-e63f-3800-b454cd4cbc0b@davenulle.org> <5834C6C2.4070102@systella.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5834C6C2.4070102@systella.fr> User-Agent: Mutt/1.7.1 (2016-10-04) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Nov 2016 13:12:59 -0000 On Tue, Nov 22, 2016 at 11:29:22PM +0100, BERTRAND Joël wrote: > > I don't understand your last sentences. What is the failure on > 'M_DONTWAIT' ? > Reading your first post, I was thinking about patching the realteak driver, as the flag 'M_DONTWAIT' is no more used when allocating mbufs. Starting from the last driver I found on the realteack website (1.92), here is a patch that make it build against FreeBSD 11-Release: --- if_re.c.orig 2016-11-23 13:36:22.034612000 +0100 +++ if_re.c 2016-11-23 13:38:15.458230000 +0100 @@ -61,6 +61,7 @@ #include #include #include +#include #include @@ -720,7 +721,7 @@ else size =MJUM9BYTES; for (i = 0; i < RE_RX_BUF_NUM; i++) { - sc->re_desc.rx_buf[i] = m_getjcl(M_DONTWAIT, MT_DATA, M_PKTHDR, size); + sc->re_desc.rx_buf[i] = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, size); if (!sc->re_desc.rx_buf[i]) { //device_printf(dev, "m_getcl fail!!!\n"); error = ENXIO; @@ -5432,7 +5433,7 @@ { struct mbuf *m_new = NULL; - m_new = m_defrag(m_head, M_DONTWAIT); + m_new = m_defrag(m_head, M_NOWAIT); if (m_new == NULL) { printf("re%d: no memory for tx list", sc->re_unit); @@ -5576,7 +5577,7 @@ sc->re_desc.tx_last_index = (sc->re_desc.tx_last_index+1)%RE_TX_BUF_NUM; txptr=&sc->re_desc.tx_desc[sc->re_desc.tx_last_index]; - ifp->if_opackets++; + if_inc_counter(ifp,IFCOUNTER_OPACKETS,1); ifp->if_drv_flags &= ~IFF_DRV_OACTIVE; } @@ -5654,7 +5655,7 @@ else size = MJUM9BYTES; - buf = m_getjcl(M_DONTWAIT, MT_DATA, M_PKTHDR, size); + buf = m_getjcl(M_NOWAIT, MT_DATA, M_PKTHDR, size); if (buf==NULL) { bError=1; goto update_desc; @@ -5719,7 +5720,7 @@ } eh = mtod(m, struct ether_header *); - ifp->if_ipackets++; + if_inc_counter(ifp,IFCOUNTER_IPACKETS,1); #ifdef _DEBUG_ printf("Rcv Packet, Len=%d \n", m->m_len); #endif @@ -5794,7 +5795,7 @@ #if OS_VER < VERSION(7,0) re_int_task(arg, 0); #else - taskqueue_enqueue_fast(taskqueue_fast, &sc->re_inttask); + taskqueue_enqueue(taskqueue_fast, &sc->re_inttask); return (FILTER_HANDLED); #endif @@ -5874,7 +5875,7 @@ #if OS_VER>=VERSION(7,0) if (CSR_READ_2(sc, RE_ISR) & RE_INTRS) { - taskqueue_enqueue_fast(taskqueue_fast, &sc->re_inttask); + taskqueue_enqueue(taskqueue_fast, &sc->re_inttask); return; } #endif I made few tests on the first machine that embedds this NIC and it seems to be working. -- David Marec From owner-freebsd-hackers@freebsd.org Thu Nov 24 21:51:57 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B803C54982 for ; Thu, 24 Nov 2016 21:51:57 +0000 (UTC) (envelope-from anastasios@mageirias.com) Received: from iserver.iopen.gr (iserver.iopen.gr [144.76.108.68]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D091613C4 for ; Thu, 24 Nov 2016 21:51:56 +0000 (UTC) (envelope-from anastasios@mageirias.com) Received: from localhost (localhost.localdomain [127.0.0.1]) by iserver.iopen.gr (Postfix) with ESMTP id C8DBB31003B6; Thu, 24 Nov 2016 23:51:47 +0200 (EET) X-Virus-Scanned: Debian amavisd-new at iserver.iopen.gr Received: from iserver.iopen.gr ([127.0.0.1]) by localhost (iserver.iopen.gr [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NoLVkXXl9kET; Thu, 24 Nov 2016 23:51:47 +0200 (EET) Received: from [192.168.1.2] (athedsl-4425452.home.otenet.gr [79.130.246.220]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: anastasios@mageirias.com) by iserver.iopen.gr (Postfix) with ESMTPSA id 1895531003A0; Thu, 24 Nov 2016 23:51:46 +0200 (EET) Subject: Re: Kernel fuzzing suite - any technical point of contact? To: Athanasios Kostopoulos , freebsd-hackers@freebsd.org References: From: Anastasios Mageirias Message-ID: Date: Thu, 24 Nov 2016 23:51:46 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Nov 2016 21:51:57 -0000 Hi Athanasios Probably this is what you are searching for. https://people.freebsd.org/~pho/stress/index.html Best Mageirias Anastasios On 11/21/2016 11:11, Athanasios Kostopoulos wrote: > Hi all, > "long time listener, first time caller". > > Is there a technical point of contact for the aforementioned project, as I > want to discuss some ideas. > > Thanks and Best Regards, > Athanasios > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@freebsd.org Sat Nov 26 10:28:00 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1485AC5574E for ; Sat, 26 Nov 2016 10:28:00 +0000 (UTC) (envelope-from tris_vern@hotmail.com) Received: from COL004-OMC3S3.hotmail.com (col004-omc3s3.hotmail.com [65.55.34.141]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CB7A89CB for ; Sat, 26 Nov 2016 10:27:59 +0000 (UTC) (envelope-from tris_vern@hotmail.com) Received: from AUS01-ME1-obe.outbound.protection.outlook.com ([65.55.34.137]) by COL004-OMC3S3.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Sat, 26 Nov 2016 02:26:53 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=MLEUw1gM3IpwNNsIiXV8PljCrJc39mjatakz2a7gaAI=; b=RtRHmt9U8wNgDbaYuATc0VOJXyok6OOa3g6yrX6QEZaFVL6hqzjPh/OZoQMED2L7YsfEvf33xogKCYyf9XvVDCHeUZhwythHAKNHVjHvNFunPaSKnw9S5nwQ093UmRp8y3sHwQeix2WXCoFVDEdExZSVOcan3jszHITQYJ7NiUDRjpnBj/EtvXDUL/8tZbSZHT7Q6qocG/uGDJ+HHh0KP52xKVfBgTcrsEL4Fo3Lf7arlBCv2J5Y6fLZy/rC9Lw0aZCMgkWNutP1jAt7oV0xpuj56UNMpEjjnQ0iHdkkj1XvZ7JPxKlj6XijbCHuIkU1+lTlp7RggAZ5cAd20wZHoQ== Received: from ME1AUS01FT008.eop-AUS01.prod.protection.outlook.com (10.152.232.54) by ME1AUS01HT010.eop-AUS01.prod.protection.outlook.com (10.152.232.115) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4; Sat, 26 Nov 2016 10:26:50 +0000 Received: from ME1PR01MB0546.ausprd01.prod.outlook.com (10.152.232.56) by ME1AUS01FT008.mail.protection.outlook.com (10.152.232.117) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4 via Frontend Transport; Sat, 26 Nov 2016 10:26:50 +0000 Received: from ME1PR01MB0546.ausprd01.prod.outlook.com ([10.162.68.22]) by ME1PR01MB0546.ausprd01.prod.outlook.com ([10.162.68.22]) with mapi id 15.01.0734.014; Sat, 26 Nov 2016 10:26:50 +0000 From: Tristan Verniquet To: freebsd hackers Subject: qsort switching to insertsort Thread-Topic: qsort switching to insertsort Thread-Index: AQHSR8zfjghRy3NqhkaB3zSchVE3aA== Date: Sat, 26 Nov 2016 10:26:50 +0000 Message-ID: Accept-Language: en-AU, en-US Content-Language: en-AU X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=hotmail.com; x-incomingtopheadermarker: OriginalChecksum:; UpperCasedChecksum:; SizeAsReceived:7238; Count:37 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [aGg+A/MuAofWIcirsUxQNiQZrscfv7l0] x-incomingheadercount: 37 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1; ME1AUS01HT010; 5:MierluyLBRA/9dHqSY9iq/qN81/2OOPHXUFFrDENRIUgKjTJpFMuaHBKRR2uNTHV3wUIoTmygpeY1lRiPTlCPx4TU/FLlb6nPeYKAZXUiMfgcutAYYfMEFjZGxGNZdxsPmwln/ZLNoR1FSDoNUDmPw==; 24:YsMEewgZp8Yc+tHJ+iXXkFslH4T7ro9PGPNBVK6WjDQSAU7EIUtIW4bQDiae4MaLEnQRm631VwlbEDVV9uoTHpNadsT5NsG/9e7/wG9rAFg=; 7:IuH4RG/rLm2P8DkroWzpcYn+GBwpjDEaABkG3YGK923JNzahX3b06LxfxDXBYIlOGw0IRy08qTGSnVQ5md0I4kV1J0myd3fAPgt9wRF4pk6QbChG2WcrCKL7BFfTs1nRKcCmcraPmpXgOm4hN0nGkLjjrvLARrd56xsLqr0eWfLQ7s2KWvJ2+eWA9qnKvp8mcPINGEmTpa2xsYO3dXol/I1tp1G0LZu9aHcfzAk7iPOkyXgmNAnuLGJx9ecGwRb6sf8l4EJErU8YaPjQvyuVXxDmdRgkeYEpbFEjtmgFdBovN0W04UitiUFw0BO7Hth+plLXJMRlFK5wFhaLfVGloUaUzfwKWw5S2hnP5/ue4E8= x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(10019020)(98900003); DIR:OUT; SFP:1102; SCL:1; SRVR:ME1AUS01HT010; H:ME1PR01MB0546.ausprd01.prod.outlook.com; FPR:; SPF:None; LANG:en; x-ms-office365-filtering-correlation-id: 47f2e4df-341b-4f75-aeed-08d415e6bc00 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(1601124038)(1603103113)(1603101340)(1601125047); SRVR:ME1AUS01HT010; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(432015012)(82015046); SRVR:ME1AUS01HT010; BCL:0; PCL:0; RULEID:; SRVR:ME1AUS01HT010; x-forefront-prvs: 0138CD935C spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Nov 2016 10:26:50.4613 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: ME1AUS01HT010 X-OriginalArrivalTime: 26 Nov 2016 10:26:53.0518 (UTC) FILETIME=[9B391EE0:01D247CF] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 10:28:00 -0000 We recently hit an issue with qsort where it was taking 2 minutes to sort 5= 00k entries. Changing to mergesort was comparatively instantaneous (which i= s what was expected). This was trusted data and shouldn't have been failing= due to pivot choice. After investigation, we discovered we'd been caught by a quirk of FreeBSD's= qsort where it will switch to insertsort (even for the whole array) if a p= ass matches a very simplistic heuristic of doing no swaps. This has previously been written up about in the following links: http://calmerthanyouare.org/2013/05/31/qsort-shootout.html and http://calmerthanyouare.org/2014/06/11/algorithmic-complexity-attacks-and-l= ibc-qsort.html The articles above focus on the security aspect of quicksort, and that all = implementations are vulnerable, however what seems to be missed in the Free= BSD case is how simple it is to hit this (really quite bad) worst case scen= ario with everyday data, as we did. This discovery is enough for us to not = want to use this version of qsort. FreeBSD does provide heapsort and mergesort as alternatives, but not merges= ort_r(). Also, as a developer, having these options gives the impression th= at choosing qsort will give me a quicksort-like implementation with the tra= de-offs that come with that, but it is not obvious that I might be given in= sertsort. The easiest way forward for us is probably to comment out the offending cod= e. But I haven't been able to find much discussion on it. I'm not sure how wel= l known the quirk is. I'm not sure of the rationality for it in the first p= lace (obviously a speedup, but whether it was considered alongside the down= falls), or what other peoples opinions are. So I thought I'd ask. I've done up a simple test case that is similar to our original usage that = shows why we were likely to always trigger an insertsort. It has 2 sort fie= lds in a 1->N relationship with the second fields sorted within the first, = but the topmost fields slightly out of order. I imagine there are lots of o= ther "almost sorted" scenarios that could end up being quite bad. http://tverniquet.com/whirlpool/test_qsort/test_qsort.c Regards, Tristan From owner-freebsd-hackers@freebsd.org Sat Nov 26 10:51:37 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A726EC567E4 for ; Sat, 26 Nov 2016 10:51:37 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (heidi.turbocat.net [88.198.202.214]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6F8C5C6E for ; Sat, 26 Nov 2016 10:51:36 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.129.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id AD2891FE024; Sat, 26 Nov 2016 11:51:34 +0100 (CET) Subject: Re: qsort switching to insertsort To: Tristan Verniquet , freebsd hackers References: From: Hans Petter Selasky Message-ID: <0bfb49b0-5d24-2766-6982-b4e49b0d5e81@selasky.org> Date: Sat, 26 Nov 2016 11:51:18 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 10:51:37 -0000 On 11/26/16 11:26, Tristan Verniquet wrote: > The easiest way forward for us is probably to comment out the offending code. > Commenting out the offending code does not help. It simply leaves for another type of dataset to provide the same behaviour. qsort() is doomed in this regard. --HPS From owner-freebsd-hackers@freebsd.org Sat Nov 26 12:28:37 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F3A7C5512B for ; Sat, 26 Nov 2016 12:28:37 +0000 (UTC) (envelope-from tris_vern@hotmail.com) Received: from BAY004-OMC3S26.hotmail.com (bay004-omc3s26.hotmail.com [65.54.190.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11F40F59 for ; Sat, 26 Nov 2016 12:28:36 +0000 (UTC) (envelope-from tris_vern@hotmail.com) Received: from AUS01-SY3-obe.outbound.protection.outlook.com ([65.54.190.187]) by BAY004-OMC3S26.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Sat, 26 Nov 2016 04:27:31 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=2O2ChWrLvboFNLjQDj2Jw8OW2l1KjHV251Pm4VCAvkc=; b=Z6WEXsD5NThM62lE4CM88cNLzBkfKbD5MINXhmRRUaauZmrNy4iQ9s8ZMrXTYy/hYj/zWkrAZssBWMuVtRmzUk/VVBOMwQmg6uEbw/iBJ/dSxGZ0H05Nvdmj5VOdQayQpABF9qGsrWaRlTI8Pq5m6q0THEujsvU1lFsBpscamqmC2Bji7V9Ahz69BxxCPpS2sLRnW+yig6v1a/TcZTT3U5BiXR3s9kQwGiLL3/vXJr0q9L1YfR/LWlstelN0RXmV0tFFifbQubA8YeHcLCJkxzqDmpkOUt+C+xesTOgaotEjZsfAYzGgDmgWEmjIDk6Xv4LARZQWjaGHikuTjItKXA== Received: from SY3AUS01FT009.eop-AUS01.prod.protection.outlook.com (10.152.234.58) by SY3AUS01HT006.eop-AUS01.prod.protection.outlook.com (10.152.234.97) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4; Sat, 26 Nov 2016 12:27:09 +0000 Received: from ME1PR01MB0546.ausprd01.prod.outlook.com (10.152.234.51) by SY3AUS01FT009.mail.protection.outlook.com (10.152.234.103) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4 via Frontend Transport; Sat, 26 Nov 2016 12:27:09 +0000 Received: from ME1PR01MB0546.ausprd01.prod.outlook.com ([10.162.68.22]) by ME1PR01MB0546.ausprd01.prod.outlook.com ([10.162.68.22]) with mapi id 15.01.0734.014; Sat, 26 Nov 2016 12:27:09 +0000 From: Tristan Verniquet To: Hans Petter Selasky , freebsd hackers Subject: Re: qsort switching to insertsort Thread-Topic: qsort switching to insertsort Thread-Index: AQHSR8zfjghRy3NqhkaB3zSchVE3aKDrFm4AgAAXcRo= Date: Sat, 26 Nov 2016 12:27:09 +0000 Message-ID: References: , <0bfb49b0-5d24-2766-6982-b4e49b0d5e81@selasky.org> In-Reply-To: <0bfb49b0-5d24-2766-6982-b4e49b0d5e81@selasky.org> Accept-Language: en-AU, en-US Content-Language: en-AU X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: selasky.org; dkim=none (message not signed) header.d=none;selasky.org; dmarc=none action=none header.from=hotmail.com; x-incomingtopheadermarker: OriginalChecksum:; UpperCasedChecksum:; SizeAsReceived:7495; Count:39 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [+5wImb4miwUI5grgp7sRocSDzhqcaUKt] x-incomingheadercount: 39 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1; SY3AUS01HT006; 5:G5k3CLhTRbCGAc/1kTC36Lo7UQTj0iWx2ST/O6jn29Ln0x3QYXgkr73Vfj6Oq4k3W/W2I1vFiTSOpLYLPmN3HbOxNJb/wkH20LF8TlAB4oxOh927kVqqk0FO7sRpQb1dKfQd/L5Vrg8Nhpb1mpuzDi9QNwmyjhyQ3kXvxl4rWFk=; 24:K44oe/k8qufHkv5c9euKZ/mgtQmBU8QpVFFJvlJTJqmizjUCg2E9B/ZI/uwXNAwQ6Yr2jvxwhsvUtU5v9brezKaUchN/dIComC7eMzT+jYA=; 7:167kFFytTVSb9OOUCm9/NmlEDm59HGQiBqnn4+yFY++ZoPrIpy5J/lAYI7r7vN1VxS52bFFRdhjAUOxm/a19dRjcjTeqN829cJtdCxiTX0xfwvFrPpKiMO3310vSDChkN9Xw5nXPVS7FFOifYzcnagM0GU+7VOgBBidx90F5zm7ogJTQu32SIaNRrtED5qEC/T5ZjqfEewsOy33uvr4UD1f7X3PdYXi6BxzYJf3YFuzDB8ptti3frBIy2Omc/WaS/740QFO3xTaMNXNCASw0ARrBY7mgMHGVP4rMQQXZ1Jf1e3Z6UNNWxr+txHN8b4DRlu23XCnSUYwkCIyWh7oj5EiEwLwBSvLjMzQ6UySGAHI= x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(10019020)(98900003); DIR:OUT; SFP:1102; SCL:1; SRVR:SY3AUS01HT006; H:ME1PR01MB0546.ausprd01.prod.outlook.com; FPR:; SPF:None; LANG:en; x-ms-office365-filtering-correlation-id: 92b1dd91-73ca-4a4f-7c6b-08d415f78ae9 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(1601124038)(1603103113)(1603101340)(1601125047); SRVR:SY3AUS01HT006; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(432015012)(82015046); SRVR:SY3AUS01HT006; BCL:0; PCL:0; RULEID:; SRVR:SY3AUS01HT006; x-forefront-prvs: 0138CD935C spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Nov 2016 12:27:09.7235 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SY3AUS01HT006 X-OriginalArrivalTime: 26 Nov 2016 12:27:32.0086 (UTC) FILETIME=[75BEF160:01D247E0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 12:28:37 -0000 > From: Hans Petter Selasky > Sent: Saturday, 26 November 2016 8:51 PM > To: Tristan Verniquet; freebsd hackers > Subject: Re: qsort switching to insertsort > =20 > On 11/26/16 11:26, Tristan Verniquet wrote: >> The easiest way forward for us is probably to comment out the offending = code. >> >=20 > Commenting out the offending code does not help. It simply leaves for=20 > another type of dataset to provide the same behaviour. qsort() is doomed= =20 > in this regard. >=20 > --HPS I can see that from, say, a security perspective, as long as a worst-case e= xists you would assume it, and so this would make no difference. But from an everyday usage where security is not such an issue, I see the t= wo worst-case triggers as being in different ball park. I would happily ass= ume I'd never meet an accidental case of triggering a qsort worst-case base= d on pivot given the pivot selection method it uses, but can no longer have= that confidence with qsort triggering an insertsort. I was kind of suspecting that this might be the reasoning behind it. For ex= ample the second link shows problems with all quicksorts. But do you not th= ink this makes a big difference in the everyday use case where qsort would = actually be used (and not avoided)? Tristan= From owner-freebsd-hackers@freebsd.org Sat Nov 26 14:35:55 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E030C56076 for ; Sat, 26 Nov 2016 14:35:55 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-yw0-x22f.google.com (mail-yw0-x22f.google.com [IPv6:2607:f8b0:4002:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DF88DE3C for ; Sat, 26 Nov 2016 14:35:54 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-yw0-x22f.google.com with SMTP id t125so82359220ywc.1 for ; Sat, 26 Nov 2016 06:35:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ZAuQZdoIt8qkJ8pSEUDAASmO3HCWc3cd8m8Ozh46rPc=; b=XZFv+Rd4eJ6HjzxQbrU2HUPaV7Djp4j2SjRqOlV5MCoI80p9Ca66m++hTNtfgwNIW4 rloBZKCnUG7BA7bqjASd+GffsEiI2+6qe+0WR9zcCcAD5nNSvvCXErzN8SEaWCQKqncn lxts8t2nKV2l+Aq+YPMm5v1ZgTzHwqG9L9eqbvpPRd54slROsx5iA9QVyUo/QKRiIWwz b7Fa++GKrhJnZSk1YGs7+aE8npNFT9+SfRN172AoPlnCbc2qLkYimYjEanbMUr98KgQB LdK2SDBuMHwKZxXbU9V7q/pZqWs2V1LoFLH0vX3/0Zzm/01cn8K8/86b3xILYjDSE3Rp 2yEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ZAuQZdoIt8qkJ8pSEUDAASmO3HCWc3cd8m8Ozh46rPc=; b=ONXc3GCsFNdiP6HtFfd50WY3ZPDtCprsm+bcdCZe8SoUjg99Qq1e5EhAjWa+0f9med jDuv6xfih0ACDAL1KlawaJd98QNuGuhCP5wP5VlX56GNaWF/vxfR0Dta+Z6vXBO1VfD+ rAfNKz5u+ogb+rsfnGUok/Utxdblf84pDYTAJ1thiyZuseULZkOK5Axj3DHNnaTtE0Hl 12KDu68TIWeP0bvUMpxNUrG6qj0TvqHdrfPuknPnHb9I58sDNSt2CB2KRs7hBlXLNj9w fEk+lIp0mxfmlIRVmITnnox0DNQjerMAVG6C6QVTOAjxHNYY/JX7EX0WFk5kO3GjEuzU VxLw== X-Gm-Message-State: AKaTC01BoHUvEBKnQar/eLfyL/2seM1hDHwh9oU1vCnbWtkF8TK9Mj809YiTASphQ/DVTcDYXDwxPW1R2KqJ4A== X-Received: by 10.129.0.67 with SMTP id 64mr16033190ywa.61.1480170953978; Sat, 26 Nov 2016 06:35:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.170.67 with HTTP; Sat, 26 Nov 2016 06:35:53 -0800 (PST) In-Reply-To: References: <0bfb49b0-5d24-2766-6982-b4e49b0d5e81@selasky.org> From: Mehmet Erol Sanliturk Date: Sat, 26 Nov 2016 06:35:53 -0800 Message-ID: Subject: Re: qsort switching to insertsort To: Tristan Verniquet Cc: Hans Petter Selasky , freebsd hackers Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 14:35:55 -0000 On Sat, Nov 26, 2016 at 4:27 AM, Tristan Verniquet wrote: > > From: Hans Petter Selasky > > Sent: Saturday, 26 November 2016 8:51 PM > > To: Tristan Verniquet; freebsd hackers > > Subject: Re: qsort switching to insertsort > > > > On 11/26/16 11:26, Tristan Verniquet wrote: > >> The easiest way forward for us is probably to comment out the offending > code. > >> > > > > Commenting out the offending code does not help. It simply leaves for > > another type of dataset to provide the same behaviour. qsort() is doomed > > in this regard. > > > > --HPS > > I can see that from, say, a security perspective, as long as a worst-case > exists you would assume it, and so this would make no difference. > > But from an everyday usage where security is not such an issue, I see the > two worst-case triggers as being in different ball park. I would happily > assume I'd never meet an accidental case of triggering a qsort worst-case > based on pivot given the pivot selection method it uses, but can no longer > have that confidence with qsort triggering an insertsort. > > I was kind of suspecting that this might be the reasoning behind it. For > example the second link shows problems with all quicksorts. But do you not > think this makes a big difference in the everyday use case where qsort > would actually be used (and not avoided)? > > Tristan > _______________________________________________ > > In quick sort , it is necessary to check worst case and switch to another sort method if it is detected . When this is not done , end result is stack overflow , etc . , but not success . Therefore , it is not possible to eliminate an alternative sort method . Mehmet Erol Sanliturk From owner-freebsd-hackers@freebsd.org Sat Nov 26 15:11:25 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C0FC2C56C23 for ; Sat, 26 Nov 2016 15:11:25 +0000 (UTC) (envelope-from tris_vern@hotmail.com) Received: from COL004-OMC3S10.hotmail.com (col004-omc3s10.hotmail.com [65.55.34.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "Microsoft IT SSL SHA2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7EEACD4 for ; Sat, 26 Nov 2016 15:11:24 +0000 (UTC) (envelope-from tris_vern@hotmail.com) Received: from AUS01-SY3-obe.outbound.protection.outlook.com ([65.55.34.137]) by COL004-OMC3S10.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Sat, 26 Nov 2016 07:10:18 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hotmail.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=pI410SNROA72UBPc7+5CAxiAeJTs94PJfJQLc22pUn0=; b=Ulnuldzzmb0UzA2VYrkHbJQnVZfB5JhTNWl9v99/RarQWFnBdnhGiCCM6sFo9A/OLUwz0lHSRH7fyIZNaLF2xjJejIEJSqJjz9qaeCuAO4PUUK8FDt5g+p2YMz++jaQHIEPI4+AJiPwZcUwQOpQ/OChZyFBqegp4t2CX0WGStdSxOH38wFefX06iCFG8vUTeK2VmafBFXGQz4zg3NLrx1o7e8svmKDXMpxOYbd+LBPTacmbzcm/Bv+373rH3jAHQ3ruPZkWS5LUGyrDhcUhvUdHRlfVwU4CwsHDjBmC9b4GnkUw6PcnIR5FTXAlEwHwY0IwianVioiBdLS0Caa20kQ== Received: from SY3AUS01FT004.eop-AUS01.prod.protection.outlook.com (10.152.234.52) by SY3AUS01HT011.eop-AUS01.prod.protection.outlook.com (10.152.234.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4; Sat, 26 Nov 2016 15:10:16 +0000 Received: from ME1PR01MB0546.ausprd01.prod.outlook.com (10.152.234.57) by SY3AUS01FT004.mail.protection.outlook.com (10.152.234.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384) id 15.1.734.4 via Frontend Transport; Sat, 26 Nov 2016 15:10:15 +0000 Received: from ME1PR01MB0546.ausprd01.prod.outlook.com ([10.162.68.22]) by ME1PR01MB0546.ausprd01.prod.outlook.com ([10.162.68.22]) with mapi id 15.01.0734.014; Sat, 26 Nov 2016 15:10:16 +0000 From: Tristan Verniquet To: freebsd hackers Subject: Re: qsort switching to insertsort Thread-Topic: qsort switching to insertsort Thread-Index: AQHSR8zfjghRy3NqhkaB3zSchVE3aKDrXEzK Date: Sat, 26 Nov 2016 15:10:16 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-AU, en-US Content-Language: en-AU X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: freebsd.org; dkim=none (message not signed) header.d=none;freebsd.org; dmarc=none action=none header.from=hotmail.com; x-incomingtopheadermarker: OriginalChecksum:; UpperCasedChecksum:; SizeAsReceived:7433; Count:39 x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [ZSPwKYqgwOGO7Ek3xZQUnIMMxCqqj+p5] x-incomingheadercount: 39 x-eopattributedmessage: 0 x-microsoft-exchange-diagnostics: 1; SY3AUS01HT011; 5:3PEweQVCcvQ0p0TcERHouLzAqsen1MnEie9825/Mk0brRvuioNONKUfRw04TB7bfAFiBJq2+MZIyGeQDkVxppvTRkWd9zWz3+ILzWd+meH/d/3QBp41aD4wxOJtCjkhesxm10uA0PTKOhPYXgRtNqQGEHQ/FpsSMhk+pieL8YXI=; 24:e0Qml8l1+irty4WZLKtjzVof1cYT5p1RMnuMRVcxz1Wty+bAb1JJU9MneE0lYBJlrg0D5Pgdhctf1tCjnPFJsNkyGT9iFcCKjxDn8dfR+fM=; 7:7Nfor8JXQokBRtdkUKBkMtXblJjf8SNWnVOSWB7tZ7Ivl2RaB6Y6RiWqx3S+0z3Zs3+TkjNfNL+NfBwn+oeVltqqKXiJRYzn4HZCeruH7lDmNWaD4kdOZiTC+m5da5udGXJMy8Apu1a/5MAbtE17Al0C2R7s2CgRtQWKN6fR+YJZgobPh+KKU8HA1rcJ4A95yKBWApGWAlmYZ2+R0nBN7XtsxwbSIICA0eu9F/fdW+oK4tPQ6WbmX4R44BUXrxZS/aRoc4U7I5ka4BCPkltKiEClBdqCAlgla4WOU/HfrDT9mY3ElJrszoYAwpD6yAIVeSuUplkTjz4fKuUi+dKCikvnennF1Oz3Cqq4muxQQwY= x-forefront-antispam-report: EFV:NLI; SFV:NSPM; SFS:(10019020)(98900003); DIR:OUT; SFP:1102; SCL:1; SRVR:SY3AUS01HT011; H:ME1PR01MB0546.ausprd01.prod.outlook.com; FPR:; SPF:None; LANG:en; x-ms-office365-filtering-correlation-id: d3ea8508-a009-4742-e039-08d4160e5411 x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(1601124038)(1603103113)(1603101340)(1601125047); SRVR:SY3AUS01HT011; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(432015012)(82015046); SRVR:SY3AUS01HT011; BCL:0; PCL:0; RULEID:; SRVR:SY3AUS01HT011; x-forefront-prvs: 0138CD935C spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: hotmail.com X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Nov 2016 15:10:16.0792 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Internet X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SY3AUS01HT011 X-OriginalArrivalTime: 26 Nov 2016 15:10:18.0557 (UTC) FILETIME=[330426D0:01D247F7] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 15:11:25 -0000 > From: owner-freebsd-hackers@freebsd.org on behalf of Tristan Verniquet > Sent: Saturday, 26 November 2016 8:26 PM > To: freebsd hackers > Subject: qsort switching to insertsort >=20 > But I haven't been able to find much discussion on it. I'm not sure how w= ell known the quirk is. I'm not sure of the rationality for it in the first= place (obviously a speedup, but whether it was considered alongside the do= wnfalls), or what other peoples opinions are. So I thought I'd ask. > Now that I've emailed I did find some previous discussion about it (linked = from the links I posted - I had originally tried searching the discussion l= ists with no success): http://www.mail-archive.com/freebsd-hackers@freebsd.org/msg155127.html The chain just starts positively but just seems to drift off.. Tristan= From owner-freebsd-hackers@freebsd.org Sat Nov 26 16:38:34 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51300C57B12 for ; Sat, 26 Nov 2016 16:38:34 +0000 (UTC) (envelope-from emorrasg@yahoo.es) Received: from nm29-vm6.bullet.mail.ir2.yahoo.com (nm29-vm6.bullet.mail.ir2.yahoo.com [212.82.97.75]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C468F15D for ; Sat, 26 Nov 2016 16:38:33 +0000 (UTC) (envelope-from emorrasg@yahoo.es) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.es; s=s2048; t=1480178198; bh=C1ctEnpFGUeowiS+S8PvrsLCxXnHDFw9tbkXeT5NGVA=; h=Date:From:To:Subject:In-Reply-To:References:From:Subject; b=XGFu45SOBEWrda66E5Pc9EQYsHiFa/Krw30cSFhK/U3HHH/vm9jndFpEWSSXwGPiZ/Wjf9J+wEBg6CpX3TnIW6ycvTCTVI1ClPXNo6mn9aLf9Pjv89aOEzLE1ox/Ij+BEdOxtV+SZrFCZevgZ8qn5gOvGRvb1dMJls6qOP+sba6MwFi8QXi9ucRqzFlc8ICbIR5vPDC2nu87DroCVGiV0yrAdYmMT2/XH7foGEqaEF4t1HOJcdoWRita8nUfkx+Ji31h8KIvVSsKxM79YGRVA28Y9e5U2tPoc49Eq7kifhvjmVEAV8SBzlXa1a4t00PcoDLEFO3Z+3VPrm32ryxsow== Received: from [212.82.98.49] by nm29.bullet.mail.ir2.yahoo.com with NNFMP; 26 Nov 2016 16:36:38 -0000 Received: from [46.228.39.71] by tm2.bullet.mail.ir2.yahoo.com with NNFMP; 26 Nov 2016 16:36:38 -0000 Received: from [127.0.0.1] by smtp108.mail.ir2.yahoo.com with NNFMP; 26 Nov 2016 16:36:38 -0000 X-Yahoo-Newman-Id: 457253.52451.bm@smtp108.mail.ir2.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: meLUhngVM1nZ2nOCy6jvv0xtt4hhr2bBrTd_IK89CvbkB7J tM4SyK2MNmcFCDwW2Ytj8zUZNYe2AnMQ.j.z8pbT9U1vazoSrySFIcsXSNY9 cYmQJBhe.b_LxTvLIzkxrd7sj0a7jGgaBlfO4F65lIKA__iAuwWtA_NP.2QW 2wQajzMHI5koASw74Sh7YxcllrSBhStvo_FFt8VAYEDYXCNTdvifZHrYF8Rb B8VrBlqLY8gXPKhR4iaQoPJMm0I4GdGDcfB8.n4hYwFl5llTVJ690N03.uc5 7LKErQm6OgTkeSfIcmL8tJZiWQ_l5KpSY4RI1Y.3SdNbR6lKJiJGlj8l7MI7 azrIZFxBb_ChIxAARgcBMkJZjyGYzbroQAh5XDl7Rd.SPaMd47Ng09IvxGa0 cQ6PietKBHFt1RjxEORyH2xPX.RRbannk19mjuKvu6sC59w8m1dJcjCUuXnE 7HvXwVoJK5xvPIEd2ioden6bPDgYcZD_yS6iSqLfLeqmm4gr9_Wwd2khBn9C sjzFJY6JsrW5t9bb9Jw6znTDERXkyMAANbY3tOIk7QCBZIohq2g-- X-Yahoo-SMTP: mX392iiswBAeJNdO_s.EW62LZDJR Date: Sat, 26 Nov 2016 17:36:11 +0100 From: Eduardo Morras To: freebsd-hackers@freebsd.org Subject: Re: qsort switching to insertsort Message-Id: <20161126173611.52a55651c31beedf51f8cf71@yahoo.es> In-Reply-To: References: <0bfb49b0-5d24-2766-6982-b4e49b0d5e81@selasky.org> X-Mailer: Sylpheed 3.5.1 (GTK+ 2.24.29; amd64-portbld-freebsd11.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 16:38:34 -0000 On Sat, 26 Nov 2016 06:35:53 -0800 Mehmet Erol Sanliturk wrote: > In quick sort , it is necessary to check worst case and switch to > another sort method if it is detected . > When this is not done , end result is stack overflow , etc . , but not > success . > Therefore , it is not possible to eliminate an alternative sort > method . And restricting the alternative sort for less than 40 elements (or other number)? It's common implementation to fallback to other sort algorithm, but doing so without restricting the number of elements can cause on corner cases, sorting all elements with alternative sort, which is bad. > > Mehmet Erol Sanliturk --- --- Eduardo Morras From owner-freebsd-hackers@freebsd.org Sat Nov 26 17:14:13 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20C0CC579D8 for ; Sat, 26 Nov 2016 17:14:13 +0000 (UTC) (envelope-from adf@idi.ntnu.no) Received: from hylle05.itea.ntnu.no (hylle05.itea.ntnu.no [IPv6:2001:700:300:3::225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D6F121BF2 for ; Sat, 26 Nov 2016 17:14:12 +0000 (UTC) (envelope-from adf@idi.ntnu.no) Received: from localhost (localhost [127.0.0.1]) by hylle05.itea.ntnu.no (Postfix) with ESMTP id A147A907151; Sat, 26 Nov 2016 18:14:08 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at hylle05.itea.ntnu.no Received: from [10.10.10.166] (unknown [109.201.137.162]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: adf) by hylle05.itea.ntnu.no (Postfix) with ESMTPSA id CE8EE907134; Sat, 26 Nov 2016 18:14:07 +0100 (CET) Subject: Re: qsort switching to insertsort Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Content-Type: text/plain; charset=utf-8 From: =?utf-8?Q?Arne_Dag_Fidjest=C3=B8l?= In-Reply-To: Date: Sat, 26 Nov 2016 18:14:06 +0100 Cc: Tristan Verniquet , Hans Petter Selasky , freebsd hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <0bfb49b0-5d24-2766-6982-b4e49b0d5e81@selasky.org> To: Mehmet Erol Sanliturk X-Mailer: Apple Mail (2.3124) X-Mailman-Approved-At: Sat, 26 Nov 2016 17:45:21 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 17:14:13 -0000 > On 26 Nov 2016, at 15:35, Mehmet Erol Sanliturk = wrote: >=20 > In quick sort , it is necessary to check worst case and switch to = another > sort method if it is detected . > When this is not done , end result is stack overflow , etc . , but not > success . > Therefore , it is not possible to eliminate an alternative sort method = . I you sort the smaller partition recursively first, and then sort the = larger partition either by tail recursion or iteration, you will only = consume O(log n) of stack, so stack overflow needn=E2=80=99t be an = issue, regardless of the input. -adf= From owner-freebsd-hackers@freebsd.org Sat Nov 26 19:51:33 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 93A98C574E8 for ; Sat, 26 Nov 2016 19:51:33 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-yw0-x233.google.com (mail-yw0-x233.google.com [IPv6:2607:f8b0:4002:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 484B21AEA for ; Sat, 26 Nov 2016 19:51:33 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: by mail-yw0-x233.google.com with SMTP id a10so85946349ywa.3 for ; Sat, 26 Nov 2016 11:51:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=poaAatZsQKPwa6JhbCqyETERRR8kAj905scUOC0Yu4c=; b=woWoPY53SYTTUu6paARrWPOQVX1YzTaL9DXuTR90xJgosC4RHsRbIM4pPmnqKQvqIe c8THam+GAXHh3ziObN6SfGku9dhWqM85RLjKO9sAVzW9XWjUWhM2Q/w+dAbCezhOgewT qn+Ptp/otvQ2dAbAkahAjforWVDRpBCzcWhDobqA2J9QpCy6vMCApjOlomQm4Beyi3yi cltgtSNb46oW0l0yQSibgav/7V9O5XjyXoKDxHNnWY/eI72XLiRPWMQNlxghdUJ1AydP ALv9dYhIeVkZg+bXgIaWm7kpU+D2ClOPWtTFTcAQ/s7DsxeFVgun2dxMoXpsHHDCebyi WFLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=poaAatZsQKPwa6JhbCqyETERRR8kAj905scUOC0Yu4c=; b=imAsSt92yx5p0iOGuyE7UqBnLHvu8abzqqONtTkb5TtWIhA/4m3uzoxPc6GZ11FT6P RVZSNIZ+Fi8mqWJXpLqicOjX7QDZxfq83UgnQoNdLQmBEkdIzfXVbBGePofrjNX3uAEt KZ76kFC6IidrZ0VnqtMNyNszc1N5jzQj1m9kLtj5h2I94fllvX/MrHO1ckRXUAciQXjj rgXj5fcyI1AQQUlMKbCtbKIpwNC88frtwExuuAtSEbQaTddkh/1dkDd47u93qb6ohKUg 5bselH8CsP7b+yTx4zGIHWI0MCBzHugd/Fe4GOlj39i3yd+cHNaFOr+ZRiaMeJFGNc52 ESlg== X-Gm-Message-State: AKaTC021sMD2+wOyxTetyCbvE6y6UcCu/V3DronR36jy8UJ3Xfez1EAnW7CsROmnR8mF1UTngNr4c8CIPCkPlg== X-Received: by 10.129.206.2 with SMTP id t2mr19625565ywi.181.1480189892347; Sat, 26 Nov 2016 11:51:32 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.170.67 with HTTP; Sat, 26 Nov 2016 11:51:31 -0800 (PST) In-Reply-To: References: <0bfb49b0-5d24-2766-6982-b4e49b0d5e81@selasky.org> From: Mehmet Erol Sanliturk Date: Sat, 26 Nov 2016 11:51:31 -0800 Message-ID: Subject: Re: qsort switching to insertsort To: =?UTF-8?Q?Arne_Dag_Fidjest=C3=B8l?= Cc: Tristan Verniquet , Hans Petter Selasky , freebsd hackers Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Nov 2016 19:51:33 -0000 On Sat, Nov 26, 2016 at 9:14 AM, Arne Dag Fidjest=C3=B8l = wrote: > > > On 26 Nov 2016, at 15:35, Mehmet Erol Sanliturk > wrote: > > > > In quick sort , it is necessary to check worst case and switch to anoth= er > > sort method if it is detected . > > When this is not done , end result is stack overflow , etc . , but not > > success . > > Therefore , it is not possible to eliminate an alternative sort method = . > > I you sort the smaller partition recursively first, and then sort the > larger partition either by tail recursion or iteration, you will only > consume O(log n) of stack, so stack overflow needn=E2=80=99t be an issue, > regardless of the input. > > -adf Important problem is caused by almost sorted values . Myself , I am counting the sorted elements first , if they exceed a large percentage of number of all elements , then I am switching to an alternative sort , otherwise a quick sort is used . This is for a simple application . In an operating system , more complex algorithms may be more useful . Mehmet Erol Sanliturk