From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 06:56:03 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C1055106566C for ; Mon, 23 Apr 2012 06:56:03 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 89F3C8FC15 for ; Mon, 23 Apr 2012 06:56:03 +0000 (UTC) Received: by obqv19 with SMTP id v19so17465213obq.13 for ; Sun, 22 Apr 2012 23:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=MFCw7DAWBYtbw+J4oXzrbt787AC6kyklN7A1sGodMNM=; b=b87ZjzG8mFHLrbct5sMtJirUI9kdvMEt56LgjncyYuUUmwsA3TcyMzeaux4okn/Fii 8P8U4SV2JJAW9SDQYtHdl4/hhxiimVFT/xShhtMsgLO9JueSGkelI4IXqxPhJn7lPyUg eZWvnXtL4z5l47F9xOdpvOh03lVUtkZ7apW4Ays2RsIm2sggye61q0Wo0cKJEwaqf7IT c0yxY+0gIhPZYnAZHIrgErcYne3Dt6WLT2YR3SF0U4RBA1TuwjibQgYsgJRxG0n/9MNZ gI/0FEKmAATDP5XGYX+YrLJW7/H8ZOaBk7pkc2Z9vmPtpl/9ZgbORC0s8uUkqLjirX1B iekw== MIME-Version: 1.0 Received: by 10.182.232.38 with SMTP id tl6mr5975973obc.16.1335164163204; Sun, 22 Apr 2012 23:56:03 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Sun, 22 Apr 2012 23:56:03 -0700 (PDT) Date: Mon, 23 Apr 2012 14:56:03 +0800 Message-ID: From: Fengwei yin To: freebsd-threads@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: jack.ren@intel.com Subject: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 06:56:03 -0000 Hi list, If this is not correct question on the list, please let me know and sorry for noise. I have a question regarding the BSD libc for SMP arch. I didn't see memory barrier used in libc. How can we make sure it's safe on SMP arch? BTW, there are memory barrier used in kernel instead. So I suppose the memory barrier is needed. Regards Yin, Fengwei From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 08:40:58 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F784106564A for ; Mon, 23 Apr 2012 08:40:58 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 551B28FC14 for ; Mon, 23 Apr 2012 08:40:58 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1SMEpk-000DXV-Cj; Mon, 23 Apr 2012 12:41:20 +0400 Date: Mon, 23 Apr 2012 12:41:20 +0400 From: Slawa Olhovchenkov To: Fengwei yin Message-ID: <20120423084120.GD76983@zxy.spb.ru> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 08:40:58 -0000 On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > Hi list, > If this is not correct question on the list, please let me know and > sorry for noise. > > I have a question regarding the BSD libc for SMP arch. I didn't see > memory barrier used in libc. > How can we make sure it's safe on SMP arch? /usr/include/machine/atomic.h: #define mb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") #define wmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") #define rmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 09:32:25 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF931106566C for ; Mon, 23 Apr 2012 09:32:25 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 940708FC08 for ; Mon, 23 Apr 2012 09:32:25 +0000 (UTC) Received: by obqv19 with SMTP id v19so17683500obq.13 for ; Mon, 23 Apr 2012 02:32:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=f1iNjodDdC9J9l+h0oKfo/K2Q5EB9PJfvR15auz99UI=; b=YcbcKgZ1T25m6hAFhmKeM4ZAXT+r/um7yHhVi3uXwAGHTNBTRcAukRsvh9W3ubZsDO ywaxXdsJrLQJSXDyDliworBZ238Nquz8xKUrFJgnhwBOPz5kePMGGxND82ttVY/xQmc6 137vtZSLvFgxw4U7eaWTZk7NgB5DynEpE66OZ9ZBOpWIPG1T8hq8eKe2BYld+CDNVw3x dMT2NgrI1k74znYSWegKjOr5BAXCXzam7rINBUYn+AaYKt3GLFpFN9kxjSxC4fMzTB51 zSAds8uKvSYwHEqFwSyh2bVNA/i/Rhpv6AMvve8d8NVwrXc4A3XoV+4ecYChDHQMrbW6 A53Q== MIME-Version: 1.0 Received: by 10.60.29.39 with SMTP id g7mr2407884oeh.6.1335173544900; Mon, 23 Apr 2012 02:32:24 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 02:32:24 -0700 (PDT) In-Reply-To: <20120423084120.GD76983@zxy.spb.ru> References: <20120423084120.GD76983@zxy.spb.ru> Date: Mon, 23 Apr 2012 17:32:24 +0800 Message-ID: From: Fengwei yin To: Slawa Olhovchenkov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 09:32:25 -0000 On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > >> Hi list, >> If this is not correct question on the list, please let me know and >> sorry for noise. >> >> I have a question regarding the BSD libc for SMP arch. I didn't see >> memory barrier used in libc. >> How can we make sure it's safe on SMP arch? > > /usr/include/machine/atomic.h: > > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : "memor= y") > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory"= ) > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory"= ) > Thanks for the information. But it looks no body use it in libc. From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 09:40:19 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id B081B1065680 for ; Mon, 23 Apr 2012 09:40:19 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id 642148FC19 for ; Mon, 23 Apr 2012 09:40:19 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1SMFlD-000Eno-CU; Mon, 23 Apr 2012 13:40:43 +0400 Date: Mon, 23 Apr 2012 13:40:43 +0400 From: Slawa Olhovchenkov To: Fengwei yin Message-ID: <20120423094043.GS32749@zxy.spb.ru> References: <20120423084120.GD76983@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 09:40:19 -0000 On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > > > >> Hi list, > >> If this is not correct question on the list, please let me know and > >> sorry for noise. > >> > >> I have a question regarding the BSD libc for SMP arch. I didn't see > >> memory barrier used in libc. > >> How can we make sure it's safe on SMP arch? > > > > /usr/include/machine/atomic.h: > > > > #define mb() š š__asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > #define wmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > #define rmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > > > Thanks for the information. But it looks no body use it in libc. I think no body in libc need memory barrier: libc don't work with peripheral, for atomic opertions used different macros. From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 11:07:27 2012 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BDEB51065676 for ; Mon, 23 Apr 2012 11:07:27 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8EC768FC26 for ; Mon, 23 Apr 2012 11:07:27 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3NB7RPl047738 for ; Mon, 23 Apr 2012 11:07:27 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3NB7QIV047735 for freebsd-threads@FreeBSD.org; Mon, 23 Apr 2012 11:07:26 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 23 Apr 2012 11:07:26 GMT Message-Id: <201204231107.q3NB7QIV047735@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 11:07:27 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o threa/166932 threads [libthr] sleepqueue-related memory leak in libthr o threa/165173 threads [build] clang buildworld breaks libthr o threa/163512 threads libc defaults to single threaded o threa/160708 threads possible security problem with RLIMIT_VMEM o threa/150959 threads [libc] Stub pthread_once in libc should call _libc_onc o threa/148515 threads Memory / syslog strangeness in FreeBSD 8.x ( possible o threa/141721 threads rtprio(1): (id|rt)prio priority resets when new thread o threa/135673 threads databases/mysql50-server - MySQL query lock-ups on 7.2 o threa/128922 threads threads hang with xorg running o threa/122923 threads 'nice' does not prevent background process from steali o threa/121336 threads lang/neko threading ok on UP, broken on SMP (FreeBSD 7 o threa/116668 threads can no longer use jdk15 with libthr on -stable SMP o threa/115211 threads pthread_atfork misbehaves in initial thread o threa/110636 threads [request] gdb(1): using gdb with multi thread applicat o threa/110306 threads apache 2.0 segmentation violation when calling gethost o threa/103975 threads Implicit loading/unloading of libpthread.so may crash o threa/101323 threads [patch] fork(2) in threaded programs broken. s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/79683 threads svctcp_create() fails if multiple threads call at the s threa/76694 threads fork cause hang in dup()/close() function in child (-l s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/34536 threads accept() blocks other threads s threa/30464 threads [patch] pthread mutex attributes -- pshared 24 problems total. From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 11:26:55 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D439106566C for ; Mon, 23 Apr 2012 11:26:55 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 4F2D58FC08 for ; Mon, 23 Apr 2012 11:26:55 +0000 (UTC) Received: by obqv19 with SMTP id v19so17853572obq.13 for ; Mon, 23 Apr 2012 04:26:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=qihIEsDuVpWLiT/2hXUQEVWVvK6/OsbnX1xkWo913Kk=; b=B2fRAxF34iooiH2iAXbvrpKPlfgUtUeBopv7ycclEKKX/TJV7+jk1pCVG4GpVJu5qz soAliX0q75JD+7RrhOiHCu53cw4OgHmL94AvYKyyWc7K+g+YLlq3U7XX/H6tYP6ZVdWT 1TbMF/QcFyOmyqvohL2RM93HAKgMIQqVYuVAAASC3mdIyMxKPn5+6Jk5gZ5MjmC5VSYN V5KuFfOnIjtvbBb/9aSydzyf4G/x1yq47n3rTw4oJ9VfLnIFGM+pLKIM/1fAEjqYwSTY how18xJLf9ud8meBLz0iD/GIhJulijNHuvYz6mTA2G/C/nY05nxPpnoUUsBdTRcygPdW d3MQ== MIME-Version: 1.0 Received: by 10.60.29.39 with SMTP id g7mr2929689oeh.6.1335180415000; Mon, 23 Apr 2012 04:26:55 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 04:26:54 -0700 (PDT) In-Reply-To: <20120423094043.GS32749@zxy.spb.ru> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> Date: Mon, 23 Apr 2012 19:26:54 +0800 Message-ID: From: Fengwei yin To: Slawa Olhovchenkov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 11:26:55 -0000 On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wro= te: >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: >> > >> >> Hi list, >> >> If this is not correct question on the list, please let me know and >> >> sorry for noise. >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't see >> >> memory barrier used in libc. >> >> How can we make sure it's safe on SMP arch? >> > >> > /usr/include/machine/atomic.h: >> > >> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : "me= mory") >> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memo= ry") >> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memo= ry") >> > >> >> Thanks for the information. But it looks no body use it in libc. > > I think no body in libc need memory barrier: libc don't work with > peripheral, for atomic opertions used different macros. If we check the usage of __sinit(), it is a typical singleton pattern which needs memory barrier to make sure no potential SMP issue. Or did I miss something here? Regards Yin, Fengwei From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 11:38:15 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1D6881065675 for ; Mon, 23 Apr 2012 11:38:15 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) by mx1.freebsd.org (Postfix) with ESMTP id BD8F88FC0A for ; Mon, 23 Apr 2012 11:38:14 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1SMHbK-000Hxx-A9; Mon, 23 Apr 2012 15:38:38 +0400 Date: Mon, 23 Apr 2012 15:38:38 +0400 From: Slawa Olhovchenkov To: Fengwei yin Message-ID: <20120423113838.GT32749@zxy.spb.ru> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 11:38:15 -0000 On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: > > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > > > >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > >> > > >> >> Hi list, > >> >> If this is not correct question on the list, please let me know and > >> >> sorry for noise. > >> >> > >> >> I have a question regarding the BSD libc for SMP arch. I didn't see > >> >> memory barrier used in libc. > >> >> How can we make sure it's safe on SMP arch? > >> > > >> > /usr/include/machine/atomic.h: > >> > > >> > #define mb() š š__asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > >> > #define wmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > >> > #define rmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > >> > > >> > >> Thanks for the information. But it looks no body use it in libc. > > > > I think no body in libc need memory barrier: libc don't work with > > peripheral, for atomic opertions used different macros. > > If we check the usage of __sinit(), it is a typical singleton pattern which > needs memory barrier to make sure no potential SMP issue. > > Or did I miss something here? What architecture with cache incoherency and FreeBSD support? From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 11:44:35 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 45EEC106566B for ; Mon, 23 Apr 2012 11:44:35 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 05F4E8FC0C for ; Mon, 23 Apr 2012 11:44:34 +0000 (UTC) Received: by obqv19 with SMTP id v19so17879749obq.13 for ; Mon, 23 Apr 2012 04:44:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=Yr+TqV8odryk8/PaHmCM25vl1xvoz8pVTnyjKvau+AM=; b=x1wqzy6zlbR3Tb+Q0ZqGwHG9iXBvELdXp43COKC/bWKEmOjzp/5prQb4dGwvvl07+E uvcHqZchqJl2i58+OeNn/LBFhPFB/zwwyt2bI4o6Uwm49jPZBdohF8K/1s7UDZ+awo2w RnHDFU0Ul+XddxGMIP0AlqQwyrjfr/ALvJ4i7AyqqlJUEbOya4PfILSJE5gkUPzvkvWM HxFsN9JkZR2I1FaE2s6rvCglIh3DBPP7SBpZqRSVOrkpsGVNaQTbAjFYLGg+HXIACC/m JAtsx87AYeXUovGNyAdkHGpXJZMhqMBgyMrhSz3Bq+AuisH+8poTCk9p9SXlh74DCN0n 9o+g== MIME-Version: 1.0 Received: by 10.60.29.39 with SMTP id g7mr3006914oeh.6.1335181474441; Mon, 23 Apr 2012 04:44:34 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 04:44:34 -0700 (PDT) In-Reply-To: <20120423113838.GT32749@zxy.spb.ru> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> Date: Mon, 23 Apr 2012 19:44:34 +0800 Message-ID: From: Fengwei yin To: Slawa Olhovchenkov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 11:44:35 -0000 On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov wrote: > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wro= te: >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: >> > >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov = wrote: >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: >> >> > >> >> >> Hi list, >> >> >> If this is not correct question on the list, please let me know an= d >> >> >> sorry for noise. >> >> >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't se= e >> >> >> memory barrier used in libc. >> >> >> How can we make sure it's safe on SMP arch? >> >> > >> >> > /usr/include/machine/atomic.h: >> >> > >> >> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : = "memory") >> >> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "m= emory") >> >> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "m= emory") >> >> > >> >> >> >> Thanks for the information. But it looks no body use it in libc. >> > >> > I think no body in libc need memory barrier: libc don't work with >> > peripheral, for atomic opertions used different macros. >> >> If we check the usage of __sinit(), it is a typical singleton pattern wh= ich >> needs memory barrier to make sure no potential SMP issue. >> >> Or did I miss something here? > > What architecture with cache incoherency and FreeBSD support? I suppose it's not related with cache inchoherency (I could be wrong). It's related with reorder of instruction by CPU. Here is the link talking about why need memory barrier for singleton: http://www.oaklib.org/docs/oak/singleton.html x86 has strict memory model and may not suffer this kind of issue. But ARM need to take care of it IMHO. Regards Yin, Fengwei From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 12:07:33 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4EAF6106564A for ; Mon, 23 Apr 2012 12:07:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id C40F08FC1E for ; Mon, 23 Apr 2012 12:07:32 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3NC7LG8052065; Mon, 23 Apr 2012 15:07:21 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3NC7KuB006329; Mon, 23 Apr 2012 15:07:20 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3NC7KIQ006328; Mon, 23 Apr 2012 15:07:20 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 23 Apr 2012 15:07:20 +0300 From: Konstantin Belousov To: Fengwei yin Message-ID: <20120423120720.GS2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="o4p1YyT9pdUjNGfy" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-threads@freebsd.org, jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 12:07:33 -0000 --o4p1YyT9pdUjNGfy Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: > On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov wrot= e: > > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > > > >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov w= rote: > >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > >> > > >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > >> >> > > >> >> >> Hi list, > >> >> >> If this is not correct question on the list, please let me know = and > >> >> >> sorry for noise. > >> >> >> > >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't = see > >> >> >> memory barrier used in libc. > >> >> >> How can we make sure it's safe on SMP arch? > >> >> > > >> >> > /usr/include/machine/atomic.h: > >> >> > > >> >> > #define mb() =9A =9A__asm __volatile("lock; addl $0,(%%esp)" : : = : "memory") > >> >> > #define wmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : : : = "memory") > >> >> > #define rmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : : : = "memory") > >> >> > > >> >> > >> >> Thanks for the information. But it looks no body use it in libc. > >> > > >> > I think no body in libc need memory barrier: libc don't work with > >> > peripheral, for atomic opertions used different macros. > >> > >> If we check the usage of __sinit(), it is a typical singleton pattern = which > >> needs memory barrier to make sure no potential SMP issue. > >> > >> Or did I miss something here? > > > > What architecture with cache incoherency and FreeBSD support? >=20 > I suppose it's not related with cache inchoherency (I could be wrong). > It's related > with reorder of instruction by CPU. >=20 > Here is the link talking about why need memory barrier for singleton: > http://www.oaklib.org/docs/oak/singleton.html >=20 > x86 has strict memory model and may not suffer this kind of issue. But > ARM need to > take care of it IMHO. Please note that __sinit is idempotent, so double-initialization is not an issue there. The only possible problematic case would be other thread executing exit and not noticing non-NULL value for __cleanup while current thread just set it. I am not sure how much real this race is. Each call to _sinit() is immediat= ely followed by a lock acquire, typically FLOCKFILE(), which enforces full barr= ier semantic due to pthread_mutex_lock call. The exit() performs __cxa_finalize= () call before checking __cleanup value, and __cxa_finalize() itself locks atexit_mutex. So the race is tiny and probably possible only for somewhat buggy applications which perform exit() while there are stdio operations in progress. Also note that some functions assign to __cleanup unconditionally. Do you see any real issue due to non-synchronized access to __cleanup ? --o4p1YyT9pdUjNGfy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+VRfgACgkQC3+MBN1Mb4gD6QCbBIrbxxcIBGlcoq5YjXrK7Qtk QYUAoNz43XBHSKfIFabK1+WaStPB5ddV =wpqq -----END PGP SIGNATURE----- --o4p1YyT9pdUjNGfy-- From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 12:33:06 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 06A5D106566C for ; Mon, 23 Apr 2012 12:33:06 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id B88148FC0A for ; Mon, 23 Apr 2012 12:33:05 +0000 (UTC) Received: by obqv19 with SMTP id v19so17949998obq.13 for ; Mon, 23 Apr 2012 05:33:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=WyIUeO1Iuod7ydcEW+/SwN772QmpofuuK9vC1/ydkvI=; b=VkGBsxCUsZC5x49QNzsvfFho95RbhQZueZ6DMWqlMkJwZidbxC7d+Ygs5U9lJdT9lT 0+qVWnmc+NCLF15QvznZOkagnlWsXYZKhCNOj1Gi+yBgSFDUgImFbQhFL1CW29nWslSH timlayWnzOiE9keJoPL+i5Va1WHFMIJ5AclFMOp7vGC3+f7aCCuP+YrvfYcz2u4QP+bP zX2BcIO6+y6cK6JBHYK6QYaC/s8KIRT0z5TVD4t78+Yx68/rQ+AqJrlanqnrDSU7HehE 4/26a5kgj9T9aWoVl3+I89prvptEUt1HDCNuMYSkYfzU4LeYKLwnjvn9KVHp6yuC2od/ ocoQ== MIME-Version: 1.0 Received: by 10.182.232.38 with SMTP id tl6mr7322531obc.16.1335184385378; Mon, 23 Apr 2012 05:33:05 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 05:33:05 -0700 (PDT) In-Reply-To: <20120423120720.GS2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> <20120423120720.GS2358@deviant.kiev.zoral.com.ua> Date: Mon, 23 Apr 2012 20:33:05 +0800 Message-ID: From: Fengwei yin To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-threads@freebsd.org, jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 12:33:06 -0000 On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov wrote: > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov wro= te: >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: >> > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov = wrote: >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: >> >> > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: >> >> >> > >> >> >> >> Hi list, >> >> >> >> If this is not correct question on the list, please let me know= and >> >> >> >> sorry for noise. >> >> >> >> >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't= see >> >> >> >> memory barrier used in libc. >> >> >> >> How can we make sure it's safe on SMP arch? >> >> >> > >> >> >> > /usr/include/machine/atomic.h: >> >> >> > >> >> >> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : := : "memory") >> >> >> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : := "memory") >> >> >> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : := "memory") >> >> >> > >> >> >> >> >> >> Thanks for the information. But it looks no body use it in libc. >> >> > >> >> > I think no body in libc need memory barrier: libc don't work with >> >> > peripheral, for atomic opertions used different macros. >> >> >> >> If we check the usage of __sinit(), it is a typical singleton pattern= which >> >> needs memory barrier to make sure no potential SMP issue. >> >> >> >> Or did I miss something here? >> > >> > What architecture with cache incoherency and FreeBSD support? >> >> I suppose it's not related with cache inchoherency (I could be wrong). >> It's related >> with reorder of instruction by CPU. >> >> Here is the link talking about why need memory barrier for singleton: >> http://www.oaklib.org/docs/oak/singleton.html >> >> x86 has strict memory model and may not suffer this kind of issue. But >> ARM need to >> take care of it IMHO. > > Please note that __sinit is idempotent, so double-initialization is not > an issue there. The only possible problematic case would be other thread > executing exit and not noticing non-NULL value for __cleanup while curren= t > thread just set it. > > I am not sure how much real this race is. Each call to _sinit() is immedi= ately > followed by a lock acquire, typically FLOCKFILE(), which enforces full ba= rrier > semantic due to pthread_mutex_lock call. The exit() performs __cxa_finali= ze() > call before checking __cleanup value, and __cxa_finalize() itself locks > atexit_mutex. So the race is tiny and probably possible only for somewhat > buggy applications which perform exit() while there are stdio operations > in progress. > > Also note that some functions assign to __cleanup unconditionally. > > Do you see any real issue due to non-synchronized access to __cleanup ? No. I didn't see real issue. I am just reviewing the code. If you don't think __sinit has issue, let's check another code: line 68 in libc/stdio/fclose.c line 133 in libc/stdio/findfp.c (function __sfp()) Which is trying to free a fp slot by assign 0 to fp->_flags. But if the instrucation could be re-ordered, another CPU could see fp->_flags is assigned to 0 before the cleanup from line 57 to 67. Let's say, if another CPU is in line 133 of __sfp(), it could see fp->_flags become 0 before it's aware of the cleanup (Line 57 to line 67 in libc/stdio/fclose.c) happen. Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c just could make sure line 70 happen after line 68. It can't impact the re-order of line 57 ~ line 68 by CPU. From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 13:03:58 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 217C6106566B for ; Mon, 23 Apr 2012 13:03:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 7CCDE8FC08 for ; Mon, 23 Apr 2012 13:03:57 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3ND3iEV060688; Mon, 23 Apr 2012 16:03:44 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3ND3hRj006679; Mon, 23 Apr 2012 16:03:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3ND3h1T006678; Mon, 23 Apr 2012 16:03:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 23 Apr 2012 16:03:43 +0300 From: Konstantin Belousov To: Fengwei yin Message-ID: <20120423130343.GT2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> <20120423120720.GS2358@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vtxr9MvRW4bhShyg" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-threads@freebsd.org, jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 13:03:58 -0000 --vtxr9MvRW4bhShyg Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote: > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov > wrote: > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov w= rote: > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > >> > > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > >> >> > > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > >> >> >> > > >> >> >> >> Hi list, > >> >> >> >> If this is not correct question on the list, please let me kn= ow and > >> >> >> >> sorry for noise. > >> >> >> >> > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn= 't see > >> >> >> >> memory barrier used in libc. > >> >> >> >> How can we make sure it's safe on SMP arch? > >> >> >> > > >> >> >> > /usr/include/machine/atomic.h: > >> >> >> > > >> >> >> > #define mb() =9A =9A__asm __volatile("lock; addl $0,(%%esp)" := : : "memory") > >> >> >> > #define wmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : := : "memory") > >> >> >> > #define rmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : := : "memory") > >> >> >> > > >> >> >> > >> >> >> Thanks for the information. But it looks no body use it in libc. > >> >> > > >> >> > I think no body in libc need memory barrier: libc don't work with > >> >> > peripheral, for atomic opertions used different macros. > >> >> > >> >> If we check the usage of __sinit(), it is a typical singleton patte= rn which > >> >> needs memory barrier to make sure no potential SMP issue. > >> >> > >> >> Or did I miss something here? > >> > > >> > What architecture with cache incoherency and FreeBSD support? > >> > >> I suppose it's not related with cache inchoherency (I could be wrong). > >> It's related > >> with reorder of instruction by CPU. > >> > >> Here is the link talking about why need memory barrier for singleton: > >> http://www.oaklib.org/docs/oak/singleton.html > >> > >> x86 has strict memory model and may not suffer this kind of issue. But > >> ARM need to > >> take care of it IMHO. > > > > Please note that __sinit is idempotent, so double-initialization is not > > an issue there. The only possible problematic case would be other thread > > executing exit and not noticing non-NULL value for __cleanup while curr= ent > > thread just set it. > > > > I am not sure how much real this race is. Each call to _sinit() is imme= diately > > followed by a lock acquire, typically FLOCKFILE(), which enforces full = barrier > > semantic due to pthread_mutex_lock call. The exit() performs __cxa_fina= lize() > > call before checking __cleanup value, and __cxa_finalize() itself locks > > atexit_mutex. So the race is tiny and probably possible only for somewh= at > > buggy applications which perform exit() while there are stdio operations > > in progress. > > > > Also note that some functions assign to __cleanup unconditionally. > > > > Do you see any real issue due to non-synchronized access to __cleanup ? >=20 > No. I didn't see real issue. I am just reviewing the code. >=20 > If you don't think __sinit has issue, let's check another code: > line 68 in libc/stdio/fclose.c > line 133 in libc/stdio/findfp.c (function __sfp()) >=20 > Which is trying to free a fp slot by assign 0 to fp->_flags. But if > the instrucation > could be re-ordered, another CPU could see fp->_flags is assigned to 0 > before the > cleanup from line 57 to 67. >=20 > Let's say, if another CPU is in line 133 of __sfp(), it could see > fp->_flags become > 0 before it's aware of the cleanup (Line 57 to line 67 in > libc/stdio/fclose.c) happen. >=20 > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c > just could make sure > line 70 happen after line 68. It can't impact the re-order of line 57 > ~ line 68 by CPU. Yes, FUNLOCKFILE() there would have no effect on the potential CPU reorderi= ng of the writes. But does the order of these writes matter at all ? Please note that __sfp() reinitializes all fields written by fclose(). Only if CPU executing fclose() is allowed to reorder operations so that the external effect of _flags =3D 0 assignment can be observed before that CPU executes other operations from fclose(), there could be a problem. This is definitely impossible on Intel, and I indeed do not know about other architectures enough to reject such possibility. The _flags member is short, so atomics cannot be used there. The easier solution, if this is indeed an issue, is to lock thread_lock around _flags =3D 0 assignment in fclose(). --vtxr9MvRW4bhShyg Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+VUy8ACgkQC3+MBN1Mb4g7pwCgnxQskpojwuEEp4/fxAeP287y 6CEAniYTpQJ3iblCQ8JnmF5i0HJOx7te =ijpf -----END PGP SIGNATURE----- --vtxr9MvRW4bhShyg-- From owner-freebsd-threads@FreeBSD.ORG Mon Apr 23 13:41:47 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A453C106566B for ; Mon, 23 Apr 2012 13:41:47 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 558178FC1F for ; Mon, 23 Apr 2012 13:41:47 +0000 (UTC) Received: by ghrr20 with SMTP id r20so7289280ghr.13 for ; Mon, 23 Apr 2012 06:41:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=1xxJUgwwNzSkFN43sdEZvfXGgv9aDVx6CKg8S1A0YYs=; b=K97fVtZd2fAebutljLPNtQLqnmVq1ZLJfcHpSlRQODqD/LVu3vzM5ZVQWzeByYGyOd IW3A3+TraqulWpHSiH2RY6dk6vwMcffFIjkZoaAh3T5Rl/jzUm9+Xi5TDzZSqB8oiH23 fqjwCAHYx7MZodcWsWRVmrj6iyhGsWIQLVyZ+aqXb9l1kQY3VfvDOW/asrWzBO/9Wn8d Y6ovV25VCNOdk6Nqk9YZpHmtGvtPtiiJ1KMR8G2KhBapV6KyjNP+Vzz05I6sRTjdfUTT Ino34AOKYsqcSKXdrNFTw8MfnvBDmabbapLhnH2li+dXU/nJlb3SS7njdAblnBS7CRjA hgKQ== MIME-Version: 1.0 Received: by 10.60.169.146 with SMTP id ae18mr18544067oec.36.1335188501619; Mon, 23 Apr 2012 06:41:41 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 06:41:41 -0700 (PDT) In-Reply-To: <20120423130343.GT2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> <20120423120720.GS2358@deviant.kiev.zoral.com.ua> <20120423130343.GT2358@deviant.kiev.zoral.com.ua> Date: Mon, 23 Apr 2012 21:41:41 +0800 Message-ID: From: Fengwei yin To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-threads@freebsd.org, jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2012 13:41:47 -0000 On Mon, Apr 23, 2012 at 9:03 PM, Konstantin Belousov wrote: > On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote: >> On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov >> wrote: >> > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: >> >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov = wrote: >> >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: >> >> > >> >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: >> >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: >> >> >> > >> >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: >> >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: >> >> >> >> > >> >> >> >> >> Hi list, >> >> >> >> >> If this is not correct question on the list, please let me k= now and >> >> >> >> >> sorry for noise. >> >> >> >> >> >> >> >> >> >> I have a question regarding the BSD libc for SMP arch. I did= n't see >> >> >> >> >> memory barrier used in libc. >> >> >> >> >> How can we make sure it's safe on SMP arch? >> >> >> >> > >> >> >> >> > /usr/include/machine/atomic.h: >> >> >> >> > >> >> >> >> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" = : : : "memory") >> >> >> >> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : = : : "memory") >> >> >> >> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : = : : "memory") >> >> >> >> > >> >> >> >> >> >> >> >> Thanks for the information. But it looks no body use it in libc= . >> >> >> > >> >> >> > I think no body in libc need memory barrier: libc don't work wit= h >> >> >> > peripheral, for atomic opertions used different macros. >> >> >> >> >> >> If we check the usage of __sinit(), it is a typical singleton patt= ern which >> >> >> needs memory barrier to make sure no potential SMP issue. >> >> >> >> >> >> Or did I miss something here? >> >> > >> >> > What architecture with cache incoherency and FreeBSD support? >> >> >> >> I suppose it's not related with cache inchoherency (I could be wrong)= . >> >> It's related >> >> with reorder of instruction by CPU. >> >> >> >> Here is the link talking about why need memory barrier for singleton: >> >> http://www.oaklib.org/docs/oak/singleton.html >> >> >> >> x86 has strict memory model and may not suffer this kind of issue. Bu= t >> >> ARM need to >> >> take care of it IMHO. >> > >> > Please note that __sinit is idempotent, so double-initialization is no= t >> > an issue there. The only possible problematic case would be other thre= ad >> > executing exit and not noticing non-NULL value for __cleanup while cur= rent >> > thread just set it. >> > >> > I am not sure how much real this race is. Each call to _sinit() is imm= ediately >> > followed by a lock acquire, typically FLOCKFILE(), which enforces full= barrier >> > semantic due to pthread_mutex_lock call. The exit() performs __cxa_fin= alize() >> > call before checking __cleanup value, and __cxa_finalize() itself lock= s >> > atexit_mutex. So the race is tiny and probably possible only for somew= hat >> > buggy applications which perform exit() while there are stdio operatio= ns >> > in progress. >> > >> > Also note that some functions assign to __cleanup unconditionally. >> > >> > Do you see any real issue due to non-synchronized access to __cleanup = ? >> >> No. I didn't see real issue. I am just reviewing the code. >> >> If you don't think __sinit has issue, let's check another code: >> =A0 =A0 =A0line 68 in libc/stdio/fclose.c >> =A0 =A0 =A0line 133 in libc/stdio/findfp.c (function __sfp()) >> >> Which is trying to free a fp slot by assign 0 to fp->_flags. But if >> the instrucation >> could be re-ordered, another CPU could see fp->_flags is assigned to 0 >> before the >> cleanup from line 57 to 67. >> >> Let's say, if another CPU is in line 133 of __sfp(), it could see >> fp->_flags become >> 0 before it's aware of the cleanup (Line 57 to line 67 in >> libc/stdio/fclose.c) happen. >> >> Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c >> just could make sure >> line 70 happen after line 68. It can't impact the re-order of line 57 >> ~ line 68 by CPU. > > Yes, FUNLOCKFILE() there would have no effect on the potential CPU reorde= ring > of the writes. =A0But does the order of these writes matter at all ? > > Please note that __sfp() reinitializes all fields written by fclose(). > Only if CPU executing fclose() is allowed to reorder operations so that > the external effect of _flags =3D 0 assignment can be observed before tha= t > CPU executes other operations from fclose(), there could be a problem. > > This is definitely impossible on Intel, and I indeed do not know about > other architectures enough to reject such possibility. The _flags member > is short, so atomics cannot be used there. The easier solution, if this > is indeed an issue, is to lock thread_lock around _flags =3D 0 assignment > in fclose(). For Intel x86, I suppose it's not a problem because of its memory model. For IA64, ARM, it could be an issue in theory. For the solution: I suppose memory barrier is enough for it. From owner-freebsd-threads@FreeBSD.ORG Tue Apr 24 13:54:51 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A7E89106566C for ; Tue, 24 Apr 2012 13:54:51 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [193.34.186.230]) by mx1.freebsd.org (Postfix) with ESMTP id B1F8B8FC08 for ; Tue, 24 Apr 2012 13:54:50 +0000 (UTC) Received: from higson.cam.lispworks.com (higson [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.3/8.14.3) with ESMTP id q3ODhfUp016214; Tue, 24 Apr 2012 14:43:41 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id q3ODhf9U032687; Tue, 24 Apr 2012 14:43:41 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id q3ODhe2C032683; Tue, 24 Apr 2012 14:43:40 +0100 Date: Tue, 24 Apr 2012 14:43:40 +0100 Message-Id: <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> From: Martin Simmons To: Konstantin Belousov In-reply-to: <20120423130343.GT2358@deviant.kiev.zoral.com.ua> (message from Konstantin Belousov on Mon, 23 Apr 2012 16:03:43 +0300) References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> <20120423120720.GS2358@deviant.kiev.zoral.com.ua> <20120423130343.GT2358@deviant.kiev.zoral.com.ua> MIME-version: 1.0 Content-type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Cc: freebsd-threads@freebsd.org, jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 13:54:51 -0000 >>>>> On Mon, 23 Apr 2012 16:03:43 +0300, Konstantin Belousov said: > > On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote: > > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov > > wrote: > > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: > > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov wrote: > > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > > >> > > > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: > > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > > >> >> > > > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > > >> >> >> > > > >> >> >> >> Hi list, > > >> >> >> >> If this is not correct question on the list, please let me know and > > >> >> >> >> sorry for noise. > > >> >> >> >> > > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't see > > >> >> >> >> memory barrier used in libc. > > >> >> >> >> How can we make sure it's safe on SMP arch? > > >> >> >> > > > >> >> >> > /usr/include/machine/atomic.h: > > >> >> >> > > > >> >> >> > #define mb() š š__asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > >> >> >> > #define wmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > >> >> >> > #define rmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > >> >> >> > > > >> >> >> > > >> >> >> Thanks for the information. But it looks no body use it in libc. > > >> >> > > > >> >> > I think no body in libc need memory barrier: libc don't work with > > >> >> > peripheral, for atomic opertions used different macros. > > >> >> > > >> >> If we check the usage of __sinit(), it is a typical singleton pattern which > > >> >> needs memory barrier to make sure no potential SMP issue. > > >> >> > > >> >> Or did I miss something here? > > >> > > > >> > What architecture with cache incoherency and FreeBSD support? > > >> > > >> I suppose it's not related with cache inchoherency (I could be wrong). > > >> It's related > > >> with reorder of instruction by CPU. > > >> > > >> Here is the link talking about why need memory barrier for singleton: > > >> http://www.oaklib.org/docs/oak/singleton.html > > >> > > >> x86 has strict memory model and may not suffer this kind of issue. But > > >> ARM need to > > >> take care of it IMHO. > > > > > > Please note that __sinit is idempotent, so double-initialization is not > > > an issue there. The only possible problematic case would be other thread > > > executing exit and not noticing non-NULL value for __cleanup while current > > > thread just set it. > > > > > > I am not sure how much real this race is. Each call to _sinit() is immediately > > > followed by a lock acquire, typically FLOCKFILE(), which enforces full barrier > > > semantic due to pthread_mutex_lock call. The exit() performs __cxa_finalize() > > > call before checking __cleanup value, and __cxa_finalize() itself locks > > > atexit_mutex. So the race is tiny and probably possible only for somewhat > > > buggy applications which perform exit() while there are stdio operations > > > in progress. > > > > > > Also note that some functions assign to __cleanup unconditionally. > > > > > > Do you see any real issue due to non-synchronized access to __cleanup ? > > > > No. I didn't see real issue. I am just reviewing the code. > > > > If you don't think __sinit has issue, let's check another code: > > line 68 in libc/stdio/fclose.c > > line 133 in libc/stdio/findfp.c (function __sfp()) > > > > Which is trying to free a fp slot by assign 0 to fp->_flags. But if > > the instrucation > > could be re-ordered, another CPU could see fp->_flags is assigned to 0 > > before the > > cleanup from line 57 to 67. > > > > Let's say, if another CPU is in line 133 of __sfp(), it could see > > fp->_flags become > > 0 before it's aware of the cleanup (Line 57 to line 67 in > > libc/stdio/fclose.c) happen. > > > > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c > > just could make sure > > line 70 happen after line 68. It can't impact the re-order of line 57 > > ~ line 68 by CPU. > > Yes, FUNLOCKFILE() there would have no effect on the potential CPU reordering > of the writes. But does the order of these writes matter at all ? > > Please note that __sfp() reinitializes all fields written by fclose(). > Only if CPU executing fclose() is allowed to reorder operations so that > the external effect of _flags = 0 assignment can be observed before that > CPU executes other operations from fclose(), there could be a problem. > > This is definitely impossible on Intel, and I indeed do not know about > other architectures enough to reject such possibility. The _flags member > is short, so atomics cannot be used there. The easier solution, if this > is indeed an issue, is to lock thread_lock around _flags = 0 assignment > in fclose(). This can be a problem, even on Intel, because the compiler can reorder the stores. E.g. if I compile the following with gcc -O4 on amd64: struct foo { int x, y; }; int foo(struct foo *p) { int x = bar(); p->y = baz(); p->x = x; } then I get the following assembly language, which sets p->x before p->y: movq %rdi, %rbx call bar movl %eax, %ebp xorl %eax, %eax call baz movl %ebp, (%rbx) movl %eax, 4(%rbx) __Martin From owner-freebsd-threads@FreeBSD.ORG Tue Apr 24 14:03:58 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 74E0D1065670 for ; Tue, 24 Apr 2012 14:03:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id D4C458FC1A for ; Tue, 24 Apr 2012 14:03:57 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3OE3n3Y043624; Tue, 24 Apr 2012 17:03:49 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3OE3mot037971; Tue, 24 Apr 2012 17:03:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3OE3msw037970; Tue, 24 Apr 2012 17:03:48 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 24 Apr 2012 17:03:48 +0300 From: Konstantin Belousov To: Martin Simmons Message-ID: <20120424140348.GY2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <20120423094043.GS32749@zxy.spb.ru> <20120423113838.GT32749@zxy.spb.ru> <20120423120720.GS2358@deviant.kiev.zoral.com.ua> <20120423130343.GT2358@deviant.kiev.zoral.com.ua> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="yDxN68y6wlbaMG9t" Content-Disposition: inline In-Reply-To: <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-threads@freebsd.org, jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 14:03:58 -0000 --yDxN68y6wlbaMG9t Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Apr 24, 2012 at 02:43:40PM +0100, Martin Simmons wrote: > >>>>> On Mon, 23 Apr 2012 16:03:43 +0300, Konstantin Belousov said: > >=20 > > On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote: > > > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov > > > wrote: > > > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: > > > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov wrote: > > > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > > > >> > > > > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: > > > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > > > >> >> > > > > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > > > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrot= e: > > > >> >> >> > > > > >> >> >> >> Hi list, > > > >> >> >> >> If this is not correct question on the list, please let m= e know and > > > >> >> >> >> sorry for noise. > > > >> >> >> >> > > > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I = didn't see > > > >> >> >> >> memory barrier used in libc. > > > >> >> >> >> How can we make sure it's safe on SMP arch? > > > >> >> >> > > > > >> >> >> > /usr/include/machine/atomic.h: > > > >> >> >> > > > > >> >> >> > #define mb() ?? ??__asm __volatile("lock; addl $0,(%%esp)"= : : : "memory") > > > >> >> >> > #define wmb() ?? __asm __volatile("lock; addl $0,(%%esp)" = : : : "memory") > > > >> >> >> > #define rmb() ?? __asm __volatile("lock; addl $0,(%%esp)" = : : : "memory") > > > >> >> >> > > > > >> >> >> > > > >> >> >> Thanks for the information. But it looks no body use it in l= ibc. > > > >> >> > > > > >> >> > I think no body in libc need memory barrier: libc don't work = with > > > >> >> > peripheral, for atomic opertions used different macros. > > > >> >> > > > >> >> If we check the usage of __sinit(), it is a typical singleton p= attern which > > > >> >> needs memory barrier to make sure no potential SMP issue. > > > >> >> > > > >> >> Or did I miss something here? > > > >> > > > > >> > What architecture with cache incoherency and FreeBSD support? > > > >> > > > >> I suppose it's not related with cache inchoherency (I could be wro= ng). > > > >> It's related > > > >> with reorder of instruction by CPU. > > > >> > > > >> Here is the link talking about why need memory barrier for singlet= on: > > > >> http://www.oaklib.org/docs/oak/singleton.html > > > >> > > > >> x86 has strict memory model and may not suffer this kind of issue.= But > > > >> ARM need to > > > >> take care of it IMHO. > > > > > > > > Please note that __sinit is idempotent, so double-initialization is= not > > > > an issue there. The only possible problematic case would be other t= hread > > > > executing exit and not noticing non-NULL value for __cleanup while = current > > > > thread just set it. > > > > > > > > I am not sure how much real this race is. Each call to _sinit() is = immediately > > > > followed by a lock acquire, typically FLOCKFILE(), which enforces f= ull barrier > > > > semantic due to pthread_mutex_lock call. The exit() performs __cxa_= finalize() > > > > call before checking __cleanup value, and __cxa_finalize() itself l= ocks > > > > atexit_mutex. So the race is tiny and probably possible only for so= mewhat > > > > buggy applications which perform exit() while there are stdio opera= tions > > > > in progress. > > > > > > > > Also note that some functions assign to __cleanup unconditionally. > > > > > > > > Do you see any real issue due to non-synchronized access to __clean= up ? > > >=20 > > > No. I didn't see real issue. I am just reviewing the code. > > >=20 > > > If you don't think __sinit has issue, let's check another code: > > > line 68 in libc/stdio/fclose.c > > > line 133 in libc/stdio/findfp.c (function __sfp()) > > >=20 > > > Which is trying to free a fp slot by assign 0 to fp->_flags. But if > > > the instrucation > > > could be re-ordered, another CPU could see fp->_flags is assigned to 0 > > > before the > > > cleanup from line 57 to 67. > > >=20 > > > Let's say, if another CPU is in line 133 of __sfp(), it could see > > > fp->_flags become > > > 0 before it's aware of the cleanup (Line 57 to line 67 in > > > libc/stdio/fclose.c) happen. > > >=20 > > > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c > > > just could make sure > > > line 70 happen after line 68. It can't impact the re-order of line 57 > > > ~ line 68 by CPU. > >=20 > > Yes, FUNLOCKFILE() there would have no effect on the potential CPU reor= dering > > of the writes. But does the order of these writes matter at all ? > >=20 > > Please note that __sfp() reinitializes all fields written by fclose(). > > Only if CPU executing fclose() is allowed to reorder operations so that > > the external effect of _flags =3D 0 assignment can be observed before t= hat > > CPU executes other operations from fclose(), there could be a problem. > >=20 > > This is definitely impossible on Intel, and I indeed do not know about > > other architectures enough to reject such possibility. The _flags member > > is short, so atomics cannot be used there. The easier solution, if this > > is indeed an issue, is to lock thread_lock around _flags =3D 0 assignme= nt > > in fclose(). >=20 > This can be a problem, even on Intel, because the compiler can reorder the > stores. E.g. if I compile the following with gcc -O4 on amd64: >=20 > struct foo { int x, y; }; >=20 > int foo(struct foo *p) > { > int x =3D bar(); > p->y =3D baz(); > p->x =3D x; > } >=20 > then I get the following assembly language, which sets p->x before p->y: >=20 > movq %rdi, %rbx > call bar > movl %eax, %ebp > xorl %eax, %eax > call baz > movl %ebp, (%rbx) > movl %eax, 4(%rbx) >=20 > __Martin Ok, as I already said, I think that the reordering is safe there. Anyway, the change below should remove all concerns. diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c index f0629e8..383040e 100644 --- a/lib/libc/stdio/fclose.c +++ b/lib/libc/stdio/fclose.c @@ -41,9 +41,12 @@ __FBSDID("$FreeBSD$"); #include #include #include "un-namespace.h" +#include #include "libc_private.h" #include "local.h" =20 +extern spinlock_t __stdio_thread_lock; + int fclose(FILE *fp) { @@ -65,7 +68,11 @@ fclose(FILE *fp) FREELB(fp); fp->_file =3D -1; fp->_r =3D fp->_w =3D 0; /* Mess up if reaccessed. */ + if (__isthreaded) + _SPINLOCK(&__stdio_thread_lock); fp->_flags =3D 0; /* Release this FILE for reuse. */ + if (__isthreaded) + _SPINUNLOCK(&__stdio_thread_lock); FUNLOCKFILE(fp); return (r); } diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c index 89c0536..bcd6f62 100644 --- a/lib/libc/stdio/findfp.c +++ b/lib/libc/stdio/findfp.c @@ -82,9 +82,9 @@ static struct glue *lastglue =3D &uglue; =20 static struct glue * moreglue(int); =20 -static spinlock_t thread_lock =3D _SPINLOCK_INITIALIZER; -#define THREAD_LOCK() if (__isthreaded) _SPINLOCK(&thread_lock) -#define THREAD_UNLOCK() if (__isthreaded) _SPINUNLOCK(&thread_lock) +spinlock_t __stdio_thread_lock =3D _SPINLOCK_INITIALIZER; +#define THREAD_LOCK() if (__isthreaded) _SPINLOCK(&__stdio_thread_lock) +#define THREAD_UNLOCK() if (__isthreaded) _SPINUNLOCK(&__stdio_thread_lock) =20 #if NOT_YET #define SET_GLUE_PTR(ptr, val) atomic_set_rel_ptr(&(ptr), (uintptr_t)(val)) --yDxN68y6wlbaMG9t Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+WssQACgkQC3+MBN1Mb4g+hACguHQ9O3LLzcvc8DuzymjOmaeg JFAAoLF1xp2cXY6dvSf7dLsk0X1X9VeY =XzVd -----END PGP SIGNATURE----- --yDxN68y6wlbaMG9t-- From owner-freebsd-threads@FreeBSD.ORG Tue Apr 24 15:10:43 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 69A70106566B for ; Tue, 24 Apr 2012 15:10:43 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 1219C8FC12 for ; Tue, 24 Apr 2012 15:10:43 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 7FEB3B943; Tue, 24 Apr 2012 11:10:42 -0400 (EDT) From: John Baldwin To: freebsd-threads@freebsd.org Date: Tue, 24 Apr 2012 11:10:13 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; ) References: <20120423084120.GD76983@zxy.spb.ru> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> <20120424140348.GY2358@deviant.kiev.zoral.com.ua> In-Reply-To: <20120424140348.GY2358@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201204241110.14017.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 24 Apr 2012 11:10:42 -0400 (EDT) Cc: jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 15:10:43 -0000 On Tuesday, April 24, 2012 10:03:48 am Konstantin Belousov wrote: > On Tue, Apr 24, 2012 at 02:43:40PM +0100, Martin Simmons wrote: > > >>>>> On Mon, 23 Apr 2012 16:03:43 +0300, Konstantin Belousov said: > > > > > > On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote: > > > > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov > > > > wrote: > > > > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote: > > > > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov wrote: > > > > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote: > > > > >> > > > > > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov wrote: > > > > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote: > > > > >> >> > > > > > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov wrote: > > > > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote: > > > > >> >> >> > > > > > >> >> >> >> Hi list, > > > > >> >> >> >> If this is not correct question on the list, please let me know and > > > > >> >> >> >> sorry for noise. > > > > >> >> >> >> > > > > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't see > > > > >> >> >> >> memory barrier used in libc. > > > > >> >> >> >> How can we make sure it's safe on SMP arch? > > > > >> >> >> > > > > > >> >> >> > /usr/include/machine/atomic.h: > > > > >> >> >> > > > > > >> >> >> > #define mb() ?? ??__asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > > > >> >> >> > #define wmb() ?? __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > > > >> >> >> > #define rmb() ?? __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > > > >> >> >> > > > > > >> >> >> > > > > >> >> >> Thanks for the information. But it looks no body use it in libc. > > > > >> >> > > > > > >> >> > I think no body in libc need memory barrier: libc don't work with > > > > >> >> > peripheral, for atomic opertions used different macros. > > > > >> >> > > > > >> >> If we check the usage of __sinit(), it is a typical singleton pattern which > > > > >> >> needs memory barrier to make sure no potential SMP issue. > > > > >> >> > > > > >> >> Or did I miss something here? > > > > >> > > > > > >> > What architecture with cache incoherency and FreeBSD support? > > > > >> > > > > >> I suppose it's not related with cache inchoherency (I could be wrong). > > > > >> It's related > > > > >> with reorder of instruction by CPU. > > > > >> > > > > >> Here is the link talking about why need memory barrier for singleton: > > > > >> http://www.oaklib.org/docs/oak/singleton.html > > > > >> > > > > >> x86 has strict memory model and may not suffer this kind of issue. But > > > > >> ARM need to > > > > >> take care of it IMHO. > > > > > > > > > > Please note that __sinit is idempotent, so double-initialization is not > > > > > an issue there. The only possible problematic case would be other thread > > > > > executing exit and not noticing non-NULL value for __cleanup while current > > > > > thread just set it. > > > > > > > > > > I am not sure how much real this race is. Each call to _sinit() is immediately > > > > > followed by a lock acquire, typically FLOCKFILE(), which enforces full barrier > > > > > semantic due to pthread_mutex_lock call. The exit() performs __cxa_finalize() > > > > > call before checking __cleanup value, and __cxa_finalize() itself locks > > > > > atexit_mutex. So the race is tiny and probably possible only for somewhat > > > > > buggy applications which perform exit() while there are stdio operations > > > > > in progress. > > > > > > > > > > Also note that some functions assign to __cleanup unconditionally. > > > > > > > > > > Do you see any real issue due to non-synchronized access to __cleanup ? > > > > > > > > No. I didn't see real issue. I am just reviewing the code. > > > > > > > > If you don't think __sinit has issue, let's check another code: > > > > line 68 in libc/stdio/fclose.c > > > > line 133 in libc/stdio/findfp.c (function __sfp()) > > > > > > > > Which is trying to free a fp slot by assign 0 to fp->_flags. But if > > > > the instrucation > > > > could be re-ordered, another CPU could see fp->_flags is assigned to 0 > > > > before the > > > > cleanup from line 57 to 67. > > > > > > > > Let's say, if another CPU is in line 133 of __sfp(), it could see > > > > fp->_flags become > > > > 0 before it's aware of the cleanup (Line 57 to line 67 in > > > > libc/stdio/fclose.c) happen. > > > > > > > > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c > > > > just could make sure > > > > line 70 happen after line 68. It can't impact the re-order of line 57 > > > > ~ line 68 by CPU. > > > > > > Yes, FUNLOCKFILE() there would have no effect on the potential CPU reordering > > > of the writes. But does the order of these writes matter at all ? > > > > > > Please note that __sfp() reinitializes all fields written by fclose(). > > > Only if CPU executing fclose() is allowed to reorder operations so that > > > the external effect of _flags = 0 assignment can be observed before that > > > CPU executes other operations from fclose(), there could be a problem. > > > > > > This is definitely impossible on Intel, and I indeed do not know about > > > other architectures enough to reject such possibility. The _flags member > > > is short, so atomics cannot be used there. The easier solution, if this > > > is indeed an issue, is to lock thread_lock around _flags = 0 assignment > > > in fclose(). > > > > This can be a problem, even on Intel, because the compiler can reorder the > > stores. E.g. if I compile the following with gcc -O4 on amd64: > > > > struct foo { int x, y; }; > > > > int foo(struct foo *p) > > { > > int x = bar(); > > p->y = baz(); > > p->x = x; > > } > > > > then I get the following assembly language, which sets p->x before p->y: > > > > movq %rdi, %rbx > > call bar > > movl %eax, %ebp > > xorl %eax, %eax > > call baz > > movl %ebp, (%rbx) > > movl %eax, 4(%rbx) > > > > __Martin > Ok, as I already said, I think that the reordering is safe there. > > Anyway, the change below should remove all concerns. > > diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c > index f0629e8..383040e 100644 > --- a/lib/libc/stdio/fclose.c > +++ b/lib/libc/stdio/fclose.c > @@ -41,9 +41,12 @@ __FBSDID("$FreeBSD$"); > #include > #include > #include "un-namespace.h" > +#include > #include "libc_private.h" > #include "local.h" > > +extern spinlock_t __stdio_thread_lock; > + > int > fclose(FILE *fp) > { > @@ -65,7 +68,11 @@ fclose(FILE *fp) > FREELB(fp); > fp->_file = -1; > fp->_r = fp->_w = 0; /* Mess up if reaccessed. */ > + if (__isthreaded) > + _SPINLOCK(&__stdio_thread_lock); > fp->_flags = 0; /* Release this FILE for reuse. */ > + if (__isthreaded) > + _SPINUNLOCK(&__stdio_thread_lock); > FUNLOCKFILE(fp); > return (r); > } > diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c > index 89c0536..bcd6f62 100644 > --- a/lib/libc/stdio/findfp.c > +++ b/lib/libc/stdio/findfp.c > @@ -82,9 +82,9 @@ static struct glue *lastglue = &uglue; > > static struct glue * moreglue(int); > > -static spinlock_t thread_lock = _SPINLOCK_INITIALIZER; > -#define THREAD_LOCK() if (__isthreaded) _SPINLOCK(&thread_lock) > -#define THREAD_UNLOCK() if (__isthreaded) _SPINUNLOCK(&thread_lock) > +spinlock_t __stdio_thread_lock = _SPINLOCK_INITIALIZER; > +#define THREAD_LOCK() if (__isthreaded) _SPINLOCK(&__stdio_thread_lock) > +#define THREAD_UNLOCK() if (__isthreaded) _SPINUNLOCK(&__stdio_thread_lock) > > #if NOT_YET > #define SET_GLUE_PTR(ptr, val) atomic_set_rel_ptr(&(ptr), (uintptr_t) (val)) Could you move the extern and THREAD_LOCK/UNLOCK macros into the stdio private header file? -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Tue Apr 24 16:58:54 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5BFE11065675; Tue, 24 Apr 2012 16:58:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id EB3D78FC0A; Tue, 24 Apr 2012 16:58:53 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3OGwh5C070035; Tue, 24 Apr 2012 19:58:43 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3OGwgrY039057; Tue, 24 Apr 2012 19:58:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3OGwgeY039056; Tue, 24 Apr 2012 19:58:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 24 Apr 2012 19:58:42 +0300 From: Konstantin Belousov To: John Baldwin Message-ID: <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> <20120424140348.GY2358@deviant.kiev.zoral.com.ua> <201204241110.14017.jhb@freebsd.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tKmWtG5s+ZnuAuMT" Content-Disposition: inline In-Reply-To: <201204241110.14017.jhb@freebsd.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 16:58:54 -0000 --tKmWtG5s+ZnuAuMT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Apr 24, 2012 at 11:10:13AM -0400, John Baldwin wrote: > Could you move the extern and THREAD_LOCK/UNLOCK macros into the stdio > private header file? This requires more glue because spinlock.h is separate from libc_private.h. I also added a comment trying to explain why assignment is braced with lock. diff --git a/lib/libc/include/libc_private.h b/lib/libc/include/libc_privat= e.h index d4b24a7..2182f46 100644 --- a/lib/libc/include/libc_private.h +++ b/lib/libc/include/libc_private.h @@ -81,6 +81,19 @@ void _rtld_error(const char *fmt, ...); #define FLOCKFILE(fp) if (__isthreaded) _FLOCKFILE(fp) #define FUNLOCKFILE(fp) if (__isthreaded) _funlockfile(fp) =20 +struct _spinlock; +extern struct _spinlock __stdio_thread_lock; +#define STDIO_THREAD_LOCK() \ +do { \ + if (__isthreaded) \ + _SPINLOCK(&__stdio_thread_lock); \ +} while (0) +#define STDIO_THREAD_UNLOCK() \ +do { \ + if (__isthreaded) \ + _SPINUNLOCK(&__stdio_thread_lock); \ +} while (0) + /* * Indexes into the pthread jump table. * diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c index f0629e8..3957b6a 100644 --- a/lib/libc/stdio/fclose.c +++ b/lib/libc/stdio/fclose.c @@ -41,6 +41,7 @@ __FBSDID("$FreeBSD$"); #include #include #include "un-namespace.h" +#include #include "libc_private.h" #include "local.h" =20 @@ -65,7 +66,20 @@ fclose(FILE *fp) FREELB(fp); fp->_file =3D -1; fp->_r =3D fp->_w =3D 0; /* Mess up if reaccessed. */ + + /* + * Lock the spinlock used to protect __sglue list walk in + * __sfp(). The __sfp() uses fp->_flags =3D=3D 0 test as an + * indication of the unused FILE. + * + * Taking the lock prevents possible compiler or processor + * reordering of the writes performed before the final _flags + * cleanup, making sure that we are done with the FILE before + * it is considered available. + */ + STDIO_THREAD_LOCK(); fp->_flags =3D 0; /* Release this FILE for reuse. */ + STDIO_THREAD_UNLOCK(); FUNLOCKFILE(fp); return (r); } diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c index 89c0536..6d0b673 100644 --- a/lib/libc/stdio/findfp.c +++ b/lib/libc/stdio/findfp.c @@ -82,9 +82,7 @@ static struct glue *lastglue =3D &uglue; =20 static struct glue * moreglue(int); =20 -static spinlock_t thread_lock =3D _SPINLOCK_INITIALIZER; -#define THREAD_LOCK() if (__isthreaded) _SPINLOCK(&thread_lock) -#define THREAD_UNLOCK() if (__isthreaded) _SPINUNLOCK(&thread_lock) +spinlock_t __stdio_thread_lock =3D _SPINLOCK_INITIALIZER; =20 #if NOT_YET #define SET_GLUE_PTR(ptr, val) atomic_set_rel_ptr(&(ptr), (uintptr_t)(val)) @@ -127,22 +125,22 @@ __sfp() /* * The list must be locked because a FILE may be updated. */ - THREAD_LOCK(); + STDIO_THREAD_LOCK(); for (g =3D &__sglue; g !=3D NULL; g =3D g->next) { for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++) if (fp->_flags =3D=3D 0) goto found; } - THREAD_UNLOCK(); /* don't hold lock while malloc()ing. */ + STDIO_THREAD_UNLOCK(); /* don't hold lock while malloc()ing. */ if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL) return (NULL); - THREAD_LOCK(); /* reacquire the lock */ + STDIO_THREAD_LOCK(); /* reacquire the lock */ SET_GLUE_PTR(lastglue->next, g); /* atomically append glue to list */ lastglue =3D g; /* not atomic; only accessed when locked */ fp =3D g->iobs; found: fp->_flags =3D 1; /* reserve this slot; caller sets real flags */ - THREAD_UNLOCK(); + STDIO_THREAD_UNLOCK(); fp->_p =3D NULL; /* no current pointer */ fp->_w =3D 0; /* nothing to read or write */ fp->_r =3D 0; @@ -183,10 +181,10 @@ f_prealloc(void) for (g =3D &__sglue; (n -=3D g->niobs) > 0 && g->next; g =3D g->next) /* void */; if ((n > 0) && ((g =3D moreglue(n)) !=3D NULL)) { - THREAD_LOCK(); + STDIO_THREAD_LOCK(); SET_GLUE_PTR(lastglue->next, g); lastglue =3D g; - THREAD_UNLOCK(); + STDIO_THREAD_UNLOCK(); } } =20 --tKmWtG5s+ZnuAuMT Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+W28EACgkQC3+MBN1Mb4gMfgCfaX0Wy+vGyU9mCRfi4p3D+WVa Hw8AoL4tyq96n3qE7t1iJ1a3g6UkP9PT =HMbW -----END PGP SIGNATURE----- --tKmWtG5s+ZnuAuMT-- From owner-freebsd-threads@FreeBSD.ORG Tue Apr 24 17:34:56 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E54AB1065672 for ; Tue, 24 Apr 2012 17:34:56 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id B9FE88FC0A for ; Tue, 24 Apr 2012 17:34:56 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 06546B95B; Tue, 24 Apr 2012 13:34:56 -0400 (EDT) From: John Baldwin To: Konstantin Belousov Date: Tue, 24 Apr 2012 13:34:37 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; ) References: <20120423084120.GD76983@zxy.spb.ru> <201204241110.14017.jhb@freebsd.org> <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> In-Reply-To: <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201204241334.37580.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 24 Apr 2012 13:34:56 -0400 (EDT) Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 17:34:57 -0000 On Tuesday, April 24, 2012 12:58:42 pm Konstantin Belousov wrote: > On Tue, Apr 24, 2012 at 11:10:13AM -0400, John Baldwin wrote: > > Could you move the extern and THREAD_LOCK/UNLOCK macros into the stdio > > private header file? > > This requires more glue because spinlock.h is separate from libc_private.h. > I also added a comment trying to explain why assignment is braced with lock. This looks great, thanks! -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Tue Apr 24 18:00:37 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 649AD106566B; Tue, 24 Apr 2012 18:00:37 +0000 (UTC) (envelope-from martin@lispworks.com) Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com [193.34.186.230]) by mx1.freebsd.org (Postfix) with ESMTP id E71878FC08; Tue, 24 Apr 2012 18:00:36 +0000 (UTC) Received: from higson.cam.lispworks.com (higson [192.168.1.7]) by lwfs1-cam.cam.lispworks.com (8.14.3/8.14.3) with ESMTP id q3OI0YCc028398; Tue, 24 Apr 2012 19:00:34 +0100 (BST) (envelope-from martin@lispworks.com) Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by higson.cam.lispworks.com (8.14.4) id q3OI0X1x007388; Tue, 24 Apr 2012 19:00:33 +0100 Received: (from martin@localhost) by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id q3OI0X89007384; Tue, 24 Apr 2012 19:00:33 +0100 Date: Tue, 24 Apr 2012 19:00:33 +0100 Message-Id: <201204241800.q3OI0X89007384@higson.cam.lispworks.com> From: Martin Simmons To: Konstantin Belousov In-reply-to: <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> (message from Konstantin Belousov on Tue, 24 Apr 2012 19:58:42 +0300) References: <20120423084120.GD76983@zxy.spb.ru> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> <20120424140348.GY2358@deviant.kiev.zoral.com.ua> <201204241110.14017.jhb@freebsd.org> <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 18:00:37 -0000 >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said: > > + > + /* > + * Lock the spinlock used to protect __sglue list walk in > + * __sfp(). The __sfp() uses fp->_flags == 0 test as an > + * indication of the unused FILE. > + * > + * Taking the lock prevents possible compiler or processor > + * reordering of the writes performed before the final _flags > + * cleanup, making sure that we are done with the FILE before > + * it is considered available. > + */ > + STDIO_THREAD_LOCK(); > fp->_flags = 0; /* Release this FILE for reuse. */ > + STDIO_THREAD_UNLOCK(); > FUNLOCKFILE(fp); > return (r); Is that assumption about reordering correct? I.e. is FreeBSD's spinlock a two-way barrier? It isn't unusual for the locking primitive to be a one-way barrier, i.e. (from Linux kernel memory-barriers.txt) "Memory operations that occur before a LOCK operation may appear to happen after it completes." See also acq and rel in atomic(9). __Martin From owner-freebsd-threads@FreeBSD.ORG Tue Apr 24 18:13:09 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BC7D61065673; Tue, 24 Apr 2012 18:13:09 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 18FF98FC0A; Tue, 24 Apr 2012 18:13:08 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3OID3Rq081651; Tue, 24 Apr 2012 21:13:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3OID2Xc039889; Tue, 24 Apr 2012 21:13:02 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3OID2Fn039888; Tue, 24 Apr 2012 21:13:02 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 24 Apr 2012 21:13:02 +0300 From: Konstantin Belousov To: Martin Simmons Message-ID: <20120424181302.GB2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> <20120424140348.GY2358@deviant.kiev.zoral.com.ua> <201204241110.14017.jhb@freebsd.org> <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> <201204241800.q3OI0X89007384@higson.cam.lispworks.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="6lA9ewhm9nShhcj2" Content-Disposition: inline In-Reply-To: <201204241800.q3OI0X89007384@higson.cam.lispworks.com> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Apr 2012 18:13:09 -0000 --6lA9ewhm9nShhcj2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote: > >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said: > >=20 > > + > > + /* > > + * Lock the spinlock used to protect __sglue list walk in > > + * __sfp(). The __sfp() uses fp->_flags =3D=3D 0 test as an > > + * indication of the unused FILE. > > + * > > + * Taking the lock prevents possible compiler or processor > > + * reordering of the writes performed before the final _flags > > + * cleanup, making sure that we are done with the FILE before > > + * it is considered available. > > + */ > > + STDIO_THREAD_LOCK(); > > fp->_flags =3D 0; /* Release this FILE for reuse. */ > > + STDIO_THREAD_UNLOCK(); > > FUNLOCKFILE(fp); > > return (r); >=20 > Is that assumption about reordering correct? I.e. is FreeBSD's spinlock a > two-way barrier? >=20 > It isn't unusual for the locking primitive to be a one-way barrier, i.e. = (from > Linux kernel memory-barriers.txt) "Memory operations that occur before a = LOCK > operation may appear to happen after it completes." See also acq and rel= in > atomic(9). Taking the lock prevents the __sfp from iterating the list until the spinlock is released. Since release makes sure that all previous memory operations become visible, the acquire in the spinlock lock provides the neccesary guarentee. --6lA9ewhm9nShhcj2 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+W7S4ACgkQC3+MBN1Mb4j2XwCfUwYZ5frhFeuNuQblrpS/xCMH lKAAoMVOuTnHmMBWsL/5YaWIBx+Me3xq =YN7b -----END PGP SIGNATURE----- --6lA9ewhm9nShhcj2-- From owner-freebsd-threads@FreeBSD.ORG Wed Apr 25 03:25:42 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 009EC106566C for ; Wed, 25 Apr 2012 03:25:41 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id B38638FC21 for ; Wed, 25 Apr 2012 03:25:41 +0000 (UTC) Received: by obbuo13 with SMTP id uo13so2378052obb.13 for ; Tue, 24 Apr 2012 20:25:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=GOLP4V6A/TYfb3BSmbejQGlYmAOvIj70bye12k6pHc4=; b=mWtiBcC9X2CcCkyMi1jOoQrlDhSeDJuNGREnzCW7JV+IJbAsHGfRlZisNBA37dP+7z aXgDsBQfzV062wIVbZfpbQrrLePxDjip1+a0GDWfnmU+xJQc3i5oSdwQNpv2T9sFySfd ktzfAOSWv+Dkcec3e+3gEBNsk6R8BmwxJ2OLim5siaZGltu+3Q6PNb7QjCFb88Qb+JUZ hseJO3kz+S5gKN+cr+IqqZfFyZvD1GfhQizFoa6K+sg6MQcIsvu72M+K3uCPJKOhK1Tb 2YO9nT9I0TWfnmBEB1g5XzEFlMiZUGs/tnItovpa90pfjqUod7XN5+IGbp99N2Zk+uTa FtVA== MIME-Version: 1.0 Received: by 10.182.232.38 with SMTP id tl6mr1431864obc.16.1335324341113; Tue, 24 Apr 2012 20:25:41 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Tue, 24 Apr 2012 20:25:40 -0700 (PDT) In-Reply-To: <20120424181302.GB2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> <20120424140348.GY2358@deviant.kiev.zoral.com.ua> <201204241110.14017.jhb@freebsd.org> <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> <201204241800.q3OI0X89007384@higson.cam.lispworks.com> <20120424181302.GB2358@deviant.kiev.zoral.com.ua> Date: Wed, 25 Apr 2012 11:25:40 +0800 Message-ID: From: Fengwei yin To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2012 03:25:42 -0000 On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov wrote: > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote: >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said: >> > >> > + >> > + =A0 /* >> > + =A0 =A0* Lock the spinlock used to protect __sglue list walk in >> > + =A0 =A0* __sfp(). =A0The __sfp() uses fp->_flags =3D=3D 0 test as an >> > + =A0 =A0* indication of the unused FILE. >> > + =A0 =A0* >> > + =A0 =A0* Taking the lock prevents possible compiler or processor >> > + =A0 =A0* reordering of the writes performed before the final _flags >> > + =A0 =A0* cleanup, making sure that we are done with the FILE before >> > + =A0 =A0* it is considered available. >> > + =A0 =A0*/ >> > + =A0 STDIO_THREAD_LOCK(); >> > =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE for reu= se. */ >> > + =A0 STDIO_THREAD_UNLOCK(); >> > =A0 =A0 FUNLOCKFILE(fp); >> > =A0 =A0 return (r); >> >> Is that assumption about reordering correct? =A0I.e. is FreeBSD's spinlo= ck a >> two-way barrier? >> >> It isn't unusual for the locking primitive to be a one-way barrier, i.e.= (from >> Linux kernel memory-barriers.txt) "Memory operations that occur before a= LOCK >> operation may appear to happen after it completes." =A0See also acq and = rel in >> atomic(9). > Taking the lock prevents the __sfp from iterating the list until the > spinlock is released. Since release makes sure that all previous memory > operations become visible, the acquire in the spinlock lock provides > the neccesary guarentee. IMHO, the lock to me is too heavy here. What about this patch? NOTE: patch just show thoughts. I didn't even check build checking. diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c index f0629e8..a26f944 100644 --- a/lib/libc/stdio/fclose.c +++ b/lib/libc/stdio/fclose.c @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$"); #include "namespace.h" #include +#include #include #include #include "un-namespace.h" @@ -65,6 +66,7 @@ fclose(FILE *fp) FREELB(fp); fp->_file =3D -1; fp->_r =3D fp->_w =3D 0; /* Mess up if reaccessed. */ + wmb(); fp->_flags =3D 0; /* Release this FILE for reuse. */ FUNLOCKFILE(fp); return (r); diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c index 89c0536..03b2945 100644 --- a/lib/libc/stdio/findfp.c +++ b/lib/libc/stdio/findfp.c @@ -129,9 +129,16 @@ __sfp() */ THREAD_LOCK(); for (g =3D &__sglue; g !=3D NULL; g =3D g->next) { - for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++) - if (fp->_flags =3D=3D 0) + for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++) { + int __flags =3D fp->_flags; + rmb(); + /* + * If could see __flags is zero here, we are sure + * the cleanup in fclose is done. + */ + if (__flags =3D=3D 0) goto found; + } } THREAD_UNLOCK(); /* don't hold lock while malloc()ing. */ if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL) From owner-freebsd-threads@FreeBSD.ORG Wed Apr 25 06:26:32 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3ABA0106566C for ; Wed, 25 Apr 2012 06:26:32 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id C9E808FC17 for ; Wed, 25 Apr 2012 06:26:31 +0000 (UTC) Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3P6QSfg046705; Wed, 25 Apr 2012 09:26:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id q3P6QRbD044397; Wed, 25 Apr 2012 09:26:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3P6QRc2044396; Wed, 25 Apr 2012 09:26:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 25 Apr 2012 09:26:27 +0300 From: Konstantin Belousov To: Fengwei yin Message-ID: <20120425062627.GI2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> <20120424140348.GY2358@deviant.kiev.zoral.com.ua> <201204241110.14017.jhb@freebsd.org> <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> <201204241800.q3OI0X89007384@higson.cam.lispworks.com> <20120424181302.GB2358@deviant.kiev.zoral.com.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="REd8r87flCTRYq/l" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2012 06:26:32 -0000 --REd8r87flCTRYq/l Content-Type: text/plain; charset=koi8-r Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote: > On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov > wrote: > > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote: > >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said: > >> > > >> > + > >> > + =9A /* > >> > + =9A =9A* Lock the spinlock used to protect __sglue list walk in > >> > + =9A =9A* __sfp(). =9AThe __sfp() uses fp->_flags =3D=3D 0 test as = an > >> > + =9A =9A* indication of the unused FILE. > >> > + =9A =9A* > >> > + =9A =9A* Taking the lock prevents possible compiler or processor > >> > + =9A =9A* reordering of the writes performed before the final _flags > >> > + =9A =9A* cleanup, making sure that we are done with the FILE before > >> > + =9A =9A* it is considered available. > >> > + =9A =9A*/ > >> > + =9A STDIO_THREAD_LOCK(); > >> > =9A =9A fp->_flags =3D 0; =9A =9A =9A =9A /* Release this FILE for r= euse. */ > >> > + =9A STDIO_THREAD_UNLOCK(); > >> > =9A =9A FUNLOCKFILE(fp); > >> > =9A =9A return (r); > >> > >> Is that assumption about reordering correct? =9AI.e. is FreeBSD's spin= lock a > >> two-way barrier? > >> > >> It isn't unusual for the locking primitive to be a one-way barrier, i.= e. (from > >> Linux kernel memory-barriers.txt) "Memory operations that occur before= a LOCK > >> operation may appear to happen after it completes." =9ASee also acq an= d rel in > >> atomic(9). > > Taking the lock prevents the __sfp from iterating the list until the > > spinlock is released. Since release makes sure that all previous memory > > operations become visible, the acquire in the spinlock lock provides > > the neccesary guarentee. >=20 > IMHO, the lock to me is too heavy here. What about this patch? >=20 > NOTE: patch just show thoughts. I didn't even check build checking. Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers over the rmb/wmb. Also, the lock is not that heavy right there, and the committed patch provides mostly zero overhead for non-threaded case. >=20 > diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c > index f0629e8..a26f944 100644 > --- a/lib/libc/stdio/fclose.c > +++ b/lib/libc/stdio/fclose.c > @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$"); >=20 > #include "namespace.h" > #include > +#include > #include > #include > #include "un-namespace.h" > @@ -65,6 +66,7 @@ fclose(FILE *fp) > FREELB(fp); > fp->_file =3D -1; > fp->_r =3D fp->_w =3D 0; /* Mess up if reaccessed. */ > + wmb(); > fp->_flags =3D 0; /* Release this FILE for reuse. */ > FUNLOCKFILE(fp); > return (r); > diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c > index 89c0536..03b2945 100644 > --- a/lib/libc/stdio/findfp.c > +++ b/lib/libc/stdio/findfp.c > @@ -129,9 +129,16 @@ __sfp() > */ > THREAD_LOCK(); > for (g =3D &__sglue; g !=3D NULL; g =3D g->next) { > - for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++) > - if (fp->_flags =3D=3D 0) > + for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++) { > + int __flags =3D fp->_flags; > + rmb(); > + /* > + * If could see __flags is zero here, we are sure > + * the cleanup in fclose is done. > + */ > + if (__flags =3D=3D 0) > goto found; > + } > } > THREAD_UNLOCK(); /* don't hold lock while malloc()ing. */ > if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL) --REd8r87flCTRYq/l Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk+XmRMACgkQC3+MBN1Mb4jXWQCfTn0XopbMM1f3hqyZQOP46LZW 1jsAn2QhZGPePVNspHwvZ5oeQvSChtWA =ANAw -----END PGP SIGNATURE----- --REd8r87flCTRYq/l-- From owner-freebsd-threads@FreeBSD.ORG Wed Apr 25 07:05:55 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A367C1065679 for ; Wed, 25 Apr 2012 07:05:55 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6286E8FC0C for ; Wed, 25 Apr 2012 07:05:55 +0000 (UTC) Received: by obbuo13 with SMTP id uo13so2643991obb.13 for ; Wed, 25 Apr 2012 00:05:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=4f126NQ7j57y8puiYv7ou8VYlXy5X3OoM3LvZyMgln0=; b=m4ftGWenoRsJqVEEsBCznTVB1DBZIXXnDTT3nHmbeOqLI2tr+GD+Fxzerrdbo3mUqN uNAXsdhWDDNgsHZTHhflTYTBGzes65xs3G9ZiiMGbYu0GGMQu98FnbyuX0Vo9kBk8MPl jXmfsJ4k+uam8PxTPV6nU0gN7QclHmbxLNenAKelvT8b7YiDvMZQWsJVStl6DELWMvnA 2mmRU5TnrtNPm0vwkOYNNKGAAwJzQiCca1n9cgKoRgiYnqs22rowaGnSJ8Rzzbr3r/uW UvJ1RRQM/WrACyZa19iIeS8tGvaWIXh6QdLQbnU8XQzDmZ65MBhMcVZ7VaU12Y6AvFfP AgKw== MIME-Version: 1.0 Received: by 10.182.113.106 with SMTP id ix10mr1508664obb.26.1335337554738; Wed, 25 Apr 2012 00:05:54 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Wed, 25 Apr 2012 00:05:54 -0700 (PDT) In-Reply-To: <20120425062627.GI2358@deviant.kiev.zoral.com.ua> References: <20120423084120.GD76983@zxy.spb.ru> <201204241343.q3ODhe2C032683@higson.cam.lispworks.com> <20120424140348.GY2358@deviant.kiev.zoral.com.ua> <201204241110.14017.jhb@freebsd.org> <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> <201204241800.q3OI0X89007384@higson.cam.lispworks.com> <20120424181302.GB2358@deviant.kiev.zoral.com.ua> <20120425062627.GI2358@deviant.kiev.zoral.com.ua> Date: Wed, 25 Apr 2012 15:05:54 +0800 Message-ID: From: Fengwei yin To: Konstantin Belousov Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2012 07:05:55 -0000 On Wed, Apr 25, 2012 at 2:26 PM, Konstantin Belousov wrote: > On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote: >> On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov >> wrote: >> > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote: >> >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said: >> >> > >> >> > + >> >> > + =A0 /* >> >> > + =A0 =A0* Lock the spinlock used to protect __sglue list walk in >> >> > + =A0 =A0* __sfp(). =A0The __sfp() uses fp->_flags =3D=3D 0 test as= an >> >> > + =A0 =A0* indication of the unused FILE. >> >> > + =A0 =A0* >> >> > + =A0 =A0* Taking the lock prevents possible compiler or processor >> >> > + =A0 =A0* reordering of the writes performed before the final _fla= gs >> >> > + =A0 =A0* cleanup, making sure that we are done with the FILE befo= re >> >> > + =A0 =A0* it is considered available. >> >> > + =A0 =A0*/ >> >> > + =A0 STDIO_THREAD_LOCK(); >> >> > =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE for = reuse. */ >> >> > + =A0 STDIO_THREAD_UNLOCK(); >> >> > =A0 =A0 FUNLOCKFILE(fp); >> >> > =A0 =A0 return (r); >> >> >> >> Is that assumption about reordering correct? =A0I.e. is FreeBSD's spi= nlock a >> >> two-way barrier? >> >> >> >> It isn't unusual for the locking primitive to be a one-way barrier, i= .e. (from >> >> Linux kernel memory-barriers.txt) "Memory operations that occur befor= e a LOCK >> >> operation may appear to happen after it completes." =A0See also acq a= nd rel in >> >> atomic(9). >> > Taking the lock prevents the __sfp from iterating the list until the >> > spinlock is released. Since release makes sure that all previous memor= y >> > operations become visible, the acquire in the spinlock lock provides >> > the neccesary guarentee. >> >> IMHO, the lock to me is too heavy here. What about this patch? >> >> NOTE: patch just show thoughts. I didn't even check build checking. > Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers > over the rmb/wmb. > There is no stand alone acq/rel APIs (Sorry, as new guy to FreeBSD, don't know too much APIs). They are bound to atomic operations. And yes, atomic_acq/rel should work also. > Also, the lock is not that heavy right there, and the committed patch > provides mostly zero overhead for non-threaded case. But lock could introduced contention in SMP case which could be avoid with rmb/wmb. I wonder whether the rmb/wmb could be defined as a compiler barrier on non-threaded case. Like STDIO_THREAD_LOCK which has version for thread/non-thread case. >> >> diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c >> index f0629e8..a26f944 100644 >> --- a/lib/libc/stdio/fclose.c >> +++ b/lib/libc/stdio/fclose.c >> @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$"); >> >> =A0#include "namespace.h" >> =A0#include >> +#include >> =A0#include >> =A0#include >> =A0#include "un-namespace.h" >> @@ -65,6 +66,7 @@ fclose(FILE *fp) >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 FREELB(fp); >> =A0 =A0 =A0 fp->_file =3D -1; >> =A0 =A0 =A0 fp->_r =3D fp->_w =3D 0; =A0 =A0/* Mess up if reaccessed. */ >> + =A0 =A0 wmb(); >> =A0 =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE for r= euse. */ >> =A0 =A0 =A0 FUNLOCKFILE(fp); >> =A0 =A0 =A0 return (r); >> diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c >> index 89c0536..03b2945 100644 >> --- a/lib/libc/stdio/findfp.c >> +++ b/lib/libc/stdio/findfp.c >> @@ -129,9 +129,16 @@ __sfp() >> =A0 =A0 =A0 =A0*/ >> =A0 =A0 =A0 THREAD_LOCK(); >> =A0 =A0 =A0 for (g =3D &__sglue; g !=3D NULL; g =3D g->next) { >> - =A0 =A0 =A0 =A0 =A0 =A0 for (fp =3D g->iobs, n =3D g->niobs; --n >=3D = 0; fp++) >> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (fp->_flags =3D=3D 0) >> + =A0 =A0 =A0 =A0 =A0 =A0 for (fp =3D g->iobs, n =3D g->niobs; --n >=3D = 0; fp++) { >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 int __flags =3D fp->_flags; >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 rmb(); >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /* >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* If could see __flags is z= ero here, we are sure >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* the cleanup in fclose is = done. >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0*/ >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (__flags =3D=3D 0) >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 goto found; >> + =A0 =A0 =A0 =A0 =A0 =A0 } >> =A0 =A0 =A0 } >> =A0 =A0 =A0 THREAD_UNLOCK(); =A0 =A0 =A0 =A0/* don't hold lock while mal= loc()ing. */ >> =A0 =A0 =A0 if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL) From owner-freebsd-threads@FreeBSD.ORG Wed Apr 25 14:15:30 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DCDDE1065675 for ; Wed, 25 Apr 2012 14:15:29 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id B00D68FC1B for ; Wed, 25 Apr 2012 14:15:29 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 0F7B8B915; Wed, 25 Apr 2012 10:15:29 -0400 (EDT) From: John Baldwin To: freebsd-threads@freebsd.org Date: Wed, 25 Apr 2012 10:12:36 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; ) References: <20120423084120.GD76983@zxy.spb.ru> <20120425062627.GI2358@deviant.kiev.zoral.com.ua> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201204251012.36754.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 25 Apr 2012 10:15:29 -0400 (EDT) Cc: jack.ren@intel.com Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2012 14:15:30 -0000 On Wednesday, April 25, 2012 3:05:54 am Fengwei yin wrote: > On Wed, Apr 25, 2012 at 2:26 PM, Konstantin Belousov > wrote: > > On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote: > >> On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov > >> wrote: > >> > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote: > >> >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said: > >> >> > > >> >> > + > >> >> > + /* > >> >> > + * Lock the spinlock used to protect __sglue list walk in > >> >> > + * __sfp(). The __sfp() uses fp->_flags == 0 test as an > >> >> > + * indication of the unused FILE. > >> >> > + * > >> >> > + * Taking the lock prevents possible compiler or processor > >> >> > + * reordering of the writes performed before the final _flags > >> >> > + * cleanup, making sure that we are done with the FILE before > >> >> > + * it is considered available. > >> >> > + */ > >> >> > + STDIO_THREAD_LOCK(); > >> >> > fp->_flags = 0; /* Release this FILE for reuse. */ > >> >> > + STDIO_THREAD_UNLOCK(); > >> >> > FUNLOCKFILE(fp); > >> >> > return (r); > >> >> > >> >> Is that assumption about reordering correct? I.e. is FreeBSD's spinlock a > >> >> two-way barrier? > >> >> > >> >> It isn't unusual for the locking primitive to be a one-way barrier, i.e. (from > >> >> Linux kernel memory-barriers.txt) "Memory operations that occur before a LOCK > >> >> operation may appear to happen after it completes." See also acq and rel in > >> >> atomic(9). > >> > Taking the lock prevents the __sfp from iterating the list until the > >> > spinlock is released. Since release makes sure that all previous memory > >> > operations become visible, the acquire in the spinlock lock provides > >> > the neccesary guarentee. > >> > >> IMHO, the lock to me is too heavy here. What about this patch? > >> > >> NOTE: patch just show thoughts. I didn't even check build checking. > > Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers > > over the rmb/wmb. > > > > There is no stand alone acq/rel APIs (Sorry, as new guy to FreeBSD, > don't know too much APIs). They are bound to atomic operations. > And yes, atomic_acq/rel should work also. Yes, you would want to use atomic_store_rel() or some such. Often doing so is much clearer than stand-alone membar's as it indicates which write has special ordering. > > Also, the lock is not that heavy right there, and the committed patch > > provides mostly zero overhead for non-threaded case. > > But lock could introduced contention in SMP case which could be avoid > with rmb/wmb. Seriously, if you are using stdio, you've already given up on performance enough to not care about contention for fclose() vs fopen(). -- John Baldwin From owner-freebsd-threads@FreeBSD.ORG Wed Apr 25 22:55:18 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 71B951065676 for ; Wed, 25 Apr 2012 22:55:18 +0000 (UTC) (envelope-from gofdt-freebsd-threads@m.gmane.org) Received: from plane.gmane.org (plane.gmane.org [80.91.229.3]) by mx1.freebsd.org (Postfix) with ESMTP id D95288FC0A for ; Wed, 25 Apr 2012 22:55:14 +0000 (UTC) Received: from list by plane.gmane.org with local (Exim 4.69) (envelope-from ) id 1SNB71-0007PK-Vz for freebsd-threads@freebsd.org; Thu, 26 Apr 2012 00:55:03 +0200 Received: from 189.32.123.86 ([189.32.123.86]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 26 Apr 2012 00:55:03 +0200 Received: from rnsanchez by 189.32.123.86 with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 26 Apr 2012 00:55:03 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-threads@freebsd.org From: Ricardo Nabinger Sanchez Date: Wed, 25 Apr 2012 22:51:08 +0000 (UTC) Lines: 19 Message-ID: References: <20120423084120.GD76983@zxy.spb.ru> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 189.32.123.86 User-Agent: Pan/0.134 (Wait for Me; GIT cb32159 master) Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2012 22:55:18 -0000 On Mon, 23 Apr 2012 12:41:20 +0400, Slawa Olhovchenkov wrote: > /usr/include/machine/atomic.h: > > #define mb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > #define wmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > #define rmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") Somewhat late on this topic, but I'd like to understand why issue a write on %esp, which would invalidate (%esp) on other cores --- thus forcing a miss on them? Instead, why not issue "mfence" (mb), "sfence" (wmb), and "lfence" (rmb)? Cheers -- Ricardo Nabinger Sanchez http://rnsanchez.wait4.org/ "Left to themselves, things tend to go from bad to worse." From owner-freebsd-threads@FreeBSD.ORG Thu Apr 26 01:20:45 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1D3BF106564A; Thu, 26 Apr 2012 01:20:45 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id C6AF88FC0A; Thu, 26 Apr 2012 01:20:44 +0000 (UTC) Received: by obcni5 with SMTP id ni5so21264obc.13 for ; Wed, 25 Apr 2012 18:20:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=WjzlvHMwpyiMQS/B6AkjFHglEnJU0EOnKxyxlD36AhE=; b=HmWEMEH2OovsMeEpM1anrapxCfxNYBTUlyJfAUP7iFTsCVxZb50AcHFFJvFZue4Y6S eGy6cderNpn+xJ73EnGAAhvTZ0hA7cE9plbelIl2FaZmle3t6z15MHXKRE76vZN07Nfx c/5wrmNAe4+dwXGgGMrsodZqZlJB5dPPn0PytsdoVW1Uyl1Y+6PMUM7smpkdahDp2fHK ljYg9XIyZWct8gz56blndS0bPAdaiw1Qu6c0IN6NRO8E3e7gtJll/fLlrNsnGybiMZhd 6C7qaJHvn2/n15yRR3sZVRu08t0f5AStbjAeoWXgs0aocgOCKyAZifKwEoYQFpLMQ6Sa zKXQ== MIME-Version: 1.0 Received: by 10.182.113.106 with SMTP id ix10mr6046003obb.26.1335403244475; Wed, 25 Apr 2012 18:20:44 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Wed, 25 Apr 2012 18:20:44 -0700 (PDT) In-Reply-To: <201204251012.36754.jhb@freebsd.org> References: <20120423084120.GD76983@zxy.spb.ru> <20120425062627.GI2358@deviant.kiev.zoral.com.ua> <201204251012.36754.jhb@freebsd.org> Date: Thu, 26 Apr 2012 09:20:44 +0800 Message-ID: From: Fengwei yin To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: jack.ren@intel.com, freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 01:20:45 -0000 On Wed, Apr 25, 2012 at 10:12 PM, John Baldwin wrote: > On Wednesday, April 25, 2012 3:05:54 am Fengwei yin wrote: >> On Wed, Apr 25, 2012 at 2:26 PM, Konstantin Belousov >> wrote: >> > On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote: >> >> On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov >> >> wrote: >> >> > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote: >> >> >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said= : >> >> >> > >> >> >> > + >> >> >> > + =A0 /* >> >> >> > + =A0 =A0* Lock the spinlock used to protect __sglue list walk i= n >> >> >> > + =A0 =A0* __sfp(). =A0The __sfp() uses fp->_flags =3D=3D 0 test= as an >> >> >> > + =A0 =A0* indication of the unused FILE. >> >> >> > + =A0 =A0* >> >> >> > + =A0 =A0* Taking the lock prevents possible compiler or process= or >> >> >> > + =A0 =A0* reordering of the writes performed before the final _= flags >> >> >> > + =A0 =A0* cleanup, making sure that we are done with the FILE b= efore >> >> >> > + =A0 =A0* it is considered available. >> >> >> > + =A0 =A0*/ >> >> >> > + =A0 STDIO_THREAD_LOCK(); >> >> >> > =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE f= or reuse. */ >> >> >> > + =A0 STDIO_THREAD_UNLOCK(); >> >> >> > =A0 =A0 FUNLOCKFILE(fp); >> >> >> > =A0 =A0 return (r); >> >> >> >> >> >> Is that assumption about reordering correct? =A0I.e. is FreeBSD's = spinlock a >> >> >> two-way barrier? >> >> >> >> >> >> It isn't unusual for the locking primitive to be a one-way barrier= , i.e. (from >> >> >> Linux kernel memory-barriers.txt) "Memory operations that occur be= fore a LOCK >> >> >> operation may appear to happen after it completes." =A0See also ac= q and rel in >> >> >> atomic(9). >> >> > Taking the lock prevents the __sfp from iterating the list until th= e >> >> > spinlock is released. Since release makes sure that all previous me= mory >> >> > operations become visible, the acquire in the spinlock lock provide= s >> >> > the neccesary guarentee. >> >> >> >> IMHO, the lock to me is too heavy here. What about this patch? >> >> >> >> NOTE: patch just show thoughts. I didn't even check build checking. >> > Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers >> > over the rmb/wmb. >> > >> >> There is no stand alone acq/rel APIs (Sorry, as new guy to FreeBSD, >> don't know too much APIs). =A0They are bound to atomic operations. >> And yes, atomic_acq/rel should work also. > > Yes, you would want to use atomic_store_rel() or some such. =A0Often doin= g > so is much clearer than stand-alone membar's as it indicates which write > has special ordering. > Thanks a lot for clarification. >> > Also, the lock is not that heavy right there, and the committed patch >> > provides mostly zero overhead for non-threaded case. >> >> But lock could introduced contention in SMP case which could be avoid >> with rmb/wmb. > > Seriously, if you are using stdio, you've already given up on performance > enough to not care about contention for fclose() vs fopen(). > This makes sense. > -- > John Baldwin From owner-freebsd-threads@FreeBSD.ORG Thu Apr 26 01:28:14 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 598B71065672 for ; Thu, 26 Apr 2012 01:28:14 +0000 (UTC) (envelope-from yfw.bsd@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1E4A98FC12 for ; Thu, 26 Apr 2012 01:28:14 +0000 (UTC) Received: by obcni5 with SMTP id ni5so31389obc.13 for ; Wed, 25 Apr 2012 18:28:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=DtorzYfPs6Wg7tZ4W7ru3KQv5XnfWPLXTxtZ2NGD5n4=; b=uwznZO6I/rC5xBWseVHYXM0f18o5oEsVVqq4xgWW+PiQM5OIhlqEMz6LZtwY/A/vEY uMabt2XWNAwaRJ97+Qrv09EHZCyNoCHbvnjBNZrBTsedFwR/8v342zsqmECGR83rBO0h 1FaQNHk4U2x+H9nbZJdgMYdkpUNwfWVoDA3ZNBUVPEd0pbTojcnTOIcv18rkeRPVZ8Pm o+wHCX8+UgAzy54huvXRam9oCviuZmSBG95z3JUO/8IMoofsvChJ7VD+g7Td+I5aqIZE 9FtXMPVT6vDOwA5EgMJvha4QU57XBIdKRGAWw/WwGajNoVYclZv/R8o1KsHZAzvjMvEq dbAg== MIME-Version: 1.0 Received: by 10.60.22.10 with SMTP id z10mr6622939oee.16.1335403693695; Wed, 25 Apr 2012 18:28:13 -0700 (PDT) Received: by 10.60.125.135 with HTTP; Wed, 25 Apr 2012 18:28:13 -0700 (PDT) In-Reply-To: References: <20120423084120.GD76983@zxy.spb.ru> Date: Thu, 26 Apr 2012 09:28:13 +0800 Message-ID: From: Fengwei yin To: Ricardo Nabinger Sanchez Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 01:28:14 -0000 On Thu, Apr 26, 2012 at 6:51 AM, Ricardo Nabinger Sanchez wrote: > On Mon, 23 Apr 2012 12:41:20 +0400, Slawa Olhovchenkov wrote: > >> /usr/include/machine/atomic.h: >> >> #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : "memo= ry") >> #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory= ") >> #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory= ") > > Somewhat late on this topic, but I'd like to understand why issue a write > on %esp, which would invalidate (%esp) on other cores --- thus forcing a > miss on them? The key here is prefix "lock:" which could be considered as "mfence" on modern CPU. > > Instead, why not issue "mfence" (mb), "sfence" (wmb), and "lfence" (rmb)? > Not all x86 family support "fence" instruction. "lock" prefix works for all= . > Cheers > > -- > Ricardo Nabinger Sanchez =A0 =A0 =A0 =A0 =A0 http://rnsanchez.wait4.org/ > =A0"Left to themselves, things tend to go from bad to worse." > > _______________________________________________ > freebsd-threads@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-threads > To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org= " From owner-freebsd-threads@FreeBSD.ORG Thu Apr 26 01:30:10 2012 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2EEDE1065670 for ; Thu, 26 Apr 2012 01:30:10 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id DC7148FC15 for ; Thu, 26 Apr 2012 01:30:09 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3Q1U93l099234 for ; Thu, 26 Apr 2012 01:30:09 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3Q1U9DO099233; Thu, 26 Apr 2012 01:30:09 GMT (envelope-from gnats) Resent-Date: Thu, 26 Apr 2012 01:30:09 GMT Resent-Message-Id: <201204260130.q3Q1U9DO099233@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-threads@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Rick Reed Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F7A61065672 for ; Thu, 26 Apr 2012 01:20:27 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id EF5DF8FC0C for ; Thu, 26 Apr 2012 01:20:26 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3Q1KQix006765 for ; Thu, 26 Apr 2012 01:20:26 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q3Q1KQCT006764; Thu, 26 Apr 2012 01:20:26 GMT (envelope-from nobody) Message-Id: <201204260120.q3Q1KQCT006764@red.freebsd.org> Date: Thu, 26 Apr 2012 01:20:26 GMT From: Rick Reed To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: threads/167308: Perf regression in thread locking on 8-stable X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 01:30:10 -0000 >Number: 167308 >Category: threads >Synopsis: Perf regression in thread locking on 8-stable >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-threads >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Apr 26 01:30:09 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Rick Reed >Release: 8.3-STABLE (cvsup @4/17/12 1300) >Organization: WhatsApp Inc. >Environment: 8.3-STABLE amd64 >Description: Rev 234373 causes a significant performance regression for our Erlang-based application. I don't know what the direct results of the change are, but it manifests as abnormal amounts of contention for a particular global lock among the scheduler threads (one per CPU) in the Erlang VM under significant load. Backing this change out results in normal contention and performance. I see there's new code using WAKE2 in HEAD & 9, but I'm not sure how you balance the correctness issue in the stated case of unlock() then destroy() versus the significant impact to performance for apps which don't do that. >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: From owner-freebsd-threads@FreeBSD.ORG Thu Apr 26 01:44:27 2012 Return-Path: Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 449A3106566B; Thu, 26 Apr 2012 01:44:27 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 17E5F8FC08; Thu, 26 Apr 2012 01:44:27 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3Q1iQSU018672; Thu, 26 Apr 2012 01:44:26 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3Q1iQnG018668; Thu, 26 Apr 2012 01:44:26 GMT (envelope-from linimon) Date: Thu, 26 Apr 2012 01:44:26 GMT Message-Id: <201204260144.q3Q1iQnG018668@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-threads@FreeBSD.org, davidxu@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: threads/167308: Performance regression in thread locking on 8-stable [regression] X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 01:44:27 -0000 Old Synopsis: Perf regression in thread locking on 8-stable New Synopsis: Performance regression in thread locking on 8-stable [regression] Responsible-Changed-From-To: freebsd-threads->davidxu Responsible-Changed-By: linimon Responsible-Changed-When: Thu Apr 26 01:41:20 UTC 2012 Responsible-Changed-Why: Over to committer of r234373. http://www.freebsd.org/cgi/query-pr.cgi?pr=167308 From owner-freebsd-threads@FreeBSD.ORG Thu Apr 26 02:45:32 2012 Return-Path: Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E1B8B106566C; Thu, 26 Apr 2012 02:45:32 +0000 (UTC) (envelope-from listlog2011@gmail.com) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CBE728FC16; Thu, 26 Apr 2012 02:45:32 +0000 (UTC) Received: from [127.0.0.1] (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3Q2jV1D074612; Thu, 26 Apr 2012 02:45:31 GMT (envelope-from listlog2011@gmail.com) Message-ID: <4F98B6C8.30703@gmail.com> Date: Thu, 26 Apr 2012 10:45:28 +0800 From: David Xu User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20120420 Thunderbird/12.0 MIME-Version: 1.0 To: linimon@FreeBSD.org References: <201204260144.q3Q1iQnG018668@freefall.freebsd.org> In-Reply-To: <201204260144.q3Q1iQnG018668@freefall.freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: davidxu@FreeBSD.org, freebsd-threads@FreeBSD.org Subject: Re: threads/167308: Performance regression in thread locking on 8-stable [regression] X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: davidxu@FreeBSD.org List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 02:45:33 -0000 On 2012/4/26 9:44, linimon@FreeBSD.org wrote: > Old Synopsis: Perf regression in thread locking on 8-stable > New Synopsis: Performance regression in thread locking on 8-stable [regression] > > Responsible-Changed-From-To: freebsd-threads->davidxu > Responsible-Changed-By: linimon > Responsible-Changed-When: Thu Apr 26 01:41:20 UTC 2012 > Responsible-Changed-Why: > Over to committer of r234373. > > http://www.freebsd.org/cgi/query-pr.cgi?pr=167308 > This is a known problem that the unlocking is not optimized when the mutex is heavily contented, I may MFC wake2 later. From owner-freebsd-threads@FreeBSD.ORG Thu Apr 26 22:43:49 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EACD3106566B for ; Thu, 26 Apr 2012 22:43:49 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) by mx1.freebsd.org (Postfix) with ESMTP id 4A84F8FC15 for ; Thu, 26 Apr 2012 22:43:49 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id 763BC3592E6; Fri, 27 Apr 2012 00:43:48 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 5953A2847A; Fri, 27 Apr 2012 00:43:48 +0200 (CEST) Date: Fri, 27 Apr 2012 00:43:48 +0200 From: Jilles Tjoelker To: Ricardo Nabinger Sanchez Message-ID: <20120426224348.GA58463@stack.nl> References: <20120423084120.GD76983@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-threads@freebsd.org Subject: Re: About the memory barrier in BSD libc X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Apr 2012 22:43:50 -0000 On Wed, Apr 25, 2012 at 10:51:08PM +0000, Ricardo Nabinger Sanchez wrote: > On Mon, 23 Apr 2012 12:41:20 +0400, Slawa Olhovchenkov wrote: > > /usr/include/machine/atomic.h: > > #define mb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > #define wmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > > #define rmb() __asm __volatile("lock; addl $0,(%%esp)" : : : "memory") > Somewhat late on this topic, but I'd like to understand why issue a write > on %esp, which would invalidate (%esp) on other cores --- thus forcing a > miss on them? The stack is usually private to a thread and is written a lot regardless. > Instead, why not issue "mfence" (mb), "sfence" (wmb), and "lfence" (rmb)? Apart from the fact that those are SSE or SSE2 instructions, the uncontended locked instructions may be faster. For example, MFENCE is a serializing instruction on AMD family 10h processors and the AMD optimization manual recommends using an uncontended locked instruction instead (though preferably one that performs a useful store). SFENCE is only useful with non-temporal stores such as MOVNTPS or stores to write-combining memory because regular stores are not reordered with respect to one another. -- Jilles Tjoelker From owner-freebsd-threads@FreeBSD.ORG Sat Apr 28 12:50:58 2012 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 39C8D106566C for ; Sat, 28 Apr 2012 12:50:58 +0000 (UTC) (envelope-from chris.hall@highwayman.com) Received: from anchor-post-1.mail.demon.net (anchor-post-1.mail.demon.net [195.173.77.132]) by mx1.freebsd.org (Postfix) with ESMTP id EE8D78FC14 for ; Sat, 28 Apr 2012 12:50:57 +0000 (UTC) Received: from [80.177.246.130] (helo=hestia.halldom.com) by anchor-post-1.mail.demon.net with esmtp (Exim 4.69) id 1SO71B-0006MB-iE for freebsd-threads@freebsd.org; Sat, 28 Apr 2012 12:44:53 +0000 Received: from hyperion.halldom.com ([80.177.246.170] helo=HYPERION) by hestia.halldom.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.76) (envelope-from ) id 1SO719-0002Ti-H8 for freebsd-threads@freebsd.org; Sat, 28 Apr 2012 12:44:51 +0000 From: "'Chris Hall'" Sender: "Chris Hall" To: Date: Sat, 28 Apr 2012 13:44:46 +0100 Organization: Highwayman Message-ID: <056201cd253c$b43a0860$1cae1920$@highwayman.com> MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Outlook 14.0 Thread-Index: Ac0lPKnGhmxEP7zpSCa0p5c0+tQlcw== Content-Language: en-gb Subject: Trying to set PTHREAD_SCOPE_SYSTEM X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Apr 2012 12:50:58 -0000 For my application (a BGP daemon) I think I should be setting all its pthreads to PTHREAD_SCOPE_SYSTEM. (This may be where I am going wrong, of course, but bear with me.) I have studied the POSIX specifications and the FreeBSD man pages, and am stuck. pthread_init_attr() is supposed to give me the default settings for everything, including the scope and PTHREAD_INHERIT_SCHED/_EXPLICIT_SCHED. First difficulty: if it is PTHREAD_INHERIT_SCHED, what are the scope, schedpolicy and schedparam set to ? Second difficulty: if I set PTHREAD_EXPLICIT_SCHED, will that change the scope/policy/param to the defaults, or leave them unchanged (in which case are they the appropriate defaults) ? Or, am I really expected to set all of scope/policy/param if I override a default PTHREAD_INHERIT_SCHED ? (If the default is PTHREAD_SCOPE_SYSTEM, am I all set having cleared PTHREAD_INHERIT_SCHED ?) Third difficulty: if I set PTHREAD_SCOPE_SYSTEM and the default is PTHREAD_SCOPE_PROCESS, do I also need to set policy/param ? If so, where do I find suitable default policy/param ? (I could copy the policy/param from the main thread... but will that be suitable for PTHREAD_SCOPE_SYSTEM ?) Fourth difficulty: if I dick about with scope/policy/param, will I need special privileges (root, say) ? Fifth difficulty: it is clearly easiest to leave this whole thing alone and ignore pthread_attr_t altogether. Is there a way to set PTHREAD_SCOPE_SYSTEM as the default for a given application from the outside ? POSIX appears to define a bunch of stuff in this area... but so much is implementation defined, or simply not defined I imagine (a) 99.9% of threaded applications do not attempt to use pthread_attr_t at all, and (b) that somewhere each implementation will define what it does ? Help !! Thanks, Chris