From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 06:56:03 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id C1055106566C
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 06:56:03 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 89F3C8FC15
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 06:56:03 +0000 (UTC)
Received: by obqv19 with SMTP id v19so17465213obq.13
	for <freebsd-threads@freebsd.org>; Sun, 22 Apr 2012 23:56:03 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:date:message-id:subject:from:to:cc:content-type;
	bh=MFCw7DAWBYtbw+J4oXzrbt787AC6kyklN7A1sGodMNM=;
	b=b87ZjzG8mFHLrbct5sMtJirUI9kdvMEt56LgjncyYuUUmwsA3TcyMzeaux4okn/Fii
	8P8U4SV2JJAW9SDQYtHdl4/hhxiimVFT/xShhtMsgLO9JueSGkelI4IXqxPhJn7lPyUg
	eZWvnXtL4z5l47F9xOdpvOh03lVUtkZ7apW4Ays2RsIm2sggye61q0Wo0cKJEwaqf7IT
	c0yxY+0gIhPZYnAZHIrgErcYne3Dt6WLT2YR3SF0U4RBA1TuwjibQgYsgJRxG0n/9MNZ
	gI/0FEKmAATDP5XGYX+YrLJW7/H8ZOaBk7pkc2Z9vmPtpl/9ZgbORC0s8uUkqLjirX1B
	iekw==
MIME-Version: 1.0
Received: by 10.182.232.38 with SMTP id tl6mr5975973obc.16.1335164163204; Sun,
	22 Apr 2012 23:56:03 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Sun, 22 Apr 2012 23:56:03 -0700 (PDT)
Date: Mon, 23 Apr 2012 14:56:03 +0800
Message-ID: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: freebsd-threads@freebsd.org
Content-Type: text/plain; charset=ISO-8859-1
Cc: jack.ren@intel.com
Subject: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 06:56:03 -0000

Hi list,
If this is not correct question on the list, please let me know and
sorry for noise.

I have a question regarding the BSD libc for SMP arch. I didn't see
memory barrier used in libc.
How can we make sure it's safe on SMP arch?

BTW, there are memory barrier used in kernel instead. So I suppose the
memory barrier is needed.

Regards
Yin, Fengwei

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 08:40:58 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 9F784106564A
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 08:40:58 +0000 (UTC)
	(envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
	by mx1.freebsd.org (Postfix) with ESMTP id 551B28FC14
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 08:40:58 +0000 (UTC)
Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD))
	(envelope-from <slw@zxy.spb.ru>)
	id 1SMEpk-000DXV-Cj; Mon, 23 Apr 2012 12:41:20 +0400
Date: Mon, 23 Apr 2012 12:41:20 +0400
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Fengwei yin <yfw.bsd@gmail.com>
Message-ID: <20120423084120.GD76983@zxy.spb.ru>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 08:40:58 -0000

On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:

> Hi list,
> If this is not correct question on the list, please let me know and
> sorry for noise.
> 
> I have a question regarding the BSD libc for SMP arch. I didn't see
> memory barrier used in libc.
> How can we make sure it's safe on SMP arch?

/usr/include/machine/atomic.h:

#define mb()    __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
#define wmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
#define rmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")


From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 09:32:25 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id CF931106566C
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 09:32:25 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 940708FC08
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 09:32:25 +0000 (UTC)
Received: by obqv19 with SMTP id v19so17683500obq.13
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 02:32:25 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=f1iNjodDdC9J9l+h0oKfo/K2Q5EB9PJfvR15auz99UI=;
	b=YcbcKgZ1T25m6hAFhmKeM4ZAXT+r/um7yHhVi3uXwAGHTNBTRcAukRsvh9W3ubZsDO
	ywaxXdsJrLQJSXDyDliworBZ238Nquz8xKUrFJgnhwBOPz5kePMGGxND82ttVY/xQmc6
	137vtZSLvFgxw4U7eaWTZk7NgB5DynEpE66OZ9ZBOpWIPG1T8hq8eKe2BYld+CDNVw3x
	dMT2NgrI1k74znYSWegKjOr5BAXCXzam7rINBUYn+AaYKt3GLFpFN9kxjSxC4fMzTB51
	zSAds8uKvSYwHEqFwSyh2bVNA/i/Rhpv6AMvve8d8NVwrXc4A3XoV+4ecYChDHQMrbW6
	A53Q==
MIME-Version: 1.0
Received: by 10.60.29.39 with SMTP id g7mr2407884oeh.6.1335173544900; Mon, 23
	Apr 2012 02:32:24 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 02:32:24 -0700 (PDT)
In-Reply-To: <20120423084120.GD76983@zxy.spb.ru>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
Date: Mon, 23 Apr 2012 17:32:24 +0800
Message-ID: <CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Slawa Olhovchenkov <slw@zxy.spb.ru>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 09:32:25 -0000

On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
>
>> Hi list,
>> If this is not correct question on the list, please let me know and
>> sorry for noise.
>>
>> I have a question regarding the BSD libc for SMP arch. I didn't see
>> memory barrier used in libc.
>> How can we make sure it's safe on SMP arch?
>
> /usr/include/machine/atomic.h:
>
> #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : "memor=
y")
> #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory"=
)
> #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory"=
)
>

Thanks for the information. But it looks no body use it in libc.

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 09:40:19 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id B081B1065680
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 09:40:19 +0000 (UTC)
	(envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
	by mx1.freebsd.org (Postfix) with ESMTP id 642148FC19
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 09:40:19 +0000 (UTC)
Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD))
	(envelope-from <slw@zxy.spb.ru>)
	id 1SMFlD-000Eno-CU; Mon, 23 Apr 2012 13:40:43 +0400
Date: Mon, 23 Apr 2012 13:40:43 +0400
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Fengwei yin <yfw.bsd@gmail.com>
Message-ID: <20120423094043.GS32749@zxy.spb.ru>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 09:40:19 -0000

On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:

> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
> >
> >> Hi list,
> >> If this is not correct question on the list, please let me know and
> >> sorry for noise.
> >>
> >> I have a question regarding the BSD libc for SMP arch. I didn't see
> >> memory barrier used in libc.
> >> How can we make sure it's safe on SMP arch?
> >
> > /usr/include/machine/atomic.h:
> >
> > #define mb() š š__asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> > #define wmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> > #define rmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> >
> 
> Thanks for the information. But it looks no body use it in libc.

I think no body in libc need memory barrier: libc don't work with
peripheral, for atomic opertions used different macros.

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 11:07:27 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id BDEB51065676
	for <freebsd-threads@FreeBSD.org>; Mon, 23 Apr 2012 11:07:27 +0000 (UTC)
	(envelope-from owner-bugmaster@FreeBSD.org)
Received: from freefall.freebsd.org (freefall.freebsd.org
	[IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id 8EC768FC26
	for <freebsd-threads@FreeBSD.org>; Mon, 23 Apr 2012 11:07:27 +0000 (UTC)
Received: from freefall.freebsd.org (localhost [127.0.0.1])
	by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3NB7RPl047738
	for <freebsd-threads@FreeBSD.org>; Mon, 23 Apr 2012 11:07:27 GMT
	(envelope-from owner-bugmaster@FreeBSD.org)
Received: (from gnats@localhost)
	by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3NB7QIV047735
	for freebsd-threads@FreeBSD.org; Mon, 23 Apr 2012 11:07:26 GMT
	(envelope-from owner-bugmaster@FreeBSD.org)
Date: Mon, 23 Apr 2012 11:07:26 GMT
Message-Id: <201204231107.q3NB7QIV047735@freefall.freebsd.org>
X-Authentication-Warning: freefall.freebsd.org: gnats set sender to
	owner-bugmaster@FreeBSD.org using -f
From: FreeBSD bugmaster <bugmaster@FreeBSD.org>
To: freebsd-threads@FreeBSD.org
Cc: 
Subject: Current problem reports assigned to freebsd-threads@FreeBSD.org
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 11:07:27 -0000

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o threa/166932 threads    [libthr] sleepqueue-related memory leak in libthr
o threa/165173 threads    [build] clang buildworld breaks libthr
o threa/163512 threads    libc defaults to single threaded
o threa/160708 threads    possible security problem with RLIMIT_VMEM
o threa/150959 threads    [libc] Stub pthread_once in libc should call _libc_onc
o threa/148515 threads    Memory / syslog strangeness in FreeBSD 8.x ( possible 
o threa/141721 threads    rtprio(1): (id|rt)prio priority resets when new thread
o threa/135673 threads    databases/mysql50-server - MySQL query lock-ups on 7.2
o threa/128922 threads    threads hang with xorg running
o threa/122923 threads    'nice' does not prevent background process from steali
o threa/121336 threads    lang/neko threading ok on UP, broken on SMP (FreeBSD 7
o threa/116668 threads    can no longer use jdk15 with libthr on -stable SMP
o threa/115211 threads    pthread_atfork misbehaves in initial thread
o threa/110636 threads    [request] gdb(1): using gdb with multi thread applicat
o threa/110306 threads    apache 2.0 segmentation violation when calling gethost
o threa/103975 threads    Implicit loading/unloading of libpthread.so may crash 
o threa/101323 threads    [patch] fork(2) in threaded programs broken.
s threa/84483  threads    problems with devel/nspr and -lc_r on 4.x
o threa/80992  threads    abort() sometimes not caught by gdb depending on threa
o threa/79683  threads    svctcp_create() fails if multiple threads call at the 
s threa/76694  threads    fork cause hang in dup()/close() function in child (-l
s threa/48856  threads    Setting SIGCHLD to SIG_IGN still leaves zombies under 
s threa/34536  threads    accept() blocks other threads
s threa/30464  threads    [patch] pthread mutex attributes -- pshared

24 problems total.


From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 11:26:55 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 8D439106566C
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 11:26:55 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 4F2D58FC08
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 11:26:55 +0000 (UTC)
Received: by obqv19 with SMTP id v19so17853572obq.13
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 04:26:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=qihIEsDuVpWLiT/2hXUQEVWVvK6/OsbnX1xkWo913Kk=;
	b=B2fRAxF34iooiH2iAXbvrpKPlfgUtUeBopv7ycclEKKX/TJV7+jk1pCVG4GpVJu5qz
	soAliX0q75JD+7RrhOiHCu53cw4OgHmL94AvYKyyWc7K+g+YLlq3U7XX/H6tYP6ZVdWT
	1TbMF/QcFyOmyqvohL2RM93HAKgMIQqVYuVAAASC3mdIyMxKPn5+6Jk5gZ5MjmC5VSYN
	V5KuFfOnIjtvbBb/9aSydzyf4G/x1yq47n3rTw4oJ9VfLnIFGM+pLKIM/1fAEjqYwSTY
	how18xJLf9ud8meBLz0iD/GIhJulijNHuvYz6mTA2G/C/nY05nxPpnoUUsBdTRcygPdW
	d3MQ==
MIME-Version: 1.0
Received: by 10.60.29.39 with SMTP id g7mr2929689oeh.6.1335180415000; Mon, 23
	Apr 2012 04:26:55 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 04:26:54 -0700 (PDT)
In-Reply-To: <20120423094043.GS32749@zxy.spb.ru>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
Date: Mon, 23 Apr 2012 19:26:54 +0800
Message-ID: <CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Slawa Olhovchenkov <slw@zxy.spb.ru>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 11:26:55 -0000

On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
>
>> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wro=
te:
>> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
>> >
>> >> Hi list,
>> >> If this is not correct question on the list, please let me know and
>> >> sorry for noise.
>> >>
>> >> I have a question regarding the BSD libc for SMP arch. I didn't see
>> >> memory barrier used in libc.
>> >> How can we make sure it's safe on SMP arch?
>> >
>> > /usr/include/machine/atomic.h:
>> >
>> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : "me=
mory")
>> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memo=
ry")
>> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memo=
ry")
>> >
>>
>> Thanks for the information. But it looks no body use it in libc.
>
> I think no body in libc need memory barrier: libc don't work with
> peripheral, for atomic opertions used different macros.

If we check the usage of __sinit(), it is a typical singleton pattern which
needs memory barrier to make sure no potential SMP issue.

Or did I miss something here?

Regards
Yin, Fengwei

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 11:38:15 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 1D6881065675
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 11:38:15 +0000 (UTC)
	(envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
	by mx1.freebsd.org (Postfix) with ESMTP id BD8F88FC0A
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 11:38:14 +0000 (UTC)
Received: from slw by zxy.spb.ru with local (Exim 4.69 (FreeBSD))
	(envelope-from <slw@zxy.spb.ru>)
	id 1SMHbK-000Hxx-A9; Mon, 23 Apr 2012 15:38:38 +0400
Date: Mon, 23 Apr 2012 15:38:38 +0400
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Fengwei yin <yfw.bsd@gmail.com>
Message-ID: <20120423113838.GT32749@zxy.spb.ru>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 11:38:15 -0000

On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:

> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
> >
> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
> >> >
> >> >> Hi list,
> >> >> If this is not correct question on the list, please let me know and
> >> >> sorry for noise.
> >> >>
> >> >> I have a question regarding the BSD libc for SMP arch. I didn't see
> >> >> memory barrier used in libc.
> >> >> How can we make sure it's safe on SMP arch?
> >> >
> >> > /usr/include/machine/atomic.h:
> >> >
> >> > #define mb() š š__asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> >> > #define wmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> >> > #define rmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> >> >
> >>
> >> Thanks for the information. But it looks no body use it in libc.
> >
> > I think no body in libc need memory barrier: libc don't work with
> > peripheral, for atomic opertions used different macros.
> 
> If we check the usage of __sinit(), it is a typical singleton pattern which
> needs memory barrier to make sure no potential SMP issue.
> 
> Or did I miss something here?

What architecture with cache incoherency and FreeBSD support?

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 11:44:35 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 45EEC106566B
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 11:44:35 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 05F4E8FC0C
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 11:44:34 +0000 (UTC)
Received: by obqv19 with SMTP id v19so17879749obq.13
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 04:44:34 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=Yr+TqV8odryk8/PaHmCM25vl1xvoz8pVTnyjKvau+AM=;
	b=x1wqzy6zlbR3Tb+Q0ZqGwHG9iXBvELdXp43COKC/bWKEmOjzp/5prQb4dGwvvl07+E
	uvcHqZchqJl2i58+OeNn/LBFhPFB/zwwyt2bI4o6Uwm49jPZBdohF8K/1s7UDZ+awo2w
	RnHDFU0Ul+XddxGMIP0AlqQwyrjfr/ALvJ4i7AyqqlJUEbOya4PfILSJE5gkUPzvkvWM
	HxFsN9JkZR2I1FaE2s6rvCglIh3DBPP7SBpZqRSVOrkpsGVNaQTbAjFYLGg+HXIACC/m
	JAtsx87AYeXUovGNyAdkHGpXJZMhqMBgyMrhSz3Bq+AuisH+8poTCk9p9SXlh74DCN0n
	9o+g==
MIME-Version: 1.0
Received: by 10.60.29.39 with SMTP id g7mr3006914oeh.6.1335181474441; Mon, 23
	Apr 2012 04:44:34 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 04:44:34 -0700 (PDT)
In-Reply-To: <20120423113838.GT32749@zxy.spb.ru>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
	<20120423113838.GT32749@zxy.spb.ru>
Date: Mon, 23 Apr 2012 19:44:34 +0800
Message-ID: <CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Slawa Olhovchenkov <slw@zxy.spb.ru>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 11:44:35 -0000

On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
>
>> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wro=
te:
>> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
>> >
>> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> =
wrote:
>> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
>> >> >
>> >> >> Hi list,
>> >> >> If this is not correct question on the list, please let me know an=
d
>> >> >> sorry for noise.
>> >> >>
>> >> >> I have a question regarding the BSD libc for SMP arch. I didn't se=
e
>> >> >> memory barrier used in libc.
>> >> >> How can we make sure it's safe on SMP arch?
>> >> >
>> >> > /usr/include/machine/atomic.h:
>> >> >
>> >> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : =
"memory")
>> >> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "m=
emory")
>> >> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "m=
emory")
>> >> >
>> >>
>> >> Thanks for the information. But it looks no body use it in libc.
>> >
>> > I think no body in libc need memory barrier: libc don't work with
>> > peripheral, for atomic opertions used different macros.
>>
>> If we check the usage of __sinit(), it is a typical singleton pattern wh=
ich
>> needs memory barrier to make sure no potential SMP issue.
>>
>> Or did I miss something here?
>
> What architecture with cache incoherency and FreeBSD support?

I suppose it's not related with cache inchoherency (I could be wrong).
It's related
with reorder of instruction by CPU.

Here is the link talking about why need memory barrier for singleton:
http://www.oaklib.org/docs/oak/singleton.html

x86 has strict memory model and may not suffer this kind of issue. But
ARM need to
take care of it IMHO.

Regards
Yin, Fengwei

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 12:07:33 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 4EAF6106564A
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 12:07:33 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200])
	by mx1.freebsd.org (Postfix) with ESMTP id C40F08FC1E
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 12:07:32 +0000 (UTC)
Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1])
	by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3NC7LG8052065;
	Mon, 23 Apr 2012 15:07:21 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1])
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id
	q3NC7KuB006329; Mon, 23 Apr 2012 15:07:20 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: (from kostik@localhost)
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3NC7KIQ006328; 
	Mon, 23 Apr 2012 15:07:20 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to
	kostikbel@gmail.com using -f
Date: Mon, 23 Apr 2012 15:07:20 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Fengwei yin <yfw.bsd@gmail.com>
Message-ID: <20120423120720.GS2358@deviant.kiev.zoral.com.ua>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
	<20120423113838.GT32749@zxy.spb.ru>
	<CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="o4p1YyT9pdUjNGfy"
Content-Disposition: inline
In-Reply-To: <CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
User-Agent: Mutt/1.4.2.3i
X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua
X-Virus-Status: Clean
X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	skuns.kiev.zoral.com.ua
Cc: freebsd-threads@freebsd.org, jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 12:07:33 -0000


--o4p1YyT9pdUjNGfy
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote:
> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrot=
e:
> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
> >
> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> w=
rote:
> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
> >> >
> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.ru=
> wrote:
> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
> >> >> >
> >> >> >> Hi list,
> >> >> >> If this is not correct question on the list, please let me know =
and
> >> >> >> sorry for noise.
> >> >> >>
> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't =
see
> >> >> >> memory barrier used in libc.
> >> >> >> How can we make sure it's safe on SMP arch?
> >> >> >
> >> >> > /usr/include/machine/atomic.h:
> >> >> >
> >> >> > #define mb() =9A =9A__asm __volatile("lock; addl $0,(%%esp)" : : =
: "memory")
> >> >> > #define wmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : : : =
"memory")
> >> >> > #define rmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : : : =
"memory")
> >> >> >
> >> >>
> >> >> Thanks for the information. But it looks no body use it in libc.
> >> >
> >> > I think no body in libc need memory barrier: libc don't work with
> >> > peripheral, for atomic opertions used different macros.
> >>
> >> If we check the usage of __sinit(), it is a typical singleton pattern =
which
> >> needs memory barrier to make sure no potential SMP issue.
> >>
> >> Or did I miss something here?
> >
> > What architecture with cache incoherency and FreeBSD support?
>=20
> I suppose it's not related with cache inchoherency (I could be wrong).
> It's related
> with reorder of instruction by CPU.
>=20
> Here is the link talking about why need memory barrier for singleton:
> http://www.oaklib.org/docs/oak/singleton.html
>=20
> x86 has strict memory model and may not suffer this kind of issue. But
> ARM need to
> take care of it IMHO.

Please note that __sinit is idempotent, so double-initialization is not
an issue there. The only possible problematic case would be other thread
executing exit and not noticing non-NULL value for __cleanup while current
thread just set it.

I am not sure how much real this race is. Each call to _sinit() is immediat=
ely
followed by a lock acquire, typically FLOCKFILE(), which enforces full barr=
ier
semantic due to pthread_mutex_lock call. The exit() performs __cxa_finalize=
()
call before checking __cleanup value, and __cxa_finalize() itself locks
atexit_mutex. So the race is tiny and probably possible only for somewhat
buggy applications which perform exit() while there are stdio operations
in progress.

Also note that some functions assign to __cleanup unconditionally.

Do you see any real issue due to non-synchronized access to __cleanup ?

--o4p1YyT9pdUjNGfy
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAk+VRfgACgkQC3+MBN1Mb4gD6QCbBIrbxxcIBGlcoq5YjXrK7Qtk
QYUAoNz43XBHSKfIFabK1+WaStPB5ddV
=wpqq
-----END PGP SIGNATURE-----

--o4p1YyT9pdUjNGfy--

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 12:33:06 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 06A5D106566C
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 12:33:06 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id B88148FC0A
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 12:33:05 +0000 (UTC)
Received: by obqv19 with SMTP id v19so17949998obq.13
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 05:33:05 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=WyIUeO1Iuod7ydcEW+/SwN772QmpofuuK9vC1/ydkvI=;
	b=VkGBsxCUsZC5x49QNzsvfFho95RbhQZueZ6DMWqlMkJwZidbxC7d+Ygs5U9lJdT9lT
	0+qVWnmc+NCLF15QvznZOkagnlWsXYZKhCNOj1Gi+yBgSFDUgImFbQhFL1CW29nWslSH
	timlayWnzOiE9keJoPL+i5Va1WHFMIJ5AclFMOp7vGC3+f7aCCuP+YrvfYcz2u4QP+bP
	zX2BcIO6+y6cK6JBHYK6QYaC/s8KIRT0z5TVD4t78+Yx68/rQ+AqJrlanqnrDSU7HehE
	4/26a5kgj9T9aWoVl3+I89prvptEUt1HDCNuMYSkYfzU4LeYKLwnjvn9KVHp6yuC2od/
	ocoQ==
MIME-Version: 1.0
Received: by 10.182.232.38 with SMTP id tl6mr7322531obc.16.1335184385378; Mon,
	23 Apr 2012 05:33:05 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 05:33:05 -0700 (PDT)
In-Reply-To: <20120423120720.GS2358@deviant.kiev.zoral.com.ua>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
	<20120423113838.GT32749@zxy.spb.ru>
	<CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
	<20120423120720.GS2358@deviant.kiev.zoral.com.ua>
Date: Mon, 23 Apr 2012 20:33:05 +0800
Message-ID: <CAPHpMumh3YpB3RDD-7g5tU6thiuNA6HTuVxmt-9_OzUiEdEXzA@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Konstantin Belousov <kostikbel@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: freebsd-threads@freebsd.org, jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 12:33:06 -0000

On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov
<kostikbel@gmail.com> wrote:
> On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote:
>> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wro=
te:
>> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
>> >
>> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> =
wrote:
>> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
>> >> >
>> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.r=
u> wrote:
>> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
>> >> >> >
>> >> >> >> Hi list,
>> >> >> >> If this is not correct question on the list, please let me know=
 and
>> >> >> >> sorry for noise.
>> >> >> >>
>> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't=
 see
>> >> >> >> memory barrier used in libc.
>> >> >> >> How can we make sure it's safe on SMP arch?
>> >> >> >
>> >> >> > /usr/include/machine/atomic.h:
>> >> >> >
>> >> >> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : :=
 : "memory")
>> >> >> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : :=
 "memory")
>> >> >> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : :=
 "memory")
>> >> >> >
>> >> >>
>> >> >> Thanks for the information. But it looks no body use it in libc.
>> >> >
>> >> > I think no body in libc need memory barrier: libc don't work with
>> >> > peripheral, for atomic opertions used different macros.
>> >>
>> >> If we check the usage of __sinit(), it is a typical singleton pattern=
 which
>> >> needs memory barrier to make sure no potential SMP issue.
>> >>
>> >> Or did I miss something here?
>> >
>> > What architecture with cache incoherency and FreeBSD support?
>>
>> I suppose it's not related with cache inchoherency (I could be wrong).
>> It's related
>> with reorder of instruction by CPU.
>>
>> Here is the link talking about why need memory barrier for singleton:
>> http://www.oaklib.org/docs/oak/singleton.html
>>
>> x86 has strict memory model and may not suffer this kind of issue. But
>> ARM need to
>> take care of it IMHO.
>
> Please note that __sinit is idempotent, so double-initialization is not
> an issue there. The only possible problematic case would be other thread
> executing exit and not noticing non-NULL value for __cleanup while curren=
t
> thread just set it.
>
> I am not sure how much real this race is. Each call to _sinit() is immedi=
ately
> followed by a lock acquire, typically FLOCKFILE(), which enforces full ba=
rrier
> semantic due to pthread_mutex_lock call. The exit() performs __cxa_finali=
ze()
> call before checking __cleanup value, and __cxa_finalize() itself locks
> atexit_mutex. So the race is tiny and probably possible only for somewhat
> buggy applications which perform exit() while there are stdio operations
> in progress.
>
> Also note that some functions assign to __cleanup unconditionally.
>
> Do you see any real issue due to non-synchronized access to __cleanup ?

No. I didn't see real issue. I am just reviewing the code.

If you don't think __sinit has issue, let's check another code:
     line 68 in libc/stdio/fclose.c
     line 133 in libc/stdio/findfp.c (function __sfp())

Which is trying to free a fp slot by assign 0 to fp->_flags. But if
the instrucation
could be re-ordered, another CPU could see fp->_flags is assigned to 0
before the
cleanup from line 57 to 67.

Let's say, if another CPU is in line 133 of __sfp(), it could see
fp->_flags become
0 before it's aware of the cleanup (Line 57 to line 67 in
libc/stdio/fclose.c) happen.

Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c
just could make sure
line 70 happen after line 68. It can't impact the re-order of line 57
~ line 68 by CPU.

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 13:03:58 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 217C6106566B
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 13:03:58 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200])
	by mx1.freebsd.org (Postfix) with ESMTP id 7CCDE8FC08
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 13:03:57 +0000 (UTC)
Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1])
	by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3ND3iEV060688;
	Mon, 23 Apr 2012 16:03:44 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1])
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id
	q3ND3hRj006679; Mon, 23 Apr 2012 16:03:43 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: (from kostik@localhost)
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3ND3h1T006678; 
	Mon, 23 Apr 2012 16:03:43 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to
	kostikbel@gmail.com using -f
Date: Mon, 23 Apr 2012 16:03:43 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Fengwei yin <yfw.bsd@gmail.com>
Message-ID: <20120423130343.GT2358@deviant.kiev.zoral.com.ua>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
	<20120423113838.GT32749@zxy.spb.ru>
	<CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
	<20120423120720.GS2358@deviant.kiev.zoral.com.ua>
	<CAPHpMumh3YpB3RDD-7g5tU6thiuNA6HTuVxmt-9_OzUiEdEXzA@mail.gmail.com>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="vtxr9MvRW4bhShyg"
Content-Disposition: inline
In-Reply-To: <CAPHpMumh3YpB3RDD-7g5tU6thiuNA6HTuVxmt-9_OzUiEdEXzA@mail.gmail.com>
User-Agent: Mutt/1.4.2.3i
X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua
X-Virus-Status: Clean
X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	skuns.kiev.zoral.com.ua
Cc: freebsd-threads@freebsd.org, jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 13:03:58 -0000


--vtxr9MvRW4bhShyg
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote:
> On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov
> <kostikbel@gmail.com> wrote:
> > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote:
> >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> w=
rote:
> >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
> >> >
> >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.ru=
> wrote:
> >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
> >> >> >
> >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb=
.ru> wrote:
> >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
> >> >> >> >
> >> >> >> >> Hi list,
> >> >> >> >> If this is not correct question on the list, please let me kn=
ow and
> >> >> >> >> sorry for noise.
> >> >> >> >>
> >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn=
't see
> >> >> >> >> memory barrier used in libc.
> >> >> >> >> How can we make sure it's safe on SMP arch?
> >> >> >> >
> >> >> >> > /usr/include/machine/atomic.h:
> >> >> >> >
> >> >> >> > #define mb() =9A =9A__asm __volatile("lock; addl $0,(%%esp)" :=
 : : "memory")
> >> >> >> > #define wmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : :=
 : "memory")
> >> >> >> > #define rmb() =9A __asm __volatile("lock; addl $0,(%%esp)" : :=
 : "memory")
> >> >> >> >
> >> >> >>
> >> >> >> Thanks for the information. But it looks no body use it in libc.
> >> >> >
> >> >> > I think no body in libc need memory barrier: libc don't work with
> >> >> > peripheral, for atomic opertions used different macros.
> >> >>
> >> >> If we check the usage of __sinit(), it is a typical singleton patte=
rn which
> >> >> needs memory barrier to make sure no potential SMP issue.
> >> >>
> >> >> Or did I miss something here?
> >> >
> >> > What architecture with cache incoherency and FreeBSD support?
> >>
> >> I suppose it's not related with cache inchoherency (I could be wrong).
> >> It's related
> >> with reorder of instruction by CPU.
> >>
> >> Here is the link talking about why need memory barrier for singleton:
> >> http://www.oaklib.org/docs/oak/singleton.html
> >>
> >> x86 has strict memory model and may not suffer this kind of issue. But
> >> ARM need to
> >> take care of it IMHO.
> >
> > Please note that __sinit is idempotent, so double-initialization is not
> > an issue there. The only possible problematic case would be other thread
> > executing exit and not noticing non-NULL value for __cleanup while curr=
ent
> > thread just set it.
> >
> > I am not sure how much real this race is. Each call to _sinit() is imme=
diately
> > followed by a lock acquire, typically FLOCKFILE(), which enforces full =
barrier
> > semantic due to pthread_mutex_lock call. The exit() performs __cxa_fina=
lize()
> > call before checking __cleanup value, and __cxa_finalize() itself locks
> > atexit_mutex. So the race is tiny and probably possible only for somewh=
at
> > buggy applications which perform exit() while there are stdio operations
> > in progress.
> >
> > Also note that some functions assign to __cleanup unconditionally.
> >
> > Do you see any real issue due to non-synchronized access to __cleanup ?
>=20
> No. I didn't see real issue. I am just reviewing the code.
>=20
> If you don't think __sinit has issue, let's check another code:
>      line 68 in libc/stdio/fclose.c
>      line 133 in libc/stdio/findfp.c (function __sfp())
>=20
> Which is trying to free a fp slot by assign 0 to fp->_flags. But if
> the instrucation
> could be re-ordered, another CPU could see fp->_flags is assigned to 0
> before the
> cleanup from line 57 to 67.
>=20
> Let's say, if another CPU is in line 133 of __sfp(), it could see
> fp->_flags become
> 0 before it's aware of the cleanup (Line 57 to line 67 in
> libc/stdio/fclose.c) happen.
>=20
> Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c
> just could make sure
> line 70 happen after line 68. It can't impact the re-order of line 57
> ~ line 68 by CPU.

Yes, FUNLOCKFILE() there would have no effect on the potential CPU reorderi=
ng
of the writes.  But does the order of these writes matter at all ?

Please note that __sfp() reinitializes all fields written by fclose().
Only if CPU executing fclose() is allowed to reorder operations so that
the external effect of _flags =3D 0 assignment can be observed before that
CPU executes other operations from fclose(), there could be a problem.

This is definitely impossible on Intel, and I indeed do not know about
other architectures enough to reject such possibility. The _flags member
is short, so atomics cannot be used there. The easier solution, if this
is indeed an issue, is to lock thread_lock around _flags =3D 0 assignment
in fclose().

--vtxr9MvRW4bhShyg
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAk+VUy8ACgkQC3+MBN1Mb4g7pwCgnxQskpojwuEEp4/fxAeP287y
6CEAniYTpQJ3iblCQ8JnmF5i0HJOx7te
=ijpf
-----END PGP SIGNATURE-----

--vtxr9MvRW4bhShyg--

From owner-freebsd-threads@FreeBSD.ORG  Mon Apr 23 13:41:47 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id A453C106566B
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 13:41:47 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com
	[209.85.160.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 558178FC1F
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 13:41:47 +0000 (UTC)
Received: by ghrr20 with SMTP id r20so7289280ghr.13
	for <freebsd-threads@freebsd.org>; Mon, 23 Apr 2012 06:41:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=1xxJUgwwNzSkFN43sdEZvfXGgv9aDVx6CKg8S1A0YYs=;
	b=K97fVtZd2fAebutljLPNtQLqnmVq1ZLJfcHpSlRQODqD/LVu3vzM5ZVQWzeByYGyOd
	IW3A3+TraqulWpHSiH2RY6dk6vwMcffFIjkZoaAh3T5Rl/jzUm9+Xi5TDzZSqB8oiH23
	fqjwCAHYx7MZodcWsWRVmrj6iyhGsWIQLVyZ+aqXb9l1kQY3VfvDOW/asrWzBO/9Wn8d
	Y6ovV25VCNOdk6Nqk9YZpHmtGvtPtiiJ1KMR8G2KhBapV6KyjNP+Vzz05I6sRTjdfUTT
	Ino34AOKYsqcSKXdrNFTw8MfnvBDmabbapLhnH2li+dXU/nJlb3SS7njdAblnBS7CRjA
	hgKQ==
MIME-Version: 1.0
Received: by 10.60.169.146 with SMTP id ae18mr18544067oec.36.1335188501619;
	Mon, 23 Apr 2012 06:41:41 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Mon, 23 Apr 2012 06:41:41 -0700 (PDT)
In-Reply-To: <20120423130343.GT2358@deviant.kiev.zoral.com.ua>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
	<20120423113838.GT32749@zxy.spb.ru>
	<CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
	<20120423120720.GS2358@deviant.kiev.zoral.com.ua>
	<CAPHpMumh3YpB3RDD-7g5tU6thiuNA6HTuVxmt-9_OzUiEdEXzA@mail.gmail.com>
	<20120423130343.GT2358@deviant.kiev.zoral.com.ua>
Date: Mon, 23 Apr 2012 21:41:41 +0800
Message-ID: <CAPHpMu=LAqmZJwKO65N0VOTVsoFpJYAvYmxndFNt4S_y5P=NQg@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Konstantin Belousov <kostikbel@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: freebsd-threads@freebsd.org, jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 23 Apr 2012 13:41:47 -0000

On Mon, Apr 23, 2012 at 9:03 PM, Konstantin Belousov
<kostikbel@gmail.com> wrote:
> On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote:
>> On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov
>> <kostikbel@gmail.com> wrote:
>> > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote:
>> >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> =
wrote:
>> >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
>> >> >
>> >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.r=
u> wrote:
>> >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
>> >> >> >
>> >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.sp=
b.ru> wrote:
>> >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
>> >> >> >> >
>> >> >> >> >> Hi list,
>> >> >> >> >> If this is not correct question on the list, please let me k=
now and
>> >> >> >> >> sorry for noise.
>> >> >> >> >>
>> >> >> >> >> I have a question regarding the BSD libc for SMP arch. I did=
n't see
>> >> >> >> >> memory barrier used in libc.
>> >> >> >> >> How can we make sure it's safe on SMP arch?
>> >> >> >> >
>> >> >> >> > /usr/include/machine/atomic.h:
>> >> >> >> >
>> >> >> >> > #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" =
: : : "memory")
>> >> >> >> > #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : =
: : "memory")
>> >> >> >> > #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : =
: : "memory")
>> >> >> >> >
>> >> >> >>
>> >> >> >> Thanks for the information. But it looks no body use it in libc=
.
>> >> >> >
>> >> >> > I think no body in libc need memory barrier: libc don't work wit=
h
>> >> >> > peripheral, for atomic opertions used different macros.
>> >> >>
>> >> >> If we check the usage of __sinit(), it is a typical singleton patt=
ern which
>> >> >> needs memory barrier to make sure no potential SMP issue.
>> >> >>
>> >> >> Or did I miss something here?
>> >> >
>> >> > What architecture with cache incoherency and FreeBSD support?
>> >>
>> >> I suppose it's not related with cache inchoherency (I could be wrong)=
.
>> >> It's related
>> >> with reorder of instruction by CPU.
>> >>
>> >> Here is the link talking about why need memory barrier for singleton:
>> >> http://www.oaklib.org/docs/oak/singleton.html
>> >>
>> >> x86 has strict memory model and may not suffer this kind of issue. Bu=
t
>> >> ARM need to
>> >> take care of it IMHO.
>> >
>> > Please note that __sinit is idempotent, so double-initialization is no=
t
>> > an issue there. The only possible problematic case would be other thre=
ad
>> > executing exit and not noticing non-NULL value for __cleanup while cur=
rent
>> > thread just set it.
>> >
>> > I am not sure how much real this race is. Each call to _sinit() is imm=
ediately
>> > followed by a lock acquire, typically FLOCKFILE(), which enforces full=
 barrier
>> > semantic due to pthread_mutex_lock call. The exit() performs __cxa_fin=
alize()
>> > call before checking __cleanup value, and __cxa_finalize() itself lock=
s
>> > atexit_mutex. So the race is tiny and probably possible only for somew=
hat
>> > buggy applications which perform exit() while there are stdio operatio=
ns
>> > in progress.
>> >
>> > Also note that some functions assign to __cleanup unconditionally.
>> >
>> > Do you see any real issue due to non-synchronized access to __cleanup =
?
>>
>> No. I didn't see real issue. I am just reviewing the code.
>>
>> If you don't think __sinit has issue, let's check another code:
>> =A0 =A0 =A0line 68 in libc/stdio/fclose.c
>> =A0 =A0 =A0line 133 in libc/stdio/findfp.c (function __sfp())
>>
>> Which is trying to free a fp slot by assign 0 to fp->_flags. But if
>> the instrucation
>> could be re-ordered, another CPU could see fp->_flags is assigned to 0
>> before the
>> cleanup from line 57 to 67.
>>
>> Let's say, if another CPU is in line 133 of __sfp(), it could see
>> fp->_flags become
>> 0 before it's aware of the cleanup (Line 57 to line 67 in
>> libc/stdio/fclose.c) happen.
>>
>> Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c
>> just could make sure
>> line 70 happen after line 68. It can't impact the re-order of line 57
>> ~ line 68 by CPU.
>
> Yes, FUNLOCKFILE() there would have no effect on the potential CPU reorde=
ring
> of the writes. =A0But does the order of these writes matter at all ?
>
> Please note that __sfp() reinitializes all fields written by fclose().
> Only if CPU executing fclose() is allowed to reorder operations so that
> the external effect of _flags =3D 0 assignment can be observed before tha=
t
> CPU executes other operations from fclose(), there could be a problem.
>
> This is definitely impossible on Intel, and I indeed do not know about
> other architectures enough to reject such possibility. The _flags member
> is short, so atomics cannot be used there. The easier solution, if this
> is indeed an issue, is to lock thread_lock around _flags =3D 0 assignment
> in fclose().

For Intel x86, I suppose it's not a problem because of its memory model.
For IA64, ARM, it could be an issue in theory.

For the solution: I suppose memory barrier is enough for it.

From owner-freebsd-threads@FreeBSD.ORG  Tue Apr 24 13:54:51 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id A7E89106566C
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 13:54:51 +0000 (UTC)
	(envelope-from martin@lispworks.com)
Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com
	[193.34.186.230])
	by mx1.freebsd.org (Postfix) with ESMTP id B1F8B8FC08
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 13:54:50 +0000 (UTC)
Received: from higson.cam.lispworks.com (higson [192.168.1.7])
	by lwfs1-cam.cam.lispworks.com (8.14.3/8.14.3) with ESMTP id
	q3ODhfUp016214; Tue, 24 Apr 2012 14:43:41 +0100 (BST)
	(envelope-from martin@lispworks.com)
Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by
	higson.cam.lispworks.com (8.14.4) id q3ODhf9U032687;
	Tue, 24 Apr 2012 14:43:41 +0100
Received: (from martin@localhost)
	by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id q3ODhe2C032683;
	Tue, 24 Apr 2012 14:43:40 +0100
Date: Tue, 24 Apr 2012 14:43:40 +0100
Message-Id: <201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
From: Martin Simmons <martin@lispworks.com>
To: Konstantin Belousov <kostikbel@gmail.com>
In-reply-to: <20120423130343.GT2358@deviant.kiev.zoral.com.ua> (message from
	Konstantin Belousov on Mon, 23 Apr 2012 16:03:43 +0300)
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
	<20120423113838.GT32749@zxy.spb.ru>
	<CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
	<20120423120720.GS2358@deviant.kiev.zoral.com.ua>
	<CAPHpMumh3YpB3RDD-7g5tU6thiuNA6HTuVxmt-9_OzUiEdEXzA@mail.gmail.com>
	<20120423130343.GT2358@deviant.kiev.zoral.com.ua>
MIME-version: 1.0
Content-type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Cc: freebsd-threads@freebsd.org, jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 13:54:51 -0000

>>>>> On Mon, 23 Apr 2012 16:03:43 +0300, Konstantin Belousov said:
> 
> On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote:
> > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov
> > <kostikbel@gmail.com> wrote:
> > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote:
> > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
> > >> >
> > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
> > >> >> >
> > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy.spb.ru> wrote:
> > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrote:
> > >> >> >> >
> > >> >> >> >> Hi list,
> > >> >> >> >> If this is not correct question on the list, please let me know and
> > >> >> >> >> sorry for noise.
> > >> >> >> >>
> > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I didn't see
> > >> >> >> >> memory barrier used in libc.
> > >> >> >> >> How can we make sure it's safe on SMP arch?
> > >> >> >> >
> > >> >> >> > /usr/include/machine/atomic.h:
> > >> >> >> >
> > >> >> >> > #define mb() š š__asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> > >> >> >> > #define wmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> > >> >> >> > #define rmb() š __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> > >> >> >> >
> > >> >> >>
> > >> >> >> Thanks for the information. But it looks no body use it in libc.
> > >> >> >
> > >> >> > I think no body in libc need memory barrier: libc don't work with
> > >> >> > peripheral, for atomic opertions used different macros.
> > >> >>
> > >> >> If we check the usage of __sinit(), it is a typical singleton pattern which
> > >> >> needs memory barrier to make sure no potential SMP issue.
> > >> >>
> > >> >> Or did I miss something here?
> > >> >
> > >> > What architecture with cache incoherency and FreeBSD support?
> > >>
> > >> I suppose it's not related with cache inchoherency (I could be wrong).
> > >> It's related
> > >> with reorder of instruction by CPU.
> > >>
> > >> Here is the link talking about why need memory barrier for singleton:
> > >> http://www.oaklib.org/docs/oak/singleton.html
> > >>
> > >> x86 has strict memory model and may not suffer this kind of issue. But
> > >> ARM need to
> > >> take care of it IMHO.
> > >
> > > Please note that __sinit is idempotent, so double-initialization is not
> > > an issue there. The only possible problematic case would be other thread
> > > executing exit and not noticing non-NULL value for __cleanup while current
> > > thread just set it.
> > >
> > > I am not sure how much real this race is. Each call to _sinit() is immediately
> > > followed by a lock acquire, typically FLOCKFILE(), which enforces full barrier
> > > semantic due to pthread_mutex_lock call. The exit() performs __cxa_finalize()
> > > call before checking __cleanup value, and __cxa_finalize() itself locks
> > > atexit_mutex. So the race is tiny and probably possible only for somewhat
> > > buggy applications which perform exit() while there are stdio operations
> > > in progress.
> > >
> > > Also note that some functions assign to __cleanup unconditionally.
> > >
> > > Do you see any real issue due to non-synchronized access to __cleanup ?
> > 
> > No. I didn't see real issue. I am just reviewing the code.
> > 
> > If you don't think __sinit has issue, let's check another code:
> >      line 68 in libc/stdio/fclose.c
> >      line 133 in libc/stdio/findfp.c (function __sfp())
> > 
> > Which is trying to free a fp slot by assign 0 to fp->_flags. But if
> > the instrucation
> > could be re-ordered, another CPU could see fp->_flags is assigned to 0
> > before the
> > cleanup from line 57 to 67.
> > 
> > Let's say, if another CPU is in line 133 of __sfp(), it could see
> > fp->_flags become
> > 0 before it's aware of the cleanup (Line 57 to line 67 in
> > libc/stdio/fclose.c) happen.
> > 
> > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c
> > just could make sure
> > line 70 happen after line 68. It can't impact the re-order of line 57
> > ~ line 68 by CPU.
> 
> Yes, FUNLOCKFILE() there would have no effect on the potential CPU reordering
> of the writes.  But does the order of these writes matter at all ?
> 
> Please note that __sfp() reinitializes all fields written by fclose().
> Only if CPU executing fclose() is allowed to reorder operations so that
> the external effect of _flags = 0 assignment can be observed before that
> CPU executes other operations from fclose(), there could be a problem.
> 
> This is definitely impossible on Intel, and I indeed do not know about
> other architectures enough to reject such possibility. The _flags member
> is short, so atomics cannot be used there. The easier solution, if this
> is indeed an issue, is to lock thread_lock around _flags = 0 assignment
> in fclose().

This can be a problem, even on Intel, because the compiler can reorder the
stores.  E.g. if I compile the following with gcc -O4 on amd64:

struct foo { int x, y; };

int foo(struct foo *p)
{
  int x = bar();
  p->y = baz();
  p->x = x;
}

then I get the following assembly language, which sets p->x before p->y:

	movq	%rdi, %rbx
	call	bar
	movl	%eax, %ebp
	xorl	%eax, %eax
	call	baz
	movl	%ebp, (%rbx)
	movl	%eax, 4(%rbx)

__Martin

From owner-freebsd-threads@FreeBSD.ORG  Tue Apr 24 14:03:58 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 74E0D1065670
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 14:03:58 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200])
	by mx1.freebsd.org (Postfix) with ESMTP id D4C458FC1A
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 14:03:57 +0000 (UTC)
Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1])
	by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3OE3n3Y043624;
	Tue, 24 Apr 2012 17:03:49 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1])
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id
	q3OE3mot037971; Tue, 24 Apr 2012 17:03:48 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: (from kostik@localhost)
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3OE3msw037970; 
	Tue, 24 Apr 2012 17:03:48 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to
	kostikbel@gmail.com using -f
Date: Tue, 24 Apr 2012 17:03:48 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Martin Simmons <martin@lispworks.com>
Message-ID: <20120424140348.GY2358@deviant.kiev.zoral.com.ua>
References: <20120423084120.GD76983@zxy.spb.ru>
	<CAPHpMu=kCwhf1RV_sYBDWDPL8368YTMLXge4L_g_F4AkTX1H5g@mail.gmail.com>
	<20120423094043.GS32749@zxy.spb.ru>
	<CAPHpMukLUeetSKpH2oiKJQ3ML_PFHEi6a0hK3_Ery=LX1YEd3g@mail.gmail.com>
	<20120423113838.GT32749@zxy.spb.ru>
	<CAPHpMumWu_aaZ4Sj5Athro6441Y+3_phbD2jxkKE-CdBf-Fd8g@mail.gmail.com>
	<20120423120720.GS2358@deviant.kiev.zoral.com.ua>
	<CAPHpMumh3YpB3RDD-7g5tU6thiuNA6HTuVxmt-9_OzUiEdEXzA@mail.gmail.com>
	<20120423130343.GT2358@deviant.kiev.zoral.com.ua>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="yDxN68y6wlbaMG9t"
Content-Disposition: inline
In-Reply-To: <201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
User-Agent: Mutt/1.4.2.3i
X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua
X-Virus-Status: Clean
X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	skuns.kiev.zoral.com.ua
Cc: freebsd-threads@freebsd.org, jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 14:03:58 -0000


--yDxN68y6wlbaMG9t
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 24, 2012 at 02:43:40PM +0100, Martin Simmons wrote:
> >>>>> On Mon, 23 Apr 2012 16:03:43 +0300, Konstantin Belousov said:
> >=20
> > On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote:
> > > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov
> > > <kostikbel@gmail.com> wrote:
> > > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote:
> > > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov <slw@zxy.spb.r=
u> wrote:
> > > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
> > > >> >
> > > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov <slw@zxy.sp=
b.ru> wrote:
> > > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
> > > >> >> >
> > > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov <slw@zxy=
.spb.ru> wrote:
> > > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin wrot=
e:
> > > >> >> >> >
> > > >> >> >> >> Hi list,
> > > >> >> >> >> If this is not correct question on the list, please let m=
e know and
> > > >> >> >> >> sorry for noise.
> > > >> >> >> >>
> > > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I =
didn't see
> > > >> >> >> >> memory barrier used in libc.
> > > >> >> >> >> How can we make sure it's safe on SMP arch?
> > > >> >> >> >
> > > >> >> >> > /usr/include/machine/atomic.h:
> > > >> >> >> >
> > > >> >> >> > #define mb() ?? ??__asm __volatile("lock; addl $0,(%%esp)"=
 : : : "memory")
> > > >> >> >> > #define wmb() ?? __asm __volatile("lock; addl $0,(%%esp)" =
: : : "memory")
> > > >> >> >> > #define rmb() ?? __asm __volatile("lock; addl $0,(%%esp)" =
: : : "memory")
> > > >> >> >> >
> > > >> >> >>
> > > >> >> >> Thanks for the information. But it looks no body use it in l=
ibc.
> > > >> >> >
> > > >> >> > I think no body in libc need memory barrier: libc don't work =
with
> > > >> >> > peripheral, for atomic opertions used different macros.
> > > >> >>
> > > >> >> If we check the usage of __sinit(), it is a typical singleton p=
attern which
> > > >> >> needs memory barrier to make sure no potential SMP issue.
> > > >> >>
> > > >> >> Or did I miss something here?
> > > >> >
> > > >> > What architecture with cache incoherency and FreeBSD support?
> > > >>
> > > >> I suppose it's not related with cache inchoherency (I could be wro=
ng).
> > > >> It's related
> > > >> with reorder of instruction by CPU.
> > > >>
> > > >> Here is the link talking about why need memory barrier for singlet=
on:
> > > >> http://www.oaklib.org/docs/oak/singleton.html
> > > >>
> > > >> x86 has strict memory model and may not suffer this kind of issue.=
 But
> > > >> ARM need to
> > > >> take care of it IMHO.
> > > >
> > > > Please note that __sinit is idempotent, so double-initialization is=
 not
> > > > an issue there. The only possible problematic case would be other t=
hread
> > > > executing exit and not noticing non-NULL value for __cleanup while =
current
> > > > thread just set it.
> > > >
> > > > I am not sure how much real this race is. Each call to _sinit() is =
immediately
> > > > followed by a lock acquire, typically FLOCKFILE(), which enforces f=
ull barrier
> > > > semantic due to pthread_mutex_lock call. The exit() performs __cxa_=
finalize()
> > > > call before checking __cleanup value, and __cxa_finalize() itself l=
ocks
> > > > atexit_mutex. So the race is tiny and probably possible only for so=
mewhat
> > > > buggy applications which perform exit() while there are stdio opera=
tions
> > > > in progress.
> > > >
> > > > Also note that some functions assign to __cleanup unconditionally.
> > > >
> > > > Do you see any real issue due to non-synchronized access to __clean=
up ?
> > >=20
> > > No. I didn't see real issue. I am just reviewing the code.
> > >=20
> > > If you don't think __sinit has issue, let's check another code:
> > >      line 68 in libc/stdio/fclose.c
> > >      line 133 in libc/stdio/findfp.c (function __sfp())
> > >=20
> > > Which is trying to free a fp slot by assign 0 to fp->_flags. But if
> > > the instrucation
> > > could be re-ordered, another CPU could see fp->_flags is assigned to 0
> > > before the
> > > cleanup from line 57 to 67.
> > >=20
> > > Let's say, if another CPU is in line 133 of __sfp(), it could see
> > > fp->_flags become
> > > 0 before it's aware of the cleanup (Line 57 to line 67 in
> > > libc/stdio/fclose.c) happen.
> > >=20
> > > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c
> > > just could make sure
> > > line 70 happen after line 68. It can't impact the re-order of line 57
> > > ~ line 68 by CPU.
> >=20
> > Yes, FUNLOCKFILE() there would have no effect on the potential CPU reor=
dering
> > of the writes.  But does the order of these writes matter at all ?
> >=20
> > Please note that __sfp() reinitializes all fields written by fclose().
> > Only if CPU executing fclose() is allowed to reorder operations so that
> > the external effect of _flags =3D 0 assignment can be observed before t=
hat
> > CPU executes other operations from fclose(), there could be a problem.
> >=20
> > This is definitely impossible on Intel, and I indeed do not know about
> > other architectures enough to reject such possibility. The _flags member
> > is short, so atomics cannot be used there. The easier solution, if this
> > is indeed an issue, is to lock thread_lock around _flags =3D 0 assignme=
nt
> > in fclose().
>=20
> This can be a problem, even on Intel, because the compiler can reorder the
> stores.  E.g. if I compile the following with gcc -O4 on amd64:
>=20
> struct foo { int x, y; };
>=20
> int foo(struct foo *p)
> {
>   int x =3D bar();
>   p->y =3D baz();
>   p->x =3D x;
> }
>=20
> then I get the following assembly language, which sets p->x before p->y:
>=20
> 	movq	%rdi, %rbx
> 	call	bar
> 	movl	%eax, %ebp
> 	xorl	%eax, %eax
> 	call	baz
> 	movl	%ebp, (%rbx)
> 	movl	%eax, 4(%rbx)
>=20
> __Martin
Ok, as I already said, I think that the reordering is safe there.

Anyway, the change below should remove all concerns.

diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c
index f0629e8..383040e 100644
--- a/lib/libc/stdio/fclose.c
+++ b/lib/libc/stdio/fclose.c
@@ -41,9 +41,12 @@ __FBSDID("$FreeBSD$");
 #include <stdio.h>
 #include <stdlib.h>
 #include "un-namespace.h"
+#include <spinlock.h>
 #include "libc_private.h"
 #include "local.h"
=20
+extern spinlock_t __stdio_thread_lock;
+
 int
 fclose(FILE *fp)
 {
@@ -65,7 +68,11 @@ fclose(FILE *fp)
 		FREELB(fp);
 	fp->_file =3D -1;
 	fp->_r =3D fp->_w =3D 0;	/* Mess up if reaccessed. */
+	if (__isthreaded)
+		_SPINLOCK(&__stdio_thread_lock);
 	fp->_flags =3D 0;		/* Release this FILE for reuse. */
+	if (__isthreaded)
+		_SPINUNLOCK(&__stdio_thread_lock);
 	FUNLOCKFILE(fp);
 	return (r);
 }
diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c
index 89c0536..bcd6f62 100644
--- a/lib/libc/stdio/findfp.c
+++ b/lib/libc/stdio/findfp.c
@@ -82,9 +82,9 @@ static struct glue *lastglue =3D &uglue;
=20
 static struct glue *	moreglue(int);
=20
-static spinlock_t thread_lock =3D _SPINLOCK_INITIALIZER;
-#define THREAD_LOCK()	if (__isthreaded) _SPINLOCK(&thread_lock)
-#define THREAD_UNLOCK()	if (__isthreaded) _SPINUNLOCK(&thread_lock)
+spinlock_t __stdio_thread_lock =3D _SPINLOCK_INITIALIZER;
+#define THREAD_LOCK()	if (__isthreaded) _SPINLOCK(&__stdio_thread_lock)
+#define THREAD_UNLOCK()	if (__isthreaded) _SPINUNLOCK(&__stdio_thread_lock)
=20
 #if NOT_YET
 #define	SET_GLUE_PTR(ptr, val)	atomic_set_rel_ptr(&(ptr), (uintptr_t)(val))

--yDxN68y6wlbaMG9t
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAk+WssQACgkQC3+MBN1Mb4g+hACguHQ9O3LLzcvc8DuzymjOmaeg
JFAAoLF1xp2cXY6dvSf7dLsk0X1X9VeY
=XzVd
-----END PGP SIGNATURE-----

--yDxN68y6wlbaMG9t--

From owner-freebsd-threads@FreeBSD.ORG  Tue Apr 24 15:10:43 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 69A70106566B
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 15:10:43 +0000 (UTC)
	(envelope-from jhb@freebsd.org)
Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net
	[IPv6:2001:470:1f10:75::2])
	by mx1.freebsd.org (Postfix) with ESMTP id 1219C8FC12
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 15:10:43 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
	by bigwig.baldwin.cx (Postfix) with ESMTPSA id 7FEB3B943;
	Tue, 24 Apr 2012 11:10:42 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: freebsd-threads@freebsd.org
Date: Tue, 24 Apr 2012 11:10:13 -0400
User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; )
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
	<20120424140348.GY2358@deviant.kiev.zoral.com.ua>
In-Reply-To: <20120424140348.GY2358@deviant.kiev.zoral.com.ua>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-15"
Content-Transfer-Encoding: 7bit
Message-Id: <201204241110.14017.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
	(bigwig.baldwin.cx); Tue, 24 Apr 2012 11:10:42 -0400 (EDT)
Cc: jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 15:10:43 -0000

On Tuesday, April 24, 2012 10:03:48 am Konstantin Belousov wrote:
> On Tue, Apr 24, 2012 at 02:43:40PM +0100, Martin Simmons wrote:
> > >>>>> On Mon, 23 Apr 2012 16:03:43 +0300, Konstantin Belousov said:
> > > 
> > > On Mon, Apr 23, 2012 at 08:33:05PM +0800, Fengwei yin wrote:
> > > > On Mon, Apr 23, 2012 at 8:07 PM, Konstantin Belousov
> > > > <kostikbel@gmail.com> wrote:
> > > > > On Mon, Apr 23, 2012 at 07:44:34PM +0800, Fengwei yin wrote:
> > > > >> On Mon, Apr 23, 2012 at 7:38 PM, Slawa Olhovchenkov 
<slw@zxy.spb.ru> wrote:
> > > > >> > On Mon, Apr 23, 2012 at 07:26:54PM +0800, Fengwei yin wrote:
> > > > >> >
> > > > >> >> On Mon, Apr 23, 2012 at 5:40 PM, Slawa Olhovchenkov 
<slw@zxy.spb.ru> wrote:
> > > > >> >> > On Mon, Apr 23, 2012 at 05:32:24PM +0800, Fengwei yin wrote:
> > > > >> >> >
> > > > >> >> >> On Mon, Apr 23, 2012 at 4:41 PM, Slawa Olhovchenkov 
<slw@zxy.spb.ru> wrote:
> > > > >> >> >> > On Mon, Apr 23, 2012 at 02:56:03PM +0800, Fengwei yin 
wrote:
> > > > >> >> >> >
> > > > >> >> >> >> Hi list,
> > > > >> >> >> >> If this is not correct question on the list, please let me 
know and
> > > > >> >> >> >> sorry for noise.
> > > > >> >> >> >>
> > > > >> >> >> >> I have a question regarding the BSD libc for SMP arch. I 
didn't see
> > > > >> >> >> >> memory barrier used in libc.
> > > > >> >> >> >> How can we make sure it's safe on SMP arch?
> > > > >> >> >> >
> > > > >> >> >> > /usr/include/machine/atomic.h:
> > > > >> >> >> >
> > > > >> >> >> > #define mb() ?? ??__asm __volatile("lock; addl $0,(%%esp)" 
: : : "memory")
> > > > >> >> >> > #define wmb() ?? __asm __volatile("lock; addl $0,(%%esp)" : 
: : "memory")
> > > > >> >> >> > #define rmb() ?? __asm __volatile("lock; addl $0,(%%esp)" : 
: : "memory")
> > > > >> >> >> >
> > > > >> >> >>
> > > > >> >> >> Thanks for the information. But it looks no body use it in 
libc.
> > > > >> >> >
> > > > >> >> > I think no body in libc need memory barrier: libc don't work 
with
> > > > >> >> > peripheral, for atomic opertions used different macros.
> > > > >> >>
> > > > >> >> If we check the usage of __sinit(), it is a typical singleton 
pattern which
> > > > >> >> needs memory barrier to make sure no potential SMP issue.
> > > > >> >>
> > > > >> >> Or did I miss something here?
> > > > >> >
> > > > >> > What architecture with cache incoherency and FreeBSD support?
> > > > >>
> > > > >> I suppose it's not related with cache inchoherency (I could be 
wrong).
> > > > >> It's related
> > > > >> with reorder of instruction by CPU.
> > > > >>
> > > > >> Here is the link talking about why need memory barrier for 
singleton:
> > > > >> http://www.oaklib.org/docs/oak/singleton.html
> > > > >>
> > > > >> x86 has strict memory model and may not suffer this kind of issue. 
But
> > > > >> ARM need to
> > > > >> take care of it IMHO.
> > > > >
> > > > > Please note that __sinit is idempotent, so double-initialization is 
not
> > > > > an issue there. The only possible problematic case would be other 
thread
> > > > > executing exit and not noticing non-NULL value for __cleanup while 
current
> > > > > thread just set it.
> > > > >
> > > > > I am not sure how much real this race is. Each call to _sinit() is 
immediately
> > > > > followed by a lock acquire, typically FLOCKFILE(), which enforces 
full barrier
> > > > > semantic due to pthread_mutex_lock call. The exit() performs 
__cxa_finalize()
> > > > > call before checking __cleanup value, and __cxa_finalize() itself 
locks
> > > > > atexit_mutex. So the race is tiny and probably possible only for 
somewhat
> > > > > buggy applications which perform exit() while there are stdio 
operations
> > > > > in progress.
> > > > >
> > > > > Also note that some functions assign to __cleanup unconditionally.
> > > > >
> > > > > Do you see any real issue due to non-synchronized access to 
__cleanup ?
> > > > 
> > > > No. I didn't see real issue. I am just reviewing the code.
> > > > 
> > > > If you don't think __sinit has issue, let's check another code:
> > > >      line 68 in libc/stdio/fclose.c
> > > >      line 133 in libc/stdio/findfp.c (function __sfp())
> > > > 
> > > > Which is trying to free a fp slot by assign 0 to fp->_flags. But if
> > > > the instrucation
> > > > could be re-ordered, another CPU could see fp->_flags is assigned to 0
> > > > before the
> > > > cleanup from line 57 to 67.
> > > > 
> > > > Let's say, if another CPU is in line 133 of __sfp(), it could see
> > > > fp->_flags become
> > > > 0 before it's aware of the cleanup (Line 57 to line 67 in
> > > > libc/stdio/fclose.c) happen.
> > > > 
> > > > Note: the mutex of FUNLOCKFILE(fp) in line 69 of libc/stdio/fclose.c
> > > > just could make sure
> > > > line 70 happen after line 68. It can't impact the re-order of line 57
> > > > ~ line 68 by CPU.
> > > 
> > > Yes, FUNLOCKFILE() there would have no effect on the potential CPU 
reordering
> > > of the writes.  But does the order of these writes matter at all ?
> > > 
> > > Please note that __sfp() reinitializes all fields written by fclose().
> > > Only if CPU executing fclose() is allowed to reorder operations so that
> > > the external effect of _flags = 0 assignment can be observed before that
> > > CPU executes other operations from fclose(), there could be a problem.
> > > 
> > > This is definitely impossible on Intel, and I indeed do not know about
> > > other architectures enough to reject such possibility. The _flags member
> > > is short, so atomics cannot be used there. The easier solution, if this
> > > is indeed an issue, is to lock thread_lock around _flags = 0 assignment
> > > in fclose().
> > 
> > This can be a problem, even on Intel, because the compiler can reorder the
> > stores.  E.g. if I compile the following with gcc -O4 on amd64:
> > 
> > struct foo { int x, y; };
> > 
> > int foo(struct foo *p)
> > {
> >   int x = bar();
> >   p->y = baz();
> >   p->x = x;
> > }
> > 
> > then I get the following assembly language, which sets p->x before p->y:
> > 
> > 	movq	%rdi, %rbx
> > 	call	bar
> > 	movl	%eax, %ebp
> > 	xorl	%eax, %eax
> > 	call	baz
> > 	movl	%ebp, (%rbx)
> > 	movl	%eax, 4(%rbx)
> > 
> > __Martin
> Ok, as I already said, I think that the reordering is safe there.
> 
> Anyway, the change below should remove all concerns.
> 
> diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c
> index f0629e8..383040e 100644
> --- a/lib/libc/stdio/fclose.c
> +++ b/lib/libc/stdio/fclose.c
> @@ -41,9 +41,12 @@ __FBSDID("$FreeBSD$");
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include "un-namespace.h"
> +#include <spinlock.h>
>  #include "libc_private.h"
>  #include "local.h"
>  
> +extern spinlock_t __stdio_thread_lock;
> +
>  int
>  fclose(FILE *fp)
>  {
> @@ -65,7 +68,11 @@ fclose(FILE *fp)
>  		FREELB(fp);
>  	fp->_file = -1;
>  	fp->_r = fp->_w = 0;	/* Mess up if reaccessed. */
> +	if (__isthreaded)
> +		_SPINLOCK(&__stdio_thread_lock);
>  	fp->_flags = 0;		/* Release this FILE for reuse. */
> +	if (__isthreaded)
> +		_SPINUNLOCK(&__stdio_thread_lock);
>  	FUNLOCKFILE(fp);
>  	return (r);
>  }
> diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c
> index 89c0536..bcd6f62 100644
> --- a/lib/libc/stdio/findfp.c
> +++ b/lib/libc/stdio/findfp.c
> @@ -82,9 +82,9 @@ static struct glue *lastglue = &uglue;
>  
>  static struct glue *	moreglue(int);
>  
> -static spinlock_t thread_lock = _SPINLOCK_INITIALIZER;
> -#define THREAD_LOCK()	if (__isthreaded) _SPINLOCK(&thread_lock)
> -#define THREAD_UNLOCK()	if (__isthreaded) _SPINUNLOCK(&thread_lock)
> +spinlock_t __stdio_thread_lock = _SPINLOCK_INITIALIZER;
> +#define THREAD_LOCK()	if (__isthreaded) _SPINLOCK(&__stdio_thread_lock)
> +#define THREAD_UNLOCK()	if (__isthreaded) 
_SPINUNLOCK(&__stdio_thread_lock)
>  
>  #if NOT_YET
>  #define	SET_GLUE_PTR(ptr, val)	atomic_set_rel_ptr(&(ptr), (uintptr_t)
(val))

Could you move the extern and THREAD_LOCK/UNLOCK macros into the stdio private 
header file?

-- 
John Baldwin

From owner-freebsd-threads@FreeBSD.ORG  Tue Apr 24 16:58:54 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 5BFE11065675;
	Tue, 24 Apr 2012 16:58:54 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200])
	by mx1.freebsd.org (Postfix) with ESMTP id EB3D78FC0A;
	Tue, 24 Apr 2012 16:58:53 +0000 (UTC)
Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1])
	by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3OGwh5C070035;
	Tue, 24 Apr 2012 19:58:43 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1])
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id
	q3OGwgrY039057; Tue, 24 Apr 2012 19:58:42 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: (from kostik@localhost)
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3OGwgeY039056; 
	Tue, 24 Apr 2012 19:58:42 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to
	kostikbel@gmail.com using -f
Date: Tue, 24 Apr 2012 19:58:42 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: John Baldwin <jhb@freebsd.org>
Message-ID: <20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
	<20120424140348.GY2358@deviant.kiev.zoral.com.ua>
	<201204241110.14017.jhb@freebsd.org>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="tKmWtG5s+ZnuAuMT"
Content-Disposition: inline
In-Reply-To: <201204241110.14017.jhb@freebsd.org>
User-Agent: Mutt/1.4.2.3i
X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua
X-Virus-Status: Clean
X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	skuns.kiev.zoral.com.ua
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 16:58:54 -0000


--tKmWtG5s+ZnuAuMT
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 24, 2012 at 11:10:13AM -0400, John Baldwin wrote:
> Could you move the extern and THREAD_LOCK/UNLOCK macros into the stdio
> private header file?

This requires more glue because spinlock.h is separate from libc_private.h.
I also added a comment trying to explain why assignment is braced with lock.

diff --git a/lib/libc/include/libc_private.h b/lib/libc/include/libc_privat=
e.h
index d4b24a7..2182f46 100644
--- a/lib/libc/include/libc_private.h
+++ b/lib/libc/include/libc_private.h
@@ -81,6 +81,19 @@ void _rtld_error(const char *fmt, ...);
 #define	FLOCKFILE(fp)		if (__isthreaded) _FLOCKFILE(fp)
 #define	FUNLOCKFILE(fp)		if (__isthreaded) _funlockfile(fp)
=20
+struct _spinlock;
+extern struct _spinlock __stdio_thread_lock;
+#define STDIO_THREAD_LOCK()				\
+do {							\
+	if (__isthreaded)				\
+		_SPINLOCK(&__stdio_thread_lock);	\
+} while (0)
+#define STDIO_THREAD_UNLOCK()				\
+do {							\
+	if (__isthreaded)				\
+		_SPINUNLOCK(&__stdio_thread_lock);	\
+} while (0)
+
 /*
  * Indexes into the pthread jump table.
  *
diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c
index f0629e8..3957b6a 100644
--- a/lib/libc/stdio/fclose.c
+++ b/lib/libc/stdio/fclose.c
@@ -41,6 +41,7 @@ __FBSDID("$FreeBSD$");
 #include <stdio.h>
 #include <stdlib.h>
 #include "un-namespace.h"
+#include <spinlock.h>
 #include "libc_private.h"
 #include "local.h"
=20
@@ -65,7 +66,20 @@ fclose(FILE *fp)
 		FREELB(fp);
 	fp->_file =3D -1;
 	fp->_r =3D fp->_w =3D 0;	/* Mess up if reaccessed. */
+
+	/*
+	 * Lock the spinlock used to protect __sglue list walk in
+	 * __sfp().  The __sfp() uses fp->_flags =3D=3D 0 test as an
+	 * indication of the unused FILE.
+	 *
+	 * Taking the lock prevents possible compiler or processor
+	 * reordering of the writes performed before the final _flags
+	 * cleanup, making sure that we are done with the FILE before
+	 * it is considered available.
+	 */
+	STDIO_THREAD_LOCK();
 	fp->_flags =3D 0;		/* Release this FILE for reuse. */
+	STDIO_THREAD_UNLOCK();
 	FUNLOCKFILE(fp);
 	return (r);
 }
diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c
index 89c0536..6d0b673 100644
--- a/lib/libc/stdio/findfp.c
+++ b/lib/libc/stdio/findfp.c
@@ -82,9 +82,7 @@ static struct glue *lastglue =3D &uglue;
=20
 static struct glue *	moreglue(int);
=20
-static spinlock_t thread_lock =3D _SPINLOCK_INITIALIZER;
-#define THREAD_LOCK()	if (__isthreaded) _SPINLOCK(&thread_lock)
-#define THREAD_UNLOCK()	if (__isthreaded) _SPINUNLOCK(&thread_lock)
+spinlock_t __stdio_thread_lock =3D _SPINLOCK_INITIALIZER;
=20
 #if NOT_YET
 #define	SET_GLUE_PTR(ptr, val)	atomic_set_rel_ptr(&(ptr), (uintptr_t)(val))
@@ -127,22 +125,22 @@ __sfp()
 	/*
 	 * The list must be locked because a FILE may be updated.
 	 */
-	THREAD_LOCK();
+	STDIO_THREAD_LOCK();
 	for (g =3D &__sglue; g !=3D NULL; g =3D g->next) {
 		for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++)
 			if (fp->_flags =3D=3D 0)
 				goto found;
 	}
-	THREAD_UNLOCK();	/* don't hold lock while malloc()ing. */
+	STDIO_THREAD_UNLOCK();	/* don't hold lock while malloc()ing. */
 	if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL)
 		return (NULL);
-	THREAD_LOCK();		/* reacquire the lock */
+	STDIO_THREAD_LOCK();	/* reacquire the lock */
 	SET_GLUE_PTR(lastglue->next, g); /* atomically append glue to list */
 	lastglue =3D g;		/* not atomic; only accessed when locked */
 	fp =3D g->iobs;
 found:
 	fp->_flags =3D 1;		/* reserve this slot; caller sets real flags */
-	THREAD_UNLOCK();
+	STDIO_THREAD_UNLOCK();
 	fp->_p =3D NULL;		/* no current pointer */
 	fp->_w =3D 0;		/* nothing to read or write */
 	fp->_r =3D 0;
@@ -183,10 +181,10 @@ f_prealloc(void)
 	for (g =3D &__sglue; (n -=3D g->niobs) > 0 && g->next; g =3D g->next)
 		/* void */;
 	if ((n > 0) && ((g =3D moreglue(n)) !=3D NULL)) {
-		THREAD_LOCK();
+		STDIO_THREAD_LOCK();
 		SET_GLUE_PTR(lastglue->next, g);
 		lastglue =3D g;
-		THREAD_UNLOCK();
+		STDIO_THREAD_UNLOCK();
 	}
 }
=20

--tKmWtG5s+ZnuAuMT
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAk+W28EACgkQC3+MBN1Mb4gMfgCfaX0Wy+vGyU9mCRfi4p3D+WVa
Hw8AoL4tyq96n3qE7t1iJ1a3g6UkP9PT
=HMbW
-----END PGP SIGNATURE-----

--tKmWtG5s+ZnuAuMT--

From owner-freebsd-threads@FreeBSD.ORG  Tue Apr 24 17:34:56 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id E54AB1065672
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 17:34:56 +0000 (UTC)
	(envelope-from jhb@freebsd.org)
Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net
	[IPv6:2001:470:1f10:75::2])
	by mx1.freebsd.org (Postfix) with ESMTP id B9FE88FC0A
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 17:34:56 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
	by bigwig.baldwin.cx (Postfix) with ESMTPSA id 06546B95B;
	Tue, 24 Apr 2012 13:34:56 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: Konstantin Belousov <kostikbel@gmail.com>
Date: Tue, 24 Apr 2012 13:34:37 -0400
User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; )
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241110.14017.jhb@freebsd.org>
	<20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
In-Reply-To: <20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-15"
Content-Transfer-Encoding: 7bit
Message-Id: <201204241334.37580.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
	(bigwig.baldwin.cx); Tue, 24 Apr 2012 13:34:56 -0400 (EDT)
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 17:34:57 -0000

On Tuesday, April 24, 2012 12:58:42 pm Konstantin Belousov wrote:
> On Tue, Apr 24, 2012 at 11:10:13AM -0400, John Baldwin wrote:
> > Could you move the extern and THREAD_LOCK/UNLOCK macros into the stdio
> > private header file?
> 
> This requires more glue because spinlock.h is separate from libc_private.h.
> I also added a comment trying to explain why assignment is braced with lock.

This looks great, thanks!

-- 
John Baldwin

From owner-freebsd-threads@FreeBSD.ORG  Tue Apr 24 18:00:37 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 649AD106566B;
	Tue, 24 Apr 2012 18:00:37 +0000 (UTC)
	(envelope-from martin@lispworks.com)
Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com
	[193.34.186.230])
	by mx1.freebsd.org (Postfix) with ESMTP id E71878FC08;
	Tue, 24 Apr 2012 18:00:36 +0000 (UTC)
Received: from higson.cam.lispworks.com (higson [192.168.1.7])
	by lwfs1-cam.cam.lispworks.com (8.14.3/8.14.3) with ESMTP id
	q3OI0YCc028398; Tue, 24 Apr 2012 19:00:34 +0100 (BST)
	(envelope-from martin@lispworks.com)
Received: from higson.cam.lispworks.com (localhost.localdomain [127.0.0.1]) by
	higson.cam.lispworks.com (8.14.4) id q3OI0X1x007388;
	Tue, 24 Apr 2012 19:00:33 +0100
Received: (from martin@localhost)
	by higson.cam.lispworks.com (8.14.4/8.14.4/Submit) id q3OI0X89007384;
	Tue, 24 Apr 2012 19:00:33 +0100
Date: Tue, 24 Apr 2012 19:00:33 +0100
Message-Id: <201204241800.q3OI0X89007384@higson.cam.lispworks.com>
From: Martin Simmons <martin@lispworks.com>
To: Konstantin Belousov <kostikbel@gmail.com>
In-reply-to: <20120424165842.GZ2358@deviant.kiev.zoral.com.ua> (message from
	Konstantin Belousov on Tue, 24 Apr 2012 19:58:42 +0300)
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
	<20120424140348.GY2358@deviant.kiev.zoral.com.ua>
	<201204241110.14017.jhb@freebsd.org>
	<20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 18:00:37 -0000

>>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said:
> 
> +
> +	/*
> +	 * Lock the spinlock used to protect __sglue list walk in
> +	 * __sfp().  The __sfp() uses fp->_flags == 0 test as an
> +	 * indication of the unused FILE.
> +	 *
> +	 * Taking the lock prevents possible compiler or processor
> +	 * reordering of the writes performed before the final _flags
> +	 * cleanup, making sure that we are done with the FILE before
> +	 * it is considered available.
> +	 */
> +	STDIO_THREAD_LOCK();
>  	fp->_flags = 0;		/* Release this FILE for reuse. */
> +	STDIO_THREAD_UNLOCK();
>  	FUNLOCKFILE(fp);
>  	return (r);

Is that assumption about reordering correct?  I.e. is FreeBSD's spinlock a
two-way barrier?

It isn't unusual for the locking primitive to be a one-way barrier, i.e. (from
Linux kernel memory-barriers.txt) "Memory operations that occur before a LOCK
operation may appear to happen after it completes."  See also acq and rel in
atomic(9).

__Martin

From owner-freebsd-threads@FreeBSD.ORG  Tue Apr 24 18:13:09 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id BC7D61065673;
	Tue, 24 Apr 2012 18:13:09 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200])
	by mx1.freebsd.org (Postfix) with ESMTP id 18FF98FC0A;
	Tue, 24 Apr 2012 18:13:08 +0000 (UTC)
Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1])
	by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3OID3Rq081651;
	Tue, 24 Apr 2012 21:13:03 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1])
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id
	q3OID2Xc039889; Tue, 24 Apr 2012 21:13:02 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: (from kostik@localhost)
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3OID2Fn039888; 
	Tue, 24 Apr 2012 21:13:02 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to
	kostikbel@gmail.com using -f
Date: Tue, 24 Apr 2012 21:13:02 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Martin Simmons <martin@lispworks.com>
Message-ID: <20120424181302.GB2358@deviant.kiev.zoral.com.ua>
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
	<20120424140348.GY2358@deviant.kiev.zoral.com.ua>
	<201204241110.14017.jhb@freebsd.org>
	<20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
	<201204241800.q3OI0X89007384@higson.cam.lispworks.com>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="6lA9ewhm9nShhcj2"
Content-Disposition: inline
In-Reply-To: <201204241800.q3OI0X89007384@higson.cam.lispworks.com>
User-Agent: Mutt/1.4.2.3i
X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua
X-Virus-Status: Clean
X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	skuns.kiev.zoral.com.ua
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 24 Apr 2012 18:13:09 -0000


--6lA9ewhm9nShhcj2
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote:
> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said:
> >=20
> > +
> > +	/*
> > +	 * Lock the spinlock used to protect __sglue list walk in
> > +	 * __sfp().  The __sfp() uses fp->_flags =3D=3D 0 test as an
> > +	 * indication of the unused FILE.
> > +	 *
> > +	 * Taking the lock prevents possible compiler or processor
> > +	 * reordering of the writes performed before the final _flags
> > +	 * cleanup, making sure that we are done with the FILE before
> > +	 * it is considered available.
> > +	 */
> > +	STDIO_THREAD_LOCK();
> >  	fp->_flags =3D 0;		/* Release this FILE for reuse. */
> > +	STDIO_THREAD_UNLOCK();
> >  	FUNLOCKFILE(fp);
> >  	return (r);
>=20
> Is that assumption about reordering correct?  I.e. is FreeBSD's spinlock a
> two-way barrier?
>=20
> It isn't unusual for the locking primitive to be a one-way barrier, i.e. =
(from
> Linux kernel memory-barriers.txt) "Memory operations that occur before a =
LOCK
> operation may appear to happen after it completes."  See also acq and rel=
 in
> atomic(9).
Taking the lock prevents the __sfp from iterating the list until the
spinlock is released. Since release makes sure that all previous memory
operations become visible, the acquire in the spinlock lock provides
the neccesary guarentee.

--6lA9ewhm9nShhcj2
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAk+W7S4ACgkQC3+MBN1Mb4j2XwCfUwYZ5frhFeuNuQblrpS/xCMH
lKAAoMVOuTnHmMBWsL/5YaWIBx+Me3xq
=YN7b
-----END PGP SIGNATURE-----

--6lA9ewhm9nShhcj2--

From owner-freebsd-threads@FreeBSD.ORG  Wed Apr 25 03:25:42 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 009EC106566C
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 03:25:41 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id B38638FC21
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 03:25:41 +0000 (UTC)
Received: by obbuo13 with SMTP id uo13so2378052obb.13
	for <freebsd-threads@freebsd.org>; Tue, 24 Apr 2012 20:25:41 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=GOLP4V6A/TYfb3BSmbejQGlYmAOvIj70bye12k6pHc4=;
	b=mWtiBcC9X2CcCkyMi1jOoQrlDhSeDJuNGREnzCW7JV+IJbAsHGfRlZisNBA37dP+7z
	aXgDsBQfzV062wIVbZfpbQrrLePxDjip1+a0GDWfnmU+xJQc3i5oSdwQNpv2T9sFySfd
	ktzfAOSWv+Dkcec3e+3gEBNsk6R8BmwxJ2OLim5siaZGltu+3Q6PNb7QjCFb88Qb+JUZ
	hseJO3kz+S5gKN+cr+IqqZfFyZvD1GfhQizFoa6K+sg6MQcIsvu72M+K3uCPJKOhK1Tb
	2YO9nT9I0TWfnmBEB1g5XzEFlMiZUGs/tnItovpa90pfjqUod7XN5+IGbp99N2Zk+uTa
	FtVA==
MIME-Version: 1.0
Received: by 10.182.232.38 with SMTP id tl6mr1431864obc.16.1335324341113; Tue,
	24 Apr 2012 20:25:41 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Tue, 24 Apr 2012 20:25:40 -0700 (PDT)
In-Reply-To: <20120424181302.GB2358@deviant.kiev.zoral.com.ua>
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
	<20120424140348.GY2358@deviant.kiev.zoral.com.ua>
	<201204241110.14017.jhb@freebsd.org>
	<20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
	<201204241800.q3OI0X89007384@higson.cam.lispworks.com>
	<20120424181302.GB2358@deviant.kiev.zoral.com.ua>
Date: Wed, 25 Apr 2012 11:25:40 +0800
Message-ID: <CAPHpMu=Lv_zYqQEjbpSpJwYsmRr04714+_-2jhPqXdYj_LNgvQ@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Konstantin Belousov <kostikbel@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2012 03:25:42 -0000

On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov
<kostikbel@gmail.com> wrote:
> On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote:
>> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said:
>> >
>> > +
>> > + =A0 /*
>> > + =A0 =A0* Lock the spinlock used to protect __sglue list walk in
>> > + =A0 =A0* __sfp(). =A0The __sfp() uses fp->_flags =3D=3D 0 test as an
>> > + =A0 =A0* indication of the unused FILE.
>> > + =A0 =A0*
>> > + =A0 =A0* Taking the lock prevents possible compiler or processor
>> > + =A0 =A0* reordering of the writes performed before the final _flags
>> > + =A0 =A0* cleanup, making sure that we are done with the FILE before
>> > + =A0 =A0* it is considered available.
>> > + =A0 =A0*/
>> > + =A0 STDIO_THREAD_LOCK();
>> > =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE for reu=
se. */
>> > + =A0 STDIO_THREAD_UNLOCK();
>> > =A0 =A0 FUNLOCKFILE(fp);
>> > =A0 =A0 return (r);
>>
>> Is that assumption about reordering correct? =A0I.e. is FreeBSD's spinlo=
ck a
>> two-way barrier?
>>
>> It isn't unusual for the locking primitive to be a one-way barrier, i.e.=
 (from
>> Linux kernel memory-barriers.txt) "Memory operations that occur before a=
 LOCK
>> operation may appear to happen after it completes." =A0See also acq and =
rel in
>> atomic(9).
> Taking the lock prevents the __sfp from iterating the list until the
> spinlock is released. Since release makes sure that all previous memory
> operations become visible, the acquire in the spinlock lock provides
> the neccesary guarentee.

IMHO, the lock to me is too heavy here. What about this patch?

NOTE: patch just show thoughts. I didn't even check build checking.

diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c
index f0629e8..a26f944 100644
--- a/lib/libc/stdio/fclose.c
+++ b/lib/libc/stdio/fclose.c
@@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$");

 #include "namespace.h"
 #include <errno.h>
+#include <machine/atomic.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include "un-namespace.h"
@@ -65,6 +66,7 @@ fclose(FILE *fp)
 		FREELB(fp);
 	fp->_file =3D -1;
 	fp->_r =3D fp->_w =3D 0;	/* Mess up if reaccessed. */
+	wmb();
 	fp->_flags =3D 0;		/* Release this FILE for reuse. */
 	FUNLOCKFILE(fp);
 	return (r);
diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c
index 89c0536..03b2945 100644
--- a/lib/libc/stdio/findfp.c
+++ b/lib/libc/stdio/findfp.c
@@ -129,9 +129,16 @@ __sfp()
 	 */
 	THREAD_LOCK();
 	for (g =3D &__sglue; g !=3D NULL; g =3D g->next) {
-		for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++)
-			if (fp->_flags =3D=3D 0)
+		for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++) {
+			int __flags =3D fp->_flags;
+			rmb();
+			/*
+			 * If could see __flags is zero here, we are sure
+			 * the cleanup in fclose is done.
+			 */
+			if (__flags =3D=3D 0)
 				goto found;
+		}
 	}
 	THREAD_UNLOCK();	/* don't hold lock while malloc()ing. */
 	if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL)

From owner-freebsd-threads@FreeBSD.ORG  Wed Apr 25 06:26:32 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 3ABA0106566C
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 06:26:32 +0000 (UTC)
	(envelope-from kostikbel@gmail.com)
Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200])
	by mx1.freebsd.org (Postfix) with ESMTP id C9E808FC17
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 06:26:31 +0000 (UTC)
Received: from skuns.kiev.zoral.com.ua (localhost [127.0.0.1])
	by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id q3P6QSfg046705;
	Wed, 25 Apr 2012 09:26:28 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1])
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5) with ESMTP id
	q3P6QRbD044397; Wed, 25 Apr 2012 09:26:27 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
Received: (from kostik@localhost)
	by deviant.kiev.zoral.com.ua (8.14.5/8.14.5/Submit) id q3P6QRc2044396; 
	Wed, 25 Apr 2012 09:26:27 +0300 (EEST)
	(envelope-from kostikbel@gmail.com)
X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to
	kostikbel@gmail.com using -f
Date: Wed, 25 Apr 2012 09:26:27 +0300
From: Konstantin Belousov <kostikbel@gmail.com>
To: Fengwei yin <yfw.bsd@gmail.com>
Message-ID: <20120425062627.GI2358@deviant.kiev.zoral.com.ua>
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
	<20120424140348.GY2358@deviant.kiev.zoral.com.ua>
	<201204241110.14017.jhb@freebsd.org>
	<20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
	<201204241800.q3OI0X89007384@higson.cam.lispworks.com>
	<20120424181302.GB2358@deviant.kiev.zoral.com.ua>
	<CAPHpMu=Lv_zYqQEjbpSpJwYsmRr04714+_-2jhPqXdYj_LNgvQ@mail.gmail.com>
Mime-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="REd8r87flCTRYq/l"
Content-Disposition: inline
In-Reply-To: <CAPHpMu=Lv_zYqQEjbpSpJwYsmRr04714+_-2jhPqXdYj_LNgvQ@mail.gmail.com>
User-Agent: Mutt/1.4.2.3i
X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua
X-Virus-Status: Clean
X-Spam-Status: No, score=-4.0 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00
	autolearn=ham version=3.2.5
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
	skuns.kiev.zoral.com.ua
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2012 06:26:32 -0000


--REd8r87flCTRYq/l
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote:
> On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov
> <kostikbel@gmail.com> wrote:
> > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote:
> >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said:
> >> >
> >> > +
> >> > + =9A /*
> >> > + =9A =9A* Lock the spinlock used to protect __sglue list walk in
> >> > + =9A =9A* __sfp(). =9AThe __sfp() uses fp->_flags =3D=3D 0 test as =
an
> >> > + =9A =9A* indication of the unused FILE.
> >> > + =9A =9A*
> >> > + =9A =9A* Taking the lock prevents possible compiler or processor
> >> > + =9A =9A* reordering of the writes performed before the final _flags
> >> > + =9A =9A* cleanup, making sure that we are done with the FILE before
> >> > + =9A =9A* it is considered available.
> >> > + =9A =9A*/
> >> > + =9A STDIO_THREAD_LOCK();
> >> > =9A =9A fp->_flags =3D 0; =9A =9A =9A =9A /* Release this FILE for r=
euse. */
> >> > + =9A STDIO_THREAD_UNLOCK();
> >> > =9A =9A FUNLOCKFILE(fp);
> >> > =9A =9A return (r);
> >>
> >> Is that assumption about reordering correct? =9AI.e. is FreeBSD's spin=
lock a
> >> two-way barrier?
> >>
> >> It isn't unusual for the locking primitive to be a one-way barrier, i.=
e. (from
> >> Linux kernel memory-barriers.txt) "Memory operations that occur before=
 a LOCK
> >> operation may appear to happen after it completes." =9ASee also acq an=
d rel in
> >> atomic(9).
> > Taking the lock prevents the __sfp from iterating the list until the
> > spinlock is released. Since release makes sure that all previous memory
> > operations become visible, the acquire in the spinlock lock provides
> > the neccesary guarentee.
>=20
> IMHO, the lock to me is too heavy here. What about this patch?
>=20
> NOTE: patch just show thoughts. I didn't even check build checking.
Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers
over the rmb/wmb.

Also, the lock is not that heavy right there, and the committed patch
provides mostly zero overhead for non-threaded case.
>=20
> diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c
> index f0629e8..a26f944 100644
> --- a/lib/libc/stdio/fclose.c
> +++ b/lib/libc/stdio/fclose.c
> @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$");
>=20
>  #include "namespace.h"
>  #include <errno.h>
> +#include <machine/atomic.h>
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include "un-namespace.h"
> @@ -65,6 +66,7 @@ fclose(FILE *fp)
>  		FREELB(fp);
>  	fp->_file =3D -1;
>  	fp->_r =3D fp->_w =3D 0;	/* Mess up if reaccessed. */
> +	wmb();
>  	fp->_flags =3D 0;		/* Release this FILE for reuse. */
>  	FUNLOCKFILE(fp);
>  	return (r);
> diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c
> index 89c0536..03b2945 100644
> --- a/lib/libc/stdio/findfp.c
> +++ b/lib/libc/stdio/findfp.c
> @@ -129,9 +129,16 @@ __sfp()
>  	 */
>  	THREAD_LOCK();
>  	for (g =3D &__sglue; g !=3D NULL; g =3D g->next) {
> -		for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++)
> -			if (fp->_flags =3D=3D 0)
> +		for (fp =3D g->iobs, n =3D g->niobs; --n >=3D 0; fp++) {
> +			int __flags =3D fp->_flags;
> +			rmb();
> +			/*
> +			 * If could see __flags is zero here, we are sure
> +			 * the cleanup in fclose is done.
> +			 */
> +			if (__flags =3D=3D 0)
>  				goto found;
> +		}
>  	}
>  	THREAD_UNLOCK();	/* don't hold lock while malloc()ing. */
>  	if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL)

--REd8r87flCTRYq/l
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)

iEYEARECAAYFAk+XmRMACgkQC3+MBN1Mb4jXWQCfTn0XopbMM1f3hqyZQOP46LZW
1jsAn2QhZGPePVNspHwvZ5oeQvSChtWA
=ANAw
-----END PGP SIGNATURE-----

--REd8r87flCTRYq/l--

From owner-freebsd-threads@FreeBSD.ORG  Wed Apr 25 07:05:55 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id A367C1065679
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 07:05:55 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 6286E8FC0C
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 07:05:55 +0000 (UTC)
Received: by obbuo13 with SMTP id uo13so2643991obb.13
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 00:05:54 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=4f126NQ7j57y8puiYv7ou8VYlXy5X3OoM3LvZyMgln0=;
	b=m4ftGWenoRsJqVEEsBCznTVB1DBZIXXnDTT3nHmbeOqLI2tr+GD+Fxzerrdbo3mUqN
	uNAXsdhWDDNgsHZTHhflTYTBGzes65xs3G9ZiiMGbYu0GGMQu98FnbyuX0Vo9kBk8MPl
	jXmfsJ4k+uam8PxTPV6nU0gN7QclHmbxLNenAKelvT8b7YiDvMZQWsJVStl6DELWMvnA
	2mmRU5TnrtNPm0vwkOYNNKGAAwJzQiCca1n9cgKoRgiYnqs22rowaGnSJ8Rzzbr3r/uW
	UvJ1RRQM/WrACyZa19iIeS8tGvaWIXh6QdLQbnU8XQzDmZ65MBhMcVZ7VaU12Y6AvFfP
	AgKw==
MIME-Version: 1.0
Received: by 10.182.113.106 with SMTP id ix10mr1508664obb.26.1335337554738;
	Wed, 25 Apr 2012 00:05:54 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Wed, 25 Apr 2012 00:05:54 -0700 (PDT)
In-Reply-To: <20120425062627.GI2358@deviant.kiev.zoral.com.ua>
References: <20120423084120.GD76983@zxy.spb.ru>
	<201204241343.q3ODhe2C032683@higson.cam.lispworks.com>
	<20120424140348.GY2358@deviant.kiev.zoral.com.ua>
	<201204241110.14017.jhb@freebsd.org>
	<20120424165842.GZ2358@deviant.kiev.zoral.com.ua>
	<201204241800.q3OI0X89007384@higson.cam.lispworks.com>
	<20120424181302.GB2358@deviant.kiev.zoral.com.ua>
	<CAPHpMu=Lv_zYqQEjbpSpJwYsmRr04714+_-2jhPqXdYj_LNgvQ@mail.gmail.com>
	<20120425062627.GI2358@deviant.kiev.zoral.com.ua>
Date: Wed, 25 Apr 2012 15:05:54 +0800
Message-ID: <CAPHpMunhHr-yGR3vBZb176xFv_gugapm7w87P-LTciEbx+JHGg@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Konstantin Belousov <kostikbel@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2012 07:05:55 -0000

On Wed, Apr 25, 2012 at 2:26 PM, Konstantin Belousov
<kostikbel@gmail.com> wrote:
> On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote:
>> On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov
>> <kostikbel@gmail.com> wrote:
>> > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote:
>> >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said:
>> >> >
>> >> > +
>> >> > + =A0 /*
>> >> > + =A0 =A0* Lock the spinlock used to protect __sglue list walk in
>> >> > + =A0 =A0* __sfp(). =A0The __sfp() uses fp->_flags =3D=3D 0 test as=
 an
>> >> > + =A0 =A0* indication of the unused FILE.
>> >> > + =A0 =A0*
>> >> > + =A0 =A0* Taking the lock prevents possible compiler or processor
>> >> > + =A0 =A0* reordering of the writes performed before the final _fla=
gs
>> >> > + =A0 =A0* cleanup, making sure that we are done with the FILE befo=
re
>> >> > + =A0 =A0* it is considered available.
>> >> > + =A0 =A0*/
>> >> > + =A0 STDIO_THREAD_LOCK();
>> >> > =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE for =
reuse. */
>> >> > + =A0 STDIO_THREAD_UNLOCK();
>> >> > =A0 =A0 FUNLOCKFILE(fp);
>> >> > =A0 =A0 return (r);
>> >>
>> >> Is that assumption about reordering correct? =A0I.e. is FreeBSD's spi=
nlock a
>> >> two-way barrier?
>> >>
>> >> It isn't unusual for the locking primitive to be a one-way barrier, i=
.e. (from
>> >> Linux kernel memory-barriers.txt) "Memory operations that occur befor=
e a LOCK
>> >> operation may appear to happen after it completes." =A0See also acq a=
nd rel in
>> >> atomic(9).
>> > Taking the lock prevents the __sfp from iterating the list until the
>> > spinlock is released. Since release makes sure that all previous memor=
y
>> > operations become visible, the acquire in the spinlock lock provides
>> > the neccesary guarentee.
>>
>> IMHO, the lock to me is too heavy here. What about this patch?
>>
>> NOTE: patch just show thoughts. I didn't even check build checking.
> Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers
> over the rmb/wmb.
>

There is no stand alone acq/rel APIs (Sorry, as new guy to FreeBSD,
don't know too much APIs).  They are bound to atomic operations.
And yes, atomic_acq/rel should work also.

> Also, the lock is not that heavy right there, and the committed patch
> provides mostly zero overhead for non-threaded case.

But lock could introduced contention in SMP case which could be avoid
with rmb/wmb.

I wonder whether the rmb/wmb could be defined as a compiler barrier on
non-threaded case. Like STDIO_THREAD_LOCK which has version
for thread/non-thread case.

>>
>> diff --git a/lib/libc/stdio/fclose.c b/lib/libc/stdio/fclose.c
>> index f0629e8..a26f944 100644
>> --- a/lib/libc/stdio/fclose.c
>> +++ b/lib/libc/stdio/fclose.c
>> @@ -38,6 +38,7 @@ __FBSDID("$FreeBSD$");
>>
>> =A0#include "namespace.h"
>> =A0#include <errno.h>
>> +#include <machine/atomic.h>
>> =A0#include <stdio.h>
>> =A0#include <stdlib.h>
>> =A0#include "un-namespace.h"
>> @@ -65,6 +66,7 @@ fclose(FILE *fp)
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 FREELB(fp);
>> =A0 =A0 =A0 fp->_file =3D -1;
>> =A0 =A0 =A0 fp->_r =3D fp->_w =3D 0; =A0 =A0/* Mess up if reaccessed. */
>> + =A0 =A0 wmb();
>> =A0 =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE for r=
euse. */
>> =A0 =A0 =A0 FUNLOCKFILE(fp);
>> =A0 =A0 =A0 return (r);
>> diff --git a/lib/libc/stdio/findfp.c b/lib/libc/stdio/findfp.c
>> index 89c0536..03b2945 100644
>> --- a/lib/libc/stdio/findfp.c
>> +++ b/lib/libc/stdio/findfp.c
>> @@ -129,9 +129,16 @@ __sfp()
>> =A0 =A0 =A0 =A0*/
>> =A0 =A0 =A0 THREAD_LOCK();
>> =A0 =A0 =A0 for (g =3D &__sglue; g !=3D NULL; g =3D g->next) {
>> - =A0 =A0 =A0 =A0 =A0 =A0 for (fp =3D g->iobs, n =3D g->niobs; --n >=3D =
0; fp++)
>> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (fp->_flags =3D=3D 0)
>> + =A0 =A0 =A0 =A0 =A0 =A0 for (fp =3D g->iobs, n =3D g->niobs; --n >=3D =
0; fp++) {
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 int __flags =3D fp->_flags;
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 rmb();
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 /*
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* If could see __flags is z=
ero here, we are sure
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0* the cleanup in fclose is =
done.
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0*/
>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 if (__flags =3D=3D 0)
>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 goto found;
>> + =A0 =A0 =A0 =A0 =A0 =A0 }
>> =A0 =A0 =A0 }
>> =A0 =A0 =A0 THREAD_UNLOCK(); =A0 =A0 =A0 =A0/* don't hold lock while mal=
loc()ing. */
>> =A0 =A0 =A0 if ((g =3D moreglue(NDYNAMIC)) =3D=3D NULL)

From owner-freebsd-threads@FreeBSD.ORG  Wed Apr 25 14:15:30 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id DCDDE1065675
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 14:15:29 +0000 (UTC)
	(envelope-from jhb@freebsd.org)
Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net
	[IPv6:2001:470:1f10:75::2])
	by mx1.freebsd.org (Postfix) with ESMTP id B00D68FC1B
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 14:15:29 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
	by bigwig.baldwin.cx (Postfix) with ESMTPSA id 0F7B8B915;
	Wed, 25 Apr 2012 10:15:29 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: freebsd-threads@freebsd.org
Date: Wed, 25 Apr 2012 10:12:36 -0400
User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; )
References: <20120423084120.GD76983@zxy.spb.ru>
	<20120425062627.GI2358@deviant.kiev.zoral.com.ua>
	<CAPHpMunhHr-yGR3vBZb176xFv_gugapm7w87P-LTciEbx+JHGg@mail.gmail.com>
In-Reply-To: <CAPHpMunhHr-yGR3vBZb176xFv_gugapm7w87P-LTciEbx+JHGg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <201204251012.36754.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
	(bigwig.baldwin.cx); Wed, 25 Apr 2012 10:15:29 -0400 (EDT)
Cc: jack.ren@intel.com
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2012 14:15:30 -0000

On Wednesday, April 25, 2012 3:05:54 am Fengwei yin wrote:
> On Wed, Apr 25, 2012 at 2:26 PM, Konstantin Belousov
> <kostikbel@gmail.com> wrote:
> > On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote:
> >> On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov
> >> <kostikbel@gmail.com> wrote:
> >> > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote:
> >> >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said:
> >> >> >
> >> >> > +
> >> >> > +   /*
> >> >> > +    * Lock the spinlock used to protect __sglue list walk in
> >> >> > +    * __sfp().  The __sfp() uses fp->_flags == 0 test as an
> >> >> > +    * indication of the unused FILE.
> >> >> > +    *
> >> >> > +    * Taking the lock prevents possible compiler or processor
> >> >> > +    * reordering of the writes performed before the final _flags
> >> >> > +    * cleanup, making sure that we are done with the FILE before
> >> >> > +    * it is considered available.
> >> >> > +    */
> >> >> > +   STDIO_THREAD_LOCK();
> >> >> >     fp->_flags = 0;         /* Release this FILE for reuse. */
> >> >> > +   STDIO_THREAD_UNLOCK();
> >> >> >     FUNLOCKFILE(fp);
> >> >> >     return (r);
> >> >>
> >> >> Is that assumption about reordering correct?  I.e. is FreeBSD's spinlock a
> >> >> two-way barrier?
> >> >>
> >> >> It isn't unusual for the locking primitive to be a one-way barrier, i.e. (from
> >> >> Linux kernel memory-barriers.txt) "Memory operations that occur before a LOCK
> >> >> operation may appear to happen after it completes."  See also acq and rel in
> >> >> atomic(9).
> >> > Taking the lock prevents the __sfp from iterating the list until the
> >> > spinlock is released. Since release makes sure that all previous memory
> >> > operations become visible, the acquire in the spinlock lock provides
> >> > the neccesary guarentee.
> >>
> >> IMHO, the lock to me is too heavy here. What about this patch?
> >>
> >> NOTE: patch just show thoughts. I didn't even check build checking.
> > Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers
> > over the rmb/wmb.
> >
> 
> There is no stand alone acq/rel APIs (Sorry, as new guy to FreeBSD,
> don't know too much APIs).  They are bound to atomic operations.
> And yes, atomic_acq/rel should work also.

Yes, you would want to use atomic_store_rel() or some such.  Often doing
so is much clearer than stand-alone membar's as it indicates which write
has special ordering.

> > Also, the lock is not that heavy right there, and the committed patch
> > provides mostly zero overhead for non-threaded case.
> 
> But lock could introduced contention in SMP case which could be avoid
> with rmb/wmb.

Seriously, if you are using stdio, you've already given up on performance
enough to not care about contention for fclose() vs fopen().

-- 
John Baldwin

From owner-freebsd-threads@FreeBSD.ORG  Wed Apr 25 22:55:18 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 71B951065676
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 22:55:18 +0000 (UTC)
	(envelope-from gofdt-freebsd-threads@m.gmane.org)
Received: from plane.gmane.org (plane.gmane.org [80.91.229.3])
	by mx1.freebsd.org (Postfix) with ESMTP id D95288FC0A
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 22:55:14 +0000 (UTC)
Received: from list by plane.gmane.org with local (Exim 4.69)
	(envelope-from <gofdt-freebsd-threads@m.gmane.org>)
	id 1SNB71-0007PK-Vz
	for freebsd-threads@freebsd.org; Thu, 26 Apr 2012 00:55:03 +0200
Received: from 189.32.123.86 ([189.32.123.86])
	by main.gmane.org with esmtp (Gmexim 0.1 (Debian))
	id 1AlnuQ-0007hv-00
	for <freebsd-threads@freebsd.org>; Thu, 26 Apr 2012 00:55:03 +0200
Received: from rnsanchez by 189.32.123.86 with local (Gmexim 0.1 (Debian))
	id 1AlnuQ-0007hv-00
	for <freebsd-threads@freebsd.org>; Thu, 26 Apr 2012 00:55:03 +0200
X-Injected-Via-Gmane: http://gmane.org/
To: freebsd-threads@freebsd.org
From: Ricardo Nabinger Sanchez <rnsanchez@wait4.org>
Date: Wed, 25 Apr 2012 22:51:08 +0000 (UTC)
Lines: 19
Message-ID: <jn9v4s$hij$1@dough.gmane.org>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
X-Complaints-To: usenet@dough.gmane.org
X-Gmane-NNTP-Posting-Host: 189.32.123.86
User-Agent: Pan/0.134 (Wait for Me; GIT cb32159 master)
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2012 22:55:18 -0000

On Mon, 23 Apr 2012 12:41:20 +0400, Slawa Olhovchenkov wrote:

> /usr/include/machine/atomic.h:
> 
> #define mb()    __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> #define wmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> #define rmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")

Somewhat late on this topic, but I'd like to understand why issue a write 
on %esp, which would invalidate (%esp) on other cores --- thus forcing a 
miss on them?

Instead, why not issue "mfence" (mb), "sfence" (wmb), and "lfence" (rmb)?

Cheers

-- 
Ricardo Nabinger Sanchez           http://rnsanchez.wait4.org/
  "Left to themselves, things tend to go from bad to worse."


From owner-freebsd-threads@FreeBSD.ORG  Thu Apr 26 01:20:45 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id 1D3BF106564A;
	Thu, 26 Apr 2012 01:20:45 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id C6AF88FC0A;
	Thu, 26 Apr 2012 01:20:44 +0000 (UTC)
Received: by obcni5 with SMTP id ni5so21264obc.13
	for <multiple recipients>; Wed, 25 Apr 2012 18:20:44 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=WjzlvHMwpyiMQS/B6AkjFHglEnJU0EOnKxyxlD36AhE=;
	b=HmWEMEH2OovsMeEpM1anrapxCfxNYBTUlyJfAUP7iFTsCVxZb50AcHFFJvFZue4Y6S
	eGy6cderNpn+xJ73EnGAAhvTZ0hA7cE9plbelIl2FaZmle3t6z15MHXKRE76vZN07Nfx
	c/5wrmNAe4+dwXGgGMrsodZqZlJB5dPPn0PytsdoVW1Uyl1Y+6PMUM7smpkdahDp2fHK
	ljYg9XIyZWct8gz56blndS0bPAdaiw1Qu6c0IN6NRO8E3e7gtJll/fLlrNsnGybiMZhd
	6C7qaJHvn2/n15yRR3sZVRu08t0f5AStbjAeoWXgs0aocgOCKyAZifKwEoYQFpLMQ6Sa
	zKXQ==
MIME-Version: 1.0
Received: by 10.182.113.106 with SMTP id ix10mr6046003obb.26.1335403244475;
	Wed, 25 Apr 2012 18:20:44 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Wed, 25 Apr 2012 18:20:44 -0700 (PDT)
In-Reply-To: <201204251012.36754.jhb@freebsd.org>
References: <20120423084120.GD76983@zxy.spb.ru>
	<20120425062627.GI2358@deviant.kiev.zoral.com.ua>
	<CAPHpMunhHr-yGR3vBZb176xFv_gugapm7w87P-LTciEbx+JHGg@mail.gmail.com>
	<201204251012.36754.jhb@freebsd.org>
Date: Thu, 26 Apr 2012 09:20:44 +0800
Message-ID: <CAPHpMukC1=fYS8L3b6xNjJhdqZSQW6Aes5RHuVfmFL5nC0n+5w@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: John Baldwin <jhb@freebsd.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: jack.ren@intel.com, freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Apr 2012 01:20:45 -0000

On Wed, Apr 25, 2012 at 10:12 PM, John Baldwin <jhb@freebsd.org> wrote:
> On Wednesday, April 25, 2012 3:05:54 am Fengwei yin wrote:
>> On Wed, Apr 25, 2012 at 2:26 PM, Konstantin Belousov
>> <kostikbel@gmail.com> wrote:
>> > On Wed, Apr 25, 2012 at 11:25:40AM +0800, Fengwei yin wrote:
>> >> On Wed, Apr 25, 2012 at 2:13 AM, Konstantin Belousov
>> >> <kostikbel@gmail.com> wrote:
>> >> > On Tue, Apr 24, 2012 at 07:00:33PM +0100, Martin Simmons wrote:
>> >> >> >>>>> On Tue, 24 Apr 2012 19:58:42 +0300, Konstantin Belousov said=
:
>> >> >> >
>> >> >> > +
>> >> >> > + =A0 /*
>> >> >> > + =A0 =A0* Lock the spinlock used to protect __sglue list walk i=
n
>> >> >> > + =A0 =A0* __sfp(). =A0The __sfp() uses fp->_flags =3D=3D 0 test=
 as an
>> >> >> > + =A0 =A0* indication of the unused FILE.
>> >> >> > + =A0 =A0*
>> >> >> > + =A0 =A0* Taking the lock prevents possible compiler or process=
or
>> >> >> > + =A0 =A0* reordering of the writes performed before the final _=
flags
>> >> >> > + =A0 =A0* cleanup, making sure that we are done with the FILE b=
efore
>> >> >> > + =A0 =A0* it is considered available.
>> >> >> > + =A0 =A0*/
>> >> >> > + =A0 STDIO_THREAD_LOCK();
>> >> >> > =A0 =A0 fp->_flags =3D 0; =A0 =A0 =A0 =A0 /* Release this FILE f=
or reuse. */
>> >> >> > + =A0 STDIO_THREAD_UNLOCK();
>> >> >> > =A0 =A0 FUNLOCKFILE(fp);
>> >> >> > =A0 =A0 return (r);
>> >> >>
>> >> >> Is that assumption about reordering correct? =A0I.e. is FreeBSD's =
spinlock a
>> >> >> two-way barrier?
>> >> >>
>> >> >> It isn't unusual for the locking primitive to be a one-way barrier=
, i.e. (from
>> >> >> Linux kernel memory-barriers.txt) "Memory operations that occur be=
fore a LOCK
>> >> >> operation may appear to happen after it completes." =A0See also ac=
q and rel in
>> >> >> atomic(9).
>> >> > Taking the lock prevents the __sfp from iterating the list until th=
e
>> >> > spinlock is released. Since release makes sure that all previous me=
mory
>> >> > operations become visible, the acquire in the spinlock lock provide=
s
>> >> > the neccesary guarentee.
>> >>
>> >> IMHO, the lock to me is too heavy here. What about this patch?
>> >>
>> >> NOTE: patch just show thoughts. I didn't even check build checking.
>> > Yes, it might be correct. But FreeBSD does prefer the acq/rel barriers
>> > over the rmb/wmb.
>> >
>>
>> There is no stand alone acq/rel APIs (Sorry, as new guy to FreeBSD,
>> don't know too much APIs). =A0They are bound to atomic operations.
>> And yes, atomic_acq/rel should work also.
>
> Yes, you would want to use atomic_store_rel() or some such. =A0Often doin=
g
> so is much clearer than stand-alone membar's as it indicates which write
> has special ordering.
>
Thanks a lot for clarification.

>> > Also, the lock is not that heavy right there, and the committed patch
>> > provides mostly zero overhead for non-threaded case.
>>
>> But lock could introduced contention in SMP case which could be avoid
>> with rmb/wmb.
>
> Seriously, if you are using stdio, you've already given up on performance
> enough to not care about contention for fclose() vs fopen().
>
This makes sense.

> --
> John Baldwin

From owner-freebsd-threads@FreeBSD.ORG  Thu Apr 26 01:28:14 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 598B71065672
	for <freebsd-threads@freebsd.org>; Thu, 26 Apr 2012 01:28:14 +0000 (UTC)
	(envelope-from yfw.bsd@gmail.com)
Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com
	[209.85.214.182])
	by mx1.freebsd.org (Postfix) with ESMTP id 1E4A98FC12
	for <freebsd-threads@freebsd.org>; Thu, 26 Apr 2012 01:28:14 +0000 (UTC)
Received: by obcni5 with SMTP id ni5so31389obc.13
	for <freebsd-threads@freebsd.org>; Wed, 25 Apr 2012 18:28:13 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	bh=DtorzYfPs6Wg7tZ4W7ru3KQv5XnfWPLXTxtZ2NGD5n4=;
	b=uwznZO6I/rC5xBWseVHYXM0f18o5oEsVVqq4xgWW+PiQM5OIhlqEMz6LZtwY/A/vEY
	uMabt2XWNAwaRJ97+Qrv09EHZCyNoCHbvnjBNZrBTsedFwR/8v342zsqmECGR83rBO0h
	1FaQNHk4U2x+H9nbZJdgMYdkpUNwfWVoDA3ZNBUVPEd0pbTojcnTOIcv18rkeRPVZ8Pm
	o+wHCX8+UgAzy54huvXRam9oCviuZmSBG95z3JUO/8IMoofsvChJ7VD+g7Td+I5aqIZE
	9FtXMPVT6vDOwA5EgMJvha4QU57XBIdKRGAWw/WwGajNoVYclZv/R8o1KsHZAzvjMvEq
	dbAg==
MIME-Version: 1.0
Received: by 10.60.22.10 with SMTP id z10mr6622939oee.16.1335403693695; Wed,
	25 Apr 2012 18:28:13 -0700 (PDT)
Received: by 10.60.125.135 with HTTP; Wed, 25 Apr 2012 18:28:13 -0700 (PDT)
In-Reply-To: <jn9v4s$hij$1@dough.gmane.org>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru> <jn9v4s$hij$1@dough.gmane.org>
Date: Thu, 26 Apr 2012 09:28:13 +0800
Message-ID: <CAPHpMukaTA5nmxFGTNgob_gLQMYZuKo6ErKZqpt3k70C91ayXA@mail.gmail.com>
From: Fengwei yin <yfw.bsd@gmail.com>
To: Ricardo Nabinger Sanchez <rnsanchez@wait4.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Cc: freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Apr 2012 01:28:14 -0000

On Thu, Apr 26, 2012 at 6:51 AM, Ricardo Nabinger Sanchez
<rnsanchez@wait4.org> wrote:
> On Mon, 23 Apr 2012 12:41:20 +0400, Slawa Olhovchenkov wrote:
>
>> /usr/include/machine/atomic.h:
>>
>> #define mb() =A0 =A0__asm __volatile("lock; addl $0,(%%esp)" : : : "memo=
ry")
>> #define wmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory=
")
>> #define rmb() =A0 __asm __volatile("lock; addl $0,(%%esp)" : : : "memory=
")
>
> Somewhat late on this topic, but I'd like to understand why issue a write
> on %esp, which would invalidate (%esp) on other cores --- thus forcing a
> miss on them?

The key here is prefix "lock:" which could be considered as "mfence" on
modern CPU.

>
> Instead, why not issue "mfence" (mb), "sfence" (wmb), and "lfence" (rmb)?
>
Not all x86 family support "fence" instruction. "lock" prefix works for all=
.

> Cheers
>
> --
> Ricardo Nabinger Sanchez =A0 =A0 =A0 =A0 =A0 http://rnsanchez.wait4.org/
> =A0"Left to themselves, things tend to go from bad to worse."
>
> _______________________________________________
> freebsd-threads@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-threads
> To unsubscribe, send any mail to "freebsd-threads-unsubscribe@freebsd.org=
"

From owner-freebsd-threads@FreeBSD.ORG  Thu Apr 26 01:30:10 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@hub.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 2EEDE1065670
	for <freebsd-threads@hub.freebsd.org>;
	Thu, 26 Apr 2012 01:30:10 +0000 (UTC)
	(envelope-from gnats@FreeBSD.org)
Received: from freefall.freebsd.org (freefall.freebsd.org
	[IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id DC7148FC15
	for <freebsd-threads@hub.freebsd.org>;
	Thu, 26 Apr 2012 01:30:09 +0000 (UTC)
Received: from freefall.freebsd.org (localhost [127.0.0.1])
	by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3Q1U93l099234
	for <freebsd-threads@freefall.freebsd.org>;
	Thu, 26 Apr 2012 01:30:09 GMT
	(envelope-from gnats@freefall.freebsd.org)
Received: (from gnats@localhost)
	by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3Q1U9DO099233;
	Thu, 26 Apr 2012 01:30:09 GMT (envelope-from gnats)
Resent-Date: Thu, 26 Apr 2012 01:30:09 GMT
Resent-Message-Id: <201204260130.q3Q1U9DO099233@freefall.freebsd.org>
Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer)
Resent-To: freebsd-threads@FreeBSD.org
Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Rick Reed <rr@whatsapp.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 0F7A61065672
	for <freebsd-gnats-submit@FreeBSD.org>;
	Thu, 26 Apr 2012 01:20:27 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id EF5DF8FC0C
	for <freebsd-gnats-submit@FreeBSD.org>;
	Thu, 26 Apr 2012 01:20:26 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3Q1KQix006765
	for <freebsd-gnats-submit@FreeBSD.org>; Thu, 26 Apr 2012 01:20:26 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id q3Q1KQCT006764;
	Thu, 26 Apr 2012 01:20:26 GMT (envelope-from nobody)
Message-Id: <201204260120.q3Q1KQCT006764@red.freebsd.org>
Date: Thu, 26 Apr 2012 01:20:26 GMT
From: Rick Reed <rr@whatsapp.com>
To: freebsd-gnats-submit@FreeBSD.org
X-Send-Pr-Version: www-3.1
Cc: 
Subject: threads/167308: Perf regression in thread locking on 8-stable
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Apr 2012 01:30:10 -0000


>Number:         167308
>Category:       threads
>Synopsis:       Perf regression in thread locking on 8-stable
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    freebsd-threads
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Thu Apr 26 01:30:09 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Rick Reed
>Release:        8.3-STABLE (cvsup @4/17/12 1300)
>Organization:
WhatsApp Inc.
>Environment:
8.3-STABLE amd64
>Description:
Rev 234373 causes a significant performance regression for our Erlang-based application.  I don't know what the direct results of the change are, but it manifests as abnormal amounts of contention for a particular global lock among the scheduler threads (one per CPU) in the Erlang VM under significant load.  Backing this change out results in normal contention and performance.

I see there's new code using WAKE2 in HEAD & 9, but I'm not sure how you balance the correctness issue in the stated case of unlock() then destroy() versus the significant impact to performance for apps which don't do that.

>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted:

From owner-freebsd-threads@FreeBSD.ORG  Thu Apr 26 01:44:27 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@hub.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 449A3106566B;
	Thu, 26 Apr 2012 01:44:27 +0000 (UTC)
	(envelope-from linimon@FreeBSD.org)
Received: from freefall.freebsd.org (freefall.freebsd.org
	[IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id 17E5F8FC08;
	Thu, 26 Apr 2012 01:44:27 +0000 (UTC)
Received: from freefall.freebsd.org (localhost [127.0.0.1])
	by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3Q1iQSU018672;
	Thu, 26 Apr 2012 01:44:26 GMT
	(envelope-from linimon@freefall.freebsd.org)
Received: (from linimon@localhost)
	by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3Q1iQnG018668;
	Thu, 26 Apr 2012 01:44:26 GMT (envelope-from linimon)
Date: Thu, 26 Apr 2012 01:44:26 GMT
Message-Id: <201204260144.q3Q1iQnG018668@freefall.freebsd.org>
To: linimon@FreeBSD.org, freebsd-threads@FreeBSD.org, davidxu@FreeBSD.org
From: linimon@FreeBSD.org
Cc: 
Subject: Re: threads/167308: Performance regression in thread locking on
	8-stable [regression]
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Apr 2012 01:44:27 -0000

Old Synopsis: Perf regression in thread locking on 8-stable
New Synopsis: Performance regression in thread locking on 8-stable [regression]

Responsible-Changed-From-To: freebsd-threads->davidxu
Responsible-Changed-By: linimon
Responsible-Changed-When: Thu Apr 26 01:41:20 UTC 2012
Responsible-Changed-Why: 
Over to committer of r234373.

http://www.freebsd.org/cgi/query-pr.cgi?pr=167308

From owner-freebsd-threads@FreeBSD.ORG  Thu Apr 26 02:45:32 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id E1B8B106566C;
	Thu, 26 Apr 2012 02:45:32 +0000 (UTC)
	(envelope-from listlog2011@gmail.com)
Received: from freefall.freebsd.org (freefall.freebsd.org
	[IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id CBE728FC16;
	Thu, 26 Apr 2012 02:45:32 +0000 (UTC)
Received: from [127.0.0.1] (localhost [127.0.0.1])
	by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3Q2jV1D074612;
	Thu, 26 Apr 2012 02:45:31 GMT (envelope-from listlog2011@gmail.com)
Message-ID: <4F98B6C8.30703@gmail.com>
Date: Thu, 26 Apr 2012 10:45:28 +0800
From: David Xu <listlog2011@gmail.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1;
	rv:12.0) Gecko/20120420 Thunderbird/12.0
MIME-Version: 1.0
To: linimon@FreeBSD.org
References: <201204260144.q3Q1iQnG018668@freefall.freebsd.org>
In-Reply-To: <201204260144.q3Q1iQnG018668@freefall.freebsd.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: davidxu@FreeBSD.org, freebsd-threads@FreeBSD.org
Subject: Re: threads/167308: Performance regression in thread locking on
 8-stable [regression]
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
Reply-To: davidxu@FreeBSD.org
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Apr 2012 02:45:33 -0000

On 2012/4/26 9:44, linimon@FreeBSD.org wrote:
> Old Synopsis: Perf regression in thread locking on 8-stable
> New Synopsis: Performance regression in thread locking on 8-stable [regression]
>
> Responsible-Changed-From-To: freebsd-threads->davidxu
> Responsible-Changed-By: linimon
> Responsible-Changed-When: Thu Apr 26 01:41:20 UTC 2012
> Responsible-Changed-Why:
> Over to committer of r234373.
>
> http://www.freebsd.org/cgi/query-pr.cgi?pr=167308
>
This is a known problem that the unlocking is not optimized when the mutex
is heavily contented, I may MFC wake2 later.


From owner-freebsd-threads@FreeBSD.ORG  Thu Apr 26 22:43:49 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id EACD3106566B
	for <freebsd-threads@freebsd.org>; Thu, 26 Apr 2012 22:43:49 +0000 (UTC)
	(envelope-from jilles@stack.nl)
Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104])
	by mx1.freebsd.org (Postfix) with ESMTP id 4A84F8FC15
	for <freebsd-threads@freebsd.org>; Thu, 26 Apr 2012 22:43:49 +0000 (UTC)
Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131])
	by mx1.stack.nl (Postfix) with ESMTP id 763BC3592E6;
	Fri, 27 Apr 2012 00:43:48 +0200 (CEST)
Received: by snail.stack.nl (Postfix, from userid 1677)
	id 5953A2847A; Fri, 27 Apr 2012 00:43:48 +0200 (CEST)
Date: Fri, 27 Apr 2012 00:43:48 +0200
From: Jilles Tjoelker <jilles@stack.nl>
To: Ricardo Nabinger Sanchez <rnsanchez@wait4.org>
Message-ID: <20120426224348.GA58463@stack.nl>
References: <CAPHpMu=DOGQ=TuFeYH7bH8hVwteT4Q3k67-mvoOFob6P3Y506w@mail.gmail.com>
	<20120423084120.GD76983@zxy.spb.ru> <jn9v4s$hij$1@dough.gmane.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <jn9v4s$hij$1@dough.gmane.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
Cc: freebsd-threads@freebsd.org
Subject: Re: About the memory barrier in BSD libc
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Apr 2012 22:43:50 -0000

On Wed, Apr 25, 2012 at 10:51:08PM +0000, Ricardo Nabinger Sanchez wrote:
> On Mon, 23 Apr 2012 12:41:20 +0400, Slawa Olhovchenkov wrote:

> > /usr/include/machine/atomic.h:

> > #define mb()    __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> > #define wmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")
> > #define rmb()   __asm __volatile("lock; addl $0,(%%esp)" : : : "memory")

> Somewhat late on this topic, but I'd like to understand why issue a write 
> on %esp, which would invalidate (%esp) on other cores --- thus forcing a 
> miss on them?

The stack is usually private to a thread and is written a lot
regardless.

> Instead, why not issue "mfence" (mb), "sfence" (wmb), and "lfence" (rmb)?

Apart from the fact that those are SSE or SSE2 instructions, the
uncontended locked instructions may be faster. For example, MFENCE is a
serializing instruction on AMD family 10h processors and the AMD
optimization manual recommends using an uncontended locked instruction
instead (though preferably one that performs a useful store).

SFENCE is only useful with non-temporal stores such as MOVNTPS or stores
to write-combining memory because regular stores are not reordered with
respect to one another.

-- 
Jilles Tjoelker

From owner-freebsd-threads@FreeBSD.ORG  Sat Apr 28 12:50:58 2012
Return-Path: <owner-freebsd-threads@FreeBSD.ORG>
Delivered-To: freebsd-threads@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 39C8D106566C
	for <freebsd-threads@freebsd.org>; Sat, 28 Apr 2012 12:50:58 +0000 (UTC)
	(envelope-from chris.hall@highwayman.com)
Received: from anchor-post-1.mail.demon.net (anchor-post-1.mail.demon.net
	[195.173.77.132])
	by mx1.freebsd.org (Postfix) with ESMTP id EE8D78FC14
	for <freebsd-threads@freebsd.org>; Sat, 28 Apr 2012 12:50:57 +0000 (UTC)
Received: from [80.177.246.130] (helo=hestia.halldom.com)
	by anchor-post-1.mail.demon.net with esmtp (Exim 4.69)
	id 1SO71B-0006MB-iE
	for freebsd-threads@freebsd.org; Sat, 28 Apr 2012 12:44:53 +0000
Received: from hyperion.halldom.com ([80.177.246.170] helo=HYPERION)
	by hestia.halldom.com with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.76)
	(envelope-from <chris.hall@highwayman.com>) id 1SO719-0002Ti-H8
	for freebsd-threads@freebsd.org; Sat, 28 Apr 2012 12:44:51 +0000
From: "'Chris Hall'" <chris.hall.list@highwayman.com>
Sender: "Chris Hall" <chris.hall@highwayman.com>
To: <freebsd-threads@freebsd.org>
Date: Sat, 28 Apr 2012 13:44:46 +0100
Organization: Highwayman
Message-ID: <056201cd253c$b43a0860$1cae1920$@highwayman.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="US-ASCII"
Content-Transfer-Encoding: 7bit
X-Mailer: Microsoft Outlook 14.0
Thread-Index: Ac0lPKnGhmxEP7zpSCa0p5c0+tQlcw==
Content-Language: en-gb
Subject: Trying to set PTHREAD_SCOPE_SYSTEM
X-BeenThere: freebsd-threads@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Threading on FreeBSD <freebsd-threads.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>, 
	<mailto:freebsd-threads-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-threads>
List-Post: <mailto:freebsd-threads@freebsd.org>
List-Help: <mailto:freebsd-threads-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-threads>,
	<mailto:freebsd-threads-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 28 Apr 2012 12:50:58 -0000

For my application (a BGP daemon) I think I should be setting all its
pthreads to PTHREAD_SCOPE_SYSTEM.  (This may be where I am going
wrong, of course, but bear with me.)

I have studied the POSIX specifications and the FreeBSD man pages, and
am stuck.

pthread_init_attr() is supposed to give me the default settings for
everything, including the scope and
PTHREAD_INHERIT_SCHED/_EXPLICIT_SCHED.

First difficulty: if it is PTHREAD_INHERIT_SCHED, what are the scope,
schedpolicy and schedparam set to ?

Second difficulty: if I set PTHREAD_EXPLICIT_SCHED, will that change
the scope/policy/param to the defaults, or leave them unchanged (in
which case are they the appropriate defaults) ?  Or, am I really
expected to set all of scope/policy/param if I override a default
PTHREAD_INHERIT_SCHED ?  (If the default is PTHREAD_SCOPE_SYSTEM, am I
all set having cleared PTHREAD_INHERIT_SCHED ?)

Third difficulty: if I set PTHREAD_SCOPE_SYSTEM and the default is
PTHREAD_SCOPE_PROCESS, do I also need to set policy/param ?  If so,
where do I find suitable default policy/param ?
(I could copy the policy/param from the main thread... but will that
be suitable for PTHREAD_SCOPE_SYSTEM ?)

Fourth difficulty: if I dick about with scope/policy/param, will I
need special privileges (root, say) ?

Fifth difficulty: it is clearly easiest to leave this whole thing
alone and ignore pthread_attr_t altogether.  Is there a way to set
PTHREAD_SCOPE_SYSTEM as the default for a given application from the
outside ?

POSIX appears to define a bunch of stuff in this area... but so much
is implementation defined, or simply not defined I imagine (a) 99.9%
of threaded applications do not attempt to use pthread_attr_t at all,
and (b) that somewhere each implementation will define what it does ?

Help !!

Thanks,

Chris