From owner-freebsd-arch@FreeBSD.ORG Sat Aug 18 12:44:54 2007 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C818716A41A for ; Sat, 18 Aug 2007 12:44:54 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (arm132.internetdsl.tpnet.pl [83.17.198.132]) by mx1.freebsd.org (Postfix) with ESMTP id EF67013C481 for ; Sat, 18 Aug 2007 12:44:53 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 8268245696; Sat, 18 Aug 2007 14:44:52 +0200 (CEST) Received: from localhost (154.81.datacomsa.pl [195.34.81.154]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id BDBC945681 for ; Sat, 18 Aug 2007 14:44:47 +0200 (CEST) Date: Sat, 18 Aug 2007 14:43:52 +0200 From: Pawel Jakub Dawidek To: freebsd-arch@FreeBSD.org Message-ID: <20070818124352.GB6498@garage.freebsd.pl> References: <20070818120056.GA6498@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="U+BazGySraz5kW0T" Content-Disposition: inline In-Reply-To: <20070818120056.GA6498@garage.freebsd.pl> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 7.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-2.6 required=3.0 tests=BAYES_00 autolearn=ham version=3.0.4 Cc: Subject: Re: Lockless uidinfo. X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 18 Aug 2007 12:44:55 -0000 --U+BazGySraz5kW0T Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Aug 18, 2007 at 02:00:56PM +0200, Pawel Jakub Dawidek wrote: > Hi. >=20 > The patch below remove per-uidinfo locks: >=20 > http://people.freebsd.org/~pjd/patches/uidinfo_lockless.patch >=20 > With the patch uidinfo is handled using atomics only and no locks > (except for the global hash lock, which is not really important, as it's > not used in the fast paths). >=20 > I needed to change ui_sbsize from rlim_t (64bit) to long, because we > don't have 64bit atomics on all archs, and because sbsize represents > size in bytes, it can't go beyond 32bit on 32bit archs (PAE might be a > bit of a problem). >=20 > I changed maxval argument in chgproccnt() from int to rlim_t, as this is > what is passed to the function. >=20 > In simple ping-pong test on unix domain socket, uidinfo lock was highly > contented: >=20 > max total wait_total count avg wait_avg cnt_hold cn= t_lock name > 1508 3242859 96267052 1467476 2 65 374553 102= 4743 /usr/src/sys/kern/kern_resource.c:1339 (sleep mutex:sleep mtxpool) >=20 > The ping-pong program you can find here: >=20 > http://people.freebsd.org/~pjd/misc/unixpingpong.c >=20 > At the end we reduced uidinfo structure size by 8 bytes and gain no > measurable performance improvements:) > Yes, I wasn't able to measure anything interesting, unfortunately, but I > still believe the patch should be committed, as I'm sure there are > workloads that will see improvements - note that uidinfo structure is > per-uid, so if there are thousands of processes running with the same > uid, they all need to fight for this one lock. > Not only contention is important, but also the fact that number of > atomic operations in chgsbsize() and chgproccnt() functions was reduced > from 2 to 1. >=20 > Ok, while writting this e-mail I came up with a better benchmark: >=20 > http://people.freebsd.org/~pjd/misc/unixpingpong2.c >=20 > This one doesn't do ping-pong between two processes, but within one > proccess only. This way we eliminate cost of context switches. I was > running 8 such processes (I tested it on a 8way machine) and here are > the results: >=20 > x ./uidinfo1.txt > + ./uidinfo2.txt > +------------------------------------------------------------------------= ------+ > |x = +| > |xx = ++| > |xx = ++| > |A| = |A| > +------------------------------------------------------------------------= ------+ > N Min Max Median Avg Stdd= ev > x 5 402742 416417 406216 408269.2 5388.34= 85 > + 5 1561566 1575987 1568964 1569767 5853.13= 99 > Difference at 95.0% confidence > 1.1615e+06 +/- 8204.54 > 284.493% +/- 2.00959% > (Student's t, pooled s =3D 5625.55) >=20 > As you can see this was just a matter of good benchmark - here we can see > 284% performance improvement, yay:) Kris asked me to verify just in case that the unit cost is not higher and here are the results from running only one ping-pong process (so no contention): x ./uidinfo_up1.txt + ./uidinfo_up2.txt +--------------------------------------------------------------------------= ----+ |x x x x x + + + + = +| | |________A_M_______| |___________A_M__________= | | +--------------------------------------------------------------------------= ----+ N Min Max Median Avg Stddev x 5 415023 420059 418405 418016.6 1878.5011 + 5 423731 430984 428084 427735.8 2623.4883 Difference at 95.0% confidence 9719.2 +/- 3327.59 2.32508% +/- 0.796043% (Student's t, pooled s =3D 2281.61) We can see 2.3% of improvement most likely because of atomic operations reduction. --=20 Pawel Jakub Dawidek http://www.wheel.pl pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --U+BazGySraz5kW0T Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD) iD8DBQFGxumIForvXbEpPzQRAoRAAKDUmAIgqAc6kx4KvXs+snHuAiqWegCg7z6C VtWabfYPU5PAeGxjX4Tg8cA= =rtmv -----END PGP SIGNATURE----- --U+BazGySraz5kW0T--