From owner-freebsd-hackers@FreeBSD.ORG Fri Jun 28 06:57:37 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8A0FC5B8; Fri, 28 Jun 2013 06:57:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 2C91B188C; Fri, 28 Jun 2013 06:57:36 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r5S6vWZl027592; Fri, 28 Jun 2013 09:57:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r5S6vWZl027592 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r5S6vWrE027591; Fri, 28 Jun 2013 09:57:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 28 Jun 2013 09:57:32 +0300 From: Konstantin Belousov To: Alexander Motin Subject: Re: b_freelist TAILQ/SLIST Message-ID: <20130628065732.GL91021@kib.kiev.ua> References: <51CCAE14.6040504@FreeBSD.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="/UjIn1MmbqnRGnkw" Content-Disposition: inline In-Reply-To: <51CCAE14.6040504@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Jun 2013 06:57:37 -0000 --/UjIn1MmbqnRGnkw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: > Hi. >=20 > While doing some profiles of GEOM/CAM IOPS scalability, on some test=20 > patterns I've noticed serious congestion with spinning on global=20 > pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is=20 > already very simple, I've tried to optimize probably the only thing=20 > possible there: switch bswlist from TAILQ to SLIST. As I can see,=20 > b_freelist field of struct buf is really used as TAILQ in some other=20 > places, so I've just added another SLIST_ENTRY field. And result=20 > appeared to be surprising -- I can no longer reproduce the issue at all.= =20 > May be it was just unlucky synchronization of specific test, but I've=20 > seen in on two different systems and rechecked results with/without=20 > patch three times. This is too unbelievable. Could it be, e.g. some cache line conflicts which cause the trashing, in fact ? Does it help if you add void *b_pad before b_freelist instead of adding b_freeslist ? >=20 > The present patch is here: > http://people.freebsd.org/~mav/buf_slist.patch >=20 > The question is how to do it better? What is the KPI/KBI policy for=20 > struct buf? I could replace b_freelist by a union and keep KBI, but=20 > partially break KPI. Or I could add another field, probably breaking=20 > KBI, but keeping KPI. Or I could do something handmade with no breakage.= =20 > Or this change is just a bad idea? The same question about using union for b_freelist/b_freeslist, does the effect of magically fixing the contention still there if b_freeslist is on the same offset as the b_freelist ? There are no K{B,P}I policy for struct buf in HEAD, just change it as it fits. --/UjIn1MmbqnRGnkw Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzTPcAAoJEJDCuSvBvK1BG4cP/3xuV0DRg/9kEyhaSXC1LNbf iSXvD/sWYTA/ttKSht5iBQaKDUR0zOFY/2Ik3NKdFMhFTGh/Wsp7+pODB9ZNd3Oe 9SRjAkzImOkwD+vJQBGsQ+Oll4Nt607gnFbQptkGZdEmArl3uqqpAglbEF8fgSws ew2yt6xLfjPv3OpPjI4YuWH40+VxJH1rEH+h2ne0Fm63EJgV+IOiNSB8JM1LWgCJ jT5TM72n5iAZPM3FlVr/oSnKjJVa4UpIfQtU9AuqDhsnGZl1ZXdgOw+eU+Er3cTq GdpzVIY/dg6qMIr20Y+BnLUKt136YNEZHXXbj1Qt32ndikNAo5VGXuCR2WeIS/Qt PdL7YfQcp3zCX/wxtkM1BFHTU2zZwAwMgL9OZkLsWLtV4H6H0/lpiwMihO3g5KNG gqxBpvuGdx1hkz3jqvHpQRMisv/HhdxokOqiZUOLoyNyOm8KMCMXIGyzbtHVIk2E P/4Ls1DhP95DQuG6Ejbj/K39owqqNt7OTcpkiwBwqVbIV0yfgVb/tQRnoO1meLJr +WU5TXRjA8WZsaarhVVPRk0oB5vBITTQott7+s6amC/JLJ6Yk6ec0kdP5iMuoSXM s6/kSrPkUvriNJ8obZxtgR+wx30oy8fWXuigvbGSknG+PICV4oreTNMa5Om2666T 8Y3Q93ljI6fDWsJqoW/1 =B8dd -----END PGP SIGNATURE----- --/UjIn1MmbqnRGnkw--