Date: Fri, 28 Jun 2013 09:57:32 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Alexander Motin <mav@FreeBSD.org> Cc: hackers@freebsd.org Subject: Re: b_freelist TAILQ/SLIST Message-ID: <20130628065732.GL91021@kib.kiev.ua> In-Reply-To: <51CCAE14.6040504@FreeBSD.org> References: <51CCAE14.6040504@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--/UjIn1MmbqnRGnkw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: > Hi. >=20 > While doing some profiles of GEOM/CAM IOPS scalability, on some test=20 > patterns I've noticed serious congestion with spinning on global=20 > pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is=20 > already very simple, I've tried to optimize probably the only thing=20 > possible there: switch bswlist from TAILQ to SLIST. As I can see,=20 > b_freelist field of struct buf is really used as TAILQ in some other=20 > places, so I've just added another SLIST_ENTRY field. And result=20 > appeared to be surprising -- I can no longer reproduce the issue at all.= =20 > May be it was just unlucky synchronization of specific test, but I've=20 > seen in on two different systems and rechecked results with/without=20 > patch three times. This is too unbelievable. Could it be, e.g. some cache line conflicts which cause the trashing, in fact ? Does it help if you add void *b_pad before b_freelist instead of adding b_freeslist ? >=20 > The present patch is here: > http://people.freebsd.org/~mav/buf_slist.patch >=20 > The question is how to do it better? What is the KPI/KBI policy for=20 > struct buf? I could replace b_freelist by a union and keep KBI, but=20 > partially break KPI. Or I could add another field, probably breaking=20 > KBI, but keeping KPI. Or I could do something handmade with no breakage.= =20 > Or this change is just a bad idea? The same question about using union for b_freelist/b_freeslist, does the effect of magically fixing the contention still there if b_freeslist is on the same offset as the b_freelist ? There are no K{B,P}I policy for struct buf in HEAD, just change it as it fits. --/UjIn1MmbqnRGnkw Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzTPcAAoJEJDCuSvBvK1BG4cP/3xuV0DRg/9kEyhaSXC1LNbf iSXvD/sWYTA/ttKSht5iBQaKDUR0zOFY/2Ik3NKdFMhFTGh/Wsp7+pODB9ZNd3Oe 9SRjAkzImOkwD+vJQBGsQ+Oll4Nt607gnFbQptkGZdEmArl3uqqpAglbEF8fgSws ew2yt6xLfjPv3OpPjI4YuWH40+VxJH1rEH+h2ne0Fm63EJgV+IOiNSB8JM1LWgCJ jT5TM72n5iAZPM3FlVr/oSnKjJVa4UpIfQtU9AuqDhsnGZl1ZXdgOw+eU+Er3cTq GdpzVIY/dg6qMIr20Y+BnLUKt136YNEZHXXbj1Qt32ndikNAo5VGXuCR2WeIS/Qt PdL7YfQcp3zCX/wxtkM1BFHTU2zZwAwMgL9OZkLsWLtV4H6H0/lpiwMihO3g5KNG gqxBpvuGdx1hkz3jqvHpQRMisv/HhdxokOqiZUOLoyNyOm8KMCMXIGyzbtHVIk2E P/4Ls1DhP95DQuG6Ejbj/K39owqqNt7OTcpkiwBwqVbIV0yfgVb/tQRnoO1meLJr +WU5TXRjA8WZsaarhVVPRk0oB5vBITTQott7+s6amC/JLJ6Yk6ec0kdP5iMuoSXM s6/kSrPkUvriNJ8obZxtgR+wx30oy8fWXuigvbGSknG+PICV4oreTNMa5Om2666T 8Y3Q93ljI6fDWsJqoW/1 =B8dd -----END PGP SIGNATURE----- --/UjIn1MmbqnRGnkw--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130628065732.GL91021>