Date: Fri, 28 Jun 2013 09:57:32 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Alexander Motin <mav@FreeBSD.org> Cc: hackers@freebsd.org Subject: Re: b_freelist TAILQ/SLIST Message-ID: <20130628065732.GL91021@kib.kiev.ua> In-Reply-To: <51CCAE14.6040504@FreeBSD.org> References: <51CCAE14.6040504@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Fri, Jun 28, 2013 at 12:26:44AM +0300, Alexander Motin wrote: > Hi. > > While doing some profiles of GEOM/CAM IOPS scalability, on some test > patterns I've noticed serious congestion with spinning on global > pbuf_mtx mutex inside getpbuf() and relpbuf(). Since that code is > already very simple, I've tried to optimize probably the only thing > possible there: switch bswlist from TAILQ to SLIST. As I can see, > b_freelist field of struct buf is really used as TAILQ in some other > places, so I've just added another SLIST_ENTRY field. And result > appeared to be surprising -- I can no longer reproduce the issue at all. > May be it was just unlucky synchronization of specific test, but I've > seen in on two different systems and rechecked results with/without > patch three times. This is too unbelievable. Could it be, e.g. some cache line conflicts which cause the trashing, in fact ? Does it help if you add void *b_pad before b_freelist instead of adding b_freeslist ? > > The present patch is here: > http://people.freebsd.org/~mav/buf_slist.patch > > The question is how to do it better? What is the KPI/KBI policy for > struct buf? I could replace b_freelist by a union and keep KBI, but > partially break KPI. Or I could add another field, probably breaking > KBI, but keeping KPI. Or I could do something handmade with no breakage. > Or this change is just a bad idea? The same question about using union for b_freelist/b_freeslist, does the effect of magically fixing the contention still there if b_freeslist is on the same offset as the b_freelist ? There are no K{B,P}I policy for struct buf in HEAD, just change it as it fits. [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRzTPcAAoJEJDCuSvBvK1BG4cP/3xuV0DRg/9kEyhaSXC1LNbf iSXvD/sWYTA/ttKSht5iBQaKDUR0zOFY/2Ik3NKdFMhFTGh/Wsp7+pODB9ZNd3Oe 9SRjAkzImOkwD+vJQBGsQ+Oll4Nt607gnFbQptkGZdEmArl3uqqpAglbEF8fgSws ew2yt6xLfjPv3OpPjI4YuWH40+VxJH1rEH+h2ne0Fm63EJgV+IOiNSB8JM1LWgCJ jT5TM72n5iAZPM3FlVr/oSnKjJVa4UpIfQtU9AuqDhsnGZl1ZXdgOw+eU+Er3cTq GdpzVIY/dg6qMIr20Y+BnLUKt136YNEZHXXbj1Qt32ndikNAo5VGXuCR2WeIS/Qt PdL7YfQcp3zCX/wxtkM1BFHTU2zZwAwMgL9OZkLsWLtV4H6H0/lpiwMihO3g5KNG gqxBpvuGdx1hkz3jqvHpQRMisv/HhdxokOqiZUOLoyNyOm8KMCMXIGyzbtHVIk2E P/4Ls1DhP95DQuG6Ejbj/K39owqqNt7OTcpkiwBwqVbIV0yfgVb/tQRnoO1meLJr +WU5TXRjA8WZsaarhVVPRk0oB5vBITTQott7+s6amC/JLJ6Yk6ec0kdP5iMuoSXM s6/kSrPkUvriNJ8obZxtgR+wx30oy8fWXuigvbGSknG+PICV4oreTNMa5Om2666T 8Y3Q93ljI6fDWsJqoW/1 =B8dd -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130628065732.GL91021>