Date: Wed, 24 Oct 2012 22:09:35 +0200 From: Andre Oppermann <andre@freebsd.org> To: Jim Harris <jim.harris@gmail.com> Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r242014 - head/sys/kern Message-ID: <50884AFF.5080203@freebsd.org> In-Reply-To: <CAJP=Hc-8U6mcYu_u_johFNzzQ_f0rw5dthZA3rTKf6R5mSEC4w@mail.gmail.com> References: <201210241836.q9OIafqo073002@svn.freebsd.org> <CAJ-VmonpdJ445hXVaoHqFgS0v7QRwqHWodQrVHm2CN9T661www@mail.gmail.com> <CAJP=Hc9XmvfW3MrDjvK15OAx1fyfjPk%2BEhqHUOzoEpChu5imtg@mail.gmail.com> <50883EA8.1010308@freebsd.org> <CAJP=Hc-8U6mcYu_u_johFNzzQ_f0rw5dthZA3rTKf6R5mSEC4w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 24.10.2012 21:49, Jim Harris wrote: > On Wed, Oct 24, 2012 at 12:16 PM, Andre Oppermann <andre@freebsd.org> wrote: > > <snip> >> >> >> See also the discussion on svn-src-all regarding global struct mtx >> alignment. >> >> Thank you for proving my point. ;) >> >> Let's go back and see how we can do this the sanest way. These are >> the options I see at the moment: >> >> 1. sprinkle __aligned(CACHE_LINE_SIZE) all over the place >> 2. use a macro like MTX_ALIGN that can be SMP/UP aware and in >> the future possibly change to a different compiler dependent >> align attribute >> 3. embed __aligned(CACHE_LINE_SIZE) into struct mtx itself so it >> automatically gets aligned in all cases, even when dynamically >> allocated. >> >> Personally I'm undecided between #2 and #3. #1 is ugly. In favor >> of #3 is that there possibly isn't any case where you'd actually >> want the mutex to share a cache line with anything else, even a data >> structure. > > I've run my same tests with #3 as you describe, and I did see further > noticeable improvement. I had a difficult time though quantifying the > effect it would have on all of the different architectures. Putting > it in ULE's tdq gained 60-70% of the overall benefit, and was well > contained. I just experimented with different specifications of alignment and couldn't get the globals aligned at all. This seems to be because of the linker not understanding or not getting passed the alignment information when linking the kernel. > I agree that sprinkling all over the place isn't pretty. But focused > investigations into specific locks (spin mutexes, default mutexes, > whatever) may find a few key additional ones that would benefit. I > started down this path with the sleepq and turnstile locks, but none > of those specifically showed noticeable improvement (at least in the > tests I was running). There's still some additional ones I want to > look at, but haven't had the time yet. This runs the very great risk of optimizing for today's available architectures and then needs rejiggling every five years. Just as you've noticed the issue with 128B alignment from the Netburst days. We never know how the next micro-architecture will behave. Micro optimizing each individual invocation of common building blocks is the wrong path to go. I'd very much prefer the alignment *and* padding control to be done in one place for all of them, either through a magic macro or compiler __attribute__(whatever). -- Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50884AFF.5080203>