From owner-svn-src-all@FreeBSD.ORG Wed Oct 24 20:09:49 2012 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4A6C95F6 for ; Wed, 24 Oct 2012 20:09:49 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 9551C8FC0A for ; Wed, 24 Oct 2012 20:09:48 +0000 (UTC) Received: (qmail 36240 invoked from network); 24 Oct 2012 21:47:34 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 24 Oct 2012 21:47:34 -0000 Message-ID: <50884AFF.5080203@freebsd.org> Date: Wed, 24 Oct 2012 22:09:35 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: Jim Harris Subject: Re: svn commit: r242014 - head/sys/kern References: <201210241836.q9OIafqo073002@svn.freebsd.org> <50883EA8.1010308@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 20:09:49 -0000 On 24.10.2012 21:49, Jim Harris wrote: > On Wed, Oct 24, 2012 at 12:16 PM, Andre Oppermann wrote: > > >> >> >> See also the discussion on svn-src-all regarding global struct mtx >> alignment. >> >> Thank you for proving my point. ;) >> >> Let's go back and see how we can do this the sanest way. These are >> the options I see at the moment: >> >> 1. sprinkle __aligned(CACHE_LINE_SIZE) all over the place >> 2. use a macro like MTX_ALIGN that can be SMP/UP aware and in >> the future possibly change to a different compiler dependent >> align attribute >> 3. embed __aligned(CACHE_LINE_SIZE) into struct mtx itself so it >> automatically gets aligned in all cases, even when dynamically >> allocated. >> >> Personally I'm undecided between #2 and #3. #1 is ugly. In favor >> of #3 is that there possibly isn't any case where you'd actually >> want the mutex to share a cache line with anything else, even a data >> structure. > > I've run my same tests with #3 as you describe, and I did see further > noticeable improvement. I had a difficult time though quantifying the > effect it would have on all of the different architectures. Putting > it in ULE's tdq gained 60-70% of the overall benefit, and was well > contained. I just experimented with different specifications of alignment and couldn't get the globals aligned at all. This seems to be because of the linker not understanding or not getting passed the alignment information when linking the kernel. > I agree that sprinkling all over the place isn't pretty. But focused > investigations into specific locks (spin mutexes, default mutexes, > whatever) may find a few key additional ones that would benefit. I > started down this path with the sleepq and turnstile locks, but none > of those specifically showed noticeable improvement (at least in the > tests I was running). There's still some additional ones I want to > look at, but haven't had the time yet. This runs the very great risk of optimizing for today's available architectures and then needs rejiggling every five years. Just as you've noticed the issue with 128B alignment from the Netburst days. We never know how the next micro-architecture will behave. Micro optimizing each individual invocation of common building blocks is the wrong path to go. I'd very much prefer the alignment *and* padding control to be done in one place for all of them, either through a magic macro or compiler __attribute__(whatever). -- Andre