From owner-svn-src-all@FreeBSD.ORG Wed Oct 24 19:49:41 2012 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EAF4BBEE; Wed, 24 Oct 2012 19:49:41 +0000 (UTC) (envelope-from jim.harris@gmail.com) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 576788FC0C; Wed, 24 Oct 2012 19:49:41 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id p27so1591476qat.13 for ; Wed, 24 Oct 2012 12:49:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P72cXlNUn+fL3jWFAQnllEHKQhqnqbpWl2sMvxH6Q5c=; b=j8zEGpnOJIPje3EmWwL0ljH4D7OhzbGYJyiixCRVhcgwdq+rbcbYVj1GhM9N4slt7w b11BSExQ5gF9vW4EbhAVy613Tiy6pcFYewXwLEHzVAhYpNChDP840iXBre8AceoLHYC9 ScAdW4oJpuoGbwYlel6y41CifBJXaPQ/RnusBbBFu10LsekER6KhkTMI+fREnRTID1Ow D7l3MX2uJ7Jg0Y2zQn9HRGl6JlZPcN0RLJwHHnmP4QSRgKNv9gPH1QUTlybTMpJE/PSt TkTnicbJgjGdVHFnRCZsJcOJM9Y6FNjABMrubqA7os7BBkFXv1vfMyVdya/jvyiMOHOC we9w== MIME-Version: 1.0 Received: by 10.224.60.17 with SMTP id n17mr8012298qah.63.1351108180424; Wed, 24 Oct 2012 12:49:40 -0700 (PDT) Received: by 10.49.35.37 with HTTP; Wed, 24 Oct 2012 12:49:40 -0700 (PDT) In-Reply-To: <50883EA8.1010308@freebsd.org> References: <201210241836.q9OIafqo073002@svn.freebsd.org> <50883EA8.1010308@freebsd.org> Date: Wed, 24 Oct 2012 12:49:40 -0700 Message-ID: Subject: Re: svn commit: r242014 - head/sys/kern From: Jim Harris To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 19:49:42 -0000 On Wed, Oct 24, 2012 at 12:16 PM, Andre Oppermann wrote: > > > See also the discussion on svn-src-all regarding global struct mtx > alignment. > > Thank you for proving my point. ;) > > Let's go back and see how we can do this the sanest way. These are > the options I see at the moment: > > 1. sprinkle __aligned(CACHE_LINE_SIZE) all over the place > 2. use a macro like MTX_ALIGN that can be SMP/UP aware and in > the future possibly change to a different compiler dependent > align attribute > 3. embed __aligned(CACHE_LINE_SIZE) into struct mtx itself so it > automatically gets aligned in all cases, even when dynamically > allocated. > > Personally I'm undecided between #2 and #3. #1 is ugly. In favor > of #3 is that there possibly isn't any case where you'd actually > want the mutex to share a cache line with anything else, even a data > structure. I've run my same tests with #3 as you describe, and I did see further noticeable improvement. I had a difficult time though quantifying the effect it would have on all of the different architectures. Putting it in ULE's tdq gained 60-70% of the overall benefit, and was well contained. I agree that sprinkling all over the place isn't pretty. But focused investigations into specific locks (spin mutexes, default mutexes, whatever) may find a few key additional ones that would benefit. I started down this path with the sleepq and turnstile locks, but none of those specifically showed noticeable improvement (at least in the tests I was running). There's still some additional ones I want to look at, but haven't had the time yet. Thanks, -Jim