From owner-freebsd-mips@FreeBSD.ORG Tue Nov 6 21:27:01 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 928549F for ; Tue, 6 Nov 2012 21:27:01 +0000 (UTC) (envelope-from freebsd@pdx.rh.CN85.ChatUSA.com) Received: from pdx.rh.CN85.ChatUSA.com (pdx.rh.CN85.ip6.chatusa.com [IPv6:2607:fa80:104::6]) by mx1.freebsd.org (Postfix) with ESMTP id 3FDE58FC12 for ; Tue, 6 Nov 2012 21:27:00 +0000 (UTC) Received: from pdx.rh.CN85.ChatUSA.com (localhost [127.0.0.1]) by pdx.rh.CN85.ChatUSA.com (8.13.3/8.13.3) with ESMTP id qA5C1sDX094613; Mon, 5 Nov 2012 04:01:54 -0800 (PST) (envelope-from freebsd@pdx.rh.CN85.ChatUSA.com) Received: (from freebsd@localhost) by pdx.rh.CN85.ChatUSA.com (8.13.3/8.13.3/Submit) id qA5C1rIo094612; Mon, 5 Nov 2012 04:01:53 -0800 (PST) (envelope-from freebsd) From: "Rodney W. Grimes" Message-Id: <201211051201.qA5C1rIo094612@pdx.rh.CN85.ChatUSA.com> Subject: Re: CACHE_LINE_SIZE macro. In-Reply-To: To: Warner Losh Date: Mon, 5 Nov 2012 04:01:53 -0800 (PST) X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII Cc: Juli Mallett , "freebsd-mips@FreeBSD.org" X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2012 21:27:01 -0000 > > On Nov 5, 2012, at 10:01 AM, Eitan Adler wrote: > > > On 5 November 2012 11:49, Warner Losh wrote: > >>> There has been some discussion recently about padding lock mutexs to > >>> the cache line size in order to avoid false sharing of CPUs. Some have > >>> claimed to see significant performance increases as a result. > >> > >> Is that an out-of-kernel interface? > >> > >> If we did that, we'd have to make it run-time settable, because there's no one right answer for arm and MIPS cpus: they are all different. > > > > The discussion ended up with using a special parameter > > CACHE_LINE_SIZE_LOCKS which is different than CACHE_LINE_SIZE. This is > > necessary for other reasons as well (CACHE_LINE_SIZE_LOCKS may take > > into account prefetching of cache lines, but CACHE_LINE_SIZE > > wouldn't). > > > > I think the "correct" thing to do here is choose a reasonable, but > > not-always-correct CACHE_LINE_SIZE_LOCKS and make CACHE_LINE_SIZE a > > per-board constant (or run time setting, or whatever works). You > > can't make it run-time settable as the padding is part of the ABI: > > > > For more details see > > http://comments.gmane.org/gmane.os.freebsd.devel.cvs/483696 > > which contains the original discussion. > > > > Note - I was not involved. > > this is a kernel-only interface, so compile time constants are fine there. What user-land visible interfaces are affected by this setting? The answer should be 'none' > Even in a kernel-only interface the answer should be ``none''. First my gut tells me that this is a synthetic bench mark anomoly and that the data is hiding the real cause and effects of what is going on. A better fix, if the data can be shown to be real, and meaningful, is to make the mutex type opaque (void *), and pass pointers around to it, and let the run time code decide how to proper cache align allocate the mutex (Probably fit 2 mutexes in 1 8 byte line and I bet it runs faster than there current ``pad it to a cache line size'' hack. I got a feeling your gona cache thrash on SMP no mater what you do.... I did not see -any- data presented that showed how this was proven to be of benifit. Why not just go out and cache align every data structure in the kernel.... :-) A benchmark well show an improvement I am sure of that. This may actually be begging for the old technique of carefull handcrafted structs that are such that things LIKE mutexes naturally end up on a proper boundary. I bet you the boundary is actually 4 byte, independent of cache line size. -- Rod Grimes freebsd@freebsd.org