From owner-freebsd-mips@FreeBSD.ORG Mon Nov 5 17:58:50 2012 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 02CF7131 for ; Mon, 5 Nov 2012 17:58:50 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9A1D68FC12 for ; Mon, 5 Nov 2012 17:58:49 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id k10so10977047iea.13 for ; Mon, 05 Nov 2012 09:58:49 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=XO6FaZW1cyVUMtFJPEZKh9ebmH2tKQk8ZI4MA7IlqUY=; b=eoJLCYhocKrqmcf/gRpLU5coOy9op6nZR+Mdf4DXuUAM+45r7t9n/VfRnh24v48Ndr lLT6Hofga4awyhTHSuqvSV7zTFvimoJwgXe/bbzDIR3lIsLVtIIh68IdLAoURSK6nMCP GYnAQUR9fZSqBpGjHRduU29YkDFvUmgZkyz9FPmz0tIZsDjiRmnCBVAGfqofKr+5IRfg BuCH7KdwjQmkMkbRtjzRUDQ2yxaoAxUoIS3ANVQ+FtvLw4D6Of0/TCSpxHokmUwDbxCK uF/djEMWsTjbgPY42xEAz40zG8ZbpaZ47NY73PPgHIQXEV+Bk5Dunduz4INvkbWXAkAe dZJg== Received: by 10.50.16.144 with SMTP id g16mr10237188igd.23.1352138328912; Mon, 05 Nov 2012 09:58:48 -0800 (PST) Received: from 53.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPS id ff4sm5711970igc.13.2012.11.05.09.58.45 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 05 Nov 2012 09:58:47 -0800 (PST) Sender: Warner Losh Subject: Re: CACHE_LINE_SIZE macro. Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <1352137087.1120.180.camel@revolution.hippie.lan> Date: Mon, 5 Nov 2012 10:58:44 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201211041828.qA4ISomC076058@pdx.rh.CN85.ChatUSA.com> <1352137087.1120.180.camel@revolution.hippie.lan> To: Ian Lepore X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQmgYmLjYOiqDo2HbWfKKL6zYG+gkeTX2leVjqS9+6D6RRK/wp5RJm/e7GkQJCi59omd/xgl Cc: "Rodney W. Grimes" , Juli Mallett , "freebsd-mips@FreeBSD.org" X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2012 17:58:50 -0000 On Nov 5, 2012, at 10:38 AM, Ian Lepore wrote: > On Mon, 2012-11-05 at 10:11 -0700, Warner Losh wrote: >> On Nov 5, 2012, at 10:01 AM, Eitan Adler wrote: >>=20 >>> On 5 November 2012 11:49, Warner Losh wrote: >>>>> There has been some discussion recently about padding lock mutexs = to >>>>> the cache line size in order to avoid false sharing of CPUs. Some = have >>>>> claimed to see significant performance increases as a result. >>>>=20 >>>> Is that an out-of-kernel interface? >>>>=20 >>>> If we did that, we'd have to make it run-time settable, because = there's no one right answer for arm and MIPS cpus: they are all = different. >>>=20 >>> The discussion ended up with using a special parameter >>> CACHE_LINE_SIZE_LOCKS which is different than CACHE_LINE_SIZE. This = is >>> necessary for other reasons as well (CACHE_LINE_SIZE_LOCKS may take >>> into account prefetching of cache lines, but CACHE_LINE_SIZE >>> wouldn't). >>>=20 >>> I think the "correct" thing to do here is choose a reasonable, but >>> not-always-correct CACHE_LINE_SIZE_LOCKS and make CACHE_LINE_SIZE a >>> per-board constant (or run time setting, or whatever works). You >>> can't make it run-time settable as the padding is part of the ABI: >>>=20 >>> For more details see >>> http://comments.gmane.org/gmane.os.freebsd.devel.cvs/483696 >>> which contains the original discussion. >>>=20 >>> Note - I was not involved. >>=20 >> this is a kernel-only interface, so compile time constants are fine = there. What user-land visible interfaces are affected by this setting? = The answer should be 'none' >>=20 >> Warner >=20 > When I commented on Attilio's recent checkins concerning padding of > locks to cache line size and the fact that the value changes per-cpu = and > we're not well-positioned to handle that right now, his main concern = was > modules matching the kernel. I had suggested making the padding > conditional on SMP (because apparently there's no benefit to the = padding > in a UP kernel), but then a module compiled for UP wouldn't work right > on an SMP kernel, and vice versa. I'm not sure why that's a problem, = my > solution to that would be "So then don't do that." Don't make these structures compile time aligned, but make them run-time = aligned is the only alternative. For arm this isn't currently a huge = issue: armv4/v5 can be set to 32. For armv6 it can be set to 64. Since = you usually don't mix kernel bits from each, that's fine. It is a = bigger deal on mips where the various 64-bit architectures have = different values. Since we compile mips and mips64 differently, the = embedded stuff that Adrian is worried about won't necessarily be = penalized. We can make these compile time. However, making it compile time makes it more optimal for some members = of the mips64 family and less efficient for others. I'd have to see = measurements to see how much. > What scares me the most is the mushy definition of what = CACHE_LINE_SIZE > really means. There's nothing about the name that says "This may not = be > the actual cache line size but it's probably close," but increasingly = I > see people talking about it as if it had such a malleable meaning. Is > that consistant with the existing uses in the code? Is it a good = idea? There's a number of places where CAHCE_LINE_SIZE are used, but mostly to = compile-time align structures. Since these tend to be 64 vs 128 vs 256 = typically, the effect is only a little increase in memory use. We don't = flush data in this sized chunk, so we aren't polluting cache more by = making this number bigger: we're just have extra padding. Most of the = interfaces that I saw aren't exposed aren't KABI things. Warner=