From owner-svn-src-user@FreeBSD.ORG Wed Oct 24 15:49:23 2012 Return-Path: Delivered-To: svn-src-user@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 9CF136A7; Wed, 24 Oct 2012 15:49:23 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-vb0-f54.google.com (mail-vb0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id BDA698FC0A; Wed, 24 Oct 2012 15:49:22 +0000 (UTC) Received: by mail-vb0-f54.google.com with SMTP id v11so872747vbm.13 for ; Wed, 24 Oct 2012 08:49:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=GsKJGvB/j6FtW5MYuo0zwrsIlatmiAHFlDXvtWFq3Po=; b=MYdaSiDbtkr0Fdck5gPPNg16JRIzx4APAQ5xNm3QnUhnac1kZCKNC+v04wd0OR+H7s N109opkE8UJYNzTzzJwWj8EcVZ0AgFAHJKaPvgqJxGDPAA+Vv4FNGTS5N8THxBYDN2sx jtozQWGP9BhKcUkaIpKgISh4Itjk8OSBsf83eivRlEuG3ILhmcmp6GHdVQHWzzypE2Ic yEXY43ZomdRDKTCh+ASxOXRIHQcXfOAPiC2iaHoyMZdl0rR3MMIIzqxTGYwp5JL9TxjW d1KhdYbO5KZwUKrwg7MwFNiSbXb7h0csdVX8VAIGyDa4GzgmiLQqWLcIdTc8C8kTbHZl 2C2g== MIME-Version: 1.0 Received: by 10.58.201.73 with SMTP id jy9mr29879900vec.29.1351093761964; Wed, 24 Oct 2012 08:49:21 -0700 (PDT) Sender: asmrookie@gmail.com Received: by 10.220.150.197 with HTTP; Wed, 24 Oct 2012 08:49:21 -0700 (PDT) In-Reply-To: <20121024154700.GI70741@FreeBSD.org> References: <201210221418.q9MEINkr026751@svn.freebsd.org> <201210241136.06154.jhb@freebsd.org> <20121024154302.GH70741@FreeBSD.org> <20121024154700.GI70741@FreeBSD.org> Date: Wed, 24 Oct 2012 16:49:21 +0100 X-Google-Sender-Auth: t5PAC9TGOc8H_l0UU8uscHwaVz8 Message-ID: Subject: Re: svn commit: r241889 - in user/andre/tcp_workqueue/sys: arm/arm cddl/compat/opensolaris/kern cddl/contrib/opensolaris/uts/common/dtrace cddl/contrib/opensolaris/uts/common/fs/zfs ddb dev/acpica dev/... From: Attilio Rao To: Gleb Smirnoff Content-Type: text/plain; charset=UTF-8 Cc: mdf@freebsd.org, src-committers@freebsd.org, Andre Oppermann , John Baldwin , svn-src-user@freebsd.org, Bruce Evans X-BeenThere: svn-src-user@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: attilio@FreeBSD.org List-Id: "SVN commit messages for the experimental " user" src tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Oct 2012 15:49:23 -0000 On Wed, Oct 24, 2012 at 4:47 PM, Gleb Smirnoff wrote: > On Wed, Oct 24, 2012 at 04:45:07PM +0100, Attilio Rao wrote: > A> On Wed, Oct 24, 2012 at 4:43 PM, Gleb Smirnoff wrote: > A> > On Wed, Oct 24, 2012 at 11:36:06AM -0400, John Baldwin wrote: > A> > J> On Wednesday, October 24, 2012 11:24:22 am Attilio Rao wrote: > A> > J> > On Wed, Oct 24, 2012 at 4:09 PM, Attilio Rao wrote: > A> > J> > > On Wed, Oct 24, 2012 at 3:45 PM, John Baldwin wrote: > A> > J> > >> On Wednesday, October 24, 2012 10:34:34 am Attilio Rao wrote: > A> > J> > >>> On Wed, Oct 24, 2012 at 3:05 PM, John Baldwin wrote: > A> > J> > >>> > On Tuesday, October 23, 2012 7:20:04 pm Andre Oppermann wrote: > A> > J> > >>> >> On 24.10.2012 00:15, mdf@FreeBSD.org wrote: > A> > J> > >>> >> > On Tue, Oct 23, 2012 at 7:41 AM, Andre Oppermann > A> > J> > >> wrote: > A> > J> > >>> >> >> Struct mtx and MTX_SYSINIT always occur as pair next to each other. > A> > J> > >>> >> > > A> > J> > >>> >> > That doesn't matter. Language basics like variable definitions should > A> > J> > >>> >> > not be obscured by macros. It either takes longer to figure out what > A> > J> > >>> >> > a variable is (because one needs to look up the definition of the > A> > J> > >>> >> > macro) or makes it almost impossible (because now e.g. cscope doesn't > A> > J> > >>> >> > know this is a variable definition. > A> > J> > >>> >> > A> > J> > >>> >> Sigh, cscope doesn't expand macros? > A> > J> > >>> >> > A> > J> > >>> >> Is there a way to do the cache line alignment in a sane way without > A> > J> > >>> >> littering __aligned(CACHE_LINE_SIZE) all over the place? > A> > J> > >>> > > A> > J> > >>> > I was hoping to do something with an anonymous union or some such like: > A> > J> > >>> > > A> > J> > >>> > union mtx_aligned { > A> > J> > >>> > struct mtx; > A> > J> > >>> > char[roundup2(sizeof(struct mtx), CACHE_LINE_SIZE)]; > A> > J> > >>> > } > A> > J> > >>> > > A> > J> > >>> > I don't know if there is a useful way to define an 'aligned mutex' type > A> > J> > >>> > that will transparently map to a 'struct mtx', e.g.: > A> > J> > >>> > > A> > J> > >>> > typedef struct mtx __aligned(CACHE_LINE_SIZE) aligned_mtx_t; > A> > J> > >>> > A> > J> > >>> Unfortunately that doesn't work as I've verified with alc@ few months ago. > A> > J> > >>> The __aligned() attribute only works with structures definition, not > A> > J> > >>> objects declaration. > A> > J> > >> > A> > J> > >> Are you saying that the typedef doesn't (I expect it doesn't), or that this > A> > J> > >> doesn't: > A> > J> > >> > A> > J> > >> struct mtx foo __aligned(CACHE_LINE_SIZE); > A> > J> > > > A> > J> > > I meant to say that such notation won't address the padding issue > A> > J> > > which is as import as the alignment. Infact, for sensitive locks, > A> > J> > > having just an aligned object is not really useful if the cacheline > A> > J> > > gets shared. > A> > J> > > In the end you will need to use explicit padding or use __aligned in > A> > J> > > the struct definition, which cannot be used as a general pattern. > A> > J> > > A> > J> > The quickest way I see this can be made general is to have a specific > A> > J> > struct defined in sys/_mutex.h like that > A> > J> > > A> > J> > struct mtx_unshare { > A> > J> > struct mtx lock; > A> > J> > char _pad[CACHE_LINE_SIZE - sizeof(struct mtx)]; > A> > J> > } __aligned(CACHE_LINE_SIZE); > A> > J> > A> > J> I think instead you want my union above that uses roundup2 in case a lock > A> > J> eats up multiple cache lines: > A> > J> > A> > J> union mtx_foo { > A> > J> struct mtx lock; > A> > J> char junk[roundup2(sizeof(struct mtx), CACHE_LINE_SIZE)]; > A> > J> } __aligned_CACHE_LINE_SIZE; > A> > J> > A> > J> > then let mtx_* functions to accept void ptrs and cast them to struct > A> > J> > mtx as long as the functions enter. > A> > J> > A> > J> Eh, that removes all compile time type checks. That seems very dubious to me. > A> > > A> > I think that we should first get benchmarking results, and only then try > A> > to evolve an API for cache aligned mutexes. > A> > > A> > As an option we can allocate mutexes from cache aligned uma zone dynamically, > A> > to avoid all these syntax acrobatics. > A> > A> There are several objections to this. Quicker that came to my mind: > A> - Some locks needs to be ready before UMA subsystem is setup > > I suppose there are not a lot of them, and those can be aligned by hand. Actually I think they are quite a few. And however the whole point of this is to avoid manual frobbing. > A> - On arches where the KVA is already scarce (i386, powerpc, etc.) this > A> is going to be completely overkill > > Do we really have that much mutexes on bss, that moving them to dynamic > allocator would affect kva? In my mind every lock that doesn't belong to a struct should be moved to be mtx_unshared, really. Attilio -- Peace can only be achieved by understanding - A. Einstein