From owner-freebsd-current@FreeBSD.ORG Tue Mar 17 20:39:00 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03CAAB56 for ; Tue, 17 Mar 2015 20:39:00 +0000 (UTC) Received: from mail-wg0-x231.google.com (mail-wg0-x231.google.com [IPv6:2a00:1450:400c:c00::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84759877 for ; Tue, 17 Mar 2015 20:38:59 +0000 (UTC) Received: by wggv3 with SMTP id v3so18300404wgg.1 for ; Tue, 17 Mar 2015 13:38:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=2FkVxEncXjhnQ+9s35oO+gc+1hFXT+65MptKZJIqZrU=; b=MRMAfOE3KaYIxnDshS419gAmknAnw1I/nTUiDoZYbiGnOuABbMS1pgD0MGgcHRcvuP f2aYcHkQzSwD0JEUnG5jXBBL5+/w+9WOeznkM7n8TSVpGvPYfvQGbYL0e87Q2jsQ6hgw bSatU6RCVT0S0Y6qUmkLuxtuG1cuD8HzVbH/mfrej71Q0IsxMcNzfbZoce9DU4/TD3Ty urusC3b/93YYUaKdmI8F6/ek+sA6ze8hYKTmn6GddJm7zrIGg/kV7umc10zp58Z0evFQ tqkUlMLT2wywwGVzM5vvLQGgKLRELJvBG99yEefM8OhfFQ01wM0KqUO37WHB3LFHWqa9 Y5cA== X-Received: by 10.180.81.7 with SMTP id v7mr903190wix.27.1426624737997; Tue, 17 Mar 2015 13:38:57 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id ga8sm158124wib.11.2015.03.17.13.38.54 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Tue, 17 Mar 2015 13:38:55 -0700 (PDT) Date: Tue, 17 Mar 2015 21:38:52 +0100 From: Mateusz Guzik To: Ryan Stone Subject: Re: [PATCH] Convert the VFS cache lock to an rmlock Message-ID: <20150317203852.GA30391@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Ryan Stone , FreeBSD Current References: <20150312173635.GB9153@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Mar 2015 20:39:00 -0000 On Fri, Mar 13, 2015 at 11:23:06AM -0400, Ryan Stone wrote: > On Thu, Mar 12, 2015 at 1:36 PM, Mateusz Guzik wrote: > > > Workloads like buildworld and the like (i.e. a lot of forks + execs) run > > into very severe contention in vm, which is orders of magnitude bigger > > than anything else. > > > > As such your result seems quite suspicious. > > > > You're right, I did mess up the testing somewhere (I have no idea how). As > you suggested, I switched to using a separate partition for the objdir, and > ran each build with a freshly newfsed filesystem. I scripted it to be sure > that I was following the same procedure with each run: > [..] > The difference is due to a significant increase in system time. Write > locks on an rmlock are extremely expensive (they involve an > smp_rendezvous), and the cost likely scales with the number of cores: > > x 32core/orig.log > + 32core/rmlock.log > +--------------------------------------------------------------------------+ > |xxx x + +++ +| > ||_MA__| |____MA______| | > +--------------------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 5 5616.63 5715.7 5641.5 5661.72 48.511545 > + 5 6502.51 6781.84 6596.5 6612.39 103.06568 > Difference at 95.0% confidence > 950.67 +/- 117.474 > 16.7912% +/- 2.07489% > (Student's t, pooled s = 80.5478) > > > At this point I'm pretty much at an impasse. The real-time behaviour is > critical to me, but a 5% performance degradation isn't likely to be > acceptable to many people. I'll see what I can do with this. Well, you can see that the entire cache is protected by one lock. It also has other shortcomings which basically ask for a complete rewrite, so I would say spending significant amount of time on it is not worth discouraged. Brief look suggests that while just protecting each hash bucket by a separate lock may require a lot of work, it should be quite doable to do it with an additional lock which makes deletions exclusive, while insertions and lookups take the same lock shared and then lock specific buckets. Also cache_purge could be micro-optimized to not take the lock if we did not have any entries for given vnode (and we don't in a lot of cases). With this in place I would not be surprised if the thing was faster even with rmlock, which would justify it until the cache is rewritten. -- Mateusz Guzik