From owner-freebsd-current@FreeBSD.ORG Thu Mar 12 17:36:42 2015 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 10616C72 for ; Thu, 12 Mar 2015 17:36:42 +0000 (UTC) Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 92D6AD8F for ; Thu, 12 Mar 2015 17:36:41 +0000 (UTC) Received: by wggy19 with SMTP id y19so17975413wgg.9 for ; Thu, 12 Mar 2015 10:36:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mail-followup-to:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=chffk/YaEq+ReThENLoqVT3vP+Y4BYC0b5nP9N8Ij5I=; b=twrO/0EzXAN6KOL0lHQaU0wlpoXPUQVYf74iFj01qjbK0aMiXVyKdnc4CORfR3L0ar iCEKCO4Foam8baGHQ0Ix39BKdCGk9FHRk1sY6gBtzlRh0croTRlFVLjgBotEXVaV1grv mq7mnb1PZbJYkVl9EtDzQDm4vOahH0M8t3dBUdHC59yoepkXzQm26a9Pwmv9WixxgIUS OajLBUuTz7XngwuQvlIdh8u2dE2boOQgXrIWd0lb1jlY80i/bsDN41VRS8hpffsRZJ/A 8OudiY7F+QaywCJ2X9TbngRi5Bd3eIsaE2RqXQy5EZO8fU9UOCaKr2ca2XFa/GJmpi3p 28yA== X-Received: by 10.180.97.226 with SMTP id ed2mr130621960wib.4.1426181800011; Thu, 12 Mar 2015 10:36:40 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id nb18sm11707297wic.3.2015.03.12.10.36.37 (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Thu, 12 Mar 2015 10:36:38 -0700 (PDT) Date: Thu, 12 Mar 2015 18:36:35 +0100 From: Mateusz Guzik To: Ryan Stone Subject: Re: [PATCH] Convert the VFS cache lock to an rmlock Message-ID: <20150312173635.GB9153@dft-labs.eu> Mail-Followup-To: Mateusz Guzik , Ryan Stone , FreeBSD Current References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: FreeBSD Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Mar 2015 17:36:42 -0000 On Thu, Mar 12, 2015 at 11:14:42AM -0400, Ryan Stone wrote: > I've just submitted a patch to Differential[1] for review that converts the > VFS cache to use an rmlock in place of the current rwlock. My main > motivation for the change is to fix a priority inversion problem that I saw > recently. A real-time priority thread attempted to acquire a write lock on > the VFS cache lock, but there was already a reader holding it. The reader > was preempted by a normal priority thread, and my real-time thread was > starved. > > [1] https://reviews.freebsd.org/D2051 > > > I was worried about the performance implications of the change, as I wasn't > sure how common write operations on the VFS cache would be. I did a -j12 > buildworld/buildkernel test on a 12-core Haswell Xeon system, as I figured > that would be a reasonable stress test that simultaneously creates lots of > small files and reads a lot of files as well. This actually wound up being > about a 10% performance *increase* (the units below are seconds of elapsed > time as measured by /usr/bin/time, so smaller is better): > > $ ministat -C 1 orig.log rmlock.log > x orig.log > + rmlock.log > +------------------------------------------------------------------------------+ > | + x > | > |++++ x x xxx > | > | |A| > |_________A___M____|| > +------------------------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 6 2710.31 2821.35 2816.75 2798.0617 43.324817 > + 5 2488.25 2500.25 2498.04 2495.756 5.0494782 > Difference at 95.0% confidence > -302.306 +/- 44.4709 > -10.8041% +/- 1.58935% > (Student's t, pooled s = 32.4674) > > The one outlier in the rwlock case does confuse me a bit. What I did was > booted a freshly-built image with the rmlock lock applied, did a git > checkout of head, and then did 5 builds in a row. The git checkout should > have had the effect of priming the disk cache with the source files. Then > I installed the stock head kernel, rebooted, and ran 5 more builds (and > then 1 more when I noticed the outlier). The fast outlier was the *first* > run, which should have been running with a cold disk cache, so I really > don't know why it would be 90 seconds faster. I do see that this run also > had about 500-600 fewer seconds spent in system time: > > x orig.log > +------------------------------------------------------------------------------+ > | > x | > |x x x > xx | > | > |_________________________A__________M_____________|| > +------------------------------------------------------------------------------+ > N Min Max Median Avg Stddev > x 6 3515.23 4121.84 4105.57 4001.71 239.61362 > > I'm not sure how much that I care, given that the rmlock is universally > faster (but maybe I should try the "cold boot" case anyway). > > If anybody had any comments or further testing that they would like to see, > please let me know. Workloads like buildworld and the like (i.e. a lot of forks + execs) run into very severe contention in vm, which is orders of magnitude bigger than anything else. As such your result seems quite suspicious. Can you describe in more detail how were you testing? Did you have a separate fs for obj tree which was mounted+unmounted before each run? I suggest you grab a machine from zoo[1] and run some tests on "bigger" hardware. A perf improvement, even slight, is definitely welcome. [1] https://wiki.freebsd.org/TestClusterOneReservations -- Mateusz Guzik