From owner-freebsd-arch@FreeBSD.ORG Wed Jun 18 20:33:21 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 075DEB70; Wed, 18 Jun 2014 20:33:21 +0000 (UTC) Received: from mail-we0-x22a.google.com (mail-we0-x22a.google.com [IPv6:2a00:1450:400c:c03::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C49E2DA4; Wed, 18 Jun 2014 20:33:19 +0000 (UTC) Received: by mail-we0-f170.google.com with SMTP id w61so1456774wes.1 for ; Wed, 18 Jun 2014 13:33:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=o/yO/cApgqWEeUutc1X/l5yHt6vHI4Dilcx2vTxtJUY=; b=f6PcJdiUtsYaiQIDgZ2V8iNipKOs7mQhnDbp6Exh8FT9YvxrJu22XakAb4dBiVBUR8 TmRrumg0/XEfcdyvfgJtbj478PBBFDm7eDTQ0dkv3LRS+VfkULzktPXhQVvzImMSkqPg cXrFB6JzrcTRxz1R5P/Wo1xalgxplKYw5iWC0JYl5VQNaiqaBMoHmZk9rKDdaNV1fDI7 TsTAknT4N2ceq2JWGgNVWkmYMvneLxFlK/9zSUdpGEL/3roic/0/kHbzKzhIhuDHEWfx 0dNLlOBxrJTli8kjrvUR6cUZNTvn+9MIL5r0UfqRaOpY6RkldLo/qDXvi7Yt2S3ZdToY XsBg== X-Received: by 10.180.189.44 with SMTP id gf12mr674304wic.14.1403123598299; Wed, 18 Jun 2014 13:33:18 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id vp5sm4308893wjc.31.2014.06.18.13.33.16 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Wed, 18 Jun 2014 13:33:16 -0700 (PDT) Date: Wed, 18 Jun 2014 22:33:14 +0200 From: Mateusz Guzik To: Adrian Chadd Subject: Re: capability races (was: Re: Seeing ENOTCAPABLE from write FDs in kqueue?) Message-ID: <20140618203314.GB7157@dft-labs.eu> References: <20140608220000.GA5030@dft-labs.eu> <20140608230059.GB5030@dft-labs.eu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20140608230059.GB5030@dft-labs.eu> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: Robert Watson , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jun 2014 20:33:21 -0000 On Mon, Jun 09, 2014 at 01:00:59AM +0200, Mateusz Guzik wrote: > So, benchmarks: > > http://people.freebsd.org/~mjg/patches/caps-race/dup2-bench.c > > 1 thread does dup2 in a loop > 7 threads fget() the same fd and fail quickly > [..] > Difference at 95.0% confidence > 1.96987e+06 +/- 46328.7 > 16.3174% +/- 0.383763% > (Student's t, pooled s = 167139) > > It would be good if someone with some time and access to higher-cpu-count > machine could test it better. > pjd suggested on irc allowing a pass with inconsistent filedescent struct and check it before we fail and before fp is accepted. I also went ahead and modified seq_* function a little to make them easier to use. This is with inconsistent struct allowed: http://people.freebsd.org/~mjg/patches/caps-race/caps-race-seq.diff And this spins to get the right one: http://people.freebsd.org/~mjg/patches/caps-race/caps-race-seqearly.diff Patches differ only in fget_unlocked function. Performnance is unfortunately still bad and an approach with inconsistent struct is the slowest. Results are reproducible well enough. Comments? Maybe we should try to pack this into one atomically replaced structure after all, but I don't have time to work on that. I checked with 2 benchmarks: 1 thread dup2 + 7 read: x pure-kernel-dup2-bench + seq-dup2-bench * seqearly-dup2-bench +-------------------------------------------------------------------------------------------------------------------------------------+ | + * | | + * | | + * | | + * x | | + * x | | ++ * x | | ++ * * x x | | +++ * * x x | | +++ * * ** * x xxx | | ++++ + ***** ** * x x xxxx xx | | + ++++++ + + ***** * **** xx xx xxxx xx | | ++++++++ +++ +****** * **** * xxxxxx xxxxxxx | | ++++++++ ++++++******* ** **** * x xxxxxx xxxxxxxx | | +++++++++ ++++++********* ******** * xx xxxxxxxxxxxxxxxx | |++ ++++++++++++++**+********** *********** x x x x xxx xxxxxxxxxxxxxxxxx x| | |___MA____| |____M_A_____| |_______A_M_____| | +-------------------------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 100 13065813 14435053 14157439 14103416 196933.44 + 100 11071952 11615178 11351083 11374348 121160.27 Difference at 95.0% confidence -2.72907e+06 +/- 45319.1 -19.3504% +/- 0.321334% (Student's t, pooled s = 163497) * 100 11517495 12111074 11774410 11828854 159139.54 Difference at 95.0% confidence -2.27456e+06 +/- 49626.4 -16.1277% +/- 0.351875% (Student's t, pooled s = 179037) 8 threads read: x pure-kernel-read-pipe-bench + seq-read-pipe-bench * seqearly-read-pipe-bench +-------------------------------------------------------------------------------------------------------------------------------------+ | * | | * | | + * x | | + * x | | + ++ * xx | | + ++ + * x xx | | + ++++ + * * xx xx | | + ++++ + + * **** * xx xxx | | + ++++ +++ * **** * xxxxxxx | | ++++++ ++++ * **** *** xxxxxxx | | ++++++ ++++ * ****** * *** x x xxxxxxx | |+ ++++++++++++ * * ******* ** ***** x xx xxxxxxxxxxx | |++++++++++++++++ + ** * ******* ** * * ***** * x xx x xxxxxxxxxxxxxxxx| |++++++++++++++++ + ++ + ** ************ * *** ****** * xxxx xxxx xxxxxxxxxxxxxxxxxx| | |___MA_____| |____M__A_______| |_____A_M___| | +-------------------------------------------------------------------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 100 20721201 21597446 21369536 21306748 206438.73 + 100 17272059 18247946 17509832 17547079 178174.14 Difference at 95.0% confidence -3.75967e+06 +/- 53448.4 -17.6454% +/- 0.250852% (Student's t, pooled s = 192825) * 100 18417084 19360802 18739503 18836380 269109.08 Difference at 95.0% confidence -2.47037e+06 +/- 66477.4 -11.5943% +/- 0.312002% (Student's t, pooled s = 239830) -- Mateusz Guzik