From owner-cvs-all@FreeBSD.ORG Thu Aug 17 20:46:10 2006 Return-Path: X-Original-To: cvs-all@FreeBSD.org Delivered-To: cvs-all@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9744F16A4DF; Thu, 17 Aug 2006 20:46:10 +0000 (UTC) (envelope-from brian@Awfulhak.org) Received: from storm.uk.FreeBSD.org (storm.uk.FreeBSD.org [194.242.157.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id B166643D7C; Thu, 17 Aug 2006 20:45:56 +0000 (GMT) (envelope-from brian@Awfulhak.org) Received: from gw.Awfulhak.org (root@gw.lan.Awfulhak.org [172.16.0.1]) by storm.uk.FreeBSD.org (8.13.6/8.13.6) with ESMTP id k7HKjrDt083653; Thu, 17 Aug 2006 21:45:54 +0100 (BST) (envelope-from brian@Awfulhak.org) Received: from demarc.ca.sophos.com (brian@localhost [127.0.0.1]) by gw.Awfulhak.org (8.13.6/8.13.6) with ESMTP id k7HKjmjL072035; Thu, 17 Aug 2006 13:45:49 -0700 (PDT) (envelope-from brian@Awfulhak.org) Date: Thu, 17 Aug 2006 13:45:46 -0700 From: Brian Somers To: Kris Kennaway Message-ID: <20060817134546.0c05e607@demarc.ca.sophos.com> In-Reply-To: <20060817163848.GA27460@xor.obsecurity.org> References: <200608041234.k74CYoc1076722@repoman.freebsd.org> <44E389AA.3000003@FreeBSD.org> <44E38E2F.4000005@FreeBSD.org> <44E391AD.3010402@FreeBSD.org> <44E39535.10701@FreeBSD.org> <20060816234236.3428ab08@dev.lan.Awfulhak.org> <20060817094501.GA98961@xor.obsecurity.org> <20060817084317.55ae1c08@demarc.ca.sophos.com> <20060817163848.GA27460@xor.obsecurity.org> X-Mailer: Sylpheed-Claws 2.3.1 (GTK+ 2.8.20; i386-portbld-freebsd6.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED,SPF_HELO_PASS, SPF_PASS autolearn=failed version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on gw.lan.Awfulhak.org Cc: Maxim Sobolev , cvs-all@FreeBSD.org, ports-committers@FreeBSD.org, G?bor K?vesd?n , cvs-ports@FreeBSD.org, Erwin Lansing Subject: Re: nullfs performance (Re: cvs commit: ports/Mk bsd.emacs.mk bsd.gnome.mk bsd.mail.mk bsd.openssl.mk bsd.port.mk bsd.port.subdir.mk bsd.python.mk bsd.ruby.mk bsd.scons.mk ports/Tools/scripts security-check.awk ports/databases/p5-DBD-Oracle Makefile ports/databases/p5-sqlrelay ...) X-BeenThere: cvs-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: CVS commit messages for the entire tree List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Aug 2006 20:46:10 -0000 On Thu, 17 Aug 2006 12:38:48 -0400 Kris Kennaway wrote: > On Thu, Aug 17, 2006 at 08:43:17AM -0700, Brian Somers wrote: > > On Thu, 17 Aug 2006 05:45:01 -0400 > > Kris Kennaway wrote: > > > > > On Wed, Aug 16, 2006 at 11:42:36PM -0700, Brian Somers wrote: > > > > [-developers elided] > > > > > > > > Interesting... I use nullfs as part of our build system here. > > > > I found that it's performance is appalling when pushed a > > > > little (running two builds on one machine, each with two > > > > nullfs mounts, two devfs and two procfs mounts gives a build > > > > time of 4.5 hours whereas a single build will finish in 1.5 > > > > hours). I've seen >6 hour builds on a loaded box - only > > > > attributable to nullfs. > > > > > > What version? > > > > The version that ships with 6.1. > > OK, that's unusual and bears further investigation then. When I > measured it I did not see this kind of dramatic performance loss > (anecdotally it doesn't fit with my own experiences either: I also use > it for parallel compilations, and I do not see anomalously low > performance compared to machines not using nullfs). > > The only thing I can think of is that the underlying filesystem is not > mpsafe (e.g. are you using UFS quotas?), in which case it's not really > nullfs to blame. We're not using quotas. > Hmm, are you sure it's 6.1-RELEASE and not a prerelease? I think the > code that was supposed to make the nullfs mount mpsafe conditional on > the mpsafety of the lower layer was broken until some time during the > release cycle, but it was fixed before the release. I integrated RELENG_6_1_0_RELEASE into our source tree. I guess the missing info might be that things get indirected somewhat. We check out code into /some/deep/directory/tree. Then, to protect against the 80 character path limitation, we create /tmp/bld.XXXXX/ and create a scratch -> /tmp/bld.XXXXX symlink in /some/deep/directory/tree. We then do various things like: mount -t nullfs /some/deep/directory/tree/src scratch/build/src mount -t nullfs /some/deep/directory/tree/obj scratch/build/obj mount -t devfs devfs scratch/dev mount -t procfs procfs scratch/proc and do a "OBJDIRPREFIX=/build/obj chroot scratch make -C /build/src". Oh, and errum, we've got debug.mpsafenet="0" in /boot/loader.conf - which is a remnant of when we were using 5.4 and the races in the socket code killed our application under load. Does the nullfs code path hit the network stack?? -- Brian Somers Don't _EVER_ lose your sense of humour !