From owner-svn-src-all@freebsd.org Thu Jul 30 06:06:49 2015 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E9E029ADC30; Thu, 30 Jul 2015 06:06:48 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id 083BC1CFA; Thu, 30 Jul 2015 06:06:47 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 65ACDD62BFF; Thu, 30 Jul 2015 16:06:39 +1000 (AEST) Date: Thu, 30 Jul 2015 16:06:39 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Warner Losh cc: Hans Petter Selasky , =?utf-8?Q?Roger_Pau_Monn=C3=A9?= , Adrian Chadd , Warner Losh , "src-committers@freebsd.org" , "svn-src-all@freebsd.org" , "svn-src-head@freebsd.org" , Shani Michaeli Subject: Re: svn commit: r285068 - in head/sys: conf modules/agp modules/geom/geom_part/geom_part_apm modules/geom/geom_part/geom_part_bsd modules/geom/geom_part/geom_part_bsd64 modules/geom/geom_part/geom_part... In-Reply-To: <2779441E-E53E-4DB4-84CF-36A0CFCB4C08@bsdimp.com> Message-ID: <20150730135259.C2050@besplex.bde.org> References: <201507030150.t631oRd0039490@repo.freebsd.org> <5596C188.10404@FreeBSD.org> <5596C7E7.5090700@FreeBSD.org> <68C8F69B-56DF-45C3-8DBB-40514CA48D85@bsdimp.com> <55B8A8CA.90701@selasky.org> <3414D44A-A22F-4693-9F84-A8E880C0B185@bsdimp.com> <55B8F547.2010008@selasky.org> <2779441E-E53E-4DB4-84CF-36A0CFCB4C08@bsdimp.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=ItbjC+Lg c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10 a=ndaoGXS1AAAA:8 a=95inkREc4AnWJPzKaJMA:9 a=PEqzJO3-QseUefnm:21 a=wRmMVqGdGscQzIu3:21 a=CjuIK1q_8ugA:10 a=Zn7JbPeRIQgA:10 X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jul 2015 06:06:49 -0000 On Wed, 29 Jul 2015, Warner Losh wrote: >> On Jul 29, 2015, at 9:46 AM, Hans Petter Selasky wrote: >> In this particular case one "find of /sys" takes 11-16 seconds over NFS, so building a single KMOD takes 16 seconds too. It's not possible to eliminate the find entirely during repeated builds? > > 16 seconds? That\xe2\x80\x99s a really slow NFS server and at least 11 seconds longer than it should take :(. 11 seconds? Building full (FreeBSD-4) kernel takes 4.5 here seconds with a tuned nfs, starting with a warm cache after "make depend". makeworld takes 130 seconds over nfs, with nfs costing about 15 seconds of that, starting with a cold cache on the client and a warm cache on the server, and doing sufficient "make depend" steps (most are optimized out in ny version to save 10% with make -j1 and much more with make -j8; "make depend" and "make install" are not parallelized so they are the slowest parts of makeworld). It is mysterious that makeworld starting with a warm cache on the client is slightly slower. This took some fighting with network latency and nfs's lack of caching of attributes (it caches everything, but checks attributes on every first-open). For find, I think this results in an RPC or two for opening every directory in the traversal, but the caching works for regular files. "find ." on /usr/src takes: 0.27 real 0.03 user 0.24 sys on the server 3.27 real 0.05 user 0.40 sys on the client (3x faster) Lookup Access Fsstat Getattr Other Total (client nfsstat delta:) 1759 408 6987 14374 1 23529 while cached du -a takes: 0.58 real 0.09 user 0.48 sys on the server 10.30 real 0.09 user 0.78 sys on the client Lookup Access Fsstat Getattr Other Total (client nfsstat delta:) 49655 6987 6986 13979 6 77613 /usr/src has 6986 directories and 49643 files. There seems to be 1 Fsstat and 2 Getattr's per directory, 1 Access per directory for du -a, and 1 Lookup per file for du -a. The caching helped more for Lookup and Access for "find .. After mounting with -nocto: find .: 1.20 real 0.04 user 0.32 sys Fsstat Other Total 6987 16 7003 du -a: 6.83 real 0.04 user 0.69 sys Lookup Access Fsstat Other Total 42666 237 6986 3 49892 There are still too many Fsstats. This reminds me of an old pessimization in opendir(). It calls fstatfs() for almost every directory to support unionfs although unionfs never workded correctly and is almost never used. At least nfs3 never cached Fsstat in any FreeBSD implementation, so "find ." has to do lots of Fsstat RPCs. The caching works perfectly to avoid almost all other RPCs. There are still too many Lookups. The Lookup count for du -a now seems to be the number of files less than the number of directories. Apparently, lookups are only cached right for directories. After waiting a bit for cache timeouts to expire, du -a does 49000+ Lookups to look up the directories too. The files get cached if you read them. This gives the silly behaviour that tar cvvf runs much faster than du -a with a warm cache, since caching actually works for it (except for Fsstat): tar cvvf /dev/zero (nocto): 2.41 real 0.17 user 1.40 sys Fsstat Other Total 6986 10 6996 I had to fudge this test a little to avoid cache timeouts. I ran it a few times and picked the fastest ones. The default cache timeouts are 3-30 seconds for files and 30-60 seconds for directories (depending on the age of the file). tar with a cold cache takes longer than that, so it takes a few runs to get everything cached. Caching everything is only possible since the caching works for reads. So the first run of tar cvvf caches all the data so that subsequent runs have a chance of completing faster than the attribute cache times out. du -a has without nocto a high variance since it takes a significant fraction of the cache timeout. I don't use nocto in production since it reduces robustness and I want to make nfs caching work better without it. Caching can also be improved and robustness unimproved by increasing the cache timeouts. These default and current settings of these are hard to determine since their documentation is incorrect and mount(8) still doesn't support retrieving any mount options that are not in old flags. The default for acdirmin is still documented as being 30. This was too large, so it was changed to 3 in the code. The man page hasn't caught up with the change. 3 seems too low. The default for acdirmax is still 60. The defaults for nametimeo and negnametimeo are only documented as manifest constants, so they they are always correct in the man page because they are invisible there. NFS_DEFAULT_NEGNAMETIMEO is too long and has value 60. For the ac* timeouts, there is a hint that the actual timeout (between the min and the max) depends on the age of the file. For negnametimeo, there is no hint that the actual timeout is often determined by changes in the directory. negnametimeo seems to be just defense against bugs in the attribute caching. I don't know of any similar timeout for data. Such a timeout would flush the data cache quite often when nothing has changed, just in case we don't detect an actual change due to buggy timestamps or buggy caching of timestamps. Bruce