Date: Wed, 28 Aug 2013 19:48:29 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Garrett Wollman <wollman@bimajority.org> Cc: freebsd-fs@freebsd.org Subject: Re: NFS on ZFS pure SSD pool Message-ID: <461209820.15034260.1377733709648.JavaMail.root@uoguelph.ca> In-Reply-To: <21022.29128.557471.157078@hergotha.csail.mit.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
Garrett Wollman wrote: > <<On Wed, 28 Aug 2013 16:49:43 -0400 (EDT), Rick Macklem > <rmacklem@uoguelph.ca> said: > > > Eric Browning wrote: > >> Sam and I applied the patch (kernel now at r254983M) and set > >> vfs.nfsd.tcphighwater=5000 in sysctl.conf and my CPU is still > >> slammed. SHould I up it to 10000? > >> > > You can try. I have no insight into where this goes, since I can't > > produce the kind of server/load where it makes any difference. (I > > have > > single core i386 (P4 or similar) to test with and I don't use ZFS > > at all.) > > I've cc'd Garrett Wollman, since he runs rather large servers and > > may > > have some insight into appropriate tuning, etc. > > 10,000 is probably way too small. We run high-peformance servers > with > vfs.nfsd.tcphighwater set between 100k and 150k, and we crank > vfs.nfsd.tcpcachetimeo down to five minutes or less. > > Just to give you an idea of how rarely this cache is actually hit: my > two main production file servers have both been up for about three > months now, and have answered billions of requests (enough for the > 32-bit signed statistics counters to wrap). One server shows 63 > hits, > with a peak TCP cache size of 150k and the other shows zero, with a > peak cache size of 64k. Another server, which serves scratch space, > has been up for a little more than a month, and in nearly two billion > accesses has yet to see a single cache hit (peak cache size 131k, > which was actually hitting the configured limit, which I've since > raised). > > -GAWollman > Yes. The cache is only hit if a client is network partitioned for long enough that it does an RPC retry over TCP. Most clients only do this now (this behaviour is required for NFSv4) when the client establishes a new TCP connection after giving up on the old one. (How quickly this occurs will depend on the client, but I am not surprised it is rare in a well maintained LAN environment.) You should get your users to do their mounts over flaky WiFi links and such, in order to make better use of the cache;-) By the way Garrett, what do you have kern.ipc.nmbclusters set to, since cache entries will use mbuf clusters normally. And Garrett, thanks for your input, rick
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?461209820.15034260.1377733709648.JavaMail.root>