Date: Tue, 18 Apr 2006 08:36:34 -0500 From: Eric Anderson <anderson@centtech.com> To: David Gilbert <dgilbert@dclg.ca> Cc: FreeBSD ISP <freebsd-isp@freebsd.org>, Francisco Reyes <lists@stringsutils.com> Subject: Re: NFS optimization Message-ID: <4444EB62.4060700@centtech.com> In-Reply-To: <17476.57714.38638.380701@canoe.dclg.ca> References: <cone.1144794037.918896.59848.1000@zoraida.natserv.net> <17475.43946.264571.52593@canoe.dclg.ca> <cone.1145294815.465429.96480.1000@zoraida.natserv.net> <17475.54375.95109.55657@canoe.dclg.ca> <4444D029.8060109@centtech.com> <17476.57714.38638.380701@canoe.dclg.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
David Gilbert wrote: >>>>>> "Eric" == Eric Anderson <anderson@centtech.com> writes: > > Eric> David Gilbert wrote: > >>> Consider that if you are "out" of nfsd's, the penalty is increased >>> latency for some small number of transactions that wait for an nfsd >>> to become available.. Even if you have tonnes of NFSd processes, >>> if disk is a limiting factor, more nfsd's won't speed the process. > > > Eric> I have found that having too little can easily cause clients to > Eric> block on nfs under peak usage times, so I tend to bump the > Eric> number way up. There's little to no harm in it. > > I have never, ever seen this behaviour. I'd go as far as to say that > it shouldn't happen. Not categorically, but NFS packets should be > entirely independant... meaning it shouldn't prefer one client's > pakcets over another unless it is massively starved for NFSd's, the > queue should be somewhat FIFO. I'm not too surprised really. If lots of nfs clients slam an nfs server simultaneously, all wanting data from different parts of the storage system, then it is very easy to stack up more requests than the nfsd's can handle if there are too few of them. We have a different kind of nfs load here than any place I've seen, so that could account for the difference too. > Eric> I usually look at my nfsd's, and see what the distribution of > Eric> run time is on them. I like to see at minimum a few (maybe 5% > Eric> or so) with 0:00.00 runtime - which (to me) means that I had > Eric> enough to service the queue, and a few extra that were bored. > Eric> For my setup, this means typically between 256 and 512 nfsd's > Eric> (with one server at 1024). > > I have run incredibly busy NFS servers (20 to 40 disks, 16 to 20 > ethernet and 100 (or more) busy diskless clients (computation cluster) > and I have never run more than 32. I've never found a performance > advantage beyond 1:1 nfsd's to disks. Well, nfs client usage patterns strongly dictate nfs server load, so the quantity of clients is important, but more so is how the clients use the data on the nfs server. We have around 1000 busy nfs clients (all 3+GHz P4's), about 900 of them in a compute cluster (some diskless, some not), nearly everything is Gig-E. My rule of thumb for nfsd's has come down to nfs clients / 4 = nfsd threads. With very fast disk subsystems, and lots of caching, and 'good' usage patterns, very few nfsd's would be needed. If you have lots of usage spikes, and a *lot* of random reads/writes, coupled with a large number of clients, you can easily see the problem I mentioned above with a small number of nfsd's. There have been a few other threads on other mailing lists (freebsd-fs I think was one) that other users have seen the same issues, and merely bumping the nfsd's gets them past the problem. Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4444EB62.4060700>