Date: Sun, 7 Jul 2013 20:31:16 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Berend de Boer <berend@pobox.com> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2 Message-ID: <50381589.2918947.1373243476714.JavaMail.root@uoguelph.ca> In-Reply-To: <580122426.2916694.1373242759482.JavaMail.root@uoguelph.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
I wrote: > Berend de Boer wrote: > > Hi All, > > > > I've just completed a round of NFS testing on FreeBSD 9.1 on AWS. > > The > > underlying file system is ZFS. I have a real nfs killer test: > > doing > > an "svn update" of a directory of 3541 files. > > > > Performance on the NFS server itself is good: checking this out > > takes > > 11 seconds (doing the same on a not really comparable Linux NFS > > server > > is 43 seconds). > > > > Doing this on a client writing to an NFS mounted home directory is > > however terrible. Really terrible. This takes 25 minutes! With > > "sync=disable" it's 16 minutes. > > > > (doing this against the underpowered Linux NFS server is about 4.5 > > minutes). > > > > The problem might be that the NFS server (nfsd) runs at 70-80% CPU. > > So > > my writing speed is cpu bound (go figure). > > > See below w.r.t. patch that reduces cpu overheads in the DRC. > > > Repeating this with nfs4 + udp: doesn't work at all, get > > input/output > > error after a few files. > > > The RFCs for NFSv4 require use of transport protocols that include > congestion control --> no UDP support. What client are you using? > (If it is a FreeBSD one, I need to patch it to make the mount fail. > Since NFSv4 over UDP was done in early testing, the client might > still have that bogus code in it.) > > > nfs3: 1m55s (nfsd cpu is in the 6%-8% range) > > > > Varying tcp/udp, lock/nolock, sync/async, or enabling zfs > > sync=standard doesn't make any particular difference. All timing > > were > > within the 1m49s - 2m3s range. > > > > So what's up with NFS4 on FreeBSD? > > > Please try this patch: > http://people.freebsd.org/rmacklem/~drc4.patch Oops, it actually is: http://people.freebsd.org/~rmacklem/drc4.patch > - After you apply the patch and boot the rebuilt kernel, the cpu > overheads should be reduced after you increase the value of > vfs.nfsd.tcphighwater. The larger you make it, the more space > (mbuf clusters and other kernel malloc'd data structures) the DRC > uses, but hopefully with reduced CPU overheads. > The plan is to commit a patch semantically similar to this to head > and > then MFC it someday. ivoras@ has a similar patch, but written in > cleaner > C. However, I've never gotten around to combining the patches into a > version for head. Someday I'll get to it, but not in time for 9.2. > > Good luck with it, rick > ps: There is also ken@'s file handle affinity patch, which is in head > (and I think stable/9), but it only works for NFSv3 at this > point. > Hopefully we'll come up with a patch for NFSv4 for it someday. > > > -- > > All the best, > > > > Berend de Boer > > > > > > ------------------------------------------------------ > > Awesome Drupal hosting: https://www.xplainhosting.com/ > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50381589.2918947.1373243476714.JavaMail.root>