Date: Tue, 9 Jul 2013 19:38:01 -0400 (EDT) From: Rick Macklem <rmacklem@uoguelph.ca> To: Berend de Boer <berend@pobox.com> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: Terrible NFS4 performance: FreeBSD 9.1 + ZFS + AWS EC2 Message-ID: <818900293.3878290.1373413081112.JavaMail.root@uoguelph.ca> In-Reply-To: <877gh0yvhm.wl%berend@pobox.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Berend de Boer wrote: > >>>>> "Berend" == Berend de Boer <berend@pobox.com> writes: > > Berend> 1. Does not effect nfs3. > > One update: with zfs sync=standard, I can get nfs3 to perform as bad > as the nfs4+patch. > Hmm, this is interesting. ken@'s file handle affinity patch works for NFSv3, but not NFSv4. If I understood his posts correctly, the fh affinity patch was needed, so ZFS's heuristic for recognizing sequential reading would function correctly. (A file handle affinity patch for NFSv4 will take some time, since all RPCs in NFSv4 are compounds, with reads/writes imbedded in them, along with other ops.) If you could do some testing where you export a UFS volume, the results might help to isolate the issue to ZFS vs nfsd. > 1. without nolock, without async, sync=standard: 8m21s > > 2. with nolock, without async, sync=standard: 5m37s > > 3. without nolock, with async, sync=standard: 4m56s. > > 4. with nolock, with async, sync=standard: 4m23s. > > 5. without nolock, without async, sync=disabled: 1m57. > > > PS: the nfs4 test was done with sync=disabled. > W.r.t. CPU overheads and the size of vfs.nfsd.tcphighwater. The size of the NFSRVCACHE_HASHSIZE was increased to 500 by the patch. 500 seems about right for a vfs.nfsd.tcphighwater of a few thousand. If you are going to use a very large value (100000), I'd suggest you increase NFSRVCACHE_HASHSIZE to something like 10000. (You have to edit sys/fs/nfs/nfsrvcache.h and recompile to change this.) I'd suggset you try something like: vfs.nfsd.tcphighwater=5000 vfs.nfsd.tcpcachetimeo=300 (5 minutes instead of 12hrs) Also, I can't remember if you've bumped up the # of nfsd threads, but I'd go for 256. nfs_server_flags="-u -t -n 256" - in your /etc/rc.conf I never use ZFS, so I can't help w.r.t. ZFS tuning, rick > > -- > All the best, > > Berend de Boer > > > ------------------------------------------------------ > Awesome Drupal hosting: https://www.xplainhosting.com/ >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?818900293.3878290.1373413081112.JavaMail.root>