Date: Wed, 28 Jan 2009 23:21:35 -0800 From: Brent Jones <brent@servuhome.net> To: freebsd-performance@freebsd.org, pathiaki2@yahoo.com Subject: Re: ZFS, NFS and Network tuning Message-ID: <ee9f3b480901282321h5b57fa49ud39265c7523e0cdf@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
I'm reviving this, as I too am seeing something eerily similar. I have made my own thread under freebsd-stable, so I will hopefully move that discussion to this list. I believe we are seeing performance problems when the FreeBSD NFS client issues FSYNC NFS instead of ASYNC, sending performance to a mere percentage of what disks and network links are capable of. Further testing tonight demonstrates that other NFSv3 and v4 clients do not issue FSYNC unless they modify attributed and close a file, or append and close a file. FreeBSD NFS client will issue FSYNCs anytime the write size (-w) is reached, instead of when just closing the file. This is not necessary, since NFSv3 and v4 TCP have provisions for safe async writes that 'guarantee' state of NFS writes. Here is the contents of what I wrote there verbatim: http://lists.freebsd.org/pipermail/freebsd-stable/2009-January/048063.html ------- Hello FreeBSD users, I am running into some performance problems with NFSv3/v4 mounts. I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS shares The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram and a FreeBSD 7.1 32bit with same hardware. The issue I am seeing, is that for certain file types, the FreeBSD NFS client will either issue an ASYNC write, or an FSYNC. However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP versions of the protocol, so that should be the default. Issuing FSYNC's for every compete block transmitted adds substantial overhead and slows everything down. The two test files I have that can reproduce this data are a file created by 'dump' which is just binary data: $ file testbinery testbinery: data ASCII text file from a Maildir format: $ file ascittest ascittest: ASCII mail text My NFS mount command lines I have tried to get all data to ASYNC write: $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/ Here is an excerpt from a snoop from the binary data file: $ snoop rpc nfs obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 (read,lookup,modify,extend,delete,execute) pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3 obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3 (read,lookup,modify,extend,delete,execute) pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend) obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3 pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for 32768 (ASYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC) And on and on.. it will acheive near full wire-speed, about 110MB/sec during the copy Here is the same snoop, only copying the ASCII mail file: $ snoop rpc nfs obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED) ascittest pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3 obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768 (FSYNC) pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC) And so on. I've reproduced this with several files, and the only difference between tests is the file type. Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the file type/contents? If so, is there a tuneable setting to make all write ASYNC? Otherwise, FSYNC'ing for every block written over NFS will cause so many IOPS on the NFS server, that performance will degrade severely. Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for any file type, if mounted with NFSv3 of NFSv4 (TCP). Any ideas? Thanks in advance! -- Brent Jones brent@servuhome.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ee9f3b480901282321h5b57fa49ud39265c7523e0cdf>