Date: Thu, 18 Oct 2012 11:16:16 -0500 From: Thomas Johnson <tom@claimlynx.com> To: freebsd-fs@freebsd.org Cc: root <root@claimlynx.com> Subject: Poor throughput using new NFS client (9.0) vs. old (8.2/9.0) Message-ID: <CAFevjsthqegWv_GXVXceabwYf9urKZ7kEW2qQw7DSQpC7Hv0BA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
We recently upgraded a number of hosts from FreeBSD 8.2 to 9.0. Almost
immediately, we received reports from users of poor performance. The
upgraded hosts are PXE-booted, with an NFS-mounted root. Additionally, they
mount a number of other NFS shares, which is where our users work from.
After a week of tweaking rsize/wsize/readahead parameters (per guidance),
it finally occurred to me that 9.0 defaults to the new NFS client and
server. I remounted the user shares using the oldnfs file type, and users
reported that performance returned to its expected level.
This is obviously a workaround, rather than a solution. We would prefer to
get our hosts using the newnfs client, since presumably oldnfs will be
deprecated at some point in the future. Is there some change that we should
have made to our NFS configuration with the upgrade to 9.0, or is it
possible that our workload is exposing some deficiency with newnfs? We tend
to deal with a huge number of tiny files (several KB in size). The NFS
server has been running 9.0 for some time (prior to the client upgrade)
without any issue. NFS is served from a zpool, backed by a Dell MD3000,
populated with 15k SAS disks. Clients and server are connected with Gig-E
links. The general hardware configuration has not changed in nearly 3 years.
As an example of the performance difference, here is some of the testing I
did while troubleshooting. Given a directory containing 5671 zip files,
with an average size of 15KB. I append all files to an existing zip file.
Using the newnfs mount, I found that this operation generally takes ~30
seconds (wall time). Switching the mount to oldnfs resulted in the same
operation taking ~10 seconds.
tom@test-1:/test-> ls 53*zip | wc -l
5671
tom@test-1:/test-> ll -h BIG*
-rw-rw-r-- 1 tom claimlynx 8.9M Oct 17 14:06 BIGGER_PILE_1.zip
tom@test-1:/test-> time zip BIGGER_PILE_1.zip 53*.zip
0.646u 0.826s 0:51.01 2.8% 199+2227k 0+2769io 0pf+0w
...reset and repeat...
0.501u 0.629s 0:30.49 3.6% 208+2319k 0+2772io 0pf+0w
...reset and repeat...
0.601u 0.522s 0:32.37 3.4% 220+2406k 0+2771io 0pf+0w
tom@test-1:/-> cd /
tom@test-1:/-> sudo umount /test
tom@test-1:/-> sudo mount -t oldnfs -o rw server:/array/test /test
tom@test-1:/-> mount | grep test
server:/array/test on /test (oldnfs)
tom@test-1:/-> cd /test
...reset and repeat...
0.470u 0.903s 0:13.09 10.4% 203+2229k 0+5107io 0pf+0w
...reset and repeat...
0.547u 0.640s 0:08.65 13.6% 231+2493k 0+5086io 0pf+0w
tom@test-1:/test-> ll -h BIG*
-rw-rw-r-- 1 tom claimlynx 92M Oct 17 14:14 BIGGER_PILE_1.zip
Thanks!
--
Thomas Johnson
ClaimLynx, Inc.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFevjsthqegWv_GXVXceabwYf9urKZ7kEW2qQw7DSQpC7Hv0BA>