Date: Tue, 30 Mar 2010 05:01:36 GMT From: Rich Ercolani <rercola@acm.jhu.edu> To: freebsd-gnats-submit@FreeBSD.org Subject: misc/145189: nfsd performs abysmally under load Message-ID: <201003300501.o2U51afE033587@www.freebsd.org> Resent-Message-ID: <201003300510.o2U5A2p6093647@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 145189 >Category: misc >Synopsis: nfsd performs abysmally under load >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Tue Mar 30 05:10:02 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Rich Ercolani >Release: 8.0 >Organization: JHU ACM >Environment: FreeBSD manticore.acm.jhu.edu 8.0-RELEASE FreeBSD 8.0-RELEASE #1: Mon Mar 22 20:13:43 EDT 2010 root@manticore.acm.jhu.edu:/usr/obj/usr/local/ncvs/src/sys/DTRACE_POLL amd64 >Description: nfsd performs abysmally on this machine under conditions in which Solaris's NFS implementation is reasonably fast, and while local IO to the same filesystems is still zippy. For instance, copying a 4GB file over NFSv3 from a ZFS filesystem with the following flags [rw,nosuid,hard,intr,nofsc,tcp,vers=3,rsize=8192,wsize=8192,sloppy,addr=X.X.X.X](Linux client, the above is the server), I achieve 2 MB/s, fluctuating between 1 and 3. (pv reports 2.23 MB/s avg) Locally, on the server, I achieve 110-140 MB/s (at the end of pv, it reports 123 MB/s avg). I'd assume network latency, but nc with no flags other than port achieves 30-50 MB/s between server and client. Latency is also abysmal - ls on a randomly chosen homedir full of files, according to time, takes: real 0m15.634s user 0m0.012s sys 0m0.097s while on the local machine: real 0m0.266s user 0m0.007s sys 0m0.000s The server in question is a 3GHz Core 2 Duo, running FreeBSD RELENG_8. The kernel conf, DTRACE_POLL, is just the stock AMD64 kernel with all of the DTRACE-related options turned on, as well as the option to enable polling in the NIC drivers, since we were wondering if that would improve our performance. We tested this with a UFS directory as well, because we were curious if this was an NFS/ZFS interaction - we still got 1-2 MB/s read speed and horrible latency while achieving fast throughput and latency local to the server, so we're reasonably certain it's not "just" ZFS, if there is indeed any interaction there. Read speed of a randomly generated 6500 MB file on UFS over NFSv3 with the same flags as above: 1-3 MB/s, averaging 2.11 MB/s Read speed of the same file, local to the server: consistently between 40-60 MB/s, averaging 61.8 MB/s [it got faster over time - presumably UFS was aggressively caching the file, or something?] Read speed of the same file over NFS again, after the local test: Amusingly, worse (768 KB/s-2.2 MB/s, with random stalls - average reported 270 KB/s(!)). >How-To-Repeat: 1) Mount multiple NFS filesystems from the server 2) Watch as your operations latency and throughput rapidly sink to near-zero >Fix: >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201003300501.o2U51afE033587>