Date: Sun, 11 Mar 2007 08:36:54 +0100 From: Ulrich Spoerlein <uspoerlein@gmail.com> To: stable@freebsd.org Subject: Weird NFS behaviour Message-ID: <20070311073654.GA1543@roadrunner.q.local>
next in thread | raw e-mail | index | archive | help
Hi,
we have performance problems with our FreeBSD 6.2 based NFS server.
Picture the following setup:
FreeBSD Client ---> Samba-Server ---> NFS-Server
all three machines are running FreeBSD 6.2 (the same image). The NFS
server is configured with 16 nfsd. sysctl.conf has
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
Now, what's the problem: The Samba-Server mounts shares via NFS. All
servers are on Gigabit Ethernet and I get read transfer rates
exceeding 50MB/s from the NFS server.
This is all good and well, but if I copy a file via scp(1) (sic!) to
the samba server into the NFS mounted directory, not only do I
seldomly exceed 12MB/s but I also get a very strange traffic pattern
on the em0 interface of the samba server. I get _twice_ as much
incoming traffic on the em0 interface as outgoing traffic.
systat -if on samba:
em0 in 24.726 MB/s 25.905 MB/s 3.046 GB
out 12.941 MB/s 13.558 MB/s 1.994 GB
systat -if on nfs-server
em0 in 11.497 MB/s 12.999 MB/s 3.727 GB
out 11.878 MB/s 13.423 MB/s 995.485 MB
To stress, this is running:
gigabit-client:# scp large-file user@samba:/mnt/nfs-share/
The wicked part is this: If I copy a file from the samba server
directly to the NFS share (not as a passthrough), I get these traffic
patterns:
systat -if on samba:
em0 in 432.724 KB/s 432.724 KB/s 3.772 GB
out 12.399 MB/s 12.399 MB/s 2.481 GB
systat -if on nfs:
em0 in 12.091 MB/s 15.791 MB/s 184.766 MB
out 440.939 KB/s 562.521 KB/s 1.339 GB
This is running:
samba:# cp large-file /mnt/nfs-share/
What on earth is causing each received NFS packet to be _bounced_ to
the samba server when using ssh, scp, smbd, etc. And not when
generating the traffic locally?
nfsstat -s is showing an increase in READ calls similar to WRITE calls
when using the samba machine as pass-through. It is showing _no_
increase in READ calls when copying the files directly.
NB: All these test were run _without_ smbd running, it's just that
this server is designated to become our samba server.
Setting vfs.nfsrv.async=1 doubled write performance, but the weird
traffic pattern remains. (Am I asking for too much trouble by setting
async NFS?)
Thanks for any pointers!
Uli
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20070311073654.GA1543>