Date: Sat, 8 Jan 2011 11:55:03 -0500 (EST) From: george+freebsd@m5p.com To: freebsd-hackers@freebsd.org Subject: NFS Performance Message-ID: <201101081655.p08Gt39N001427@m5p.com>
next in thread | raw e-mail | index | archive | help
Among four machines on my network, I'm observing startling differences in NFS performance. All machines are AMD64, and rpc_statd, rpc_lockd, and amd are enabled on all four machines. wonderland: hw.model: AMD Athlon(tm) II Dual-Core M32 hw.physmem: 293510758 ethernet: 100Mb/s partition 1: FreeBSD 8.1-STABLE partition 2: FreeBSD 7.3-STABLE scollay: hw.model: AMD Sempron(tm) 140 Processor hw.physmem: 186312294 ethernet: 1000Mb/s FreeBSD 8.1-PRERELEASE sullivan: hw.model: AMD Athlon(tm) 64 X2 Dual Core Processor 4800+ hw.physmem: 4279980032 ethernet 1000Mb/s FreeBSD 7.2-RELEASE mattapan: hw.model: AMD Sempron(tm) Processor 2600+ hw.physmem: 456380416 ethernet: 1000Mb/s FreeBSD 7.1-RELEASE Observed bytes per second (dd if=filename of=/dev/null bs=65536): Source machine: mattapan scollay sullivan Destination machine: wonderland/7.3 870K 5.2M 1.8M wonderland/8.1 496K 690K 420K mattapan 38M 28M scollay 33M 33M sullivan 38M 5M There is one 10/100/1000Mb/s ethernet switch between the various pairs of machines. I'm startled by the numbers for wonderland, first because of how much the 100Mb/s interface slows things down, but even more because of how much difference there is on the identical hardware between FreeBSD 7 and FreeBSD 8. Even more annoying when running 8.1 on wonderland, NFS simply locks up at random for roughly a minute and a half under high load (such as when firefox does a gazillion locked references to my places.sqlite file), leading to entertaining log message clusters such as: Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:17:41 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:17:47 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:17:47 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:18:01 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:18:01 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:18:02 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:18:08 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:18:08 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:18:09 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:20:21 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:20:21 wonderland last message repeated 2 times Dec 29 08:20:22 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:20:22 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:20:36 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:20:36 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:21:05 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:21:10 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:22:20 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:22:22 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:22 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:24 wonderland last message repeated 2 times Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:22:24 wonderland last message repeated 2 times Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:24 wonderland kernel: nfs server home:/usr: is alive again Dec 29 08:22:27 wonderland kernel: nfs server home:/usr: not responding Dec 29 08:22:27 wonderland kernel: nfs server home:/usr: is alive again The time stamps on the "not responding" messages are lies; the lack of response in each case was roughly a minute and a half earlier. "home" equals mattapan in the earlier table. During the periods of NFS lack of response, mattapan is still completely functional, and a concurrent ssh session proceeds without difficulty. I can also make this happen (though not as easily) on NFS from sullivan to wonderland, but I haven't seen it yet from scollay to wonderland. This problem never occurs when wonderland is running 7.3. Any suggestions? -- George Mitchell
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201101081655.p08Gt39N001427>