Date: Wed, 6 Feb 2008 09:11:49 +0300 From: Nicholas Kulikov <lostar@rambler.ru> To: freebsd-net@freebsd.org Subject: 1 Gb network NFS issues Message-ID: <6FBCC65B-734C-4CF5-9F44-2ECECBFD18EC@rambler.ru>
next in thread | raw e-mail | index | archive | help
Hello all,
I'm trying setup NFS in 1 Gb home network and found some strange issues:
Current NFS performance is about 30-35 Mb/s but few days ago i had
40-45 Mb/s. I've made some changes in server configuration (moved to
FreeBSD 6.3 and changed network driver to nfe) and I can't catch what
is wrong...
Network structure:
NFS server (FreeBSD 6.3)
|
| 1 Gb link
|
NFS client (Mac OS X 10.5.1)
Server configuration
-----------------------------
nfe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 8000
options=48<VLAN_MTU,POLLING>
Client configuration
------------------------
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 8000
media: autoselect (100baseTX <full-duplex>) status: active
client mount share by using the following parameters: tcp vers=3
rsize=40960 wsize=40960 readahead=16 rdirplus nolocks intr noatime
I've tried copy big file (about 3 Gb) and tcdump-ed client and server
packets:
07:11:08.350744 IP (tos 0x0, ttl 64, id 51461, offset 0, flags [DF],
proto TCP (6), length 176) 192.168.254.249.nfs >
192.168.254.254.2468561782: reply ok 124 access attr: REG 660 ids
1001/1006 sz 3958224896 nlink 1 rdev 133/134414584 fsid 5e fileid
2013005 a/m/ctime 1199479211.000000 1199479280.000000
1201346449.000000 c 001f
07:11:08.350793 IP (tos 0x0, ttl 64, id 46719, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.254.49158 >
192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0xd9e5),
22844:22844(0) ack 27581 win 65535 <nop,nop,timestamp 67511239 30749984>
07:11:08.350988 IP (tos 0x0, ttl 64, id 9526, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561783 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 0
07:11:08.351069 IP (tos 0x0, ttl 64, id 8470, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561784 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
32768
07:11:08.351145 IP (tos 0x0, ttl 64, id 65343, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561785 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
65536
07:11:08.351242 IP (tos 0x0, ttl 64, id 11022, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561786 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
98304
07:11:08.351314 IP (tos 0x0, ttl 64, id 59209, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561787 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
131072
07:11:08.351383 IP (tos 0x0, ttl 64, id 39733, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561788 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
163840
07:11:08.351450 IP (tos 0x0, ttl 64, id 33325, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561789 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
196608
07:11:08.351516 IP (tos 0x0, ttl 64, id 50481, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561790 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
229376
07:11:08.351582 IP (tos 0x0, ttl 64, id 50464, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561791 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
262144
07:11:08.351658 IP (tos 0x0, ttl 64, id 37215, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561792 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
294912
07:11:08.351739 IP (tos 0x0, ttl 64, id 6496, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561793 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
327680
07:11:08.351750 IP (tos 0x0, ttl 64, id 49495, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.249.nfsd >
192.168.254.254.49158: ., cksum 0x15e5 (correct), 27581:27581(0) ack
23076 win 49943 <nop,nop,timestamp 30749985 67511239>
07:11:08.351754 IP (tos 0x0, ttl 64, id 36632, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.249.nfsd >
192.168.254.254.49158: ., cksum 0x1501 (correct), 27581:27581(0) ack
23308 win 49939 <nop,nop,timestamp 30749985 67511239>
07:11:08.351758 IP (tos 0x0, ttl 64, id 21313, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.249.nfsd >
192.168.254.254.49158: ., cksum 0x141c (correct), 27581:27581(0) ack
23540 win 49936 <nop,nop,timestamp 30749985 67511239>
07:11:08.351783 IP (tos 0x0, ttl 64, id 50759, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.249.nfsd >
192.168.254.254.49158: ., cksum 0x1338 (correct), 27581:27581(0) ack
23772 win 49932 <nop,nop,timestamp 30749985 67511239>
07:11:08.351801 IP (tos 0x0, ttl 64, id 56942, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.249.nfsd >
192.168.254.254.49158: ., cksum 0x1254 (correct), 27581:27581(0) ack
24004 win 49928 <nop,nop,timestamp 30749985 67511239>
07:11:08.351818 IP (tos 0x0, ttl 64, id 13319, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561794 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
360448
07:11:08.351904 IP (tos 0x0, ttl 64, id 17022, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561795 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
393216
07:11:08.351981 IP (tos 0x0, ttl 64, id 45600, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561796 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
425984
07:11:08.351999 IP (tos 0x0, ttl 64, id 31024, offset 0, flags [DF],
proto TCP (6), length 6196) 192.168.254.249.nfs >
192.168.254.254.2468561783: reply ok 6144 read REG 660 ids 1001/1006
sz 3958224896 nlink 1 rdev 133/134414584 fsid 5e fileid 2013005 a/m/
ctime 1199479211.000000 1199479280.000000 1201346449.000000 32768 bytes
07:11:08.352014 IP (tos 0x0, ttl 64, id 13078, offset 0, flags [DF],
proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.0:
reply ok 6144
07:11:08.352074 IP (tos 0x0, ttl 64, id 9792, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561797 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
458752
07:11:08.352085 IP (tos 0x0, ttl 64, id 61207, offset 0, flags [DF],
proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.0:
reply ok 6144
07:11:08.352143 IP (tos 0x0, ttl 64, id 63816, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.254.49158 >
192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0x8b18),
24584:24584(0) ack 46013 win 65535 <nop,nop,timestamp 67511239 30749985>
07:11:08.352162 IP (tos 0x0, ttl 64, id 60945, offset 0, flags [DF],
proto TCP (6), length 52) 192.168.254.254.49158 >
192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0x8b18),
24584:24584(0) ack 46013 win 65535 <nop,nop,timestamp 67511239 30749985>
07:11:08.352177 IP (tos 0x0, ttl 64, id 3701, offset 0, flags [DF],
proto TCP (6), length 168) 192.168.254.254.2468561798 >
192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @
491520
07:11:08.352198 IP (tos 0x0, ttl 64, id 3073, offset 0, flags [DF],
proto TCP (6), length 6196) 192.168.254.249.nfs >
192.168.254.254.3681401976: reply Unknown rpc response code=3722304989
6144
07:11:08.352203 IP (tos 0x0, ttl 64, id 2122, offset 0, flags [DF],
proto TCP (6), length 6196) 192.168.254.249.nfs >
192.168.254.254.3067833788: reply Unknown rpc response code=3722304989
6144
07:11:08.352207 IP (tos 0x0, ttl 64, id 61294, offset 0, flags [DF],
proto TCP (6), length 2232) 192.168.254.249.nfs >
192.168.254.254.3067833788: reply Unknown rpc response code=3722304989
2180
There are strange errors:
192.168.254.254.3681401976: reply Unknown rpc response
Looks like server successfully sent first piece of data
"07:11:08.351999...length 6196) 192.168.254.249.nfs >
192.168.254.254.2468561783: reply ok 6144...32768 bytes" and then
report about errors in the following packets:
length 6196) 192.168.254.249.nfs > 192.168.254.254.3681401976: reply
Unknown rpc response code=3722304989 6144
length 6196) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply
Unknown rpc response code=3722304989 6144
length 2232) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply
Unknown rpc response code=3722304989 2180
There is a lot of such errors in tcpdump logs.
NFS3 specification do not contains this error code description...
p.s. And why server is limited with read/write data size 32760? Is
there a possibility to change it?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6FBCC65B-734C-4CF5-9F44-2ECECBFD18EC>