From owner-freebsd-net@FreeBSD.ORG Wed Feb 6 06:29:28 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ECEFE16A420 for ; Wed, 6 Feb 2008 06:29:27 +0000 (UTC) (envelope-from lostar@rambler.ru) Received: from mxb.rambler.ru (mxb.rambler.ru [81.19.66.30]) by mx1.freebsd.org (Postfix) with ESMTP id 2046C13C46B for ; Wed, 6 Feb 2008 06:29:27 +0000 (UTC) (envelope-from lostar@rambler.ru) Received: from maild.rambler.ru (maild.rambler.ru [81.19.66.33]) by mxb.rambler.ru (Postfix) with ESMTP id 7133D1B9779 for ; Wed, 6 Feb 2008 09:11:44 +0300 (MSK) Received: from [172.16.1.69] (unknown [212.45.28.21]) (Authenticated sender: lostar@rambler.ru) by maild.rambler.ru (Postfix) with ESMTP id 20C4984437 for ; Wed, 6 Feb 2008 09:11:43 +0300 (MSK) Message-Id: <6FBCC65B-734C-4CF5-9F44-2ECECBFD18EC@rambler.ru> From: Nicholas Kulikov To: freebsd-net@freebsd.org Mime-Version: 1.0 (Apple Message framework v915) Date: Wed, 6 Feb 2008 09:11:49 +0300 X-Mailer: Apple Mail (2.915) Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: 1 Gb network NFS issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Feb 2008 06:29:28 -0000 Hello all, I'm trying setup NFS in 1 Gb home network and found some strange issues: Current NFS performance is about 30-35 Mb/s but few days ago i had 40-45 Mb/s. I've made some changes in server configuration (moved to FreeBSD 6.3 and changed network driver to nfe) and I can't catch what is wrong... Network structure: NFS server (FreeBSD 6.3) | | 1 Gb link | NFS client (Mac OS X 10.5.1) Server configuration ----------------------------- nfe0: flags=8843 mtu 8000 options=48 Client configuration ------------------------ en0: flags=8863 mtu 8000 media: autoselect (100baseTX ) status: active client mount share by using the following parameters: tcp vers=3 rsize=40960 wsize=40960 readahead=16 rdirplus nolocks intr noatime I've tried copy big file (about 3 Gb) and tcdump-ed client and server packets: 07:11:08.350744 IP (tos 0x0, ttl 64, id 51461, offset 0, flags [DF], proto TCP (6), length 176) 192.168.254.249.nfs > 192.168.254.254.2468561782: reply ok 124 access attr: REG 660 ids 1001/1006 sz 3958224896 nlink 1 rdev 133/134414584 fsid 5e fileid 2013005 a/m/ctime 1199479211.000000 1199479280.000000 1201346449.000000 c 001f 07:11:08.350793 IP (tos 0x0, ttl 64, id 46719, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.254.49158 > 192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0xd9e5), 22844:22844(0) ack 27581 win 65535 07:11:08.350988 IP (tos 0x0, ttl 64, id 9526, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561783 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 0 07:11:08.351069 IP (tos 0x0, ttl 64, id 8470, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561784 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 32768 07:11:08.351145 IP (tos 0x0, ttl 64, id 65343, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561785 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 65536 07:11:08.351242 IP (tos 0x0, ttl 64, id 11022, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561786 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 98304 07:11:08.351314 IP (tos 0x0, ttl 64, id 59209, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561787 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 131072 07:11:08.351383 IP (tos 0x0, ttl 64, id 39733, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561788 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 163840 07:11:08.351450 IP (tos 0x0, ttl 64, id 33325, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561789 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 196608 07:11:08.351516 IP (tos 0x0, ttl 64, id 50481, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561790 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 229376 07:11:08.351582 IP (tos 0x0, ttl 64, id 50464, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561791 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 262144 07:11:08.351658 IP (tos 0x0, ttl 64, id 37215, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561792 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 294912 07:11:08.351739 IP (tos 0x0, ttl 64, id 6496, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561793 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 327680 07:11:08.351750 IP (tos 0x0, ttl 64, id 49495, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.249.nfsd > 192.168.254.254.49158: ., cksum 0x15e5 (correct), 27581:27581(0) ack 23076 win 49943 07:11:08.351754 IP (tos 0x0, ttl 64, id 36632, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.249.nfsd > 192.168.254.254.49158: ., cksum 0x1501 (correct), 27581:27581(0) ack 23308 win 49939 07:11:08.351758 IP (tos 0x0, ttl 64, id 21313, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.249.nfsd > 192.168.254.254.49158: ., cksum 0x141c (correct), 27581:27581(0) ack 23540 win 49936 07:11:08.351783 IP (tos 0x0, ttl 64, id 50759, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.249.nfsd > 192.168.254.254.49158: ., cksum 0x1338 (correct), 27581:27581(0) ack 23772 win 49932 07:11:08.351801 IP (tos 0x0, ttl 64, id 56942, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.249.nfsd > 192.168.254.254.49158: ., cksum 0x1254 (correct), 27581:27581(0) ack 24004 win 49928 07:11:08.351818 IP (tos 0x0, ttl 64, id 13319, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561794 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 360448 07:11:08.351904 IP (tos 0x0, ttl 64, id 17022, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561795 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 393216 07:11:08.351981 IP (tos 0x0, ttl 64, id 45600, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561796 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 425984 07:11:08.351999 IP (tos 0x0, ttl 64, id 31024, offset 0, flags [DF], proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.2468561783: reply ok 6144 read REG 660 ids 1001/1006 sz 3958224896 nlink 1 rdev 133/134414584 fsid 5e fileid 2013005 a/m/ ctime 1199479211.000000 1199479280.000000 1201346449.000000 32768 bytes 07:11:08.352014 IP (tos 0x0, ttl 64, id 13078, offset 0, flags [DF], proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.0: reply ok 6144 07:11:08.352074 IP (tos 0x0, ttl 64, id 9792, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561797 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 458752 07:11:08.352085 IP (tos 0x0, ttl 64, id 61207, offset 0, flags [DF], proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.0: reply ok 6144 07:11:08.352143 IP (tos 0x0, ttl 64, id 63816, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.254.49158 > 192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0x8b18), 24584:24584(0) ack 46013 win 65535 07:11:08.352162 IP (tos 0x0, ttl 64, id 60945, offset 0, flags [DF], proto TCP (6), length 52) 192.168.254.254.49158 > 192.168.254.249.nfsd: ., cksum 0x7f70 (incorrect (-> 0x8b18), 24584:24584(0) ack 46013 win 65535 07:11:08.352177 IP (tos 0x0, ttl 64, id 3701, offset 0, flags [DF], proto TCP (6), length 168) 192.168.254.254.2468561798 > 192.168.254.249.nfs: 116 read fh 1143,951878/33632261 32768 bytes @ 491520 07:11:08.352198 IP (tos 0x0, ttl 64, id 3073, offset 0, flags [DF], proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.3681401976: reply Unknown rpc response code=3722304989 6144 07:11:08.352203 IP (tos 0x0, ttl 64, id 2122, offset 0, flags [DF], proto TCP (6), length 6196) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply Unknown rpc response code=3722304989 6144 07:11:08.352207 IP (tos 0x0, ttl 64, id 61294, offset 0, flags [DF], proto TCP (6), length 2232) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply Unknown rpc response code=3722304989 2180 There are strange errors: 192.168.254.254.3681401976: reply Unknown rpc response Looks like server successfully sent first piece of data "07:11:08.351999...length 6196) 192.168.254.249.nfs > 192.168.254.254.2468561783: reply ok 6144...32768 bytes" and then report about errors in the following packets: length 6196) 192.168.254.249.nfs > 192.168.254.254.3681401976: reply Unknown rpc response code=3722304989 6144 length 6196) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply Unknown rpc response code=3722304989 6144 length 2232) 192.168.254.249.nfs > 192.168.254.254.3067833788: reply Unknown rpc response code=3722304989 2180 There is a lot of such errors in tcpdump logs. NFS3 specification do not contains this error code description... p.s. And why server is limited with read/write data size 32760? Is there a possibility to change it?