Date: Wed, 20 Feb 2008 07:06:13 -0600 From: Eric Anderson <anderson@freebsd.org> To: Valerio Daelli <valerio.daelli@gmail.com> Cc: "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org> Subject: Re: Bad performance of 7.0 nfs client with Solaris nfs server Message-ID: <47BC25C5.1000300@freebsd.org> In-Reply-To: <27dbfc8c0802200323r13f69905l4940d0d5accd1eb1@mail.gmail.com> References: <27dbfc8c0802190243y113d3059yd0c602850a4dbd6b@mail.gmail.com> <47BB33AD.1050005@FreeBSD.org> <27dbfc8c0802200323r13f69905l4940d0d5accd1eb1@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Valerio Daelli wrote: > On Feb 19, 2008 8:53 PM, Kris Kennaway <kris@freebsd.org> wrote: >> Valerio Daelli wrote: >>> Hi list >>> >>> we have a FreeBSD 7.0 NFS client (csup today, built world and kernel). >>> It mounts a Solaris 10 NFS share. >>> We have bad performance with 7.0 (3MB/s). >>> We have tried both UDP and TCP mounts, both sync and async. >>> This is our mount: >>> >>> nest.xx.xx:/data/export/hosts/bsd7.xx.xx/ /mnt/nest.xx.xx nfs >>> noatime,async,-i,rw,-T,-3 >>> >>> Both our server (7.0 and Solaris 10) are Gigabit Ethernet, both are HP >>> Proliant DL360 i386 (NIC bge0): >>> >>> --- >>> FreeBSD 7.0: >>> >>> [eldon@bsd7 ~]$ uname -a >>> FreeBSD bsd7.xx.xx 7.0-RC2 FreeBSD 7.0-RC2 #1: Mon Feb 18 17:46:46 CET >>> 2008 root@bsd7.xx.xx:/usr/obj/usr/src/sys/BSD7 i386 >>> >>> --- >>> This is our performance with iozone: >>> command line: >>> >>> iozone -+q 1 -i 0 -i 1 -n 2048 -g 2G -Raceb iozone.xls -f >>> /mnt/nest.xx.xx/iozone.bsd7/iozone.tmp >>> >>> FreeBSD 7: >>> >>> File stride size set to 17 * record size. >>> random >>> random bkwd record stride >>> KB reclen write rewrite read reread read >>> write read rewrite read fwrite frewrite fread freread >>> 2048 1024 109883 101289 769058 779880 >>> 2048 2048 3812 3674 760479 767066 >>> 4096 1024 111156 106788 724692 728040 >>> 4096 2048 3336 2241 157132 733417 >>> 4096 4096 2829 3364 699351 699807 >>> >>> >>> As you can see, while with record length less than 1024KB the speed is >>> 'fast', with record of 2048 or more (I've tried with record much >>> bigger) we get only >>> 3 MB/s. >>> Is this a known issue? If you need more details please contact me, I >>> am willing to do more tests to risolve this problem. >> Can you characterize what is different about the NFS traffic when >> FreeBSD and Solaris are the clients? Does other network traffic to the >> server perform well? 2048 bytes records are > the 1500 byte MTU, so you >> will be invoking packet fragmentation and reassembly. Maybe you are >> getting packet loss for some reason. What does netstat say about >> interface errors on the bge, and for the protocol layers? >> >> Follow-ups set to performance@ >> >> Kris >> > > Hi > thanks for responding. I did a couple of test on 7.0, and I've not > seen error on interfaces. > Attached are netstat -s ad netstat -i. > > > TCP mount > --- > root@bsd7:~ netstat -i > Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll > bge0 1500 <Link#1> 00:0b:cd:37:45:e8 45444 0 110188 0 0 > bge0 1500 xx.xx.xx.xx bsd7 45427 - 110183 - - > ---tcp: > 1536 packets sent > 396 data packets (68516 bytes) > 0 data packets (0 bytes) retransmitted > 0 data packets unnecessarily retransmitted > 0 resends initiated by MTU discovery > 1021 ack-only packets (6 delayed) > 0 URG only packets > 0 window probe packets > 101 window update packets > 18 control packets > 2555 packets received > 413 acks (for 68487 bytes) > 1 duplicate ack > 0 acks for unsent data > 2173 packets (763594 bytes) received in-sequence > 0 completely duplicate packets (0 bytes) > 0 old duplicate packets > 0 packets with some dup. data (0 bytes duped) > 0 out-of-order packets (0 bytes) > 0 packets (0 bytes) of data after window > 0 window probes > 0 window update packets > 7 packets received after close > 0 discarded for bad checksums > 0 discarded for bad header offset fields > 0 discarded because packet too short > 0 discarded due to memory problems > 10 connection requests > 1 connection accept > 0 bad connection attempts > 0 listen queue overflows > 0 ignored RSTs in the windows > 11 connections established (including accepts) > 10 connections closed (including 1 drop) > 7 connections updated cached RTT on close > 7 connections updated cached RTT variance on close > 0 connections updated cached ssthresh on close > 0 embryonic connections dropped > 413 segments updated rtt (of 358 attempts) > 0 retransmit timeouts > 0 connections dropped by rexmit timeout > 0 persist timeouts > 0 connections dropped by persist timeout > 0 Connections (fin_wait_2) dropped because of timeout > 0 keepalive timeouts > 0 keepalive probes sent > 0 connections dropped by keepalive > 0 correct ACK header predictions > 2122 correct data packet header predictions > 1 syncache entrie added > 0 retransmitted > 0 dupsyn > 0 dropped > 1 completed > 0 bucket overflow > 0 cache overflow > 0 reset > 0 stale > 0 aborted > 0 badack > 0 unreach > 0 zone failures > 1 cookie sent > 0 cookies received > 0 SACK recovery episodes > 0 segment rexmits in SACK recovery episodes > 0 byte rexmits in SACK recovery episodes > 0 SACK options (SACK blocks) received > 0 SACK options (SACK blocks) sent > 0 SACK scoreboard overflow > [...] > ip: > 99904 total packets received > 0 bad header checksums > 0 with size smaller than minimum > 0 with data size < data length > 0 with ip length > max ip packet size > 0 with header length < data size > 0 with data length < header length > 0 with bad options > 0 with incorrect version number > 61441 fragments received > 0 fragments dropped (dup or out of space) > 1 fragment dropped after timeout > 10240 packets reassembled ok > 48703 packets for this host > 0 packets for unknown/unsupported protocol > 0 packets forwarded (0 packets fast forwarded) > 0 packets not forwardable > 0 packets received for unknown multicast group > 0 redirects sent > 47693 packets sent from this host > 0 packets sent with fabricated ip header > 0 output packets dropped due to no bufs, etc. > 0 output packets discarded due to no route > 34819 output datagrams fragmented > 208914 fragments created > 0 datagrams that can't be fragmented > 0 tunneling packets that can't find gif > 0 datagrams with bad address in header > > [...] > > > UDP mount > --- > root@bsd7:~ netstat -i > Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll > bge0 1500 <Link#1> 00:0b:cd:37:45:e8 63685 1 119978 0 0 > bge0 1500 xx.xx.xx.xx bsd7 63628 - 119973 - - > --- > udp: > 25880 datagrams received > 0 with incomplete header > 0 with bad data length field > 0 with bad checksum > 0 with no checksum > 0 dropped due to no socket > 7 broadcast/multicast datagrams undelivered > 0 dropped due to full socket buffers > 0 not for hashed pcb > 25873 delivered > 25882 datagrams output > 0 times multicast source filter matched > [...] > ip: > 62379 total packets received > 0 bad header checksums > 0 with size smaller than minimum > 0 with data size < data length > 0 with ip length > max ip packet size > 0 with header length < data size > 0 with data length < header length > 0 with bad options > 0 with incorrect version number > 41473 fragments received > 0 fragments dropped (dup or out of space) > 1 fragment dropped after timeout > 6912 packets reassembled ok > 27818 packets for this host > 0 packets for unknown/unsupported protocol > 0 packets forwarded (0 packets fast forwarded) > 0 packets not forwardable > 0 packets received for unknown multicast group > 0 redirects sent > 27056 packets sent from this host > 0 packets sent with fabricated ip header > 0 output packets dropped due to no bufs, etc. > 0 output packets discarded due to no route > 18438 output datagrams fragmented > 110628 fragments created > 0 datagrams that can't be fragmented > 0 tunneling packets that can't find gif > 0 datagrams with bad address in header > [...] > > Then I tried with different versions of FreeBSD and I discovered that: > -FreeBSD 6.2-p3 has good performance with reclen 2048 and bad with > 4096 and 8192: > > --- > root@litio:~ uname -a > FreeBSD litio.xx.xx 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #1: Thu Mar > 15 14:52:08 CET 2007 root@litio.xx.xx:/usr/obj/usr/src/sys/LITIO > amd64 > > TCP > KB reclen write rewrite read reread read > write read rewrite read fwrite frewrite fread freread > 2048 1024 90041 97352 82001 1019428 > 2048 2048 87030 88329 1009366 1053046 > 4096 1024 96139 95218 74186 1032306 > 4096 2048 103038 99700 79950 1003190 > 4096 4096 2897 3354 970821 522038 > 8192 1024 97978 100020 80681 946699 > 8192 2048 95260 104290 82395 1032127 > 8192 4096 3300 3169 149505 860882 > 8192 8192 3371 3090 998300 1012987 > 16384 1024 100072 98446 80503 962737 > 16384 2048 99360 105249 80983 924453 > 16384 4096 3162 3173 106750 936178 > 16384 8192 3233 3223 150394 998913 > 16384 16384 3250 3316 955004 944153 > > UDP similar > > > -FreeBSD 6.2-p4 and 6.2-p6 has the same performance as 7.0: > --- > > root@xtl:~ uname -a > FreeBSD xtl.xx.xx 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #0: Wed Jul 25 > 14:32:07 CEST 2007 root@xtl.xx.xx:/usr/obj/usr/src/sys/XTL i386 > > TCP > random > random bkwd record stride > KB reclen write rewrite read reread read > write read rewrite read fwrite frewrite fread freread > 2048 1024 15544 14560 846597 841044 > 2048 2048 2769 2649 852225 851211 > 4096 1024 17366 16851 833229 835417 > 4096 2048 2777 2457 49038 841226 > 4096 4096 2527 2813 847868 847492 > > > > > root@rubidio:~ uname -a > FreeBSD rubidio.xx.xx 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #0: Tue > May 8 11:24:10 CEST 2007 > root@rubidio.xx.xx:/usr/obj/usr/src/sys/PROXYCARP1 amd64 > > TCP > > random > random bkwd record stride > KB reclen write rewrite read reread read > write read rewrite read fwrite frewrite fread freread > 2048 1024 93010 89366 996997 970635 > 2048 2048 94740 90820 943347 946048 > 4096 2048 100036 94668 81065 927548 > 4096 4096 3274 3319 1074493 1047309 > 8192 2048 100990 98212 80574 1054050 > 8192 4096 3078 3071 149890 990428 > 8192 8192 3074 3055 992029 995391 > > > > --- > Now I will try with a Solaris client and with rsync as protocol. > Other suggestions? > Bye > If possible, it might help a lot to send the output (or a snippet of it) or post it somewhere on the web, of a tshark output from the client. You might try UDP mounting the NFS export also, just to try it. Eric
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?47BC25C5.1000300>