From owner-freebsd-performance@FreeBSD.ORG Wed Feb 20 11:48:49 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBFAE16A406 for ; Wed, 20 Feb 2008 11:48:49 +0000 (UTC) (envelope-from valerio.daelli@gmail.com) Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.234]) by mx1.freebsd.org (Postfix) with ESMTP id 927BF13C478 for ; Wed, 20 Feb 2008 11:48:49 +0000 (UTC) (envelope-from valerio.daelli@gmail.com) Received: by wr-out-0506.google.com with SMTP id 68so2104512wri.3 for ; Wed, 20 Feb 2008 03:48:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=bxGDfQ0d79iN7KGSkZKEl3AKTsvxYgB2mBcIGjsNPqs=; b=rPxU/8jjN3HeFbrqTMknWc28pp7rVDTMSm8kWEmxgQ7BJQ3McVkRQ+tfwTPGbAI5O6CQb99KW4WuU6WN0XDvugMT9kAUDmAaxB7LBTVLjrRjcnRYU/UOYBHhMfJgd0yrAKF7ItJldfc7JuV93KdHoX9bL05XyUbhMYBu55jNs6g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=x6CCjUp/9PIPfF/n/4UXWGZ6uV3vO2fllx4476APMd5pSvKrsgJRd7BrGORfsZUXBT2GZricEivJXYJH+cgHMQYrmZpRPMg33+ki1Tf8imRSF2FrGi1RgeKy6Q73oIlPd2ieOCw9/dBWOJcIx8nwV6joKHFyrqHfi1Iq6CvwX8w= Received: by 10.143.31.4 with SMTP id i4mr4347682wfj.87.1203506636334; Wed, 20 Feb 2008 03:23:56 -0800 (PST) Received: by 10.143.158.2 with HTTP; Wed, 20 Feb 2008 03:23:56 -0800 (PST) Message-ID: <27dbfc8c0802200323r13f69905l4940d0d5accd1eb1@mail.gmail.com> Date: Wed, 20 Feb 2008 12:23:56 +0100 From: "Valerio Daelli" To: "freebsd-performance@freebsd.org" In-Reply-To: <47BB33AD.1050005@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <27dbfc8c0802190243y113d3059yd0c602850a4dbd6b@mail.gmail.com> <47BB33AD.1050005@FreeBSD.org> Subject: Re: Bad performance of 7.0 nfs client with Solaris nfs server X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Feb 2008 11:48:50 -0000 On Feb 19, 2008 8:53 PM, Kris Kennaway wrote: > > Valerio Daelli wrote: > > Hi list > > > > we have a FreeBSD 7.0 NFS client (csup today, built world and kernel). > > It mounts a Solaris 10 NFS share. > > We have bad performance with 7.0 (3MB/s). > > We have tried both UDP and TCP mounts, both sync and async. > > This is our mount: > > > > nest.xx.xx:/data/export/hosts/bsd7.xx.xx/ /mnt/nest.xx.xx nfs > > noatime,async,-i,rw,-T,-3 > > > > Both our server (7.0 and Solaris 10) are Gigabit Ethernet, both are HP > > Proliant DL360 i386 (NIC bge0): > > > > --- > > FreeBSD 7.0: > > > > [eldon@bsd7 ~]$ uname -a > > FreeBSD bsd7.xx.xx 7.0-RC2 FreeBSD 7.0-RC2 #1: Mon Feb 18 17:46:46 CET > > 2008 root@bsd7.xx.xx:/usr/obj/usr/src/sys/BSD7 i386 > > > > --- > > This is our performance with iozone: > > command line: > > > > iozone -+q 1 -i 0 -i 1 -n 2048 -g 2G -Raceb iozone.xls -f > > /mnt/nest.xx.xx/iozone.bsd7/iozone.tmp > > > > FreeBSD 7: > > > > File stride size set to 17 * record size. > > random > > random bkwd record stride > > KB reclen write rewrite read reread read > > write read rewrite read fwrite frewrite fread freread > > 2048 1024 109883 101289 769058 779880 > > 2048 2048 3812 3674 760479 767066 > > 4096 1024 111156 106788 724692 728040 > > 4096 2048 3336 2241 157132 733417 > > 4096 4096 2829 3364 699351 699807 > > > > > > As you can see, while with record length less than 1024KB the speed is > > 'fast', with record of 2048 or more (I've tried with record much > > bigger) we get only > > 3 MB/s. > > Is this a known issue? If you need more details please contact me, I > > am willing to do more tests to risolve this problem. > > Can you characterize what is different about the NFS traffic when > FreeBSD and Solaris are the clients? Does other network traffic to the > server perform well? 2048 bytes records are > the 1500 byte MTU, so you > will be invoking packet fragmentation and reassembly. Maybe you are > getting packet loss for some reason. What does netstat say about > interface errors on the bge, and for the protocol layers? > > Follow-ups set to performance@ > > Kris > Hi thanks for responding. I did a couple of test on 7.0, and I've not seen error on interfaces. Attached are netstat -s ad netstat -i. TCP mount --- root@bsd7:~ netstat -i Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll bge0 1500 00:0b:cd:37:45:e8 45444 0 110188 0 0 bge0 1500 xx.xx.xx.xx bsd7 45427 - 110183 - - ---tcp: 1536 packets sent 396 data packets (68516 bytes) 0 data packets (0 bytes) retransmitted 0 data packets unnecessarily retransmitted 0 resends initiated by MTU discovery 1021 ack-only packets (6 delayed) 0 URG only packets 0 window probe packets 101 window update packets 18 control packets 2555 packets received 413 acks (for 68487 bytes) 1 duplicate ack 0 acks for unsent data 2173 packets (763594 bytes) received in-sequence 0 completely duplicate packets (0 bytes) 0 old duplicate packets 0 packets with some dup. data (0 bytes duped) 0 out-of-order packets (0 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 7 packets received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 0 discarded due to memory problems 10 connection requests 1 connection accept 0 bad connection attempts 0 listen queue overflows 0 ignored RSTs in the windows 11 connections established (including accepts) 10 connections closed (including 1 drop) 7 connections updated cached RTT on close 7 connections updated cached RTT variance on close 0 connections updated cached ssthresh on close 0 embryonic connections dropped 413 segments updated rtt (of 358 attempts) 0 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections dropped by persist timeout 0 Connections (fin_wait_2) dropped because of timeout 0 keepalive timeouts 0 keepalive probes sent 0 connections dropped by keepalive 0 correct ACK header predictions 2122 correct data packet header predictions 1 syncache entrie added 0 retransmitted 0 dupsyn 0 dropped 1 completed 0 bucket overflow 0 cache overflow 0 reset 0 stale 0 aborted 0 badack 0 unreach 0 zone failures 1 cookie sent 0 cookies received 0 SACK recovery episodes 0 segment rexmits in SACK recovery episodes 0 byte rexmits in SACK recovery episodes 0 SACK options (SACK blocks) received 0 SACK options (SACK blocks) sent 0 SACK scoreboard overflow [...] ip: 99904 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 61441 fragments received 0 fragments dropped (dup or out of space) 1 fragment dropped after timeout 10240 packets reassembled ok 48703 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 47693 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 34819 output datagrams fragmented 208914 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header [...] UDP mount --- root@bsd7:~ netstat -i Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll bge0 1500 00:0b:cd:37:45:e8 63685 1 119978 0 0 bge0 1500 xx.xx.xx.xx bsd7 63628 - 119973 - - --- udp: 25880 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 with no checksum 0 dropped due to no socket 7 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 25873 delivered 25882 datagrams output 0 times multicast source filter matched [...] ip: 62379 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 41473 fragments received 0 fragments dropped (dup or out of space) 1 fragment dropped after timeout 6912 packets reassembled ok 27818 packets for this host 0 packets for unknown/unsupported protocol 0 packets forwarded (0 packets fast forwarded) 0 packets not forwardable 0 packets received for unknown multicast group 0 redirects sent 27056 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 18438 output datagrams fragmented 110628 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header [...] Then I tried with different versions of FreeBSD and I discovered that: -FreeBSD 6.2-p3 has good performance with reclen 2048 and bad with 4096 and 8192: --- root@litio:~ uname -a FreeBSD litio.xx.xx 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #1: Thu Mar 15 14:52:08 CET 2007 root@litio.xx.xx:/usr/obj/usr/src/sys/LITIO amd64 TCP KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 2048 1024 90041 97352 82001 1019428 2048 2048 87030 88329 1009366 1053046 4096 1024 96139 95218 74186 1032306 4096 2048 103038 99700 79950 1003190 4096 4096 2897 3354 970821 522038 8192 1024 97978 100020 80681 946699 8192 2048 95260 104290 82395 1032127 8192 4096 3300 3169 149505 860882 8192 8192 3371 3090 998300 1012987 16384 1024 100072 98446 80503 962737 16384 2048 99360 105249 80983 924453 16384 4096 3162 3173 106750 936178 16384 8192 3233 3223 150394 998913 16384 16384 3250 3316 955004 944153 UDP similar -FreeBSD 6.2-p4 and 6.2-p6 has the same performance as 7.0: --- root@xtl:~ uname -a FreeBSD xtl.xx.xx 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #0: Wed Jul 25 14:32:07 CEST 2007 root@xtl.xx.xx:/usr/obj/usr/src/sys/XTL i386 TCP random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 2048 1024 15544 14560 846597 841044 2048 2048 2769 2649 852225 851211 4096 1024 17366 16851 833229 835417 4096 2048 2777 2457 49038 841226 4096 4096 2527 2813 847868 847492 root@rubidio:~ uname -a FreeBSD rubidio.xx.xx 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #0: Tue May 8 11:24:10 CEST 2007 root@rubidio.xx.xx:/usr/obj/usr/src/sys/PROXYCARP1 amd64 TCP random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 2048 1024 93010 89366 996997 970635 2048 2048 94740 90820 943347 946048 4096 2048 100036 94668 81065 927548 4096 4096 3274 3319 1074493 1047309 8192 2048 100990 98212 80574 1054050 8192 4096 3078 3071 149890 990428 8192 8192 3074 3055 992029 995391 --- Now I will try with a Solaris client and with rsync as protocol. Other suggestions? Bye Valerio Daelli