Date: Wed, 20 Feb 2008 12:23:56 +0100 From: "Valerio Daelli" <valerio.daelli@gmail.com> To: "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org> Subject: Re: Bad performance of 7.0 nfs client with Solaris nfs server Message-ID: <27dbfc8c0802200323r13f69905l4940d0d5accd1eb1@mail.gmail.com> In-Reply-To: <47BB33AD.1050005@FreeBSD.org> References: <27dbfc8c0802190243y113d3059yd0c602850a4dbd6b@mail.gmail.com> <47BB33AD.1050005@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Feb 19, 2008 8:53 PM, Kris Kennaway <kris@freebsd.org> wrote:
>
> Valerio Daelli wrote:
> > Hi list
> >
> > we have a FreeBSD 7.0 NFS client (csup today, built world and kernel).
> > It mounts a Solaris 10 NFS share.
> > We have bad performance with 7.0 (3MB/s).
> > We have tried both UDP and TCP mounts, both sync and async.
> > This is our mount:
> >
> > nest.xx.xx:/data/export/hosts/bsd7.xx.xx/ /mnt/nest.xx.xx nfs
> > noatime,async,-i,rw,-T,-3
> >
> > Both our server (7.0 and Solaris 10) are Gigabit Ethernet, both are HP
> > Proliant DL360 i386 (NIC bge0):
> >
> > ---
> > FreeBSD 7.0:
> >
> > [eldon@bsd7 ~]$ uname -a
> > FreeBSD bsd7.xx.xx 7.0-RC2 FreeBSD 7.0-RC2 #1: Mon Feb 18 17:46:46 CET
> > 2008 root@bsd7.xx.xx:/usr/obj/usr/src/sys/BSD7 i386
> >
> > ---
> > This is our performance with iozone:
> > command line:
> >
> > iozone -+q 1 -i 0 -i 1 -n 2048 -g 2G -Raceb iozone.xls -f
> > /mnt/nest.xx.xx/iozone.bsd7/iozone.tmp
> >
> > FreeBSD 7:
> >
> > File stride size set to 17 * record size.
> > random
> > random bkwd record stride
> > KB reclen write rewrite read reread read
> > write read rewrite read fwrite frewrite fread freread
> > 2048 1024 109883 101289 769058 779880
> > 2048 2048 3812 3674 760479 767066
> > 4096 1024 111156 106788 724692 728040
> > 4096 2048 3336 2241 157132 733417
> > 4096 4096 2829 3364 699351 699807
> >
> >
> > As you can see, while with record length less than 1024KB the speed is
> > 'fast', with record of 2048 or more (I've tried with record much
> > bigger) we get only
> > 3 MB/s.
> > Is this a known issue? If you need more details please contact me, I
> > am willing to do more tests to risolve this problem.
>
> Can you characterize what is different about the NFS traffic when
> FreeBSD and Solaris are the clients? Does other network traffic to the
> server perform well? 2048 bytes records are > the 1500 byte MTU, so you
> will be invoking packet fragmentation and reassembly. Maybe you are
> getting packet loss for some reason. What does netstat say about
> interface errors on the bge, and for the protocol layers?
>
> Follow-ups set to performance@
>
> Kris
>
Hi
thanks for responding. I did a couple of test on 7.0, and I've not
seen error on interfaces.
Attached are netstat -s ad netstat -i.
TCP mount
---
root@bsd7:~ netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
bge0 1500 <Link#1> 00:0b:cd:37:45:e8 45444 0 110188 0 0
bge0 1500 xx.xx.xx.xx bsd7 45427 - 110183 - -
---tcp:
1536 packets sent
396 data packets (68516 bytes)
0 data packets (0 bytes) retransmitted
0 data packets unnecessarily retransmitted
0 resends initiated by MTU discovery
1021 ack-only packets (6 delayed)
0 URG only packets
0 window probe packets
101 window update packets
18 control packets
2555 packets received
413 acks (for 68487 bytes)
1 duplicate ack
0 acks for unsent data
2173 packets (763594 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
0 old duplicate packets
0 packets with some dup. data (0 bytes duped)
0 out-of-order packets (0 bytes)
0 packets (0 bytes) of data after window
0 window probes
0 window update packets
7 packets received after close
0 discarded for bad checksums
0 discarded for bad header offset fields
0 discarded because packet too short
0 discarded due to memory problems
10 connection requests
1 connection accept
0 bad connection attempts
0 listen queue overflows
0 ignored RSTs in the windows
11 connections established (including accepts)
10 connections closed (including 1 drop)
7 connections updated cached RTT on close
7 connections updated cached RTT variance on close
0 connections updated cached ssthresh on close
0 embryonic connections dropped
413 segments updated rtt (of 358 attempts)
0 retransmit timeouts
0 connections dropped by rexmit timeout
0 persist timeouts
0 connections dropped by persist timeout
0 Connections (fin_wait_2) dropped because of timeout
0 keepalive timeouts
0 keepalive probes sent
0 connections dropped by keepalive
0 correct ACK header predictions
2122 correct data packet header predictions
1 syncache entrie added
0 retransmitted
0 dupsyn
0 dropped
1 completed
0 bucket overflow
0 cache overflow
0 reset
0 stale
0 aborted
0 badack
0 unreach
0 zone failures
1 cookie sent
0 cookies received
0 SACK recovery episodes
0 segment rexmits in SACK recovery episodes
0 byte rexmits in SACK recovery episodes
0 SACK options (SACK blocks) received
0 SACK options (SACK blocks) sent
0 SACK scoreboard overflow
[...]
ip:
99904 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
61441 fragments received
0 fragments dropped (dup or out of space)
1 fragment dropped after timeout
10240 packets reassembled ok
48703 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
47693 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
34819 output datagrams fragmented
208914 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
[...]
UDP mount
---
root@bsd7:~ netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
bge0 1500 <Link#1> 00:0b:cd:37:45:e8 63685 1 119978 0 0
bge0 1500 xx.xx.xx.xx bsd7 63628 - 119973 - -
---
udp:
25880 datagrams received
0 with incomplete header
0 with bad data length field
0 with bad checksum
0 with no checksum
0 dropped due to no socket
7 broadcast/multicast datagrams undelivered
0 dropped due to full socket buffers
0 not for hashed pcb
25873 delivered
25882 datagrams output
0 times multicast source filter matched
[...]
ip:
62379 total packets received
0 bad header checksums
0 with size smaller than minimum
0 with data size < data length
0 with ip length > max ip packet size
0 with header length < data size
0 with data length < header length
0 with bad options
0 with incorrect version number
41473 fragments received
0 fragments dropped (dup or out of space)
1 fragment dropped after timeout
6912 packets reassembled ok
27818 packets for this host
0 packets for unknown/unsupported protocol
0 packets forwarded (0 packets fast forwarded)
0 packets not forwardable
0 packets received for unknown multicast group
0 redirects sent
27056 packets sent from this host
0 packets sent with fabricated ip header
0 output packets dropped due to no bufs, etc.
0 output packets discarded due to no route
18438 output datagrams fragmented
110628 fragments created
0 datagrams that can't be fragmented
0 tunneling packets that can't find gif
0 datagrams with bad address in header
[...]
Then I tried with different versions of FreeBSD and I discovered that:
-FreeBSD 6.2-p3 has good performance with reclen 2048 and bad with
4096 and 8192:
---
root@litio:~ uname -a
FreeBSD litio.xx.xx 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #1: Thu Mar
15 14:52:08 CET 2007 root@litio.xx.xx:/usr/obj/usr/src/sys/LITIO
amd64
TCP
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
2048 1024 90041 97352 82001 1019428
2048 2048 87030 88329 1009366 1053046
4096 1024 96139 95218 74186 1032306
4096 2048 103038 99700 79950 1003190
4096 4096 2897 3354 970821 522038
8192 1024 97978 100020 80681 946699
8192 2048 95260 104290 82395 1032127
8192 4096 3300 3169 149505 860882
8192 8192 3371 3090 998300 1012987
16384 1024 100072 98446 80503 962737
16384 2048 99360 105249 80983 924453
16384 4096 3162 3173 106750 936178
16384 8192 3233 3223 150394 998913
16384 16384 3250 3316 955004 944153
UDP similar
-FreeBSD 6.2-p4 and 6.2-p6 has the same performance as 7.0:
---
root@xtl:~ uname -a
FreeBSD xtl.xx.xx 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #0: Wed Jul 25
14:32:07 CEST 2007 root@xtl.xx.xx:/usr/obj/usr/src/sys/XTL i386
TCP
random
random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
2048 1024 15544 14560 846597 841044
2048 2048 2769 2649 852225 851211
4096 1024 17366 16851 833229 835417
4096 2048 2777 2457 49038 841226
4096 4096 2527 2813 847868 847492
root@rubidio:~ uname -a
FreeBSD rubidio.xx.xx 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #0: Tue
May 8 11:24:10 CEST 2007
root@rubidio.xx.xx:/usr/obj/usr/src/sys/PROXYCARP1 amd64
TCP
random
random bkwd record stride
KB reclen write rewrite read reread read
write read rewrite read fwrite frewrite fread freread
2048 1024 93010 89366 996997 970635
2048 2048 94740 90820 943347 946048
4096 2048 100036 94668 81065 927548
4096 4096 3274 3319 1074493 1047309
8192 2048 100990 98212 80574 1054050
8192 4096 3078 3071 149890 990428
8192 8192 3074 3055 992029 995391
---
Now I will try with a Solaris client and with rsync as protocol.
Other suggestions?
Bye
Valerio Daelli
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?27dbfc8c0802200323r13f69905l4940d0d5accd1eb1>