Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 20 Feb 2008 12:23:56 +0100
From:      "Valerio Daelli" <valerio.daelli@gmail.com>
To:        "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org>
Subject:   Re: Bad performance of 7.0 nfs client with Solaris nfs server
Message-ID:  <27dbfc8c0802200323r13f69905l4940d0d5accd1eb1@mail.gmail.com>
In-Reply-To: <47BB33AD.1050005@FreeBSD.org>
References:  <27dbfc8c0802190243y113d3059yd0c602850a4dbd6b@mail.gmail.com> <47BB33AD.1050005@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Feb 19, 2008 8:53 PM, Kris Kennaway <kris@freebsd.org> wrote:
>
> Valerio Daelli wrote:
> > Hi list
> >
> > we have a FreeBSD 7.0 NFS client (csup today, built world and kernel).
> > It mounts a Solaris 10 NFS share.
> > We have bad performance with 7.0 (3MB/s).
> > We have tried both UDP and TCP mounts, both sync and async.
> > This is our mount:
> >
> > nest.xx.xx:/data/export/hosts/bsd7.xx.xx/ /mnt/nest.xx.xx nfs
> > noatime,async,-i,rw,-T,-3
> >
> > Both our server (7.0 and Solaris 10) are Gigabit Ethernet, both are HP
> > Proliant DL360 i386 (NIC bge0):
> >
> > ---
> > FreeBSD 7.0:
> >
> > [eldon@bsd7 ~]$ uname -a
> > FreeBSD bsd7.xx.xx 7.0-RC2 FreeBSD 7.0-RC2 #1: Mon Feb 18 17:46:46 CET
> > 2008     root@bsd7.xx.xx:/usr/obj/usr/src/sys/BSD7  i386
> >
> > ---
> > This is our performance with iozone:
> > command line:
> >
> > iozone -+q 1 -i 0 -i 1 -n 2048 -g 2G -Raceb iozone.xls -f
> > /mnt/nest.xx.xx/iozone.bsd7/iozone.tmp
> >
> > FreeBSD 7:
> >
> >       File stride size set to 17 * record size.
> >                                                             random
> > random    bkwd  record  stride
> >               KB  reclen   write rewrite    read    reread    read
> > write    read rewrite    read   fwrite frewrite   fread  freread
> >             2048    1024  109883  101289   769058   779880
> >             2048    2048    3812    3674   760479   767066
> >             4096    1024  111156  106788   724692   728040
> >             4096    2048    3336    2241   157132   733417
> >             4096    4096    2829    3364   699351   699807
> >
> >
> > As you can see, while with record length less than 1024KB the speed is
> > 'fast', with record of 2048 or more (I've tried with record much
> > bigger) we get only
> > 3 MB/s.
> > Is this a known issue? If you need more details please contact me, I
> > am willing to do more tests to risolve this problem.
>
> Can you characterize what is different about the NFS traffic when
> FreeBSD and Solaris are the clients?  Does other network traffic to the
> server perform well?  2048 bytes records are > the 1500 byte MTU, so you
> will be invoking packet fragmentation and reassembly.  Maybe you are
> getting packet loss for some reason.  What does netstat say about
> interface errors on the bge, and for the protocol layers?
>
> Follow-ups set to performance@
>
> Kris
>

Hi
thanks for responding. I did a couple of test on 7.0, and I've not
seen error on interfaces.
Attached are netstat -s ad netstat -i.


TCP mount
---
root@bsd7:~ netstat -i
Name    Mtu Network       Address              Ipkts Ierrs    Opkts Oerrs  Coll
bge0   1500 <Link#1>      00:0b:cd:37:45:e8    45444     0   110188     0     0
bge0   1500 xx.xx.xx.xx bsd7                 45427     -   110183     -     -
---tcp:
	1536 packets sent
		396 data packets (68516 bytes)
		0 data packets (0 bytes) retransmitted
		0 data packets unnecessarily retransmitted
		0 resends initiated by MTU discovery
		1021 ack-only packets (6 delayed)
		0 URG only packets
		0 window probe packets
		101 window update packets
		18 control packets
	2555 packets received
		413 acks (for 68487 bytes)
		1 duplicate ack
		0 acks for unsent data
		2173 packets (763594 bytes) received in-sequence
		0 completely duplicate packets (0 bytes)
		0 old duplicate packets
		0 packets with some dup. data (0 bytes duped)
		0 out-of-order packets (0 bytes)
		0 packets (0 bytes) of data after window
		0 window probes
		0 window update packets
		7 packets received after close
		0 discarded for bad checksums
		0 discarded for bad header offset fields
		0 discarded because packet too short
		0 discarded due to memory problems
	10 connection requests
	1 connection accept
	0 bad connection attempts
	0 listen queue overflows
	0 ignored RSTs in the windows
	11 connections established (including accepts)
	10 connections closed (including 1 drop)
		7 connections updated cached RTT on close
		7 connections updated cached RTT variance on close
		0 connections updated cached ssthresh on close
	0 embryonic connections dropped
	413 segments updated rtt (of 358 attempts)
	0 retransmit timeouts
		0 connections dropped by rexmit timeout
	0 persist timeouts
		0 connections dropped by persist timeout
	0 Connections (fin_wait_2) dropped because of timeout
	0 keepalive timeouts
		0 keepalive probes sent
		0 connections dropped by keepalive
	0 correct ACK header predictions
	2122 correct data packet header predictions
	1 syncache entrie added
		0 retransmitted
		0 dupsyn
		0 dropped
		1 completed
		0 bucket overflow
		0 cache overflow
		0 reset
		0 stale
		0 aborted
		0 badack
		0 unreach
		0 zone failures
	1 cookie sent
	0 cookies received
	0 SACK recovery episodes
	0 segment rexmits in SACK recovery episodes
	0 byte rexmits in SACK recovery episodes
	0 SACK options (SACK blocks) received
	0 SACK options (SACK blocks) sent
	0 SACK scoreboard overflow
[...]
ip:
	99904 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	61441 fragments received
	0 fragments dropped (dup or out of space)
	1 fragment dropped after timeout
	10240 packets reassembled ok
	48703 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	0 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	47693 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	34819 output datagrams fragmented
	208914 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header

[...]


UDP mount
---
root@bsd7:~ netstat -i
Name    Mtu Network       Address              Ipkts Ierrs    Opkts Oerrs  Coll
bge0   1500 <Link#1>      00:0b:cd:37:45:e8    63685     1   119978     0     0
bge0   1500 xx.xx.xx.xx bsd7                 63628     -   119973     -     -
---
udp:
	25880 datagrams received
	0 with incomplete header
	0 with bad data length field
	0 with bad checksum
	0 with no checksum
	0 dropped due to no socket
	7 broadcast/multicast datagrams undelivered
	0 dropped due to full socket buffers
	0 not for hashed pcb
	25873 delivered
	25882 datagrams output
	0 times multicast source filter matched
[...]
ip:
	62379 total packets received
	0 bad header checksums
	0 with size smaller than minimum
	0 with data size < data length
	0 with ip length > max ip packet size
	0 with header length < data size
	0 with data length < header length
	0 with bad options
	0 with incorrect version number
	41473 fragments received
	0 fragments dropped (dup or out of space)
	1 fragment dropped after timeout
	6912 packets reassembled ok
	27818 packets for this host
	0 packets for unknown/unsupported protocol
	0 packets forwarded (0 packets fast forwarded)
	0 packets not forwardable
	0 packets received for unknown multicast group
	0 redirects sent
	27056 packets sent from this host
	0 packets sent with fabricated ip header
	0 output packets dropped due to no bufs, etc.
	0 output packets discarded due to no route
	18438 output datagrams fragmented
	110628 fragments created
	0 datagrams that can't be fragmented
	0 tunneling packets that can't find gif
	0 datagrams with bad address in header
[...]

Then I tried with different versions of FreeBSD and I discovered that:
-FreeBSD 6.2-p3 has good performance with reclen 2048 and bad with
4096 and 8192:

---
root@litio:~ uname -a
FreeBSD litio.xx.xx 6.2-RELEASE-p3 FreeBSD 6.2-RELEASE-p3 #1: Thu Mar
15 14:52:08 CET 2007     root@litio.xx.xx:/usr/obj/usr/src/sys/LITIO
amd64

TCP
              KB  reclen   write rewrite    read    reread    read
write    read rewrite    read   fwrite frewrite   fread  freread
            2048    1024   90041   97352    82001  1019428
            2048    2048   87030   88329  1009366  1053046
            4096    1024   96139   95218    74186  1032306
            4096    2048  103038   99700    79950  1003190
            4096    4096    2897    3354   970821   522038
            8192    1024   97978  100020    80681   946699
            8192    2048   95260  104290    82395  1032127
            8192    4096    3300    3169   149505   860882
            8192    8192    3371    3090   998300  1012987
           16384    1024  100072   98446    80503   962737
           16384    2048   99360  105249    80983   924453
           16384    4096    3162    3173   106750   936178
           16384    8192    3233    3223   150394   998913
           16384   16384    3250    3316   955004   944153

UDP similar


-FreeBSD 6.2-p4 and 6.2-p6 has the same performance as 7.0:
---

root@xtl:~ uname -a
FreeBSD xtl.xx.xx 6.2-RELEASE-p6 FreeBSD 6.2-RELEASE-p6 #0: Wed Jul 25
14:32:07 CEST 2007     root@xtl.xx.xx:/usr/obj/usr/src/sys/XTL  i386

TCP
                                                            random
random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read
write    read rewrite    read   fwrite frewrite   fread  freread
            2048    1024   15544   14560   846597   841044
            2048    2048    2769    2649   852225   851211
            4096    1024   17366   16851   833229   835417
            4096    2048    2777    2457    49038   841226
            4096    4096    2527    2813   847868   847492




root@rubidio:~ uname -a
FreeBSD rubidio.xx.xx 6.2-RELEASE-p4 FreeBSD 6.2-RELEASE-p4 #0: Tue
May  8 11:24:10 CEST 2007
root@rubidio.xx.xx:/usr/obj/usr/src/sys/PROXYCARP1  amd64

TCP

                                                            random
random    bkwd  record  stride
              KB  reclen   write rewrite    read    reread    read
write    read rewrite    read   fwrite frewrite   fread  freread
            2048    1024   93010   89366   996997   970635
            2048    2048   94740   90820   943347   946048
            4096    2048  100036   94668    81065   927548
            4096    4096    3274    3319  1074493  1047309
            8192    2048  100990   98212    80574  1054050
            8192    4096    3078    3071   149890   990428
            8192    8192    3074    3055   992029   995391



---
Now I will try with a Solaris client and with rsync as protocol.
Other suggestions?
Bye

Valerio Daelli



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?27dbfc8c0802200323r13f69905l4940d0d5accd1eb1>