Date: Tue, 20 Nov 2001 11:01:12 -0800 (PST) From: Alexander Haderer <alexander.haderer@charite.de> To: freebsd-gnats-submit@FreeBSD.org Subject: misc/32141: NFS V3/tcp server: low read performance Message-ID: <200111201901.fAKJ1C283740@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 32141
>Category: misc
>Synopsis: NFS V3/tcp server: low read performance
>Confidential: no
>Severity: non-critical
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Tue Nov 20 11:10:00 PST 2001
>Closed-Date:
>Last-Modified:
>Originator: Alexander Haderer
>Release: 4.2 Release, 4.3 Release 4.4 stable
>Organization:
Charite Berlin - Germany
>Environment:
Server:
FreeBSD xxx.charite.de 4.3-RELEASE FreeBSD 4.3-RELEASE #0: Tue Nov 20 13:11:00 CET 2001 root@marvinci.str.charite.de:/usr/src/sys/compile/GENERIC_MARVIN_NO_SCSI i386
FreeBSD yyy.charite.de 4.4-STABLE FreeBSD 4.4-STABLE #0: Tue Nov 20 11:39:34 CET 2001 root@greenaway.str.charite.de:/opt/src/sys/compile/GENERIC i386
Client:
FreeBSD zzz.charite.de 4.2-RELEASE FreeBSD 4.2-RELEASE #0: Mon Nov 20 13:02:55 GMT 2000 jkh@bento.FreeBSD.org:/usr/src/sys/compile/GENERIC i386
...
>Description:
Mounting a NFS volume with NFS V3/TCP from an FreeBSD 4.3+ server
gives low read transferrates (80 kbyte/s) when mounting with
FreeBSD 4.2/4.3 clients.
The server:
(Same effect with 4.3 Release / 4.4 Release / 4.4 stable)
A FreeBSD 4.3+ machine (x86) exports a filesystem.
The installation is out of the box with no tuning done. One
ethernet card is installed and works properly: tcpblast shows
the throughput one expects for an unloaded network (1 Mbyte/s or
11 Mbyte/s for 10/100 Mbit/s network in both directions).
The ethernet config (10/100 hdx/fdx) is done manually. Connection
to the clients is a) via 10Mbit Ethernet hub or b) via switched VLAN
100Mbit/fdx set manually. Neither the server nor the clients have CPU
or network load.
At client-side: "wire-test server" shows that the server talks
v2/v3 and udp/tcp in all 4 variants.
(A) FreeBSD 4.2/4.3 Release Client, mount without options:
Mounting the exported filesystem with
"mount server:/export/scratch /mnt"
and copying a large file (e.g. 3MB generic kernel) from remote
to local filesystem shows the performace for reading
(cp /mnt/file /usr/tmp) and writing is as expected:
about 70% .. 90% of tcpdumps bandwitdh; writing slower
than reading; copying files that were not just copyied before
because of cache effects; no tcpdump running.
Using ftp instead of cp to get the file gives similar results.
(B) FreeBSD 4.2/4.3 Release Client, mount with options:
Mounting the exported filesystem with
"mount_nfs -3 -T server:/export/scratch /mnt"
forces NFS v3 over TCP. Now copying a large file shows that
performace for writing is just as before. But performance for
reading breaks down to 80 kbyte/s, regardless if we use 10Mbit
or 100Mbit network. Switching down to nfs v2 tcp brings no change.
Using ftp instead of cp to get the file gives max. transferrate.
BUT changing the server OS to 4.2 Release gives us the performance
expected, too.
(C) Solaris 2.6 client, mount without options:
mounting the filesystem without options gives performance as
expected (r/w).
(D) Solaris 2.6 client, mount with options:
Forcing NFS v3 TCP gives no change: good performance. Setting
rsize also gives:
32768 = good performance
smaller = goes downto 20%
Things tried out / checked / not appropriate here:
- odd network routing config: no, our's fine, other protocols
work
- wrong fdx/hdx settings: ok here: everything set manually and tested
with tcpdump and ftp'ing files around
- high cpu / network load, lot of clients: no clients, no load here
- /kernel reports some buffer trouble in /var/log/messages:
no messages here.
- multiplatform with odd NFS implemenentations: no, FreeBSD only
- black box network equipment: nope, only a switch or stupid
10 mbit HUB between client/server
- unreliable network (wlan) or dialin network: no, ethernet only
- old HW: no, P500+ with PCI Ethernetcards 3com 3c905ctx
- other HW conflicts: no, VGA, Ethernet, Onboard IDE, ser+par only
- broken ipfw settings: no ipfw here
- rc.conf: tcp_extensions YES/NO plus server reboot: no change
- Changing the Server from 4.4 Release to 4.4 Stable: no change
- make all traffic go via stupid 10mbit hub to avoid vlan/switch:
no change
- mismatch because of incomplete cvsup: no, Release setup from CDROM
Please feel free to contact me for detailed debug output, session logs,
tcpdumps, etc...
>How-To-Repeat:
A FreeBSD 4.3 Release / 4.4 Release NFS server exports a filesystem,
a 4.{2,3} Release client mounts the filesystem with
'mount_nfs -3 -T server:/export/filesys /mnt'
At the client: copying big files from the server to the client
gives 80kbyte/s tranfserrate while writing to the server goes fast.
This problem also appears with amd using autonegotiation or forcing
amd to use v3/tcp (this way we discovered the 80kbyte/s reading).
>Fix:
When mounting the remote FS via NFS v3 TCP also set the rsize
within the mount command:
'mount_nfs -3 -T -r 4096 server:/export/filesys /mnt'
>Release-Note:
>Audit-Trail:
>Unformatted:
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-bugs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200111201901.fAKJ1C283740>