Date: Wed, 17 Nov 2004 23:57:41 +0100 From: Emanuel Strobl <Emanuel.Strobl@gmx.net> To: freebsd-stable@freebsd.org Cc: freebsd-current@freebsd.org Subject: serious networking (em) performance (ggate and NFS) problem Message-ID: <200411172357.47735.Emanuel.Strobl@gmx.net>
next in thread | raw e-mail | index | archive | help
--nextPart8094321.VeYSp2vXmv
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Dear best guys,
I really love 5.3 in many ways but here're some unbelievable transfer rates=
,=20
after I went out and bought a pair of Intel GigaBit Ethernet Cards to solve=
=20
my performance problem (*laugh*):
(In short, see *** below)
Tests were done with two Intel GigaBit Ethernet cards (82547EI, 32bit PCI=20
Desktop adapter MT) connected directly without a switch/hub and "device=20
polling" compiled into a custom kernel with HZ set to 256 and=20
kern.polling.enabled set to "1":
LOCAL:
(/samsung is ufs2 on /dev/ad4p1, a SAMSUNG SP080N2)
test3:~#7: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D16k
^C10524+0 records in
10524+0 records out
172425216 bytes transferred in 3.284735 secs (52492882 bytes/sec)
=2D>
^^^^^^^^ ~ 52MB/s
NFS(udp,polling):
(/samsung is nfs on test3:/samsung, via em0, x-over, polling enabled)
test2:/#21: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D16k
^C1858+0 records in
1857+0 records out
30425088 bytes transferred in 8.758475 secs (3473788 bytes/sec)
=2D> ^^^^^^^ ~ 3,4MB/s
This example shows that using NFS over GigaBit Ethernet decimates performan=
ce=20
by the factor of 15, in words fifteen!
GGATE with MTU 16114 and polling:
test2:/dev#28: ggatec create 10.0.0.2 /dev/ad4p1
ggate0
test2:/dev#29: mount /dev/ggate0 /samsung/
test2:/dev#30: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D16k
^C2564+0 records in
2563+0 records out
41992192 bytes transferred in 15.908581 secs (2639594 bytes/sec)
=2D> ^^^^^^^ ~ 2,6MB/s
GGATE without polling and MTU 16114:
test2:~#12: ggatec create 10.0.0.2 /dev/ad4p1
ggate0
test2:~#13: mount /dev/ggate0 /samsung/
test2:~#14: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D128k
^C1282+0 records in
1281+0 records out
167903232 bytes transferred in 11.274768 secs (14891945 bytes/sec)
=2D> ^^^^^^^^ ~ 15MB/s
=2E....and with 1m blocksize:
test2:~#17: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D1m
^C61+0 records in
60+0 records out
62914560 bytes transferred in 4.608726 secs (13651182 bytes/sec)
=2D> ^^^^^^^^ ~ 13,6MB/s
I can't imagine why there seems to be a absolute limit of 15MB/s that can b=
e=20
transfered over the network
But it's even worse, here two excerpts of NFS (udp) with jumbo Frames=20
(mtu=3D16114):
test2:~#23: mount 10.0.0.2:/samsung /samsung/
test2:~#24: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D1m
^C89+0 records in
88+0 records out
92274688 bytes transferred in 13.294708 secs (6940708 bytes/sec)
=2D> ^^^^^^^ ~7MB/s
=2E....and with 64k blocksize:
test2:~#25: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D64k
^C848+0 records in
847+0 records out
55508992 bytes transferred in 8.063415 secs (6884055 bytes/sec)
And with TCP-NFS (and Jumbo Frames):
test2:~#30: mount_nfs -T 10.0.0.2:/samsung /samsung/
test2:~#31: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D64k
^C1921+0 records in
1920+0 records out
125829120 bytes transferred in 7.461226 secs (16864403 bytes/sec)
=2D> ^^^^^^^^ ~ 17MB/s
Again NFS (udp) but with MTU 1500:
test2:~#9: mount_nfs 10.0.0.2:/samsung /samsung/
test2:~#10: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D8k
^C12020+0 records in
12019+0 records out
98459648 bytes transferred in 10.687460 secs (9212633 bytes/sec)
=2D> ^^^^^^^ ~ 10MB/s
And TCP-NFS with MTU 1500:
test2:~#12: mount_nfs -T 10.0.0.2:/samsung /samsung/
test2:~#13: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D8k
^C19352+0 records in
19352+0 records out
158531584 bytes transferred in 12.093529 secs (13108794 bytes/sec)
=2D> ^^^^^^^^ ~ 13MB/s
GGATE with default MTU of 1500, polling disabled:
test2:~#14: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D64k
^C971+0 records in
970+0 records out
63569920 bytes transferred in 6.274578 secs (10131346 bytes/sec)
=2D> ^^^^^^^^ ~ 10M/s
Conclusion:
***
=2D It seems that GEOM_GATE is less efficient with GigaBit (em) than NFS vi=
a TCP=20
is.
=2D em seems to have problems with MTU greater than 1500
=2D UDP seems to have performance disadvantages over TCP regarding NFS whic=
h=20
should be vice versa AFAIK
=2D polling and em (GbE) with HZ=3D256 is definitly no good idea, even 10Ba=
se-2=20
can compete
=2D NFS over TCP with MTU of 16114 gives the maximum transferrate for large=
=20
files over GigaBit Ethernet with a value of 17MB/s, a quarter of what I'd=20
expect with my test equipment.
=2D overall network performance (regarding large file transfers) is horrible
Please, if anybody has the knowledge to dig into these problems, let me kno=
w=20
if I can do any tests to help getting ggate and NFS useful in fast 5.3-stab=
le=20
environments.
Best regards,
=2DHarry
--nextPart8094321.VeYSp2vXmv
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (FreeBSD)
iD8DBQBBm9drBylq0S4AzzwRAljpAJ0VnUnFyfiS6xz2NyErHaS1mdnz0QCaAuyR
wSvMpSbvASM/3eTX2OEQ3tU=
=wF3T
-----END PGP SIGNATURE-----
--nextPart8094321.VeYSp2vXmv--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200411172357.47735.Emanuel.Strobl>