From owner-freebsd-current@FreeBSD.ORG Wed Nov 17 22:57:51 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB02A16A4FB for ; Wed, 17 Nov 2004 22:57:51 +0000 (GMT) Received: from mail.gmx.net (mail.gmx.net [213.165.64.20]) by mx1.FreeBSD.org (Postfix) with SMTP id 91D0243D2F for ; Wed, 17 Nov 2004 22:57:50 +0000 (GMT) (envelope-from Emanuel.Strobl@gmx.net) Received: (qmail 30150 invoked by uid 65534); 17 Nov 2004 22:57:49 -0000 Received: from flb.schmalzbauer.de (EHLO cale.flintsbach.schmalzbauer.de) (62.245.232.135) by mail.gmx.net (mp010) with SMTP; 17 Nov 2004 23:57:49 +0100 X-Authenticated: #301138 From: Emanuel Strobl To: freebsd-stable@freebsd.org Date: Wed, 17 Nov 2004 23:57:41 +0100 User-Agent: KMail/1.7 X-OS: FreeBSD X-Birthday: 10/06/72 X-Address: Munich, 80686 X-Tel: +49 89 18947781 X-CelPhone: +49 173 9967781 X-Country: Germany MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart8094321.VeYSp2vXmv"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200411172357.47735.Emanuel.Strobl@gmx.net> cc: freebsd-current@freebsd.org Subject: serious networking (em) performance (ggate and NFS) problem X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 17 Nov 2004 22:57:52 -0000 --nextPart8094321.VeYSp2vXmv Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Dear best guys, I really love 5.3 in many ways but here're some unbelievable transfer rates= ,=20 after I went out and bought a pair of Intel GigaBit Ethernet Cards to solve= =20 my performance problem (*laugh*): (In short, see *** below) Tests were done with two Intel GigaBit Ethernet cards (82547EI, 32bit PCI=20 Desktop adapter MT) connected directly without a switch/hub and "device=20 polling" compiled into a custom kernel with HZ set to 256 and=20 kern.polling.enabled set to "1": LOCAL: (/samsung is ufs2 on /dev/ad4p1, a SAMSUNG SP080N2) test3:~#7: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D16k ^C10524+0 records in 10524+0 records out 172425216 bytes transferred in 3.284735 secs (52492882 bytes/sec) =2D> ^^^^^^^^ ~ 52MB/s NFS(udp,polling): (/samsung is nfs on test3:/samsung, via em0, x-over, polling enabled) test2:/#21: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D16k ^C1858+0 records in 1857+0 records out 30425088 bytes transferred in 8.758475 secs (3473788 bytes/sec) =2D> ^^^^^^^ ~ 3,4MB/s This example shows that using NFS over GigaBit Ethernet decimates performan= ce=20 by the factor of 15, in words fifteen! GGATE with MTU 16114 and polling: test2:/dev#28: ggatec create 10.0.0.2 /dev/ad4p1 ggate0 test2:/dev#29: mount /dev/ggate0 /samsung/ test2:/dev#30: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D16k ^C2564+0 records in 2563+0 records out 41992192 bytes transferred in 15.908581 secs (2639594 bytes/sec) =2D> ^^^^^^^ ~ 2,6MB/s GGATE without polling and MTU 16114: test2:~#12: ggatec create 10.0.0.2 /dev/ad4p1 ggate0 test2:~#13: mount /dev/ggate0 /samsung/ test2:~#14: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D128k ^C1282+0 records in 1281+0 records out 167903232 bytes transferred in 11.274768 secs (14891945 bytes/sec) =2D> ^^^^^^^^ ~ 15MB/s =2E....and with 1m blocksize: test2:~#17: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D1m ^C61+0 records in 60+0 records out 62914560 bytes transferred in 4.608726 secs (13651182 bytes/sec) =2D> ^^^^^^^^ ~ 13,6MB/s I can't imagine why there seems to be a absolute limit of 15MB/s that can b= e=20 transfered over the network But it's even worse, here two excerpts of NFS (udp) with jumbo Frames=20 (mtu=3D16114): test2:~#23: mount 10.0.0.2:/samsung /samsung/ test2:~#24: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D1m ^C89+0 records in 88+0 records out 92274688 bytes transferred in 13.294708 secs (6940708 bytes/sec) =2D> ^^^^^^^ ~7MB/s =2E....and with 64k blocksize: test2:~#25: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D64k ^C848+0 records in 847+0 records out 55508992 bytes transferred in 8.063415 secs (6884055 bytes/sec) And with TCP-NFS (and Jumbo Frames): test2:~#30: mount_nfs -T 10.0.0.2:/samsung /samsung/ test2:~#31: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D64k ^C1921+0 records in 1920+0 records out 125829120 bytes transferred in 7.461226 secs (16864403 bytes/sec) =2D> ^^^^^^^^ ~ 17MB/s Again NFS (udp) but with MTU 1500: test2:~#9: mount_nfs 10.0.0.2:/samsung /samsung/ test2:~#10: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D8k ^C12020+0 records in 12019+0 records out 98459648 bytes transferred in 10.687460 secs (9212633 bytes/sec) =2D> ^^^^^^^ ~ 10MB/s And TCP-NFS with MTU 1500: test2:~#12: mount_nfs -T 10.0.0.2:/samsung /samsung/ test2:~#13: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D8k ^C19352+0 records in 19352+0 records out 158531584 bytes transferred in 12.093529 secs (13108794 bytes/sec) =2D> ^^^^^^^^ ~ 13MB/s GGATE with default MTU of 1500, polling disabled: test2:~#14: dd if=3D/dev/zero of=3D/samsung/testfile bs=3D64k ^C971+0 records in 970+0 records out 63569920 bytes transferred in 6.274578 secs (10131346 bytes/sec) =2D> ^^^^^^^^ ~ 10M/s Conclusion: *** =2D It seems that GEOM_GATE is less efficient with GigaBit (em) than NFS vi= a TCP=20 is. =2D em seems to have problems with MTU greater than 1500 =2D UDP seems to have performance disadvantages over TCP regarding NFS whic= h=20 should be vice versa AFAIK =2D polling and em (GbE) with HZ=3D256 is definitly no good idea, even 10Ba= se-2=20 can compete =2D NFS over TCP with MTU of 16114 gives the maximum transferrate for large= =20 files over GigaBit Ethernet with a value of 17MB/s, a quarter of what I'd=20 expect with my test equipment. =2D overall network performance (regarding large file transfers) is horrible Please, if anybody has the knowledge to dig into these problems, let me kno= w=20 if I can do any tests to help getting ggate and NFS useful in fast 5.3-stab= le=20 environments. Best regards, =2DHarry --nextPart8094321.VeYSp2vXmv Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQBBm9drBylq0S4AzzwRAljpAJ0VnUnFyfiS6xz2NyErHaS1mdnz0QCaAuyR wSvMpSbvASM/3eTX2OEQ3tU= =wF3T -----END PGP SIGNATURE----- --nextPart8094321.VeYSp2vXmv--