Date: Fri, 11 Aug 2006 11:04:19 -0700 From: "David Christensen" <davidch@broadcom.com> To: "Dima Roshin" <roshind@mail.ru>, freebsd-net@freebsd.org Subject: RE: Gigabit ethernet questions? Message-ID: <09BFF2FA5EAB4A45B6655E151BBDD90301B7B6B9@NT-IRVA-0750.brcm.ad.broadcom.com> In-Reply-To: <E1GAmM1-000I0z-00.roshind-mail-ru@f49.mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
> Greeting colleagues. I've got two DL-360(pciX bus) servers,=20 > with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname=20 > is 6.1-RELEASE-p3. The bge interfaces of the both servers are=20 > connected with each other with a cat6 patchcord. > Here are my settings: > kernel config: > options DEVICE_POLLING=20 > options HZ=3D1000 # >=20 > sysctl.conf: > kern.polling.enable=3D1 > net.inet.ip.intr_queue_maxlen=3D5000 > kern.ipc.maxsockbuf=3D8388608 > net.inet.tcp.sendspace=3D3217968 > net.inet.tcp.recvspace=3D3217968 > net.inet.tcp.rfc1323=3D1 >=20 > bge1: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000 > options=3D5b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING> > inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 > ether 00:17:a4:3a:e1:81 > media: Ethernet autoselect (1000baseTX <full-duplex>) > status: active > (note mtu 9000) >=20 > and here are tests results: >=20 > netperf: >=20 > TCP STREAM TEST to 192.168.0.1 > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec >=20 > 6217968 6217968 6217968 10.22 320.04 >=20 > UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1 > Socket Message Elapsed Messages > Size Size Time Okay Errors Throughput > bytes bytes secs # # 10^6bits/sec >=20 > 9216 9216 10.00 118851 1724281 876.20 > 41600 10.00 0 0.00] >=20 >=20 >=20 > iperf: > gate2# iperf -s -N > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 3.07 MByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.0.2 port 5001 connected with 192.168.0.1=20 > port 52597 > [ 4] 0.0-10.1 sec 384 MBytes 319 Mbits/sec >=20 > Also I can say, that I've managed to achieve about 500mbit.s=20 > by tuning tcp window with -w key in iperf. >=20 > How can we explain such a low tcp performance? What else is=20 > to tune? Is there somebody who achieved gigabit speed with=20 > tcp on freebsd? You're test is non-optimal for the 5704 since the ports are linked=20 together back-to-back. In a dual port configuration such as the 5704 each port must arbitrate for access to the PCI bus. Due to an errata for the 5704, the BGE_PCIDMAWCTL_ONEDMA_ATONCE bit is set which allows only one port access to the PCI bus at a time for the duration of the DMA transaction, rather than allowing the two ports to interleave DMAs. Your test configuration is a worst case scenario since both ports are=20 active in both directions at the same time. If you can change your=20 test to use a second system you should see the TCP performance rise substantially. Dave
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?09BFF2FA5EAB4A45B6655E151BBDD90301B7B6B9>