Date: Fri, 11 Aug 2006 11:04:19 -0700 From: "David Christensen" <davidch@broadcom.com> To: "Dima Roshin" <roshind@mail.ru>, freebsd-net@freebsd.org Subject: RE: Gigabit ethernet questions? Message-ID: <09BFF2FA5EAB4A45B6655E151BBDD90301B7B6B9@NT-IRVA-0750.brcm.ad.broadcom.com> In-Reply-To: <E1GAmM1-000I0z-00.roshind-mail-ru@f49.mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
> Greeting colleagues. I've got two DL-360(pciX bus) servers, > with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname > is 6.1-RELEASE-p3. The bge interfaces of the both servers are > connected with each other with a cat6 patchcord. > Here are my settings: > kernel config: > options DEVICE_POLLING > options HZ=1000 # > > sysctl.conf: > kern.polling.enable=1 > net.inet.ip.intr_queue_maxlen=5000 > kern.ipc.maxsockbuf=8388608 > net.inet.tcp.sendspace=3217968 > net.inet.tcp.recvspace=3217968 > net.inet.tcp.rfc1323=1 > > bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 9000 > options=5b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,POLLING> > inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 > ether 00:17:a4:3a:e1:81 > media: Ethernet autoselect (1000baseTX <full-duplex>) > status: active > (note mtu 9000) > > and here are tests results: > > netperf: > > TCP STREAM TEST to 192.168.0.1 > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec > > 6217968 6217968 6217968 10.22 320.04 > > UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1 > Socket Message Elapsed Messages > Size Size Time Okay Errors Throughput > bytes bytes secs # # 10^6bits/sec > > 9216 9216 10.00 118851 1724281 876.20 > 41600 10.00 0 0.00] > > > > iperf: > gate2# iperf -s -N > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 3.07 MByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.0.2 port 5001 connected with 192.168.0.1 > port 52597 > [ 4] 0.0-10.1 sec 384 MBytes 319 Mbits/sec > > Also I can say, that I've managed to achieve about 500mbit.s > by tuning tcp window with -w key in iperf. > > How can we explain such a low tcp performance? What else is > to tune? Is there somebody who achieved gigabit speed with > tcp on freebsd? You're test is non-optimal for the 5704 since the ports are linked together back-to-back. In a dual port configuration such as the 5704 each port must arbitrate for access to the PCI bus. Due to an errata for the 5704, the BGE_PCIDMAWCTL_ONEDMA_ATONCE bit is set which allows only one port access to the PCI bus at a time for the duration of the DMA transaction, rather than allowing the two ports to interleave DMAs. Your test configuration is a worst case scenario since both ports are active in both directions at the same time. If you can change your test to use a second system you should see the TCP performance rise substantially. Dave
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?09BFF2FA5EAB4A45B6655E151BBDD90301B7B6B9>