From owner-freebsd-net@FreeBSD.ORG Wed Aug 9 12:12:06 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 73F9816A4DF for ; Wed, 9 Aug 2006 12:12:06 +0000 (UTC) (envelope-from roshind@mail.ru) Received: from f60.mail.ru (f60.mail.ru [194.67.57.94]) by mx1.FreeBSD.org (Postfix) with ESMTP id F258943D6B for ; Wed, 9 Aug 2006 12:12:05 +0000 (GMT) (envelope-from roshind@mail.ru) Received: from mail by f60.mail.ru with local id 1GAmuq-000CR8-00 for freebsd-net@freebsd.org; Wed, 09 Aug 2006 16:12:04 +0400 Received: from [81.94.128.242] by win.mail.ru with HTTP; Wed, 09 Aug 2006 16:12:04 +0400 From: Dima Roshin To: freebsd-net@freebsd.org Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [81.94.128.242] Date: Wed, 09 Aug 2006 16:12:04 +0400 In-Reply-To: =?koi8-r?Q?<44D9CCEA.2050103=40ide.resurscentrum.se>?= Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Subject: Re[2]: Gigabit ethernet questions? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Dima Roshin List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Aug 2006 12:12:06 -0000 Thanks Jon, I did it on both sides, thats much better now: gate1# sysctl kern.polling.idle_poll=1 kern.polling.idle_poll: 0 -> 1 gate1# sysctl net.inet.ip.fastforwarding net.inet.ip.fastforwarding: 0 gate1# sysctl net.inet.ip.fastforwarding=1 net.inet.ip.fastforwarding: 0 -> 1 gate1# iperf -c 192.168.0.2 -N -w 196000 ------------------------------------------------------------ Client connecting to 192.168.0.2, TCP port 5001 TCP window size: 192 KByte (WARNING: requested 191 KByte) ------------------------------------------------------------ [ 3] local 192.168.0.1 port 63941 connected with 192.168.0.2 port 5001 [ 3] 0.0-10.0 sec 762 MBytes 639 Mbits/sec But there is still some bottleneck, and I can't understand where. -----Original Message----- From: Jon Otterholm To: Dima Roshin Date: Wed, 09 Aug 2006 13:54:18 +0200 Subject: Re: Gigabit ethernet questions? > > Dima Roshin wrote: > > Greeting colleagues. I've got two DL-360(pciX bus) servers, with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname is 6.1-RELEASE-p3. The bge interfaces of the both servers are connected with each other with a cat6 patchcord. > > Here are my settings: > > kernel config: > > options DEVICE_POLLING > > options HZ=1000 # > > > > sysctl.conf: > > kern.polling.enable=1 > > net.inet.ip.intr_queue_maxlen=5000 > > kern.ipc.maxsockbuf=8388608 > > net.inet.tcp.sendspace=3217968 > > net.inet.tcp.recvspace=3217968 > > net.inet.tcp.rfc1323=1 > > > > bge1: flags=8843 mtu 9000 > > options=5b > > inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 > > ether 00:17:a4:3a:e1:81 > > media: Ethernet autoselect (1000baseTX ) > > status: active > > (note mtu 9000) > > > > and here are tests results: > > > > netperf: > > > > TCP STREAM TEST to 192.168.0.1 > > Recv Send Send > > Socket Socket Message Elapsed > > Size Size Size Time Throughput > > bytes bytes bytes secs. 10^6bits/sec > > > > 6217968 6217968 6217968 10.22 320.04 > > > > UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1 > > Socket Message Elapsed Messages > > Size Size Time Okay Errors Throughput > > bytes bytes secs # # 10^6bits/sec > > > > 9216 9216 10.00 118851 1724281 876.20 > > 41600 10.00 0 0.00] > > > > > > > > iperf: > > gate2# iperf -s -N > > ------------------------------------------------------------ > > Server listening on TCP port 5001 > > TCP window size: 3.07 MByte (default) > > ------------------------------------------------------------ > > [ 4] local 192.168.0.2 port 5001 connected with 192.168.0.1 port 52597 > > [ 4] 0.0-10.1 sec 384 MBytes 319 Mbits/sec > > > > Also I can say, that I've managed to achieve about 500mbit.s by tuning tcp window with -w key in iperf. > > > > How can we explain such a low tcp performance? What else is to tune? Is there somebody who achieved gigabit speed with tcp on freebsd? > > > > Thanks. > > > > > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > You also need kern.polling.idle_poll=1 and maybe > net.inet.ip.fastforwarding=1 though I never noticed any difference with > that one enabled. I got about 840Mbit/s routed through a dell 1850 > (EMT64 running AMD64) with em-interfaces (I only used one physical IF > though with 2 VLAN-if). > >