From owner-freebsd-net@FreeBSD.ORG Thu Aug 10 05:57:06 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 92B8416A4DF for ; Thu, 10 Aug 2006 05:57:06 +0000 (UTC) (envelope-from mv@thebeastie.org) Received: from p4.roq.com (ns1.ecoms.com [207.44.130.137]) by mx1.FreeBSD.org (Postfix) with ESMTP id 28CF443D46 for ; Thu, 10 Aug 2006 05:57:06 +0000 (GMT) (envelope-from mv@thebeastie.org) Received: from p4.roq.com (localhost.roq.com [127.0.0.1]) by p4.roq.com (Postfix) with ESMTP id C47A94CE86 for ; Thu, 10 Aug 2006 05:57:49 +0000 (GMT) Received: from vaulte.jumbuck.com (ppp166-27.static.internode.on.net [150.101.166.27]) by p4.roq.com (Postfix) with ESMTP id D6C1B4CE85 for ; Thu, 10 Aug 2006 05:57:48 +0000 (GMT) Received: from vaulte.jumbuck.com (localhost [127.0.0.1]) by vaulte.jumbuck.com (Postfix) with ESMTP id 7B5C98A069; Thu, 10 Aug 2006 15:57:03 +1000 (EST) Received: from [192.168.46.102] (unknown [192.168.46.250]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by vaulte.jumbuck.com (Postfix) with ESMTP id 76DCB8A037; Thu, 10 Aug 2006 15:57:03 +1000 (EST) Message-ID: <44DACAAF.5020306@thebeastie.org> Date: Thu, 10 Aug 2006 15:57:03 +1000 From: Michael Vince User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.7.13) Gecko/20060727 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Dima Roshin References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-net@freebsd.org Subject: Re: Gigabit ethernet questions? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Aug 2006 05:57:06 -0000 JICUDN, I have been using nc,dd and systat to check TCP performance on my servers, the good thing about it is it requires little setup and gives results fast. For example on host A start the nc server. nc -4kl 3000 > /dev/null Then start another one on hostb sending data via nc from /dev/zero cat /dev/zero | dd bs=1m | nc hosta 3000 ^C0+226041 records in 0+226040 records out 14813757440 bytes transferred in 246.184967 secs (60173282 bytes/sec) You can take out the dd and just use systat -if depending on what you like. Mike Dima Roshin wrote: >Thanks Jon, I did it on both sides, thats much better now: > >gate1# sysctl kern.polling.idle_poll=1 >kern.polling.idle_poll: 0 -> 1 >gate1# sysctl net.inet.ip.fastforwarding >net.inet.ip.fastforwarding: 0 >gate1# sysctl net.inet.ip.fastforwarding=1 >net.inet.ip.fastforwarding: 0 -> 1 >gate1# iperf -c 192.168.0.2 -N -w 196000 >------------------------------------------------------------ >Client connecting to 192.168.0.2, TCP port 5001 >TCP window size: 192 KByte (WARNING: requested 191 KByte) >------------------------------------------------------------ >[ 3] local 192.168.0.1 port 63941 connected with 192.168.0.2 port 5001 >[ 3] 0.0-10.0 sec 762 MBytes 639 Mbits/sec > >But there is still some bottleneck, and I can't understand where. > > > >-----Original Message----- >From: Jon Otterholm >To: Dima Roshin >Date: Wed, 09 Aug 2006 13:54:18 +0200 >Subject: Re: Gigabit ethernet questions? > > > >>Dima Roshin wrote: >> >> >>> Greeting colleagues. I've got two DL-360(pciX bus) servers, with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname is 6.1-RELEASE-p3. The bge interfaces of the both servers are connected with each other with a cat6 patchcord. >>> Here are my settings: >>>kernel config: >>>options DEVICE_POLLING >>>options HZ=1000 # >>> >>>sysctl.conf: >>>kern.polling.enable=1 >>>net.inet.ip.intr_queue_maxlen=5000 >>>kern.ipc.maxsockbuf=8388608 >>>net.inet.tcp.sendspace=3217968 >>>net.inet.tcp.recvspace=3217968 >>>net.inet.tcp.rfc1323=1 >>> >>>bge1: flags=8843 mtu 9000 >>> options=5b >>> inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 >>> ether 00:17:a4:3a:e1:81 >>> media: Ethernet autoselect (1000baseTX ) >>> status: active >>>(note mtu 9000) >>> >>>and here are tests results: >>> >>>netperf: >>> >>>TCP STREAM TEST to 192.168.0.1 >>>Recv Send Send >>>Socket Socket Message Elapsed >>>Size Size Size Time Throughput >>>bytes bytes bytes secs. 10^6bits/sec >>> >>>6217968 6217968 6217968 10.22 320.04 >>> >>>UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1 >>>Socket Message Elapsed Messages >>>Size Size Time Okay Errors Throughput >>>bytes bytes secs # # 10^6bits/sec >>> >>> 9216 9216 10.00 118851 1724281 876.20 >>>41600 10.00 0 0.00] >>> >>> >>> >>>iperf: >>>gate2# iperf -s -N >>>------------------------------------------------------------ >>>Server listening on TCP port 5001 >>>TCP window size: 3.07 MByte (default) >>>------------------------------------------------------------ >>>[ 4] local 192.168.0.2 port 5001 connected with 192.168.0.1 port 52597 >>>[ 4] 0.0-10.1 sec 384 MBytes 319 Mbits/sec >>> >>>Also I can say, that I've managed to achieve about 500mbit.s by tuning tcp window with -w key in iperf. >>> >>>How can we explain such a low tcp performance? What else is to tune? Is there somebody who achieved gigabit speed with tcp on freebsd? >>> >>>Thanks. >>> >>> >>> >>>_______________________________________________ >>>freebsd-net@freebsd.org mailing list >>>http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >>> >>> >>You also need kern.polling.idle_poll=1 and maybe >>net.inet.ip.fastforwarding=1 though I never noticed any difference with >>that one enabled. I got about 840Mbit/s routed through a dell 1850 >>(EMT64 running AMD64) with em-interfaces (I only used one physical IF >>though with 2 VLAN-if). >> >> >> >> >_______________________________________________ >freebsd-net@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-net >To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >