From owner-freebsd-net@FreeBSD.ORG Fri Aug 11 18:04:36 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F00D316A4DE for ; Fri, 11 Aug 2006 18:04:35 +0000 (UTC) (envelope-from davidch@broadcom.com) Received: from MMS3.broadcom.com (mms3.broadcom.com [216.31.210.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5B87F43D6E for ; Fri, 11 Aug 2006 18:04:34 +0000 (GMT) (envelope-from davidch@broadcom.com) Received: from 10.10.64.154 by MMS3.broadcom.com with ESMTP (Broadcom SMTP Relay (Email Firewall v6.2.0)); Fri, 11 Aug 2006 11:04:20 -0700 X-Server-Uuid: B238DE4C-2139-4D32-96A8-DD564EF2313E Received: by mail-irva-10.broadcom.com (Postfix, from userid 47) id BA8242AE; Fri, 11 Aug 2006 11:04:20 -0700 (PDT) Received: from mail-irva-8.broadcom.com (mail-irva-8 [10.10.64.221]) by mail-irva-10.broadcom.com (Postfix) with ESMTP id 67A082AE; Fri, 11 Aug 2006 11:04:20 -0700 (PDT) Received: from mail-irva-12.broadcom.com (mail-irva-12.broadcom.com [10.10.64.146]) by mail-irva-8.broadcom.com (MOS 3.7.5a-GA) with ESMTP id ECI18952; Fri, 11 Aug 2006 11:04:19 -0700 (PDT) Received: from NT-IRVA-0750.brcm.ad.broadcom.com (nt-irva-0750 [10.8.194.64]) by mail-irva-12.broadcom.com (Postfix) with ESMTP id DFBA569CA4; Fri, 11 Aug 2006 11:04:19 -0700 (PDT) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Fri, 11 Aug 2006 11:04:19 -0700 Message-ID: <09BFF2FA5EAB4A45B6655E151BBDD90301B7B6B9@NT-IRVA-0750.brcm.ad.broadcom.com> In-Reply-To: Thread-Topic: Gigabit ethernet questions? Thread-Index: Aca7qB2iA0jVLfN2ShSQlECDAkgv/gBxjm0Q From: "David Christensen" To: "Dima Roshin" , freebsd-net@freebsd.org X-TMWD-Spam-Summary: SEV=1.1; DFV=A2006081106; IFV=2.0.6,4.0-7; RPD=4.00.0004; RPDID=303030312E30413031303230342E34344443433545442E303034382D412D; ENG=IBF; TS=20060811180422; CAT=NONE; CON=NONE; X-MMS-Spam-Filter-ID: A2006081106_4.00.0004_2.0.6,4.0-7 X-WSS-ID: 68C2192E1OO12185780-01-01 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Cc: Subject: RE: Gigabit ethernet questions? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Aug 2006 18:04:36 -0000 > Greeting colleagues. I've got two DL-360(pciX bus) servers,=20 > with BCM5704 NetXtreme Dual Gigabit Adapters(bge). The Uname=20 > is 6.1-RELEASE-p3. The bge interfaces of the both servers are=20 > connected with each other with a cat6 patchcord. > Here are my settings: > kernel config: > options DEVICE_POLLING=20 > options HZ=3D1000 # >=20 > sysctl.conf: > kern.polling.enable=3D1 > net.inet.ip.intr_queue_maxlen=3D5000 > kern.ipc.maxsockbuf=3D8388608 > net.inet.tcp.sendspace=3D3217968 > net.inet.tcp.recvspace=3D3217968 > net.inet.tcp.rfc1323=3D1 >=20 > bge1: flags=3D8843 mtu 9000 > options=3D5b > inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255 > ether 00:17:a4:3a:e1:81 > media: Ethernet autoselect (1000baseTX ) > status: active > (note mtu 9000) >=20 > and here are tests results: >=20 > netperf: >=20 > TCP STREAM TEST to 192.168.0.1 > Recv Send Send > Socket Socket Message Elapsed > Size Size Size Time Throughput > bytes bytes bytes secs. 10^6bits/sec >=20 > 6217968 6217968 6217968 10.22 320.04 >=20 > UDP UNIDIRECTIONAL SEND TEST to 192.168.0.1 > Socket Message Elapsed Messages > Size Size Time Okay Errors Throughput > bytes bytes secs # # 10^6bits/sec >=20 > 9216 9216 10.00 118851 1724281 876.20 > 41600 10.00 0 0.00] >=20 >=20 >=20 > iperf: > gate2# iperf -s -N > ------------------------------------------------------------ > Server listening on TCP port 5001 > TCP window size: 3.07 MByte (default) > ------------------------------------------------------------ > [ 4] local 192.168.0.2 port 5001 connected with 192.168.0.1=20 > port 52597 > [ 4] 0.0-10.1 sec 384 MBytes 319 Mbits/sec >=20 > Also I can say, that I've managed to achieve about 500mbit.s=20 > by tuning tcp window with -w key in iperf. >=20 > How can we explain such a low tcp performance? What else is=20 > to tune? Is there somebody who achieved gigabit speed with=20 > tcp on freebsd? You're test is non-optimal for the 5704 since the ports are linked=20 together back-to-back. In a dual port configuration such as the 5704 each port must arbitrate for access to the PCI bus. Due to an errata for the 5704, the BGE_PCIDMAWCTL_ONEDMA_ATONCE bit is set which allows only one port access to the PCI bus at a time for the duration of the DMA transaction, rather than allowing the two ports to interleave DMAs. Your test configuration is a worst case scenario since both ports are=20 active in both directions at the same time. If you can change your=20 test to use a second system you should see the TCP performance rise substantially. Dave