From owner-freebsd-net@FreeBSD.ORG Tue May 22 03:06:27 2007 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D3FA716A46C for ; Tue, 22 May 2007 03:06:27 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: from ug-out-1314.google.com (ug-out-1314.google.com [66.249.92.172]) by mx1.freebsd.org (Postfix) with ESMTP id 4750413C484 for ; Tue, 22 May 2007 03:06:27 +0000 (UTC) (envelope-from kip.macy@gmail.com) Received: by ug-out-1314.google.com with SMTP id 78so92197ugn for ; Mon, 21 May 2007 20:06:26 -0700 (PDT) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=lmdNdch3h0lMYHtmuckwecjio67cCDK4dL6B5ntsweNQS+xO1zAjvVJXe0dlM7wSHm0VHvUbvizCuRSgNAHuv1/1N+dFM4+SMru+J7uFQiyb9u1flKXq2JAUiSsPkysG1erwdq6tm9W8gDMP0G/7W94nCuVZECdS0/SWP6B1VFQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=KPX3cnpbv4RujYykgFQdE9Le5HCE8kFqc4XPQ9j9rwnLGlzj+vxImk7ERxy4oxUvhwq1ULLrqRdAKysN5GD+9Gryc7va0FQR4/fVj+7w+vccbwnrCDZxZqtJWTrAIY9DFiN0w5GSWa5ERWkptJZe67wmvAyeThP4s5W3pgFqvDE= Received: by 10.78.171.20 with SMTP id t20mr1294799hue.1179803185973; Mon, 21 May 2007 20:06:25 -0700 (PDT) Received: by 10.78.107.13 with HTTP; Mon, 21 May 2007 20:06:25 -0700 (PDT) Message-ID: Date: Mon, 21 May 2007 20:06:25 -0700 From: "Kip Macy" To: root In-Reply-To: <4651DB6E.5090102@jim-liesl.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4651DB6E.5090102@jim-liesl.org> Cc: freebsd-net@freebsd.org Subject: Re: asymetric speeds over gigE link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 May 2007 03:06:27 -0000 One more thing to remember is to raise the socket size on the linux sender. I found that I got very poor tx performance on linux running netperf without making the socket buffers fairy large. -Kip On 5/21/07, root wrote: > Summary: Using iperf to measure TCP net speed between a linux and > freebsd box over gigE, I see significant speed difference depending on > the data direction. Pushing data from the freebsd box to the linux box, > I average about 500Gb/s. Pushing data from the linux box to the freebsd > box, I see about 300Gb/s. It would seem that there is something > limiting either the transmit side of the linux box or the recv side of > the freebsd box. Any suggestions on how to narrow down who's tuning > needs more work would also be welcome. Just for grins, I tried upping > the txqueuelen on the linux box to 1500, but no help. > > I'd like to get closer to the theoretical speed of this link (I'd be > happy if I could get 600-700 both ways), but the real puzzler is the > difference. The client is the linux box @192.168.1.104 and the iperf > server is the freebsd box @ 192.168.1.15 . Neither box is cpu or net > busy during testing. Both systems only have one nic. I played with > larger window sizes, but it only had a minor affect. I went with the > Intel Pro/1000's based on their good rep in both the linux and freebsd > communities. > > Client connecting to 192.168.1.15, TCP port 5001 > TCP window size: 128 KByte (default) > ------------------------------------------------------------ > [ 6] local 192.168.1.104 port 34788 connected with 192.168.1.15 port 5001 > [ 6] 0.0-10.0 sec 369 MBytes 309 Mbits/sec > [ 5] local 192.168.1.104 port 5001 connected with 192.168.1.15 port 52963 > [ 5] 0.0-10.0 sec 597 MBytes 500 Mbits/sec > > > Switch is the Netgear GS105 (5 port, supposedly wire speed, cables are > Belkin 5e), both systems are on this switch. > > FreeBSD box: > FreeBSD 6.1-RELEASE-p10 > 1.9Ghz Athlon / 1 gig of main mem > Abit/nforce2 MB+chipset (onboard nic disabled in bios) > Intel Pro/1000GT NIC > sysctl.conf: > kern.ipc.maxsockbuf=8192000 > net.inet.tcp.sendspace=262144 > net.inet.tcp.recvspace=262144 > sysctl reports: > kern.ipc.nmbjumbo16: 0 > kern.ipc.nmbjumbo9: 0 > kern.ipc.nmbjumbop: 0 > kern.ipc.nmbclusters: 25600 > net.inet.tcp.rfc1323: 1 > > em0: flags=8843 mtu 1500 > options=b > inet6 fe80::20e:cff:feda:1a3c%em0 prefixlen 64 scopeid 0x1 > inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255 > ether 00:0e:0c:da:1a:3c > media: Ethernet autoselect (1000baseTX ) > status: active > > Linux box: > Linux emperor 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686 > GNU/Linux > Soyo Dragon+ MB/ 1.9 Ghz Athlon/ 1 gig main mem. > I run tcp_tune.sh at boot on the linux box > #!/bin/bash > echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_rmem > echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_wmem > echo "131072 262144 8192000" > /proc/sys/net/ipv4/tcp_mem > > /proc/sys/net/ipv4/tcp_window_scaling is 1 > /proc/sys/net/ipv4/tcp_timestamps is 1 > > eth1 Link encap:Ethernet HWaddr 00:0E:0C:DA:1A:3B > inet addr:192.168.1.104 Bcast:192.168.1.255 Mask:255.255.255.0 > inet6 addr: fe80::20e:cff:feda:1a3b/64 Scope:Link > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > RX packets:8139539 errors:0 dropped:0 overruns:0 frame:0 > TX packets:5638407 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:1000 > RX bytes:147638720 (140.7 MiB) TX bytes:1109677958 (1.0 GiB) > Base address:0xd800 Memory:e2020000-e2040000 > > Settings for eth1: > Supported ports: [ TP ] > Supported link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Full > Supports auto-negotiation: Yes > Advertised link modes: 10baseT/Half 10baseT/Full > 100baseT/Half 100baseT/Full > 1000baseT/Full > Advertised auto-negotiation: Yes > Speed: 1000Mb/s > Duplex: Full > Port: Twisted Pair > PHYAD: 0 > Transceiver: internal > Auto-negotiation: on > Supports Wake-on: umbg > Wake-on: g > Current message level: 0x00000007 (7) > Link detected: yes > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >