From owner-freebsd-net@FreeBSD.ORG Mon May 21 18:15:15 2007 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E803D16A469 for ; Mon, 21 May 2007 18:15:14 +0000 (UTC) (envelope-from root@jim-liesl.org) Received: from qsmtp1.mc.surewest.net (qsmtp.mc.surewest.net [66.60.130.145]) by mx1.freebsd.org (Postfix) with SMTP id C74C013C4B9 for ; Mon, 21 May 2007 18:15:12 +0000 (UTC) (envelope-from root@jim-liesl.org) Received: (qmail 29145 invoked from network); 21 May 2007 10:48:32 -0700 Received: by simscan 1.1.0 ppid: 29110, pid: 29111, t: 5.6864s scanners: regex: 1.1.0 attach: 1.1.0 clamav: 0.90.1/m:43 spam: 3.1.7-deb Received: from unknown (HELO daemon.jim-liesl.org) (66.60.173.44) by qsmtp1 with SMTP; 21 May 2007 10:48:26 -0700 Received: from daemon.jim-liesl.org (localhost.static.surewest.net [127.0.0.1]) by daemon.jim-liesl.org (Postfix) with ESMTP id 8E4975C1D for ; Mon, 21 May 2007 10:48:26 -0700 (PDT) Received: from [127.0.0.1] (daemon.static.surewest.net [192.168.1.15]) by daemon.jim-liesl.org (Postfix) with ESMTP id 539C25C1C for ; Mon, 21 May 2007 10:48:26 -0700 (PDT) Message-ID: <4651DB6E.5090102@jim-liesl.org> Date: Mon, 21 May 2007 10:48:30 -0700 From: root User-Agent: Thunderbird 1.5.0.10 (Windows/20070221) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on qsmtp1.surewest.net X-Spam-Level: X-Spam-Status: No, score=-2.5 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.0.3 Subject: asymetric speeds over gigE link X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 May 2007 18:15:15 -0000 Summary: Using iperf to measure TCP net speed between a linux and freebsd box over gigE, I see significant speed difference depending on the data direction. Pushing data from the freebsd box to the linux box, I average about 500Gb/s. Pushing data from the linux box to the freebsd box, I see about 300Gb/s. It would seem that there is something limiting either the transmit side of the linux box or the recv side of the freebsd box. Any suggestions on how to narrow down who's tuning needs more work would also be welcome. Just for grins, I tried upping the txqueuelen on the linux box to 1500, but no help. I'd like to get closer to the theoretical speed of this link (I'd be happy if I could get 600-700 both ways), but the real puzzler is the difference. The client is the linux box @192.168.1.104 and the iperf server is the freebsd box @ 192.168.1.15 . Neither box is cpu or net busy during testing. Both systems only have one nic. I played with larger window sizes, but it only had a minor affect. I went with the Intel Pro/1000's based on their good rep in both the linux and freebsd communities. Client connecting to 192.168.1.15, TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [ 6] local 192.168.1.104 port 34788 connected with 192.168.1.15 port 5001 [ 6] 0.0-10.0 sec 369 MBytes 309 Mbits/sec [ 5] local 192.168.1.104 port 5001 connected with 192.168.1.15 port 52963 [ 5] 0.0-10.0 sec 597 MBytes 500 Mbits/sec Switch is the Netgear GS105 (5 port, supposedly wire speed, cables are Belkin 5e), both systems are on this switch. FreeBSD box: FreeBSD 6.1-RELEASE-p10 1.9Ghz Athlon / 1 gig of main mem Abit/nforce2 MB+chipset (onboard nic disabled in bios) Intel Pro/1000GT NIC sysctl.conf: kern.ipc.maxsockbuf=8192000 net.inet.tcp.sendspace=262144 net.inet.tcp.recvspace=262144 sysctl reports: kern.ipc.nmbjumbo16: 0 kern.ipc.nmbjumbo9: 0 kern.ipc.nmbjumbop: 0 kern.ipc.nmbclusters: 25600 net.inet.tcp.rfc1323: 1 em0: flags=8843 mtu 1500 options=b inet6 fe80::20e:cff:feda:1a3c%em0 prefixlen 64 scopeid 0x1 inet 192.168.1.15 netmask 0xffffff00 broadcast 192.168.1.255 ether 00:0e:0c:da:1a:3c media: Ethernet autoselect (1000baseTX ) status: active Linux box: Linux emperor 2.6.17-11-generic #2 SMP Tue Mar 13 23:32:38 UTC 2007 i686 GNU/Linux Soyo Dragon+ MB/ 1.9 Ghz Athlon/ 1 gig main mem. I run tcp_tune.sh at boot on the linux box #!/bin/bash echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_rmem echo "10000 131072 262144">/proc/sys/net/ipv4/tcp_wmem echo "131072 262144 8192000" > /proc/sys/net/ipv4/tcp_mem /proc/sys/net/ipv4/tcp_window_scaling is 1 /proc/sys/net/ipv4/tcp_timestamps is 1 eth1 Link encap:Ethernet HWaddr 00:0E:0C:DA:1A:3B inet addr:192.168.1.104 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::20e:cff:feda:1a3b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:8139539 errors:0 dropped:0 overruns:0 frame:0 TX packets:5638407 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:147638720 (140.7 MiB) TX bytes:1109677958 (1.0 GiB) Base address:0xd800 Memory:e2020000-e2040000 Settings for eth1: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: umbg Wake-on: g Current message level: 0x00000007 (7) Link detected: yes