From owner-freebsd-current@FreeBSD.ORG Fri Oct 26 16:37:31 2007 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC27216A417 for ; Fri, 26 Oct 2007 16:37:30 +0000 (UTC) (envelope-from mattjreimer@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.188]) by mx1.freebsd.org (Postfix) with ESMTP id C230113C4AC for ; Fri, 26 Oct 2007 16:37:30 +0000 (UTC) (envelope-from mattjreimer@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so743159rvb for ; Fri, 26 Oct 2007 09:37:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:subject:from:to:content-type:date:message-id:mime-version:x-mailer:content-transfer-encoding; bh=OrM9Vb5x1jqxKcmgV838Y8+5mdVqywDms8TO5Tqsu6c=; b=aeWDTbYXXwPw945KjYct3dJWwazjCkg1PKC0/aK3zYczwkoykft7l1zBAT2aTSek2UDLw/jfBJ8hIme+DO5GOXOm28fruiXuAdWY3d6NUZe8nn/L7KCHR7ifoNn7oXIEPn2/mkxm94pB5MYyslAgx7TB0lNC39Gp6DAw7Cy7kFs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:subject:from:to:content-type:date:message-id:mime-version:x-mailer:content-transfer-encoding; b=LFxGqDgMD2aQ7NDQn6VJIPzHzybNWoJF6qkzGtYhA3DtR3/6in0YGSGHB0nwQxTCl3v4AQRzb5oiHaMLTkLeJ1UT1V0btUh/S3vI7ttT9kB+nD1B5RONYasFoajJsZYN1tH/DVP9cVM3kJZ1C2bRLu7R55zxUBPCW/5Vv0TWIGU= Received: by 10.141.52.5 with SMTP id e5mr1605973rvk.1193415063128; Fri, 26 Oct 2007 09:11:03 -0700 (PDT) Received: from ?70.56.77.194? ( [70.56.77.194]) by mx.google.com with ESMTPS id l31sm6917443rvb.2007.10.26.09.11.01 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 26 Oct 2007 09:11:02 -0700 (PDT) From: Matthew Reimer To: current@freebsd.org Content-Type: text/plain Date: Fri, 26 Oct 2007 09:10:55 -0700 Message-Id: <1193415055.3378.1.camel@bilbo.vpop.net> Mime-Version: 1.0 X-Mailer: Evolution 2.10.3 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Subject: Network throughput problems in RELENG_7 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Oct 2007 16:37:31 -0000 I'm seeing a problem where a much faster quad-core host running RELENG_7 serves many fewer netrate/http requests per second (175/sec) than an old, busy, UP 6.0 host (828/sec). The problem seems to be related to latency and connection setup, as it shows up dramatically over a link with 50-60 ms latency. Can you help? "Gandalf" is the problem RELENG_7 host connected via em0 on a 100baseTX full-duplex switch port. "Boromir" is a 6.0 host for reference, connected via bge0 to another 100baseTX full-duplex port on the same switch. "Theoden" is a 6.2 host remote (50-60 ms away) from gandalf and boromir. Running tools/netrate/http against tools/netrate/httpd on gandalf and boromir, serving a document with 35 bytes of content, I get this (details including tcpdumps following my sig): dest dest source boromir gandalf ------ ------- ------- theoden 828 175 boromir - 18092 gandalf 11493 - The problem occurs using ab as well, but is much less pronounced when -k is used (enabling keepalive and presumably HTTP pipelining). This makes me wonder if the problem might have something to do with connection setups. I've verified that the switch port and the interface are configured correctly (100baseTX full-duplex). I've also tried toggling polling and the following sysctls, with no significant change: net.inet.tcp.tso net.inet.tcp.newreno net.inet.tcp.sack.enable net.inet.tcp.inflight.enable net.inet.tcp.syncookies_only net.inet.tcp.syncookies What else can I try? Thanks for any help you can give. Matt theoden-gandalf =============== theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2 216.193.240.4 1280 /short.html 187 transfers/second 0 errors/second tcpdump: http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate.dump.gz --- theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.4 1280 /short.html 175 transfers/second 0 errors/second tcpdump: http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate-n100.dump.gz theoden-boromir =============== theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2 216.193.240.2 1280 /short.html 272 transfers/second 0 errors/second tcpdump: http://boromir.vpop.net/~mreimer/theoden-boromir-netrate.dump.gz --- theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.2 1280 /short.html 828 transfers/second 0 errors/second tcpdump: http://boromir.vpop.net/~mreimer/theoden-boromir-netrate-n100.dump.gz boromir-gandalf =============== boromir: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.4 1280 /short.html 18092 transfers/second 0 errors/second gandalf-boromir =============== gandalf: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.2 1280 /short.html 11493 transfers/second 0 errors/second boromir sysctl and ifconfig: http://boromir.vpop.net/~mreimer/boromir.sysctl.gz # ifconfig bge0 bge0: flags=8843 mtu 1500 options=1a inet6 fe80::2e0:81ff:fe60:3925%bge0 prefixlen 64 scopeid 0x1 inet 216.193.240.33 netmask 0xffffffff broadcast 216.193.240.33 inet 216.193.240.34 netmask 0xffffffff broadcast 216.193.240.34 inet 216.193.240.35 netmask 0xffffffff broadcast 216.193.240.35 inet 216.193.240.36 netmask 0xffffffff broadcast 216.193.240.36 inet 216.193.240.37 netmask 0xffffffff broadcast 216.193.240.37 inet 216.193.240.17 netmask 0xffffffff broadcast 216.193.240.17 inet 216.193.240.38 netmask 0xffffffff broadcast 216.193.240.38 inet 216.193.240.39 netmask 0xffffffff broadcast 216.193.240.39 inet 216.193.240.2 netmask 0xfffff800 broadcast 216.193.247.255 inet 216.193.240.40 netmask 0xffffffff broadcast 216.193.240.40 ether 00:e0:81:60:39:25 media: Ethernet autoselect (100baseTX ) status: active gandalf sysctl and ifconfig: http://gandalf.vpop.net/~mreimer/gandalf.sysctl.gz # ifconfig em0 em0: flags=8843 metric 0 mtu 1500 options=1cb ether 00:30:48:33:96:a4 inet 216.193.240.4 netmask 0xfffff800 broadcast 216.193.247.255 inet 216.193.240.65 netmask 0xffffffff broadcast 216.193.240.65 inet 216.193.240.66 netmask 0xffffffff broadcast 216.193.240.66 inet 216.193.240.67 netmask 0xffffffff broadcast 216.193.240.67 inet 216.193.240.68 netmask 0xffffffff broadcast 216.193.240.68 media: Ethernet 100baseTX status: active