From owner-freebsd-net@FreeBSD.ORG Thu Oct 25 22:39:46 2007 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D63016A469 for ; Thu, 25 Oct 2007 22:39:46 +0000 (UTC) (envelope-from mattjreimer@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.190]) by mx1.freebsd.org (Postfix) with ESMTP id DA4AF13C4B5 for ; Thu, 25 Oct 2007 22:39:45 +0000 (UTC) (envelope-from mattjreimer@gmail.com) Received: by rv-out-0910.google.com with SMTP id l15so539846rvb for ; Thu, 25 Oct 2007 15:39:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:subject:from:to:content-type:date:message-id:mime-version:x-mailer:content-transfer-encoding; bh=OrM9Vb5x1jqxKcmgV838Y8+5mdVqywDms8TO5Tqsu6c=; b=i/2814UGpcss3akhbnTizYRLmYA+Q4fAzsQeU6MG9exIx4hZxwm6b4iKsbuqrvCC6RVzaDAFfRS6Fyc8kBLrDixLafpY8gZtv38047QA3QnddTZgBWS/aFGsroO3eqQyjzIIDALH9HWcKg7CzXg5l1zCjMI8vgoSew1GpLHxxEc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:subject:from:to:content-type:date:message-id:mime-version:x-mailer:content-transfer-encoding; b=hk/y9dOYROCKPVk2Nnx5zzA3VNyRVWdKiqYq3PcrAeIoQx8fWp8FmxMSN4C1fd32KbHHct670McJstAjAmPLX3Gz1QMs8257fBsyn8Y+y8T8kj8FfAe/bp+WqHawd/zAJf4OWOkgt3oNI9uQPDAirTFd20cr+i2IDIrUpI7jLz0= Received: by 10.141.177.2 with SMTP id e2mr1213052rvp.1193350485679; Thu, 25 Oct 2007 15:14:45 -0700 (PDT) Received: from ?70.56.77.194? ( [70.56.77.194]) by mx.google.com with ESMTPS id l27sm4380775rvb.2007.10.25.15.14.44 (version=TLSv1/SSLv3 cipher=RC4-MD5); Thu, 25 Oct 2007 15:14:44 -0700 (PDT) From: Matthew Reimer To: net@freebsd.org Content-Type: text/plain Date: Thu, 25 Oct 2007 15:14:37 -0700 Message-Id: <1193350477.20932.53.camel@bilbo.vpop.net> Mime-Version: 1.0 X-Mailer: Evolution 2.10.3 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Subject: Throughput problems in RELENG_7 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Oct 2007 22:39:46 -0000 I'm seeing a problem where a much faster quad-core host running RELENG_7 serves many fewer netrate/http requests per second (175/sec) than an old, busy, UP 6.0 host (828/sec). The problem seems to be related to latency and connection setup, as it shows up dramatically over a link with 50-60 ms latency. Can you help? "Gandalf" is the problem RELENG_7 host connected via em0 on a 100baseTX full-duplex switch port. "Boromir" is a 6.0 host for reference, connected via bge0 to another 100baseTX full-duplex port on the same switch. "Theoden" is a 6.2 host remote (50-60 ms away) from gandalf and boromir. Running tools/netrate/http against tools/netrate/httpd on gandalf and boromir, serving a document with 35 bytes of content, I get this (details including tcpdumps following my sig): dest dest source boromir gandalf ------ ------- ------- theoden 828 175 boromir - 18092 gandalf 11493 - The problem occurs using ab as well, but is much less pronounced when -k is used (enabling keepalive and presumably HTTP pipelining). This makes me wonder if the problem might have something to do with connection setups. I've verified that the switch port and the interface are configured correctly (100baseTX full-duplex). I've also tried toggling polling and the following sysctls, with no significant change: net.inet.tcp.tso net.inet.tcp.newreno net.inet.tcp.sack.enable net.inet.tcp.inflight.enable net.inet.tcp.syncookies_only net.inet.tcp.syncookies What else can I try? Thanks for any help you can give. Matt theoden-gandalf =============== theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2 216.193.240.4 1280 /short.html 187 transfers/second 0 errors/second tcpdump: http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate.dump.gz --- theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.4 1280 /short.html 175 transfers/second 0 errors/second tcpdump: http://gandalf.vpop.net/~mreimer/theoden-gandalf-netrate-n100.dump.gz theoden-boromir =============== theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -s 2 216.193.240.2 1280 /short.html 272 transfers/second 0 errors/second tcpdump: http://boromir.vpop.net/~mreimer/theoden-boromir-netrate.dump.gz --- theoden:~$ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.2 1280 /short.html 828 transfers/second 0 errors/second tcpdump: http://boromir.vpop.net/~mreimer/theoden-boromir-netrate-n100.dump.gz boromir-gandalf =============== boromir: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.4 1280 /short.html 18092 transfers/second 0 errors/second gandalf-boromir =============== gandalf: $ /usr/obj/usr/src/tools/tools/netrate/http/http -n 100 -s 2 216.193.240.2 1280 /short.html 11493 transfers/second 0 errors/second boromir sysctl and ifconfig: http://boromir.vpop.net/~mreimer/boromir.sysctl.gz # ifconfig bge0 bge0: flags=8843 mtu 1500 options=1a inet6 fe80::2e0:81ff:fe60:3925%bge0 prefixlen 64 scopeid 0x1 inet 216.193.240.33 netmask 0xffffffff broadcast 216.193.240.33 inet 216.193.240.34 netmask 0xffffffff broadcast 216.193.240.34 inet 216.193.240.35 netmask 0xffffffff broadcast 216.193.240.35 inet 216.193.240.36 netmask 0xffffffff broadcast 216.193.240.36 inet 216.193.240.37 netmask 0xffffffff broadcast 216.193.240.37 inet 216.193.240.17 netmask 0xffffffff broadcast 216.193.240.17 inet 216.193.240.38 netmask 0xffffffff broadcast 216.193.240.38 inet 216.193.240.39 netmask 0xffffffff broadcast 216.193.240.39 inet 216.193.240.2 netmask 0xfffff800 broadcast 216.193.247.255 inet 216.193.240.40 netmask 0xffffffff broadcast 216.193.240.40 ether 00:e0:81:60:39:25 media: Ethernet autoselect (100baseTX ) status: active gandalf sysctl and ifconfig: http://gandalf.vpop.net/~mreimer/gandalf.sysctl.gz # ifconfig em0 em0: flags=8843 metric 0 mtu 1500 options=1cb ether 00:30:48:33:96:a4 inet 216.193.240.4 netmask 0xfffff800 broadcast 216.193.247.255 inet 216.193.240.65 netmask 0xffffffff broadcast 216.193.240.65 inet 216.193.240.66 netmask 0xffffffff broadcast 216.193.240.66 inet 216.193.240.67 netmask 0xffffffff broadcast 216.193.240.67 inet 216.193.240.68 netmask 0xffffffff broadcast 216.193.240.68 media: Ethernet 100baseTX status: active