From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 18:50:06 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1F6FDE7B for ; Wed, 21 Nov 2012 18:50:06 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id EC1A88FC0C for ; Wed, 21 Nov 2012 18:50:05 +0000 (UTC) Received: from JRE-MBP-2.local (c-50-143-149-146.hsd1.ca.comcast.net [50.143.149.146]) (authenticated bits=0) by vps1.elischer.org (8.14.5/8.14.5) with ESMTP id qALInwIh083667 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Wed, 21 Nov 2012 10:49:58 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <50AD2251.3040904@freebsd.org> Date: Wed, 21 Nov 2012 10:49:53 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:16.0) Gecko/20121026 Thunderbird/16.0.2 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> In-Reply-To: <50ACF62C.8000408@mpeters.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:50:06 -0000 On 11/21/12 7:41 AM, Marc Peters wrote: > Hi list, > > we are experiencing low throughput on interncontinental connections with > our FreeBSD Servers. We made several tests and are wondering, why this > would be. The first tests were on an IPSEC VPN between our datacenter in > DE and Santa Clara, CA. We are connected with two gigabit uplinks in > each DC. Pushing data by scp between our FreeBSD servers takes ages. > Starting with several MB/s it drops to 60-70KB/s: > > [root@freebsd ~]# ls -alh test.tgz > -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz > [root@freebsd ~]# scp test.tgz 172.16.3.10:. > Password: > test.tgz 28% 17MB 75.3KB/s 09:32 ETA > > > For comparision, we did a similiar test with Linux, which didn't show > this behaviour: > > root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: > root@172.16.4.50's password: > jdk-6u33-linux-x64.bin 100% > 69MB 3.4MB/s 00:20 > root@linux:~# > > > Otherwise, the servers are really fast, when copying data to a machine > nearby: > > [root@freebsd ~]# ls -alh test > -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test > [root@freebsd ~]# scp test 172.16.3.11: > Password: > test 100% 1000MB 38.5MB/s 00:26 > > > Intercontinental ftp downloads are the same: > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso > FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > > FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s > > [root@freebsd ~]# fetch > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C > > > Linux: > > root@linux:~# wget > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > --2012-11-21 15:07:57-- > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > => `FreeBSD-9.1-RC3-amd64-disc1.iso' > Resolving ftp1.de.freebsd.org... 137.226.34.42 > Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. > Logging in as anonymous ... Logged in! > ==> SYST ... done. ==> PWD ... done. > ==> TYPE I ... done. ==> CWD (1) > /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. > ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 > ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. > Length: 718800896 (686M) (unauthoritative) > > 100%[=====================================================================>] > 718,800,896 19.1M/s in 61s > > 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' > saved [718800896] > > > Doing some googling brought up a lot of tuning hints, but nothing worked > for us. We tweaked some sysctls: > > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.sendbuf_inc=16384 > net.inet.tcp.recvbuf_inc=524288 > net.inet.tcp.hostcache.expire=1 > > but to no good. Disabling MSI and TSO4 for the card didn't change > anything, too. > > The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and > pciconf -lvc at the end of this mail). > > Can someone hit me with a cluestick to get the BSDs on speed? you really do need to get a tcpdump of the transfer under slow conditions and a SIFTR output to match. What is the ping time between the hosts. that will allow you to work out how large a window you should have. > > marc > > PS: The version is FreeBSD-RC2 amd64, because we need the patch for > process migration on the CPUs which didn't make it 9.0 or an errata, as > we were the only ones, hitting this bug (so kib@ said). >