From owner-freebsd-net@FreeBSD.ORG Thu Nov 22 09:07:44 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E6ED3760 for ; Thu, 22 Nov 2012 09:07:44 +0000 (UTC) (envelope-from marc@mpeters.org) Received: from mail.mpeters.org (mail.mpeters.org [78.46.104.142]) by mx1.freebsd.org (Postfix) with ESMTP id 66F398FC15 for ; Thu, 22 Nov 2012 09:07:44 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail.mpeters.org (Postfix) with ESMTP id 185F113200F for ; Thu, 22 Nov 2012 10:07:43 +0100 (CET) X-Virus-Scanned: amavisd-new at mpeters.org Received: from mail.mpeters.org ([127.0.0.1]) by localhost (mail.mpeters.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PvwONG0F50oH for ; Thu, 22 Nov 2012 10:07:41 +0100 (CET) Received: from [192.168.0.204] (unknown [62.159.86.18]) by mail.mpeters.org (Postfix) with ESMTPSA id E3A9513200E for ; Thu, 22 Nov 2012 10:07:40 +0100 (CET) Message-ID: <50ADEB5C.6060401@mpeters.org> Date: Thu, 22 Nov 2012 10:07:40 +0100 From: Marc Peters User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:16.0) Gecko/20121104 Thunderbird/16.0.1 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections References: <50ACF62C.8000408@mpeters.org> <50AD2251.3040904@freebsd.org> In-Reply-To: <50AD2251.3040904@freebsd.org> X-Enigmail-Version: 1.5a1pre Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Nov 2012 09:07:45 -0000 On 11/21/2012 07:49 PM, Julian Elischer wrote: > On 11/21/12 7:41 AM, Marc Peters wrote: >> Hi list, >> >> we are experiencing low throughput on interncontinental connections with >> our FreeBSD Servers. We made several tests and are wondering, why this >> would be. The first tests were on an IPSEC VPN between our datacenter in >> DE and Santa Clara, CA. We are connected with two gigabit uplinks in >> each DC. Pushing data by scp between our FreeBSD servers takes ages. >> Starting with several MB/s it drops to 60-70KB/s: >> >> [root@freebsd ~]# ls -alh test.tgz >> -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz >> [root@freebsd ~]# scp test.tgz 172.16.3.10:. >> Password: >> test.tgz 28% 17MB 75.3KB/s 09:32 ETA >> >> >> For comparision, we did a similiar test with Linux, which didn't show >> this behaviour: >> >> root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: >> root@172.16.4.50's password: >> jdk-6u33-linux-x64.bin 100% >> 69MB 3.4MB/s 00:20 >> root@linux:~# >> >> >> Otherwise, the servers are really fast, when copying data to a machine >> nearby: >> >> [root@freebsd ~]# ls -alh test >> -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test >> [root@freebsd ~]# scp test 172.16.3.11: >> Password: >> test 100% 1000MB 38.5MB/s 00:26 >> >> >> Intercontinental ftp downloads are the same: >> >> [root@freebsd ~]# fetch >> ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso >> >> FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps >> >> [root@freebsd ~]# fetch >> ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> >> FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s >> >> [root@freebsd ~]# fetch >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C >> >> >> Linux: >> >> root@linux:~# wget >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> --2012-11-21 15:07:57-- >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> => `FreeBSD-9.1-RC3-amd64-disc1.iso' >> Resolving ftp1.de.freebsd.org... 137.226.34.42 >> Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. >> Logging in as anonymous ... Logged in! >> ==> SYST ... done. ==> PWD ... done. >> ==> TYPE I ... done. ==> CWD (1) >> /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. >> ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 >> ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. >> Length: 718800896 (686M) (unauthoritative) >> >> 100%[=====================================================================>] >> >> 718,800,896 19.1M/s in 61s >> >> 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' >> saved [718800896] >> >> >> Doing some googling brought up a lot of tuning hints, but nothing worked >> for us. We tweaked some sysctls: >> >> kern.ipc.maxsockbuf=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.sendbuf_inc=16384 >> net.inet.tcp.recvbuf_inc=524288 >> net.inet.tcp.hostcache.expire=1 >> >> but to no good. Disabling MSI and TSO4 for the card didn't change >> anything, too. >> >> The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and >> pciconf -lvc at the end of this mail). >> >> Can someone hit me with a cluestick to get the BSDs on speed? > you really do need to get a tcpdump of the transfer under slow > conditions and a SIFTR output to match. > What is the ping time between the hosts. The ping times are okay, as for the distance (DE to US): ping 172.16.3.10 PING 172.16.3.10 (172.16.3.10) 56(84) bytes of data. 64 bytes from 172.16.3.10: icmp_req=1 ttl=62 time=155 ms 64 bytes from 172.16.3.10: icmp_req=2 ttl=62 time=156 ms 64 bytes from 172.16.3.10: icmp_req=3 ttl=62 time=155 ms 64 bytes from 172.16.3.10: icmp_req=4 ttl=62 time=156 ms 64 bytes from 172.16.3.10: icmp_req=5 ttl=62 time=156 ms > that will allow you to work out > how large a window you should have. What exactly do you mean by that? The MTU, we should use on the hosts? >> >> marc >> >> PS: The version is FreeBSD-RC2 amd64, because we need the patch for >> process migration on the CPUs which didn't make it 9.0 or an errata, as >> we were the only ones, hitting this bug (so kib@ said). >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"