Date: Wed, 21 Nov 2012 10:49:53 -0800 From: Julian Elischer <julian@freebsd.org> To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections Message-ID: <50AD2251.3040904@freebsd.org> In-Reply-To: <50ACF62C.8000408@mpeters.org> References: <50ACF62C.8000408@mpeters.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/21/12 7:41 AM, Marc Peters wrote: > Hi list, > > we are experiencing low throughput on interncontinental connections with > our FreeBSD Servers. We made several tests and are wondering, why this > would be. The first tests were on an IPSEC VPN between our datacenter in > DE and Santa Clara, CA. We are connected with two gigabit uplinks in > each DC. Pushing data by scp between our FreeBSD servers takes ages. > Starting with several MB/s it drops to 60-70KB/s: > > [root@freebsd ~]# ls -alh test.tgz > -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz > [root@freebsd ~]# scp test.tgz 172.16.3.10:. > Password: > test.tgz 28% 17MB 75.3KB/s 09:32 ETA > > > For comparision, we did a similiar test with Linux, which didn't show > this behaviour: > > root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: > root@172.16.4.50's password: > jdk-6u33-linux-x64.bin 100% > 69MB 3.4MB/s 00:20 > root@linux:~# > > > Otherwise, the servers are really fast, when copying data to a machine > nearby: > > [root@freebsd ~]# ls -alh test > -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test > [root@freebsd ~]# scp test 172.16.3.11: > Password: > test 100% 1000MB 38.5MB/s 00:26 > > > Intercontinental ftp downloads are the same: > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso > FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps > > [root@freebsd ~]# fetch > ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > > FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s > > [root@freebsd ~]# fetch > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C > > > Linux: > > root@linux:~# wget > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > --2012-11-21 15:07:57-- > ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso > => `FreeBSD-9.1-RC3-amd64-disc1.iso' > Resolving ftp1.de.freebsd.org... 137.226.34.42 > Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. > Logging in as anonymous ... Logged in! > ==> SYST ... done. ==> PWD ... done. > ==> TYPE I ... done. ==> CWD (1) > /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. > ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 > ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. > Length: 718800896 (686M) (unauthoritative) > > 100%[=====================================================================>] > 718,800,896 19.1M/s in 61s > > 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' > saved [718800896] > > > Doing some googling brought up a lot of tuning hints, but nothing worked > for us. We tweaked some sysctls: > > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.sendbuf_inc=16384 > net.inet.tcp.recvbuf_inc=524288 > net.inet.tcp.hostcache.expire=1 > > but to no good. Disabling MSI and TSO4 for the card didn't change > anything, too. > > The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and > pciconf -lvc at the end of this mail). > > Can someone hit me with a cluestick to get the BSDs on speed? you really do need to get a tcpdump of the transfer under slow conditions and a SIFTR output to match. What is the ping time between the hosts. that will allow you to work out how large a window you should have. > > marc > > PS: The version is FreeBSD-RC2 amd64, because we need the patch for > process migration on the CPUs which didn't make it 9.0 or an errata, as > we were the only ones, hitting this bug (so kib@ said). >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50AD2251.3040904>