Date: Thu, 22 Nov 2012 10:07:40 +0100 From: Marc Peters <marc@mpeters.org> To: freebsd-net@freebsd.org Subject: Re: Low Bandwidth on intercontinental connections Message-ID: <50ADEB5C.6060401@mpeters.org> In-Reply-To: <50AD2251.3040904@freebsd.org> References: <50ACF62C.8000408@mpeters.org> <50AD2251.3040904@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/21/2012 07:49 PM, Julian Elischer wrote: > On 11/21/12 7:41 AM, Marc Peters wrote: >> Hi list, >> >> we are experiencing low throughput on interncontinental connections with >> our FreeBSD Servers. We made several tests and are wondering, why this >> would be. The first tests were on an IPSEC VPN between our datacenter in >> DE and Santa Clara, CA. We are connected with two gigabit uplinks in >> each DC. Pushing data by scp between our FreeBSD servers takes ages. >> Starting with several MB/s it drops to 60-70KB/s: >> >> [root@freebsd ~]# ls -alh test.tgz >> -rw-r----- 1 root wheel 58M Oct 5 2010 test.tgz >> [root@freebsd ~]# scp test.tgz 172.16.3.10:. >> Password: >> test.tgz 28% 17MB 75.3KB/s 09:32 ETA >> >> >> For comparision, we did a similiar test with Linux, which didn't show >> this behaviour: >> >> root@linux:~# scp jdk-6u33-linux-x64.bin 172.16.4.50: >> root@172.16.4.50's password: >> jdk-6u33-linux-x64.bin 100% >> 69MB 3.4MB/s 00:20 >> root@linux:~# >> >> >> Otherwise, the servers are really fast, when copying data to a machine >> nearby: >> >> [root@freebsd ~]# ls -alh test >> -rw-r--r-- 1 root wheel 1G Nov 21 13:43 test >> [root@freebsd ~]# scp test 172.16.3.11: >> Password: >> test 100% 1000MB 38.5MB/s 00:26 >> >> >> Intercontinental ftp downloads are the same: >> >> [root@freebsd ~]# fetch >> ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-bootonly.iso >> >> FreeBSD-9.1-RC3-amd64-bootonly.iso 100% of 146 MB 46 MBps >> >> [root@freebsd ~]# fetch >> ftp://ftp1.us.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> >> FreeBSD-9.1-RC3-amd64-disc1.iso 100% of 685 MB 36 MBps 00m00s >> >> [root@freebsd ~]# fetch >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> FreeBSD-9.1-RC3-amd64-disc1.iso 0% of 685 MB 13 kBps 14h49m^C >> >> >> Linux: >> >> root@linux:~# wget >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> --2012-11-21 15:07:57-- >> ftp://ftp1.de.freebsd.org/pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1/FreeBSD-9.1-RC3-amd64-disc1.iso >> >> => `FreeBSD-9.1-RC3-amd64-disc1.iso' >> Resolving ftp1.de.freebsd.org... 137.226.34.42 >> Connecting to ftp1.de.freebsd.org|137.226.34.42|:21... connected. >> Logging in as anonymous ... Logged in! >> ==> SYST ... done. ==> PWD ... done. >> ==> TYPE I ... done. ==> CWD (1) >> /pub/FreeBSD/releases/amd64/amd64/ISO-IMAGES/9.1 ... done. >> ==> SIZE FreeBSD-9.1-RC3-amd64-disc1.iso ... 718800896 >> ==> PASV ... done. ==> RETR FreeBSD-9.1-RC3-amd64-disc1.iso ... done. >> Length: 718800896 (686M) (unauthoritative) >> >> 100%[=====================================================================>] >> >> 718,800,896 19.1M/s in 61s >> >> 2012-11-21 15:09:01 (11.2 MB/s) - `FreeBSD-9.1-RC3-amd64-disc1.iso' >> saved [718800896] >> >> >> Doing some googling brought up a lot of tuning hints, but nothing worked >> for us. We tweaked some sysctls: >> >> kern.ipc.maxsockbuf=16777216 >> net.inet.tcp.sendbuf_max=16777216 >> net.inet.tcp.recvbuf_max=16777216 >> net.inet.tcp.sendbuf_inc=16384 >> net.inet.tcp.recvbuf_inc=524288 >> net.inet.tcp.hostcache.expire=1 >> >> but to no good. Disabling MSI and TSO4 for the card didn't change >> anything, too. >> >> The machines are all HP DL360G7 with bce cards (find dmesg, ifconfig and >> pciconf -lvc at the end of this mail). >> >> Can someone hit me with a cluestick to get the BSDs on speed? > you really do need to get a tcpdump of the transfer under slow > conditions and a SIFTR output to match. > What is the ping time between the hosts. The ping times are okay, as for the distance (DE to US): ping 172.16.3.10 PING 172.16.3.10 (172.16.3.10) 56(84) bytes of data. 64 bytes from 172.16.3.10: icmp_req=1 ttl=62 time=155 ms 64 bytes from 172.16.3.10: icmp_req=2 ttl=62 time=156 ms 64 bytes from 172.16.3.10: icmp_req=3 ttl=62 time=155 ms 64 bytes from 172.16.3.10: icmp_req=4 ttl=62 time=156 ms 64 bytes from 172.16.3.10: icmp_req=5 ttl=62 time=156 ms > that will allow you to work out > how large a window you should have. What exactly do you mean by that? The MTU, we should use on the hosts? >> >> marc >> >> PS: The version is FreeBSD-RC2 amd64, because we need the patch for >> process migration on the CPUs which didn't make it 9.0 or an errata, as >> we were the only ones, hitting this bug (so kib@ said). >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50ADEB5C.6060401>