From owner-freebsd-net@freebsd.org Thu Aug 27 13:14:33 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A9F2E9C48C3 for ; Thu, 27 Aug 2015 13:14:33 +0000 (UTC) (envelope-from lidl@pix.net) Received: from hydra.pix.net (hydra.pix.net [IPv6:2001:470:e254::4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.pix.net", Issuer "Pix.Com Technologies, LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 5DA4FFEE for ; Thu, 27 Aug 2015 13:14:33 +0000 (UTC) (envelope-from lidl@pix.net) Received: from torb.pix.net (verizon.pix.net [71.178.232.3]) (authenticated bits=0) by hydra.pix.net (8.15.2/8.15.2) with ESMTPA id t7RDEVXT049108; Thu, 27 Aug 2015 09:14:31 -0400 (EDT) (envelope-from lidl@pix.net) X-Authentication-Warning: hydra.pix.net: Host verizon.pix.net [71.178.232.3] claimed to be torb.pix.net To: freebsd-net@freebsd.org References: <55DCF080.7080208@stankevitz.com> Subject: Re: ssh over WAN: TCP window too small From: Kurt Lidl Message-ID: <55DF0D37.5060003@pix.net> Date: Thu, 27 Aug 2015 09:14:31 -0400 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <55DCF080.7080208@stankevitz.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 13:14:33 -0000 Chris Stankevitz wrote: > Hi, > > # cat /dev/urandom | ssh root at host 'cat > /dev/null' > > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). > iperf with default options gets the window opened to 500 KBytes > (yielding 35 Mbps). > > Both sides of the connection: FreeBSD 10.1 w/default sshd options > (except I permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? > > Can anyone recommend some tools/tricks to figure out what in FreeBSD > and/or base SSH is limiting the send/recv buffer and/or TCP window? I know this response is a little late to the party, but... I spent a bit of time last year tuning my FreeBSD 10.1 host to be able to transfer a bunch of data between the east coast of the US and the west cost. My WAN link was more like 70ms @ 75 Mbps, so not too different than yours. The other end of the connection was also a FreeBSD 10.1 host. I have the following in my /etc/sysctl.conf - and I get pretty much all 75Mbps when I scp or rsync a file: # tcp options for long-haul speedups kern.ipc.maxsockbuf=4194304 # (2 * default 2097152) net.inet.tcp.mssdflt=1448 # (default 576) net.inet.tcp.sendbuf_max=4194304 # (2 * default 2097152) net.inet.tcp.recvbuf_max=4194304 # (2 * default 2097152) net.inet.tcp.syncache.rexmtlimit=1 # (default 3) net.inet.tcp.recvspace=262144 # (4 * default 65,536) net.inet.tcp.sendspace=262144 # (4 * default 65,536) net.inet.tcp.sendbuf_inc=65536 # (8 * default 8192) net.inet.tcp.recvbuf_inc=131072 # (8 * default 16384) One thing that was noticed - it can take a really, really, really long time for the TCP window to open up the whole way with the default net.inet.tcp.sendbuf_inc setting! -Kurt