From owner-freebsd-net@freebsd.org Wed Aug 26 01:03:30 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EFE39C220A for ; Wed, 26 Aug 2015 01:03:30 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B9AA6F1 for ; Wed, 26 Aug 2015 01:03:29 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t7Q13NpX021967 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 25 Aug 2015 18:03:23 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t7Q13NNm021966; Tue, 25 Aug 2015 18:03:23 -0700 (PDT) (envelope-from jmg) Date: Tue, 25 Aug 2015 18:03:23 -0700 From: John-Mark Gurney To: Chris Stankevitz Cc: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Message-ID: <20150826010323.GN33167@funkthat.com> References: <55DCF080.7080208@stankevitz.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <55DCF080.7080208@stankevitz.com> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Tue, 25 Aug 2015 18:03:23 -0700 (PDT) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2015 01:03:30 -0000 Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: > # cat /dev/urandom | ssh root@host 'cat > /dev/null' Don't use this for testing... use /dev/zero or some other device that can produce data faster than this... > I use the above ssh command over a high-BDP WAN link (80 ms @ 100 Mbps). > tcpdump shows I am TCP window limited to 64 KBytes (yielding 5 Mbps). > iperf with default options gets the window opened to 500 KBytes > (yielding 35 Mbps). > > Both sides of the connection: FreeBSD 10.1 w/default sshd options > (except I permit root login). In particular, HPN is not disabled. > > Can anyone explain my abysmally small TCP window? Looks like ssh is propbably hard setting the send/recv buffers to values that are too small... So, our SSH does have the HPN patches: https://www.psc.edu/index.php/hpn-ssh and the README says: BUFFER SIZES: - if HPN is disabled the receive buffer size will be set to the OpenSSH default of 64K. You can read more at: https://svnweb.freebsd.org/base/stable/10/crypto/openssh/README.hpn?annotate=256281 Looks like there are undocumented options like TCPRcvBuf that you can use to adjust the recv buffer window... It looks like OpenSSH hard sets the buffer sizes for some reason... On FreeBSD, these should never be set unless the option is provided and you know what you are doing.. We have code that will auto grow buffer sizes properly so that slow connections won't use up too much buffer space... > Can anyone recommend some tools/tricks to figure out what in FreeBSD > and/or base SSH is limiting the send/recv buffer and/or TCP window? Seems like from looking at the code, things should "just work", so not sure why you are seeing the smaller window size... In a quick test of mine, I'm seeing a buffer size of ~520k from my MacOSX box, and ~776k from my 9.2-R box... Server in both cases is a June -CURRENT... netstat -xAanfinet is helpful on this... Hope this helps! -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."