From owner-freebsd-net@freebsd.org Fri Aug 28 16:35:42 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 86B749C5BEF for ; Fri, 28 Aug 2015 16:35:42 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194]) by mx1.freebsd.org (Postfix) with ESMTP id 740BE3A7 for ; Fri, 28 Aug 2015 16:35:42 +0000 (UTC) (envelope-from chris@stankevitz.com) Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net [209.203.101.124]) (using TLSv1.2 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mango.stankevitz.com (Postfix) with ESMTPSA id 1D47E1653; Fri, 28 Aug 2015 09:35:42 -0700 (PDT) Message-ID: <55E08DDD.1040804@stankevitz.com> Date: Fri, 28 Aug 2015 09:35:41 -0700 From: Chris Stankevitz User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com> <55DD2A98.2010605@stankevitz.com> <20150826082457.GQ33167@funkthat.com> <55DE2FDF.5030707@stankevitz.com> <20150826223215.GS33167@funkthat.com> In-Reply-To: <20150826223215.GS33167@funkthat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Aug 2015 16:35:42 -0000 John-Mark, I'm going to rearrange the conversation a bit in the quotes below: On 8/26/15 3:32 PM, John-Mark Gurney wrote: > So, I looked at what getsockopt SO_RCVBUF returns, and it returns: > case SO_RCVBUF: > optval = so->so_rcv.sb_hiwat; > > Which is NOT S-BMAX, so it looks like OpenSSH only ever gets 66052 bytes > for the buffer... I believe that explains everything we are seeing. HPN developers thought SO_RCVBUF would return S-BMAX but it instead returns S-HIWA. > If it's decided to base it's waiting for ack on SO_RCVBUF, then this > is probably where the issue is... > > Try setting HPNBufferSize to something resonably large, like 1MB, or > above your bandwidth-delay product to see if this will make a difference.. > Per: > Conditions: HPNBufferSize SET, TCPRcvBufPoll enabled, TCPRcvBuf NOT Set > Result: HPN Buffer Size = grows to HPNBufferSize > The buffer will grow up to the maximum size specified here. Setting HPNBufferSize to 1 MB does not change S-BCNT which remains stuck at ~60KB. Although I might not necessarily expect a change. From README.hpn: HPNBufferSize: This is the default buffer size the HPN functionality uses when interacting with non-HPN SSH installations. > It would be interesting to know how long from the read of stdin (and is > it really reading stdin in 4k blocks? If so, that should be fixed) > to the write out the socket... Basicly, how long does it take to encrypt > the data... > >> Note in the above the blocking call to select at time 6.592065 that >> takes ~0.1 second. This is the reason my connection is slow. The >> content appears to be encrypted... I presume it's an application-level >> ssh ack. I posted the in-context kdump at: https://www.stankevitz.com/ssh-ktrace.txt stdin is read in 4k blocks... but each "cycle" reads 3 of these blocks. I define as "cycle" as the period beginning and ending at a 0.1 second select(2) sleep. Looks like encryption of a 4KB block takes 10 microseconds, which means: for every 0.1 seconds in select(2), .000030 seconds are encrypting. >> net.inet.tcp.sendspace: 32768 >> net.inet.tcp.recvspace: 65536 > > Try increasing these and reporting back... the buf_auto should override > those, but this may be limiting it... Okay. Now we're getting somewhere. Increase sendspace on "sending client": no change Increase recvspace on "sending client": no change Increase sendspace on "receiving server": no change Increase recvspace on "receiving server": sending client's S-BCNT increases proportionally! I'm going to try Kurts parameters now... Chris