From owner-freebsd-net@freebsd.org  Fri Aug 28 19:14:32 2015
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEE4C9C47ED
 for <freebsd-net@mailman.ysv.freebsd.org>;
 Fri, 28 Aug 2015 19:14:32 +0000 (UTC)
 (envelope-from chris@stankevitz.com)
Received: from mango.stankevitz.com (mango.stankevitz.com [208.79.93.194])
 by mx1.freebsd.org (Postfix) with ESMTP id DF1801148
 for <freebsd-net@freebsd.org>; Fri, 28 Aug 2015 19:14:32 +0000 (UTC)
 (envelope-from chris@stankevitz.com)
Received: from Chriss-MacBook-Pro.local (209-203-101-124.static.twtelecom.net
 [209.203.101.124])
 (using TLSv1.2 with cipher AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mango.stankevitz.com (Postfix) with ESMTPSA id CB7501678;
 Fri, 28 Aug 2015 12:14:32 -0700 (PDT)
Message-ID: <55E0B318.2090709@stankevitz.com>
Date: Fri, 28 Aug 2015 12:14:32 -0700
From: Chris Stankevitz <chris@stankevitz.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10;
 rv:31.0) Gecko/20100101 Thunderbird/31.7.0
MIME-Version: 1.0
To: Kurt Lidl <lidl@pix.net>, freebsd-net@freebsd.org
Subject: Re: ssh over WAN: TCP window too small
References: <55DCF080.7080208@stankevitz.com> <55DF0D37.5060003@pix.net>
In-Reply-To: <55DF0D37.5060003@pix.net>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.20
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 28 Aug 2015 19:14:33 -0000

On 8/27/15 6:14 AM, Kurt Lidl wrote:
> # tcp options for long-haul speedups
> kern.ipc.maxsockbuf=4194304             # (2 * default 2097152)
> net.inet.tcp.mssdflt=1448               # (default 576)
> net.inet.tcp.sendbuf_max=4194304        # (2 * default 2097152)
> net.inet.tcp.recvbuf_max=4194304        # (2 * default 2097152)
>
> net.inet.tcp.syncache.rexmtlimit=1      # (default 3)
> net.inet.tcp.recvspace=262144           # (4 * default 65,536)
> net.inet.tcp.sendspace=262144           # (4 * default 65,536)
>
> net.inet.tcp.sendbuf_inc=65536          # (8 * default 8192)
> net.inet.tcp.recvbuf_inc=131072         # (8 * default 16384)


Kurt,

Thank you.  FYI the default for sendspace is 32768 (not 65536).

With these parameters my S-BCNT increases from ~60K to ~200K when doing 
`ssh < /dev/zero`.  Amusingly, something somewhere limits 
bytes-in-flight to 172,888 bytes (I believe the ssh sending client is 
limiting -- although I'm not sure why).  This effectively caps my 
bandwidth to 20 Mbps.

With iperf this limitation does not exist -- nor does the need to tune 
these values (except for buf_inc which can get the ball rolling faster 
as you pointed out).

Chris