Date: Tue, 25 Aug 2015 19:55:20 -0700 From: Chris Stankevitz <chris@stankevitz.com> To: John-Mark Gurney <jmg@funkthat.com> Cc: freebsd-net@freebsd.org Subject: Re: ssh over WAN: TCP window too small Message-ID: <55DD2A98.2010605@stankevitz.com> In-Reply-To: <20150826010323.GN33167@funkthat.com> References: <55DCF080.7080208@stankevitz.com> <20150826010323.GN33167@funkthat.com>
next in thread | previous in thread | raw e-mail | index | archive | help
John-Mark, Thank you for your reply. On 8/25/15 6:03 PM, John-Mark Gurney wrote: > Chris Stankevitz wrote this message on Tue, Aug 25, 2015 at 15:47 -0700: >> # cat /dev/urandom | ssh root@host 'cat > /dev/null' > > Don't use this for testing... use /dev/zero or some other device > that can produce data faster than this... Okay. As I'm sure you can imagine, I used urandom to avoid compression artifacts. My urandom produces data at ~300 Mbps... but I will use /dev/zero from now on. > So, our SSH does have the HPN patches: > https://www.psc.edu/index.php/hpn-ssh > > and the README says: > BUFFER SIZES: > - if HPN is disabled the receive buffer size will be set to the OpenSSH default > of 64K. Yes... I spent some time reading that document and fretting over whether or not HPN was really incorporated in my setup. I "confirmed" that it was available and enabled by setting "HPNDisabled no" and restarting sshd (on both sides) without complaint. I'm half-tempted to build from ports to be certain. > Looks like there are undocumented options like TCPRcvBuf that you can > use to adjust the recv buffer window... According to the HPN README the default (which I am using) is the "system wide TCP receive buffer size". I don't know what value that is or where it comes from (net.inet.tcp.???). I will experiment with TCPRcvBuf. > We have code that will auto grow > buffer sizes properly so that slow connections won't use up too much > buffer space... That is what I expected, although I believe openssh tries thwart/limit this by requesting particular buffer sizes (I'm really unqualified to talk about this). And it is my understanding that HPN undoes these limitations although I'm not sure if it opens the door to FreeBSD having full control or uses its own voodoo. > In a quick test of mine, I'm seeing a buffer size of ~520k from my > MacOSX box, and ~776k from my 9.2-R box... Server in both cases is > a June -CURRENT Thank you for those numbers. Since my system is basically stock, I wonder if my bad behavior is an artifact of something on my network. Did you invoke ssh more or less as "cat /dev/zero | ssh root@host 'cat > /dev/null'"? Are you quoting S-BCNT numbers? > netstat -xAanfinet is helpful on this... That is brilliant! I was using pcap and wireshark to deduce some of that info. I include my sender and receiver netstat's below for the ssh-ing /dev/zero. It differs from iperf (which works well), most notably in S-BCNT (~1MB for iperf, ~64kB for ssh). I think in my case the question is: - who is keeping S-BCNT so low (openssh, HPN, or FreeBSD)? - Is the limitation introduced by the sending or receiving system? - what is the mechanism by which S-BCNT grows when using ssh over long/fat pipes? Thank you again, Chris SSH Sender Recv-Q 0 Send-Q 50132 R-MBUF 0 S-MBUF 16 R-CLUS 0 S-CLUS 14 R-HIWA 66052 S-HIWA 82852 R-LOWA 1 S-LOWA 2048 R-BCNT 0 S-BCNT 57344 R-BMAX 528416 S-BMAX 662816 rexmt 0.29 persist 0 keep 7199.98 2msl 0 delack 0 rcvtime 0.01 SSH Receiver Recv-Q 0 Send-Q 36 R-MBUF 0 S-MBUF 1 R-CLUS 0 S-CLUS 0 R-HIWA 66052 S-HIWA 33700 R-LOWA 1 S-LOWA 2048 R-BCNT 0 S-BCNT 256 R-BMAX 528416 S-BMAX 269600 rexmt 0.24 persist 0 keep 7199.96 2msl 0 delack 0.06 rcvtime 0.03
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55DD2A98.2010605>