From owner-freebsd-net@freebsd.org Thu Aug 27 23:17:32 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 249DD9C327B for ; Thu, 27 Aug 2015 23:17:32 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ob0-x232.google.com (mail-ob0-x232.google.com [IPv6:2607:f8b0:4003:c01::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DDFA1164A for ; Thu, 27 Aug 2015 23:17:31 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: by obkg7 with SMTP id g7so29308552obk.3 for ; Thu, 27 Aug 2015 16:17:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=miTuGhkVBKNKUovcaODTjGReUo1940+k9wTxWtdNW50=; b=rOzhTPl6DdRslDjfRGNe0/dwtUJvgIBmAx223m4Dz5dO6DIoY48jsxggjyxob0tJxb o5680SsgaGvEmKW/h7Ck2FzUWWHED+h+8+Q9n0cRNg9Ye6s7loSPJBrqWfY9QixOExbk mgRcPtdPi8U8xIuoa5JH6gK0rOf62Q+D5HYIPUD6i6DLyZWJ7llQ63SGTj8boCgp4Q/o kMw0tJtAW4XmzRbkOIIpeY90DzoteoOIxFGydJ+oOCi6JSEEZjZG6nIlK9PVPytIUEx+ 5bBklzmXkf2dGVTtkbRv9KESpJSjT9l5xjYR3nXL43ISGU7RiA9nwvHYi4cCm25/znHZ frRQ== MIME-Version: 1.0 X-Received: by 10.182.97.10 with SMTP id dw10mr4345237obb.60.1440717450657; Thu, 27 Aug 2015 16:17:30 -0700 (PDT) Sender: kob6558@gmail.com Received: by 10.202.173.83 with HTTP; Thu, 27 Aug 2015 16:17:30 -0700 (PDT) In-Reply-To: <55DD0453.3040803@stankevitz.com> References: <55DCF080.7080208@stankevitz.com> <27420EDC-5816-4B9E-A834-E4A035B8411C@lists.zabbadoz.net> <55DD0453.3040803@stankevitz.com> Date: Thu, 27 Aug 2015 13:17:30 -1000 X-Google-Sender-Auth: jeQjGc-LGBpBOxq56kw6F2lI_js Message-ID: Subject: Re: ssh over WAN: TCP window too small From: Kevin Oberman To: Chris Stankevitz Cc: "Bjoern A. Zeeb" , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 23:17:32 -0000 On Tue, Aug 25, 2015 at 2:12 PM, Chris Stankevitz wrote: > On 8/25/15 4:11 PM, Bjoern A. Zeeb wrote: > >> >> On 25 Aug 2015, at 22:47 , Chris Stankevitz wrote: >>> >>> Can anyone recommend some tools/tricks to figure out what in FreeBSD >>> and/or >>> >> >> base SSH is limiting the send/recv buffer and/or TCP window? > >> >> if you have the memory, try these sysctls: >> >> kern.ipc.maxsockbuf=146800640 >> net.inet.tcp.recvbuf_max=67108864 >> net.inet.tcp.sendbuf_max=67108864 >> > > Bjoern, > > Thank you for the reply. Before your suggestion my sysctls are: > > kern.ipc.maxsockbuf=2097152 > net.inet.tcp.recvbuf_max=2097152 > net.inet.tcp.sendbuf_max=2097152 > > Each of these is much larger than the limit I am experiencing (~64,000). > So I [naively] expect changing these values will have no effect. Let me > try... > > ... okay... sure enough the sysctrl changes you suggest did not change the > 64,000 bytes-in-flight limit I am experiencing. > > Thanks for the idea (and keep 'em coming!), > > Chris > My former employer, ESnet, was heavily involved in moving very large amounts of data (petabytes) over very long (intercontinental), very fat (100G) pipes. In an effort to improve customer satisfaction they have done extensive research into the issues involved and have published much of it at http://fasterdata.es.net. In particular, they have documented the issues with ssh over long latency links at http://fasterdata.es.net/data-transfer-tools/say-no-to-scp/. It is VERY hard to get good performance on high latency links in the bast of cases and, unfortunately, ssh/scp makes it not the best of cases. -- Kevin Oberman, Goat herder and Retired Network Engineer