From nobody Tue Jun 27 06:34:34 2023 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qqw101pXzz4hq34; Tue, 27 Jun 2023 06:34:28 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "www.zefox.com", Issuer "www.zefox.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qqw0z5qLYz4134; Tue, 27 Jun 2023 06:34:27 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Authentication-Results: mx1.freebsd.org; none Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.17.1/8.15.2) with ESMTPS id 35R6YZV3080875 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Mon, 26 Jun 2023 23:34:35 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.17.1/8.15.2/Submit) id 35R6YYcl080874; Mon, 26 Jun 2023 23:34:34 -0700 (PDT) (envelope-from fbsd) Date: Mon, 26 Jun 2023 23:34:34 -0700 From: bob prohaska To: Mark Millard Cc: Jamie Landeg-Jones , freebsd-net@freebsd.org, freebsd-arm@freebsd.org Subject: Re: -current dropping ssh connections Message-ID: References: <202306212305.35LN5ITH069587@donotpassgo.dyslexicfish.net> <202306222238.35MMcoQm017939@donotpassgo.dyslexicfish.net> <601E35C4-CDE1-47A3-B2D2-29501AB5FB93@yahoo.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <601E35C4-CDE1-47A3-B2D2-29501AB5FB93@yahoo.com> X-Rspamd-Queue-Id: 4Qqw0z5qLYz4134 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7065, ipnet:50.1.16.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Mon, Jun 26, 2023 at 07:38:22PM -0700, Mark Millard wrote: > On Jun 26, 2023, at 19:00, bob prohaska wrote: > > > On Thu, Jun 22, 2023 at 11:38:50PM +0100, Jamie Landeg-Jones wrote: > >> bob prohaska wrote: > >> > >>> That seems worth a try. > >>> The notion of an ssh escape (~. in this case) finding its way into the data stream is new to me. > >> > >> Thinking again, that looks like corruption coming down the ssh connection. > >> For the ssh escape char to affect anything (note it needs to be preceeded > >> by a new line) it would have to be sent up the line. > >> > >> For an example, assuming ssh has the default escape char, look at the difference between: > >> > >> printf 'pwd;\n~.;echo sleeping.; sleep 5' | ssh -tt user@host > >> > >> and with the same printf, but no escape char: > >> > >> printf 'pwd;\n~.;echo sleeping.; sleep 5' | ssh -tt -e none user@host > >> > >> You can see how the former closes the connection due to the \n~. > >> > >> (The -tt forces a terminal/interactive session to be set up - normally, as we are > >> piping input to ssh in this case, the terminal isnt set up, and the escape character > >> isn't used - it's only recongnised in interactive sesions by default) > >> > >> I personally have "EscapeChar none" in my ssh_config, but I suspect this > >> is probably not the issue here, still, can't hurt to try it! > > > > Indeed, connection was dropped with escape character set to none. > > There does seem to be a some dependence on system load. Connections > > survive at light or no load and drop when the sshd side is busy. > > So how busy is the disk media (really the USB bus) from > somewhat before the failures happen to the failure point? > The question applies separately to both systems involved > in the ssh session. > I've no quantitative measure. Is there any command I could add to the swap activity logger that can report USB load? > I'll remind that on a RPi2B v1.1, RPi2B v1.2, or RPi3B* > the Ethernet and the 4 USB ports all share the one USB2 > path at an internal stage for the USB subsystem, if I > remember right. > > Saturating that USB2 path with data transfer bytes > to/from disk media might be able to block other activity > long enough to cause the Ethernet communications to have > problems for all I know. (Amount of data queued for > transfer would be relevant.) > > This might be a reason to avoid settings for the > combination vm.pfault_oom_attempts and vm.pfault_oom_wait > to values that could contribute to a sustained period of > saturating the USB channel. > At this point the vm.* settings are default, with no warnings on the serial console apparent.. > (I have no detailed knowledge of how the tradeoffs work > for the competing uses of the USB channel. So I can not > validly claim to know that the above is realistic.) > It is tempting to think USB saturation would account for the trouble. A tool similar to gstat, but for USB as a whole, would be helpful if it exists. It's perhaps notable that the dropped connections are seen in connection with usb-serial adapters more than interactive connections running shells. Thanks for reading! bob prohaska