From nobody Sun Feb 13 00:50:21 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9D3CD194B9CC for ; Sun, 13 Feb 2022 00:50:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jx80X3Ymmz4gfB for ; Sun, 13 Feb 2022 00:50:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1644713428; bh=ZzU0lW+e3SDAMJ77Bfcj9fefML85XPlBc40hOFUYloU=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=jmCnRPmOlj/Q+cvovdTaDAQMD6yoDG8lBpy6TeP+7lHecciNmHBFCAjklrAU54U7jn8rQVvk2poJ7yfD6pUaavMlptg7fFA/xdirm4Isl9nQsVrdZ2K0tj8MD9S57utNdyzsxlPWN08sRI/R89JtAJ8T8DzJ75I5/hUrNaNQe+oXLWcdg113A3iRYQJ9SLdUCFeVPkOPhF7A2O/J/xqwN2jAZbzmmwsnELZf/4NZrfF+AJnGRNnkrMdHWeiBCmF+IRkxDLVhPiR2xinZk7Ur1BKd0vrefn+JFI5wDqBAXI4ib9MH6KO0vJxrcbkDLmj4a7Kx/KyINSfz/vXwUdgo6g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1644713428; bh=b0GJ2rSE+Qy0lgg5v9pP12qDQRZgLJU4dUk/z9K86ng=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=dx1X66dMhE2oJG2QgHsuFRl8K3HqUxn97Tci9Oft4ePP+k8n2+LDKKU7YNv2AnnVo548Ixt+zTTsuJTMoa60UvYaBeyZEroiqHE6mm9BNiELJqB3gw/aStrF1iTCbHeBB8XOgae8XmdVZfcwWo1JMvG2Kk/Htc2QlYWYQGa6xtFtRFSWTg7MWrZl3o4qBHLVvix//BAaoWMWVh34WXuv2n2a8rbHw+lGa6HNXKZCh8oQPuR0BBlmqK+AJ5Ulfh+G8CVQQ0kaXuoP8RKG8YkJU47UfLI5H/yxg8bIgQMz4i0J3/5SSl6DI8mNPdaIGzLEUCiX3Al771dpwXoBrF/niw== X-YMail-OSG: Oo4QfHAVM1nKmsrms9fwglMzzRPMUWbP1gH78on4j.D5vIeqgiYO81BZ3H0oTBg gSgKcXl9B7wUYLIMcj2P6Grj8Ep0E0JfPMk6fFo.0EV7kU9um1WX1QdtMGB_XN1.Xn35evq0Hgn0 ORXKCbqSnew6nPnpxgCxVcGTUbalxL7tVU_g3fgeAvCkGVhEwVwWqCPIYFCRuFAoX3vZuL0x_Zn3 XFh4.gMwRkC0FcEL2g_hNDRxGscKt2lSjDholN_iKpq7m90UN31SHVVEJlQ5gZEfbZYwofNrZO9y mu6uQZa4A0T7_SGaUsTzmBki1Wz6r_oh7s2aIK9.N2h7thCWuIDjsYeC8IFklLWhbU6VfWCW5sQB y9RZZT8i3YgWbdoC7tfQL80hwsU3mokkIxqsBzcHMG4FK5cPMGXOdxuwDhLRzDofuSdFOmts.OXk KbTotgcCjFEwSyJYqNCTl1HeNjIWg7sBxkhrJ0N4OsR_T2DchMJbGTvFddUSUUXY.Hra7k2tNqsU sX5yoVHfNav_liyhvy3PrpJZc8sBwF813EvROq3_stlK0tbemupqyINyx1JtLmtsaMrAnrAQ5QEM Ode_c.ol.rJIrtuIo6QuumS3IFeZUJV3U.mRp01.XY4PJ9v1DRvaMxydBzv3S_UeKy9ihFx_tIPG Uf25WKxQBYYfwRiC0nftmO8Z53XvKSIim67EH61wYem1p66x6cO.q7TOg92TFFZXjqEitNGDJnLR nq_tcghNTfb5GCTzVEUJgnf1jBJ.UgLb_pgkB8pPdYo3mfFS6e3pp5DzOkZpdPWlO7xPAzOIHVrb wSvSdPjzQ_h53t.J4iSUDAkmLovL8X5eO4E3uQem_GxNiOnnNTtjpEuqjTerBUfrKZvOXLOCNhFg l2e3QzzXA4ITOLKmcmGiL1FiQreD1tpsZ020QnQGzes3u5Q5_wYj0A0gpQWX4X2l_VclCUVjuZS5 g0LSiQIsk74nKvI1hUm6RUoNgBUqYG5iC16_HO5TYeVY5n2cSMyEjZ8MdZc.Q3LGdj0xJYAQPnie Rw7fJsFTV7C7xsK_DpMPTamBuK3j2Gb_jvSXaRV7Z9DJT_63ceDncZMxZJsjkO2wJPEEQUGOP_2o YWFlAx5.0ccseOsGDIp8UTp_f_zYBkgANGxd8lXi3VGXHxF7ZdS0m2J2ftZQGiI98xdPjSG0Ob4p FICIa6cVz3aVZgA6IV6duAuOw8UD.zOtHEgEaCRcXVzdxLNy4nx7WrnSh0sn3pyOyw1lzgbVYfq1 ha8sTcXO3cV1FCvoUQu70D8uinKD.NJ6974usZQE_OmMhARl_zSiACxifsh0EQJ34iY1cdJBVTfs IIyzXM1.wJ6NVKE0OSgaf3IfsUpOQy34DOKZ5Mht01scNLfAIg_GgGqMuqriK4HVBOGvVxy8p4EM eTIiKWsvYFnLk_l2x3gDAOZWORzN0Wqq.boE3LBv_9.Kf0Vjm66vWLuZ3ok3LtCbesSPRLAFFwPr u_umEEBpaxqgxbpdp81_1LWo7OtI.ivbXqpH1bFa0hgEaTrJhNiAQ.Y8yVn7vFpyrEbiowLRPJrl I.tIcyf3OsV8GbOmnqEhXRSMq.Rw4KG6q.ztSQkK4P7462QcRCafOwInUrHwUYYlybfXmIx3Uuba bJif2Q2GgoTpzJIFS9CyKnG69HgisKmzEr3EC83gb7H1WHm4_q8_1ugA_szzp7bm0ekS7SY1wOWP WHTpuC8mQGa3pSUwKNLN4oHMVudbiyYg9d5TgFYfK3MEKbi295CZ_4xPOBzg6BM07KFSZS.aN.Lb 9.9jBC0nfwVYmVxRG8FWO7ZKE8RQqTYrDONz.KC_WkOKGS9oqSBOQ2zqNEYQZH3otHduOydA00OU tTb90te6OsyIF0tUaAH7k34x1Z08es60lA5t_ckJAPAXrL264vNPqD0hlq8nAYQcCxc9lRt4dVJ0 9sMPpC65A0pAUqqLiyWTsxGH8PKD0aBi0xSixlfXzk.pICbKweWz_V4rcPvIJbhkmwnaGGeZMvyu fk7Eui_AVrOHdWXDos27_BVvaD40fgfd3W63R2IRULz2KsEMIKoVhB8Wx5qjwAEYzLapruWOUajJ PU4bgKldsIpX2imVVOEKVOHGb8lFm5engl9tNCoj0XuEONTifsWsY0h.gmt4LgJ2XTpFFExdlfMh r.Tu4oObGFO_aACa1ljiycPT8DOx4aN7AZtVI0OlLcJ4PHMCOmMIMDwnM3kjC8aLhNkN287Ue2.o INfcxj7ZFKO72DB2YTCQ.FU.dImggOtE_8QI.Nht0AiOj0Jp1mzNin6QKPxxoc9fT0BkTBbY0ggb bsTQfUgSl6Pfi5mRNfiW7vqfk0oMSWQAXquOhYhVgHAevkQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Sun, 13 Feb 2022 00:50:28 +0000 Received: by kubenode550.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 610d5e8c6eb6651d1b47202bc370ae40; Sun, 13 Feb 2022 00:50:23 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Pi3 answers ssh only if outbound ping is running on -current From: Mark Millard In-Reply-To: <34F5C092-7C35-4CBB-9CC3-99E373D1F785@yahoo.com> Date: Sat, 12 Feb 2022 16:50:21 -0800 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20220212185618.GA37391@www.zefox.net> <34F5C092-7C35-4CBB-9CC3-99E373D1F785@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Jx80X3Ymmz4gfB X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=jmCnRPmO; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; MLMMJ_DEST(0.00)[freebsd-arm]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N Status: O Content-Length: 10702 Lines: 271 On 2022-Feb-12, at 13:32, Mark Millard wrote: > On 2022-Feb-12, at 10:56, bob prohaska wrote: >=20 >> For a few weeks now a Pi3 running -current will not respond to >> an incoming ssh connection unless an outbound ping process is = running. >>=20 >> Once the outbound ping is started via the serial console, incoming >> ssh connections are answered normally. Uname -a reports >> FreeBSD www.zefox.org 14.0-CURRENT FreeBSD 14.0-CURRENT #10 = main-n253073-6db44b0158c: Sat Feb 12 04:30:21 PST 2022 = bob@www.zefox.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 >>=20 >> A Pi4 running -current of a few days ago exhibits no such problems. >>=20 >> Another Pi3 running stable/13 has been behaving in the same way. >>=20 >> Both Pi3s successfully set time via ntp on reboot and will >> very briefly (one or two minutes) prompt for an ssh password, >> but no further progress is made and the login attempt times out. >> If the ssh login is attempted a second time, not even a password >> prompt comes back. >>=20 >> Ping times (to an adjacent machine on the same subnet are >> 64 bytes from 50.1.20.26: icmp_seq=3D2 ttl=3D64 time=3D0.978 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D3 ttl=3D64 time=3D0.967 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D4 ttl=3D64 time=3D1.088 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D5 ttl=3D64 time=3D0.983 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D6 ttl=3D64 time=3D1.007 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D7 ttl=3D64 time=3D1.075 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D8 ttl=3D64 time=3D1.020 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D9 ttl=3D64 time=3D1.044 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D10 ttl=3D64 time=3D1.026 ms >> 64 bytes from 50.1.20.26: icmp_seq=3D11 ttl=3D64 time=3D0.908 ms >>=20 >> That might be considered slow, but the correspondent machine >> is only a Pi2 running=20 >> FreeBSD www.zefox.com 14.0-CURRENT FreeBSD 14.0-CURRENT #3 = main-71d2d5adfe: Tue Dec 21 00:23:51 PST 2021 = bob@www.zefox.com:/usr/obj/usr/freebsd-src/arm.armv7/sys/GENERIC arm >>=20 >> If the outbound ping is started, an incoming ssh connection = established >> and the outbound ping subsequently stopped the running ssh connection >> silently freezes; no disconnect, but no response, not even echo. Some >> tens of seconds later, all inputs were responded to. Tried a second = time, >> the stoppage recurred, restarting the outbound ping eventually = restored >> responsiveness. >>=20 >> With the outbound ping stopped, an inbound ssh attempt silently = failed: >>=20 >> bob@raspberrypi:~ $ ssh -vvv 50.1.20.28 >> OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1, OpenSSL 1.1.1d 10 Sep 2019 >> debug1: Reading configuration data /etc/ssh/ssh_config >> debug1: /etc/ssh/ssh_config line 19: Applying options for * >> debug2: resolve_canonicalize: hostname 50.1.20.28 is address >> debug2: ssh_connect_direct >> debug1: Connecting to 50.1.20.28 [50.1.20.28] port 22. >> [enter key echoed] >> debug1: connect to address 50.1.20.28 port 22: Connection timed out >> ssh: connect to host 50.1.20.28 port 22: Connection timed out >> bob@raspberrypi:~ $ =20 >>=20 >> Thanks for reading and any insights. If I've omitted useful=20 >> details or tests please indicate. >>=20 >=20 > You have made multiple reports to the arm list for this issue > without anyone having managed to help. This report does have > more comparative context, which might help someone help. >=20 > It may be time to try other lists like freebsd-net and, > possibly, freebsd-hackers or freebsd-stable or > freebsd-current . >=20 > However, the best thing no matter where you go would be > to (approximately) bisect toward the back-to-back FreeBSD > version-pair on, say, stable/13 at which the the problem > goes from not-there to happening. ( stable/13 changes > slower and so has fewer versions to deal with. Also its > KBI may grow but is constrained to otherwise be more > stable [ relative to releng/13.0 ]. So you are less > likely to run into version compatibility problems > for the below suggestion.) >=20 > I'd recommend using kernel and world materials from: >=20 > https://artifact.ci.freebsd.org/snapshot/stable-13/?C=3DM&O=3DD >=20 > on a separate microsd card updated from a normal context, > avoiding builds. Remember that older stable/13 worlds can > run on newer kernels generally. So you might only need to > update the kernel after getting an initial, somewhat older > context in place. (It is not obvious if it is a kernel-only > problem or not.) If it is a kernel problem, you might be > able to put down a releng/13.0 world and never change it > during the approximate bisect activity. >=20 > For what https://artifact.ci.freebsd.org/snapshot/ has > available, this avoids having to build the versions. > It also allows checking if your builds are behaving > differently than the official snapshots do. >=20 > https://artifact.ci.freebsd.org/snapshot/ may not be able > to get you to the back-to-back FreeBSD version-pair: the > range might be wider. Sometimes the wider range is enough > by inspection of the types of commmits in the range. So > I'd report whatever range you find wihtout having done > any builds. >=20 > I'll note that I have no problem with connecting via ssh > to a RPi3B running my build of (line split for readability): >=20 > # uname -apKU > FreeBSD Rock64_RPi_4_3_2v1p2 14.0-CURRENT FreeBSD 14.0-CURRENT #28 > main-n252475-e76c0108990b-dirty: Sat Jan 15 23:39:27 PST 2022 > = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA53-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA53 > arm64 aarch64 1400047 1400047 >=20 > I have no stable/13 context set up for a RPi3B, only > stable/13's that have an untuned ZFS context. Still, > I wonder if that might operate well enough to test > the issue, despite the 1 GiByte of RAM limitation. I > may test that later today. Other than needing to put in place my u-boot.bin build that has usb_pgood_delay=3D2000 built-in, I had no trouble with booting and ssh'ing in to (line split for readability): # uname -apKU FreeBSD CA72_4c8G_ZFS 13.0-STABLE FreeBSD 13.0-STABLE #25 stable/13-n249004-a5f698599560-dirty: Sun Jan 16 15:07:11 PST 2022 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/13S-CA72-nodbg-clang/usr/13S-src/arm64.= aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1300524 1300524 # ~/fbsd-based-on-what-commit.sh -C /usr/13S-src/ branch: stable/13 merge-base: a5f69859956049b5153b0e1b67f8f4a99622dc6f merge-base: CommitDate: 2022-01-15 12:55:32 +0000 a5f698599560 (HEAD -> stable/13, freebsd/stable/13) Ignore = debugger-injected signals left after detaching n249004 (--first-parent --count for merge-base) SIDE NOTE After the above, my patched top reports: Mem: 32504Ki Active, 214888Ki Inact, 393248Ki Wired, 40960B Buf, = 321468Ki Free, 75516Ki MaxObsActive, 394108Ki MaxObsWired, 469624Ki = MaxObs(Act+Wir+Lndry) ARC: 316408Ki Total, 201575Ki MFU, 111090Ki MRU, 143360B Anon, 1024Ki = Header, 2551Ki Other 259140Ki Compressed, 346379Ki Uncompressed, 1.34:1 Ratio Swap: 3584Mi Total, 3584Mi Free, 75516Ki MaxObs(Act+Lndry+SwapUsed), = 469624Ki MaxObs(Act+Wir+Lndry+SwapUsed) So it is not an environment I'd want to do buildworld buildkernel on. But it looks to be usable for less memory intensive activities. END SIDE NOTE So I've looked and found (from today): = https://artifact.ci.freebsd.org/snapshot/stable-13/371633ece3ae88e3b3d7a02= 8c372d4ac4f72b503/arm64/aarch64/kernel.txz and downloaded it. Then I decided to try it with my normal boot media, leaving world as it is. So: # ls -Tld /boot/ker* drwxr-xr-x 2 root wheel 680 Jan 16 16:49:24 2022 /boot/kernel drwxr-xr-x 2 root wheel 680 Jan 4 23:08:57 2022 /boot/kernel.old # mv /boot/kernel /boot/kernorm # tar -xpf kernel.txz -C / # ls -Tld /boot/ker* drwxr-xr-x 2 root wheel 679 Feb 12 11:14:27 2022 /boot/kernel drwxr-xr-x 2 root wheel 680 Jan 4 23:08:57 2022 /boot/kernel.old drwxr-xr-x 2 root wheel 680 Jan 16 16:49:24 2022 /boot/kernorm (I choose to not replace the system's debug information --that is not stored under /boot/ but in with world files. So I did not download or install kernel-dbg.txz .) So now a reboot with loader defaults (for that boot environment in my context) will use the kernel that I got from: https://artifact.ci.freebsd.org/snapshot/stable-13/. . . [Hmm. Looks like the u-boot.bin is not sufficient to be sure that shutdown -r now will boot the RPi3B. =46rom power-on seems to boot so far. I might need another built-in setting added (or more) in order to allow the RPi3B to shutdown -r now well for the USB3 NVMe based SSD media that I'm using.] Still no trouble connecting and logging-in via ssh. For reference (line split for readability): # uname -apKU FreeBSD CA72_4c8G_ZFS 13.0-STABLE FreeBSD 13.0-STABLE #0 371633e: Sat Feb 12 19:06:49 UTC 2022 = root@FreeBSD-stable-13-aarch64-build.jail.ci.FreeBSD.org:/usr/obj/usr/src/= arm64.aarch64/sys/GENERIC arm64 aarch64 1300525 1300524 (I do not have matching source at this point.) Recommended experiment . . . Since I have a context working based on the kernel in: = https://artifact.ci.freebsd.org/snapshot/stable-13/371633ece3ae88e3b3d7a02= 8c372d4ac4f72b503/arm64/aarch64/kernel.txz I recommend that you try that exact same kernel in your stable/13 context. I recommend renaming the existing /boot/kernel before expanding the kernel.txz into / and so causing a new /boot/kernel/ to be filled in. If that makes things work after rebooting, then your kernel can be blamed. (More investigation to know more about what is going on in your kernel build.) But if the above does not make things work, that points to investigating alternate worlds from: https://artifact.ci.freebsd.org/snapshot/stable-13/. . . That is a messier context. I only do that with media that I can delete everything on, such as an independent microsd card: chflags -R noschg /mnt/ ; rm -fr /mnt/ ; various tar -xpf ???.txz -C /mnt/ commands --while not booted from the microsd card. Repeat for each snapshot tried. There is a bias to the world not being newer than the kernel. But since stable/13 's 371633ece3ae seems to work in my context, you might be able to hold the kernel invariant and just try different world versions in this messier context. Also: You might be be to find: https://artifact.ci.freebsd.org/snapshot/stable-13/. . . materials for the specific builds that you have been working with and do comparison/contrast with the behavior of your builds that had issues. Note: The above does not consider other networking configuration issues --that might not even be on RPi* devices. I'm not networking literate overall. =3D=3D=3D Mark Millard marklmi at yahoo.com