From owner-freebsd-stable@freebsd.org Fri May 21 05:15:23 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8D020650FB6 for ; Fri, 21 May 2021 05:15:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-20.consmr.mail.gq1.yahoo.com (sonic301-20.consmr.mail.gq1.yahoo.com [98.137.64.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FmZYk26Q4z3mKv for ; Fri, 21 May 2021 05:15:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1621574120; bh=TEn4m8EJiHNbqWAr+k8mcuRm9faARQooV00sHtB+J0i=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=aNKPkx8DBja82D6sMGYlKdASwJ8Impay2BJCECAUYuolQQIE2zWXNxBgVcp3kXpwQ15kYcIIrRwcOrsv81P5+7a77IvZsvjpkSKAbVz5DDWQjoQ5Hzwvl5+wzJYxTIE92EGc7Vk/sVcCI0aelLwiKJoinGQGr/wKVbfo4UWpaFA8jxJL8nwYd7vehYX4ee5beWd93rlpx04lxRcLWDVNUbcPdm+PqGiFoWPCSaDYL1Zzz197Y/KOaQ84DMoEUA89TIl/BTD63ZtjFBHwEi7vwm9aQCOeSKaj2FUICwRuUika8UK6CteY18flswoFxOL/MaxZsUEpTlmJsyDMUbt44Q== X-YMail-OSG: EDNx4sMVM1mNJPPHRVsc7gFosJfQF0P2EwgJSRp0cLG_7SD1YJ_eJSrjakJ0MCi wFobT6V1isLANl910yrsmiB5Jm5WsSnkgcTvYHRu3WJd1YhpYIulv0rVQ6cq0SiAvPhowFwuGAj8 1yQ68MIRDJoxcpQhCADB2XRWJcbU9ku.stFjlc80XxPM86Q8AmWyEPFbVpf8TK9I2dmfT6S5Fy8V cgQZGi1VZ48EEg7m13.uqJl7tAWgKY4x3g0PX3XZK8fFc.qIiVeC1vLmKJTk_BoAI8fNgsUY5lVs UksXz0yfchShQeit6ffX0Lv7DEIGCFE888QrhD1s8sQujOb9vv_BMkP4OvV3njD7uCsvgeh_kiQN Hv2g8UdtgAcBDTXEkJAaqmvvRcpBf93EqoFqp0hrMcFGvN82T3Pw2W7ipi1BJzl2tJ8tjaPKsc8w TUxaCHngaTJqlPiK2..qV45bCn81FmAMQOW37UYvxm4zf4fk4Xymfd9r.ZjFKodUxxokIRBGtMKe 4BEpNM_W4WtUN2fZQ_R9nUR9GGDS9a56ZmzFAbtlHodAW2AmnBwZscoBuq.y05eDZPrcHMfTKcu3 qF22ztLEtiyyiWOM6.I9gF0Qkxge3ljNDBzFc0qD5pyOjong_7uDnfgc0_Sfx1qv_A4OibfeveIn 7rNzOF6Q9Xqlows7eg5gHpNxB8h_BQH.vB9lN1aXWvFi.xG0cOUvp6IK3Wx35atSjiLaWJbD.qLn _5URgSVuyUEYI2s_pB1syHyIPh81Z5trrINaXXUdWk4HmX50z_pXGwwH35qFUTqaTo10Vqgei_DR X2brUJe.c6NXlpS5yGoQGRIXQQFs3_mA5YRAIzClcAsXk.HDscaND_JuR73TGpMTjOCeTwomdGpu _DjcmEHqgkodOhByeJ7wXoTJnzFuF7Cakc9QxU9906KiWlaDLQP8H7yHFaZZsW0.7xnbN4SmeSeU 0ZB7di6NE3cQnr.uCCn66kctsTRDwhs7QlpHKHmrm_2vO3OT.8CqEt7IFF_7asN.aQR2sYePTVIo S411PiG5MFQNYyhUnercIJAEPS5WnPgIdnVbjl_74HFWHF4SbMH9zAyY3rF3STmmE1LGwpApvjw8 5sZdDW7HWuTQDeVC0oRvVey0vFH8dXU2avf_e9Be.Q0mWoWekPxxSMe.8xjS0LcJGz95BFIG6lZX KSTlctJBMA8v4Xxe9iHcjAFzjR6eDby6GMz4S17wU2KZgVADk6lWqyvtCGvow.2LQ83oiMFzMNwI jLjvGOk7BLN042AlgKbbxD8b8H1S8AtPL9h5j2XbdUqnde0qjp5sJKBxKil8n8._w.ZyU7mC8eWX Zt184RkbDcuOLBj7sjhHq1almdiG3Sz1BDTxJpvP48hGiUUfkxKbHpgDiLyKX91Ut4aUNpmREGQP ZfD95gLFZ8Hs9dUNPRz_LwB3_BW0EbnALHRx.d0QZLhFOz87cR_EcKm3kMlbrXUW_7nasE0Q2OC3 Gr8xyH2Irm8M2YkbCz.8N8Afdqr3Jx5qm3feEClMu_oiHC5LjAnFZMgOTMks2MNahl97EJE.yyNP _Ofjjel_CLCQpyvBoiSeZ0o6iPLmITXZRgbueXEKVYRASMekmpTMbqhFKCJrnNo4len0JnYN1Vl2 eoCoW27UE2dDFpDrsiU1ap_3eFVOawVQCmKUcWM5DcnddFQwASZuVM2awXenN1xfspXpyS1zyyz9 VQsPPtX56g0ubqPjq_r25i3AEYzUke5f4KbUKsys0KxswBvDGg9YbjPUnXnxVuQ3Wnl0X_JjQ9aR ZJ0phCyd_OnuzhSPF4A9pqSIbLZwcMwtPVTL5d.554FPRFNxYqrN1eq99YieaD8Q597kZRTK9ccw pn47_tyk7ZM.8VzWmG82ABtPgMoUM1RgrFQtqLfM3d9E2DF82XbRuFLBxJyrq2Gk6vpw4FT0pLTi GVqOOnz5GZwfyNghBNRh0hAwk3GnyFjS6EQpVbKev_vJjHsZP4rZiSZA7Re6ACyNMUfi.B7F7x7d sUWaTIafn86cFpXboq1pT70x2SjINeVQdjG0kEeXHkY_WcH1qhVCeczLg6YrL_SMYoG0vGSnscH2 r49wBd6saveWG7z7ZQ4QDzg3TTuCkalhhQNLqzI4fJaFbcUkARYw1hGKmNVua5TTiFN1LlwRYtdS w.63tJIS_Fl40zdYyrvEjtygEFel2PBtOqg4t9_Mrk3H0x0yi0S0.gAtF5MHEnAZy1TurxSSmmCS XtHDpozcvCzU4kzczMELz7b75Q.8MsOfOdo679Ow27IxRZtKYoQfMb6gQAT7XDUfTXBpcAQvJjHd wjDq3K4.XSWdhHSqNKwziBGdLK_XBrMVQZvydt1959W3PzJOWxYNyKgEDb7qXg4ET3r8a3d5re2M yskWZ7sBczlDRmh3TkvDSjI3heQMMMECtDJCPRix_0Aldg9nfY0YvubSLgYJcfFgdZtLvK99yaIz O56MmzpcBUJjj8ADCGABhXNNVDY76VM0XmITT7YTa9cVDNDlCfWUGzMxCRzXHAsiuKLjB4LuBVh3 jm5kdjzCztM.nn.Gm2RWkJ5EPGTYm_hz9eURLkeVofs9dq41nQ2EFxRj7qaWR6s.6klBAGnzBGHZ dhZOR8CoBMxEIijHIWw-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Fri, 21 May 2021 05:15:20 +0000 Received: by kubenode541.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 5821523a78ec2819da1c1df682fd7761; Fri, 21 May 2021 05:15:15 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) From: Mark Millard In-Reply-To: Date: Thu, 20 May 2021 22:15:13 -0700 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <623369D9-5EE5-4FEF-B9AD-56499E8F1C09.ref@yahoo.com> <623369D9-5EE5-4FEF-B9AD-56499E8F1C09@yahoo.com> <04D7264A-206B-4281-B452-779B01EA3327@yahoo.com> <34E915B3-30DF-408C-A931-C39188F3EB0F@yahoo.com> To: Rick Macklem X-Mailer: Apple Mail (2.3654.80.0.2.43) X-Rspamd-Queue-Id: 4FmZYk26Q4z3mKv X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.64.146:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.64.146:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.146:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.146:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 May 2021 05:15:23 -0000 [Direct drive connection to machine: no problem.] On 2021-May-20, at 21:40, Mark Millard wrote: > [main test example and main/releng/13 mixed example] >=20 > On 2021-May-20, at 20:36, Mark Millard wrote: >=20 >> [stable/13 test: example ends up being odder. That might >> allow eliminating some potential alternatives.] >>=20 >> On 2021-May-20, at 19:38, Mark Millard wrote: >>>=20 >>> On 2021-May-20, at 18:09, Rick Macklem wrote: >>>>=20 >>>> Oh, one additional thing that I'll dare to top post... >>>> r367492 broke the TCP upcalls that the NFS server uses, such >>>> that intermittent hangs of NFS mounts to FreeBSD13 servers can = occur. >>>> This has not yet been resolved in "main" etc and could explain >>>> why an RPC could time out for a soft mount. >>>=20 >>> See later notes that I added: soft mount is not required >>> to see the problem. >>>=20 >>>> You can revert the patch in r367492 to avoid the problem. >>>=20 >>> If I understand right, you are indicating that this would >>> not apply to the non-soft mount case that I got. >>>=20 >>>> Disabling TSO, LRO are also de-facto standard things to do when >>>> you observe weird NFS behaviour, because they are often broken >>>> in various network device drivers. >>>=20 >>> I'll have to figure out how to experiment with such. Things >>> are at defaults rather generally on the systems. I'm not >>> literate in the subject areas. >>>=20 >>> I'm the only user of the machines and network. It is not >>> outward facing. It is a rather small EtherNet network. >>>=20 >>>> rick >>>>=20 >>>> ________________________________________ >>>> From: owner-freebsd-stable@freebsd.org = on behalf of Rick Macklem = >>>> Sent: Thursday, May 20, 2021 8:55 PM >>>> To: FreeBSD-STABLE Mailing List; Mark Millard >>>> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result = over nfs (in a zfs file systems context) >>>>=20 >>>> Mark Millard wrote: >>>>> [I warn that I'm a fairly minimal user of NFS >>>>> mounts, not knowing all that much. I'm mostly >>>>> reporting this in case it ends up as evidence >>>>> via eventually matching up with others observing >>>>> possibly related oddities.] >>>>>=20 >>>>> I got the following odd sequence (that I've >>>>> mixed notes into). It involved a diff -r over NFS >>>>> showing differences (files missing) and then a >>>>> later diff finding matches for the same files, >>>>> no file system changes made on either machine. >>>>> I'm unable to reproduce the oddity on demand. >>>>>=20 >>>>> Note: A larger scope diff -r originally returned the >>>>> below as well, but doing the narrower diff -r did >>>>> repeat the result and that is what I show. (I >>>>> make no use of devel/ice .) >>>>>=20 >>>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >>>>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD >>> . . . >>>>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py >>>>>=20 >>>>> Note: The above was not expected. So I tried: >>>>>=20 >>>>> # ls -Tld /mnt/devel/ice/files/* >>>>> -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 = /mnt/devel/ice/files/Make.rules.FreeBSD >>> . . . >>>>> -rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 = /mnt/devel/ice/files/patch-scripts-TestUtil.py >>>>>=20 >>>>> Note: So that indicated that the files were there on the >>>>> machine that /mnt references. So attempting the original >>>>> diff -r again: >>>>>=20 >>>>> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >>>>> # >>>>>=20 >>>>> (Empty difference.) >>>>>=20 >>>>> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*" >>>>> the odd result of the diff -r no longer happened: no >>>>> differences reported. >>>>>=20 >>>>>=20 >>>>>=20 >>>>> For reference (both machines reported): >>>>>=20 >>>>> . . . >>>>> The original mount command was on CA72_16Gp_ZFS: >>>>>=20 >>>>> # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/ >>>> The likely explanation for this is your use of a "soft" mount. >>>> - If the NFS server is slow to respond or there is a temporary = network issue, >>>> the RPC request can time out and then the >>>> syscall can fail with EINT/ETIMEDOUT. Since almost nothing, = including the >>>> readdir(3) libc functions expect syscalls to fail this way... >>>> Then the cached directory is messed up. >>>> Doing the "ls" read the directory again and fixed the problem. >>>>=20 >>>> Try to reproduce it for a mount without the "soft" option. >>>> (If a mount point is hung, due to an unresponsive server "umount -N = /mnt" >>>> can usually get rid of it.) >>>> Personally, I thought "soft" was a bad idea when Sun introduced it = in NFS in 1985 >>>> and I still feel that way. >>>> --> If you can reproduce it without "soft" then I can't explain it. >>>> To be honest, the directory reading/caching code in the NFSv3 = client >>>> hasn't changed significantly in literally decades, as far as I = can remember. >>>=20 >>> Well . . . trying an even wider scope diff than >>> the original . . . >>>=20 >>> # umount /mnt/ >>> # mount -onoatime 192.168.1.170:/usr/ports/ /mnt/ >>> # diff -r /usr/ports/ /mnt/ | more >>> Only in /mnt/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_ >>> Only in /usr/ports/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cp= p__js__src25.cpp >>> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD >>> Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules >>> Only in /usr/ports/devel/ice/files: patch-cpp-Makefile >>> . . . >>> Only in /usr/ports/devel/ice/files: = patch-python-test-Slice-unicodePaths-run.py >>> Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py >>> Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py >>> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py >>>=20 >>> So the devel/ice files showed up again. >>>=20 >>> But 2 other lines show up, one finding a file supposedly only >>> on /mnt/. . . >>>=20 >>> QUOTE >>> Only in /mnt/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_ >>> END QUOTE >>>=20 >>> That seems to be a truncated file name. Looking directly on the = machine that >>> /mnt/ references (hitting tab at the end of the partial name to show = a >>> list): >>>=20 >>> # ls -Tld = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_ >>> = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_gen-config.sh =20 >>> = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_js-confdefs.h =20 >>> = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_Unified__cpp__js__src0.cpp =20 >>> = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_Unified__cpp__js__src1.cpp =20 >>> . . . =20 >>> = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_Unified__cpp__js__src9.cpp =20 >>> = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_include_js-config.h =20 >>>=20 >>> The other machine agrees (machine-local usage). >>>=20 >>> The other of the 2 new names is one of the matches to the prefix: >>>=20 >>> QUOTE >>> Only in /usr/ports/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cp= p__js__src25.cpp >>> END QUOTE >>>=20 >>> For reference: I've not gotten any console messages about >>> anything during these. >>>=20 >>>> One additional thing to note is that cached directory contents are = invalidated >>>> when the directory's ctime changes. >>>=20 >>> I'm not aware of anything that should have been touching the >>> /usr/ports file systems on either machine any time near my >>> diff activities. (I'm the only system user.) >>>=20 >>>> I am not sure how/if/when ZFS changes a >>>> directory's ctime. However, if it was badly broken, I'd hear about = this a lot. >>>> (If the ZFS change to ZoL has changed its ctime handling, that = might also explain it >>>> and I'll be hearing a lot more soon as FreeBSD13 becomes adopted. I = never use ZFS and, >>>> as such, never test with it.) >>>=20 >>> I recently decided to try using bectl, which lead to my recent >>> ZFS-based system experiments. >>>=20 >>> This means I can boot the stable/13 or main [so: 14] that >>> I last built and try the same experiments with the same >>> /usr/ports file sysystems. releng/13 's release/13.0.0 , >>> stable/13 , and main are all non-debug builds as stands. I >>> could add debug builds of any or all, but it would take >>> a while. (aarch64 4-core Cortex-A72 contexts.) >>>=20 >>>> --> For UFS, if you use mtime, directory caching does not work as = well, which is >>>> why the client directory caching code uses ctime and not mtime = to detect that >>>> a directory has changed and cached directory blocks need to be = invalidated. >>>>=20 >>>> Jason Bacon did report a directory reading issue some months ago = that never >>>> quite got resolved, although I recall he said he couldn't reproduce = it after a >>>> system update, so he thought it was related to some local change he = had made. >>>> (I can't remember his email or I'd add him to the cc so he could = remind me what >>>> his case was. I do recall it being somewhat reproducible and = happened for both >>>> UFS and ZFS.) >>>>> The network is just a local EtherNet. >>>>=20 >>>=20 >>=20 >>=20 >> stable/13 got similar "diff -r /usr/ports/ /mnt/ | more" results but >> /mnt/devel/electron12/files indications of the = /usr/ports/devel/ice/files >> ones. It did again start with: >>=20 >> Only in /mnt/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_ >> Only in /usr/ports/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cp= p__js__src25.cpp >>=20 >> for this rather wide range diff -r . It continued with: >>=20 >> Only in /mnt/devel/electron12/files:=20 >> Only in /mnt/devel/electron12/files: package.json >> Only in /mnt/devel/electron12/files: = patch-apps_ui_views_app__window__frame__view.cc >> Only in /mnt/devel/electron12/files: = patch-ash_display_mirror__window__controller.cc >> Only in /mnt/devel/electron12/files: patch-base_BUILD.gn >> . . . >>=20 >> It finished with: >>=20 >> Only in /mnt/devel/electron12/files: yarn.lock >> Only in /mnt/devel/electron12/files: = =D6=8F=DC=A62^H >> Only in /mnt/www/chromium/files: patch-chrome_browser_chrome__browser >> Only in /usr/ports/www/chromium/files: = patch-chrome_browser_chrome__browser__main__posix.cc >>=20 >>=20 >> That last is the only /usr/ports/ prefixed path this time: the >> only one where it was under /mnt/ that something appeared to >> be missing. >>=20 >> It appears that the file name on the line after the yarn.lock >> line is garbage with no matching file present when using ls >> on the system that /mnt/ references. >>=20 >> Locally on each machine the devel/electron12/files/* files >> are shown by ls as present ( through yarn.lock ). >>=20 >> NOTE: >> I find it odd that the local /usr/ports/ ended up being where >> most of the files were reported as missing, instead of under >> /mnt/ : Wrong side for a network/network-protocol issue? >>=20 >>=20 >> For reference (David W. indicated I should look at ifconfig >> for figuring out controlling TSO and such so I figured I'd >> show the default ifconfig output): >>=20 >> # ifconfig >> lo0: flags=3D8049 metric 0 mtu 16384 >> options=3D680003= >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 >> inet 127.0.0.1 netmask 0xff000000 >> groups: lo >> nd6 options=3D21 >> ue0: flags=3D8843 metric 0 = mtu 1500 >> = options=3D68009b >> ether REPLACED >> inet 192.168.1.148 netmask 0xffffff00 broadcast 192.168.1.255 >> media: Ethernet autoselect (1000baseT ) >> status: active >> nd6 options=3D29 >>=20 >> # ifconfig >> genet0: flags=3D8843 metric 0 = mtu 1500 >> = options=3D68000b= >> ether REPLACED >> inet6 REPLACED%genet0 prefixlen 64 scopeid 0x1 >> inet6 REPLACED prefixlen 64 autoconf >> inet 192.168.1.170 netmask 0xffffff00 broadcast 192.168.1.255 >> media: Ethernet autoselect (1000baseT ) >> status: active >> nd6 options=3D23 >> lo0: flags=3D8049 metric 0 mtu 16384 >> options=3D680003= >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >> inet 127.0.0.1 netmask 0xff000000 >> groups: lo >> nd6 options=3D21 >>=20 >>=20 >> # uname -apKU >> FreeBSD CA72_16Gp_ZFS 13.0-STABLE FreeBSD 13.0-STABLE #1 = stable/13-n245474-fb34817c686c-dirty: Sat May 1 02:27:02 PDT 2021 = root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13S-CA72-nodbg-clang/usr/13S-src/arm64.= aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1300504 1300504 >>=20 >> # ~/fbsd-based-on-what-commit.sh=20 >> branch: stable/13 >> merge-base: fb34817c686cc130449325499870e36979899801 >> merge-base: CommitDate: 2021-05-01 00:56:57 +0000 >> fb34817c686c (HEAD -> stable/13, freebsd/stable/13) param.h: bump = __FreeBSD_version for commits efe7f12cd37b and 9781105bea58 >> n245474 (--first-parent --count for merge-base) >>=20 >> # uname -apKU >> FreeBSD CA72_4c8G_ZFS 13.0-STABLE FreeBSD 13.0-STABLE #1 = stable/13-n245474-fb34817c686c-dirty: Sat May 1 02:27:02 PDT 2021 = root@CA72_4c8G_ZFS:/usr/obj/BUILDs/13S-CA72-nodbg-clang/usr/13S-src/arm64.= aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1300504 1300504 >>=20 >> # ~/fbsd-based-on-what-commit.sh=20 >> branch: stable/13 >> merge-base: fb34817c686cc130449325499870e36979899801 >> merge-base: CommitDate: 2021-05-01 00:56:57 +0000 >> fb34817c686c (HEAD -> stable/13, freebsd/stable/13) param.h: bump = __FreeBSD_version for commits efe7f12cd37b and 9781105bea58 >> n245474 (--first-parent --count for merge-base) >=20 > Both systems running main: >=20 > # diff -r /usr/ports/ /mnt/ | more > Only in /mnt/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_ > Only in /usr/ports/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cp= p__js__src25.cpp > Only in /mnt/devel/electron12/files:=20 > Only in /mnt/devel/electron12/files:=20 > Only in /mnt/devel/electron12/files: patch-chrome2 > Only in /usr/ports/devel/electron12/files: = patch-chrome_browser_media_webrtc_webrtc__logging__controller.cc > Only in /usr/ports/devel/electron12/files: = patch-chrome_browser_ui_webui_settings_appearance__handler.h > Only in /usr/ports/devel/electron12/files: = patch-components_previews_core_previews__features.cc > Only in /usr/ports/devel/electron12/files: = patch-ui_compositor_compositor.cc > Only in /mnt/devel/electron12/files: = =D6=8F=DC=A62^H >=20 > (That was all that was listed.) >=20 > # uname -apKU > FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 = main-n246411-a6ca7519f89c-dirty: Sat May 1 19:07:50 PDT 2021 = root@CA72_4c8G_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400013 1400013 >=20 > # ~/fbsd-based-on-what-commit.sh=20 > branch: main > merge-base: a6ca7519f89c52e9fab205cded0f2bf32d914cd6 > merge-base: CommitDate: 2021-05-01 00:58:11 +0000 > a6ca7519f89c (HEAD -> main, freebsd/main, freebsd/HEAD) powerpc64: = Optimize radix trap handling a little more > n246411 (--first-parent --count for merge-base) >=20 > # uname -apKU > FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #1 = main-n246411-a6ca7519f89c-dirty: Sat May 1 19:07:50 PDT 2021 = root@CA72_4c8G_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400013 1400013 >=20 > # ~/fbsd-based-on-what-commit.sh=20 > branch: main > merge-base: a6ca7519f89c52e9fab205cded0f2bf32d914cd6 > merge-base: CommitDate: 2021-05-01 00:58:11 +0000 > a6ca7519f89c (HEAD -> main, freebsd/main, freebsd/HEAD) powerpc64: = Optimize radix trap handling a little more > n246411 (--first-parent --count for merge-base) >=20 >=20 >=20 > I tried main on the /usr/ side with releng/13 's release/13.0.0 > where /mnt/ references and got: >=20 > # diff -r /usr/ports/ /mnt/ | more > Only in /mnt/devel/electron12/files: package.json > Only in /mnt/devel/electron12/files: = patch-apps_ui_views_app__window__frame__view.cc > Only in /mnt/devel/electron12/files: = patch-ash_display_mirror__window__controller.cc > Only in /mnt/devel/electron12/files: patch-base_BUILD.gn > . . . > Only in /mnt/devel/electron12/files: = patch-weblayer_browser_system__network__context__manager.cc > Only in /mnt/devel/electron12/files: = patch-weblayer_common_weblayer__paths.cc > Only in /mnt/devel/electron12/files: yarn.lock > Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD > Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules > Only in /usr/ports/devel/ice/files: patch-cpp-Makefile > . . . > Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py > Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py > Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py > Only in /mnt/games: 0ad > Only in /mnt/games: 0verkill > Only in /mnt/games: 2048 > . . . > Only in /mnt/games: zaz > Only in /mnt/games: zhlt > Only in /mnt/games: ztrack >=20 > No obvious garbage or truncated names. Another mix of > /mnt/ vs. /usr/ being the "missing" side. >=20 > NOTE: > So far I do not see an obvious reason to prefer any > specific one of releng/13 vs. stable/13 vs. main > at either end of the connection for the vintages that > I happen to have in place for them. >=20 Just to be sure, I shutdown the machine that /mnt was referencing and moved the drive to the other machine, directly connected. # zpool import -f -N -R /mnt -t zroot zptmp # zfs mount zptmp/usr/ports # diff -r /usr/ports/ /mnt/usr/ports/ | more # So: The diff -r works for this context. The remote status is somehow involved in producing the type of problem. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)