From owner-freebsd-stable@freebsd.org Fri May 21 02:39:06 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0EF8C64E7A6 for ; Fri, 21 May 2021 02:39:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-20.consmr.mail.gq1.yahoo.com (sonic302-20.consmr.mail.gq1.yahoo.com [98.137.68.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FmW5P0pgCz3M5l for ; Fri, 21 May 2021 02:39:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1621564743; bh=6ODviWNaEPcesWcCWNmFoJn80K2HbB7gVy5Ltp5KgvB=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=UcrojeTh0SMkt2/XFPnkMi3vetHKWcma9irvbERbaJ6bBjTGk2954t35j3xvFqcI52Kh/j1MEa329Q2RsQ2vV99BcdaUtt3iWLBQiBWsoRSl/SrWsie9cwLAaLorLMe+gtVdRqPid/oO/t2wVw88+wq2mwQROQptZ2cnrpCUplKqdxyUfRTGBDG/oCN+jGnSUkd5vQA7KJrC7G6MTu4CQRsMXEjFSnWxDD3ZBJVTsLQsbowbM9ZhukReNt9r8BqTPqx63vInCWz7EddpGqDSiqQ+gr0m+pvE2+5sjXKpNV1VFJWnu2DZRu3r+bmOynvaUrZiP3qx+lrsNMxlr9uvUw== X-YMail-OSG: Rrrz8SgVM1mmAUNT7dgqyXDRSX20Mfub7NVYZbRexx3S.zrdU50l1v2H.ISvshU oZilixt7HEbARCDJ0Oi7c6l7APERYz3XZjkRVyiXaJKcToLk9GwqsR3IPfTRGFts2ghiiloAqrnI kT_CYyQ40lCrCHqERFUGtfrutpEIVsjPBfn__f.k3r6OD9qlXInYmS4XIK3HlnJDqtYeilDPQFM0 3sKP.czidKdeFPLxhgxnJhj0bArL7ynwzbj0RZQVh4bXu3AwZKo6jy4LnWSsJfnIQczqxLOl4x9G U5E.rVBBmC2.MmFHL.QkRHgy7FV6g7BlQ01VysXBZoJ8xNvWxzZ2YI4peor.IvQAWfG2fg6b_Dts UesBROFbASrP9KRkPl_MpQ8lVVHXHVXlUu_tN_4KaNC7Aa93lwRD.xdbjwGIwHEBWm5r.7TmTdnF GcBm8EnlMl81C5CKxU2K_9BnL1Y00Q4sMDt5SIWARWiqQhkPM0Xw.YSUhyTRu_me17.1OSb_7QA1 wnIcCfUi_w_mRfUr1oWXOPaAv1Ea1fFPTpw2yP8dc.QXdSYPNpaHbx9.LwAEmZ9pL2Kn6qWz.QwE 4A0VBncO_W6_3DByzlbDIJ8DPuokD8yeOhpvIA3u6w1ba5v2c6XoC_kPPRm0wHY9z0wvh2evqhvl rHJPt5DLldSweuUehPf3DoSDpuQ3q1huGN3o1ZpSYSYF9Lxg91z7PWEbP2tu8_joAearwH5ndqM4 uINzrzpnfDlmC2aJ46_gT5JuxojN.W9fmWJ1va_2PWISMyx3SllrOtOhv057ftIr4jQE8nJEg3Dr 8.scHKninQ0FGuPrEJs9D6f7379mICX0rGIOQklQJ2scGHMa.r6vq1lXoVZamu0WYak.Wv30bKXX ItOL2aVPY9fifqkmZyEHt5jSi_I5JU2wB9dl498ZB0nbmP.2kNwIzgUkKSIxEuYUFqPvzoNryCfz VhVhoK91rwJ1DWkOvk3gU9J7SyWzDtntQ5jmvzP8AA_ZQ5QytMCiMa1N0.crVYjgO8cmspTjfYXj .vKfJX6_4FQITb9GH2zETaLPx2zZCS47ikZnHQGIjyVuAI1wAg_shff1FhbImU1mv6zXgjdp4U45 P_7VZNWzEBhVthad36ucV0FCp3OvuxGybpgMTHqX3aXsTa0h.lrsbYZaVbqQr21WCHo6On03nOTI hA2WDa38Vnt3BYCsRjgL.TvCjH3C1K82kQwSq4Or4.K1badW8x0kVatpa_GGF0svpsKtHHtsShU2 L6.5gMFQVr1XeUFIWkr7BXBB10_S8CEOQUb_rq1CX2JV3gqG336cj.jl_cjWIN17Fne.Vt3Vdoog Lft4UpZk9tGIm99YI6vTW19N1NggkZmNA4PdUxEKm.a2Wz1e8ewb.H13sEeA_TEtGN6k4Wg5CBcm gbFMbIsNBix7rWczFy5Hc_Ed2fyrXzKjQipV7_6KaDqDpLJo1OP2NfTIL5oj7OZ_aQej3bgyt3gk dOTHN95L.c4UVKcy9_oeG4V5QQZKKInm1LT2FAIPLS.MrAGLYPysNvMHiBak3MPeIiqNBpo2mWC9 A0wOi1lAC3nAoXUdv5gPlUin2sewNJXPuOoyAlKGIRflLp3RGknlE0V.bRE7.8..bMrPLRouT1wO 4W_JAl05BANgIChnCEe.DicwsXQjKm09uvbyhCx2PzM1_HPPlUc4Mdq8P2nYYLV0w9wwBFy.PX9n EoOtHYbU43EWrqp9LnpuYWpuOM9VH3_.T3gj98Nfb.tAdXZ8ujuScjtjNK7DGr1nHNirRJpmGW8m kwRVLtYkZeSugDHPp1JDT7YbasCD8YJZclQAqa3RzgJYOddOpz9xmu2_PyJNL1g1tJVcUmSBxqf9 DuqNMfgO_I0jHzN3QwZ6w5jmjKUe7VBl.3QzsmThEPc1fJVT2ByimZaKpteybMlzTkPCsEAjplEN C4_RG.xm7ZX6iXt_EXARCfzKISgRu0UG2qhmk5v.9OnpstXd9vueNxs_L2RbwAVvlbnNr_sl1SpH 78or9cu5LspZBTnBXN0lEFaEx.Ike6IRk.ECxpxbS_jK3d4FZi2VSckumOyk9ij6D_jdlldV8jep whQzDn.DrIjU23muflbSChV..CZlH3Dk0Ikf43asGeSt3T_BQX4wQ96Mlj2jJuLWHSQiMnb54ZT5 sYM10yKqRveshGoPmXMAhmuG9YCgyOWxqmhrEgAHmLodJB7yYz86Em_Feb3gUqqbeHKHcHfZyCfq xEwIIBstmpN1OqyGmVhpFHNI_ACog_BDGjJKeg54ygyOQYF4e7xB3bEqUJXjuiszuyEk39Yo2FjW KnDDVsA9Xj_tZSULGQLpmheGFCaD7CEYlNSBQbRSt_z7Yl4.g2pvFwdMPUbyaAg_nVf4FNpRN5Ck GvHusPqMEjikvSacvc9XsBE62F1nnYBKBiHvKVb12hSqI0yjVrFKiH84EFt_Bo6BHHrT4Tx04wrH K0IPy..4qpkpcbmnnM4c4vnZZmEe5Y32TPjrO7IlwbjLDaINX22utg0YkMFTR9ymUlGqY2A5aAMr qnkQYyxvj0qHyL3aac7mvtHN7QNuPvduF6ZT82FlJL_.xKLMMYP34YBOn6.Ef9sYiRYLnndzVVU7 bHYv5RtHOEf0GALx4xqT.xt7FAwD_lcnOf1rL_NGMMi3euXpXm9Mo9g4N2S8tVrmCgS_9KM0- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Fri, 21 May 2021 02:39:03 +0000 Received: by kubenode540.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 63bcf4cbaf5a70e51079d0748ef9bf0f; Fri, 21 May 2021 02:38:58 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.80.0.2.43\)) Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) From: Mark Millard In-Reply-To: Date: Thu, 20 May 2021 19:38:57 -0700 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <04D7264A-206B-4281-B452-779B01EA3327@yahoo.com> References: <623369D9-5EE5-4FEF-B9AD-56499E8F1C09.ref@yahoo.com> <623369D9-5EE5-4FEF-B9AD-56499E8F1C09@yahoo.com> To: Rick Macklem X-Mailer: Apple Mail (2.3654.80.0.2.43) X-Rspamd-Queue-Id: 4FmW5P0pgCz3M5l X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.146:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.68.146:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.146:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.146:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 May 2021 02:39:06 -0000 > On 2021-May-20, at 18:09, Rick Macklem wrote: >=20 > Oh, one additional thing that I'll dare to top post... > r367492 broke the TCP upcalls that the NFS server uses, such > that intermittent hangs of NFS mounts to FreeBSD13 servers can occur. > This has not yet been resolved in "main" etc and could explain > why an RPC could time out for a soft mount. See later notes that I added: soft mount is not required to see the problem. > You can revert the patch in r367492 to avoid the problem. If I understand right, you are indicating that this would not apply to the non-soft mount case that I got. > Disabling TSO, LRO are also de-facto standard things to do when > you observe weird NFS behaviour, because they are often broken > in various network device drivers. I'll have to figure out how to experiment with such. Things are at defaults rather generally on the systems. I'm not literate in the subject areas. I'm the only user of the machines and network. It is not outward facing. It is a rather small EtherNet network. > rick >=20 > ________________________________________ > From: owner-freebsd-stable@freebsd.org = on behalf of Rick Macklem = > Sent: Thursday, May 20, 2021 8:55 PM > To: FreeBSD-STABLE Mailing List; Mark Millard > Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over = nfs (in a zfs file systems context) >=20 > Mark Millard wrote: >> [I warn that I'm a fairly minimal user of NFS >> mounts, not knowing all that much. I'm mostly >> reporting this in case it ends up as evidence >> via eventually matching up with others observing >> possibly related oddities.] >>=20 >> I got the following odd sequence (that I've >> mixed notes into). It involved a diff -r over NFS >> showing differences (files missing) and then a >> later diff finding matches for the same files, >> no file system changes made on either machine. >> I'm unable to reproduce the oddity on demand. >>=20 >> Note: A larger scope diff -r originally returned the >> below as well, but doing the narrower diff -r did >> repeat the result and that is what I show. (I >> make no use of devel/ice .) >>=20 >> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >> Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD . . . >> Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py >>=20 >> Note: The above was not expected. So I tried: >>=20 >> # ls -Tld /mnt/devel/ice/files/* >> -rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 = /mnt/devel/ice/files/Make.rules.FreeBSD . . . >> -rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 = /mnt/devel/ice/files/patch-scripts-TestUtil.py >>=20 >> Note: So that indicated that the files were there on the >> machine that /mnt references. So attempting the original >> diff -r again: >>=20 >> # diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more >> # >>=20 >> (Empty difference.) >>=20 >> Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*" >> the odd result of the diff -r no longer happened: no >> differences reported. >>=20 >>=20 >>=20 >> For reference (both machines reported): >>=20 >> . . . >> The original mount command was on CA72_16Gp_ZFS: >>=20 >> # mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/ > The likely explanation for this is your use of a "soft" mount. > - If the NFS server is slow to respond or there is a temporary network = issue, > the RPC request can time out and then the > syscall can fail with EINT/ETIMEDOUT. Since almost nothing, = including the > readdir(3) libc functions expect syscalls to fail this way... > Then the cached directory is messed up. > Doing the "ls" read the directory again and fixed the problem. >=20 > Try to reproduce it for a mount without the "soft" option. > (If a mount point is hung, due to an unresponsive server "umount -N = /mnt" > can usually get rid of it.) > Personally, I thought "soft" was a bad idea when Sun introduced it in = NFS in 1985 > and I still feel that way. > --> If you can reproduce it without "soft" then I can't explain it. > To be honest, the directory reading/caching code in the NFSv3 = client > hasn't changed significantly in literally decades, as far as I = can remember. Well . . . trying an even wider scope diff than the original . . . # umount /mnt/ # mount -onoatime 192.168.1.170:/usr/ports/ /mnt/ # diff -r /usr/ports/ /mnt/ | more Only in /mnt/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_ Only in /usr/ports/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cp= p__js__src25.cpp Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules Only in /usr/ports/devel/ice/files: patch-cpp-Makefile . . . Only in /usr/ports/devel/ice/files: = patch-python-test-Slice-unicodePaths-run.py Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py So the devel/ice files showed up again. But 2 other lines show up, one finding a file supposedly only on /mnt/. . . QUOTE Only in /mnt/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_ END QUOTE That seems to be a truncated file name. Looking directly on the machine = that /mnt/ references (hitting tab at the end of the partial name to show a list): # ls -Tld = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_ = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_gen-config.sh =20 = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_js-confdefs.h =20 = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_Unified__cpp__js__src0.cpp =20 = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_Unified__cpp__js__src1.cpp =20 . . . =20 = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_build_Unified__cpp__js__src9.cpp =20 = /usr/ports/databases/mongodb42/files/aarch64/patch-src_third__party_mozjs-= 60_platform_aarch64_freebsd_include_js-config.h =20 The other machine agrees (machine-local usage). The other of the 2 new names is one of the matches to the prefix: QUOTE Only in /usr/ports/databases/mongodb42/files/aarch64: = patch-src_third__party_mozjs-60_platform_aarch64_freebsd_build_Unified__cp= p__js__src25.cpp END QUOTE For reference: I've not gotten any console messages about anything during these. > One additional thing to note is that cached directory contents are = invalidated > when the directory's ctime changes. I'm not aware of anything that should have been touching the /usr/ports file systems on either machine any time near my diff activities. (I'm the only system user.) > I am not sure how/if/when ZFS changes a > directory's ctime. However, if it was badly broken, I'd hear about = this a lot. > (If the ZFS change to ZoL has changed its ctime handling, that might = also explain it > and I'll be hearing a lot more soon as FreeBSD13 becomes adopted. I = never use ZFS and, > as such, never test with it.) I recently decided to try using bectl, which lead to my recent ZFS-based system experiments. This means I can boot the stable/13 or main [so: 14] that I last built and try the same experiments with the same /usr/ports file sysystems. releng/13 's release/13.0.0 , stable/13 , and main are all non-debug builds as stands. I could add debug builds of any or all, but it would take a while. (aarch64 4-core Cortex-A72 contexts.) > --> For UFS, if you use mtime, directory caching does not work as = well, which is > why the client directory caching code uses ctime and not mtime = to detect that > a directory has changed and cached directory blocks need to be = invalidated. >=20 > Jason Bacon did report a directory reading issue some months ago that = never > quite got resolved, although I recall he said he couldn't reproduce it = after a > system update, so he thought it was related to some local change he = had made. > (I can't remember his email or I'd add him to the cc so he could = remind me what > his case was. I do recall it being somewhat reproducible and happened = for both > UFS and ZFS.) >> The network is just a local EtherNet. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)