Date: Fri, 21 May 2021 00:55:17 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>, Mark Millard <marklmi@yahoo.com> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) Message-ID: <YQXPR0101MB0968B29934D7BD73FCA73907DD299@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <623369D9-5EE5-4FEF-B9AD-56499E8F1C09@yahoo.com> References: <623369D9-5EE5-4FEF-B9AD-56499E8F1C09.ref@yahoo.com>, <623369D9-5EE5-4FEF-B9AD-56499E8F1C09@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Mark Millard wrote:=0A=
>[I warn that I'm a fairly minimal user of NFS=0A=
>mounts, not knowing all that much. I'm mostly=0A=
>reporting this in case it ends up as evidence=0A=
>via eventually matching up with others observing=0A=
>possibly related oddities.]=0A=
>=0A=
>I got the following odd sequence (that I've=0A=
>mixed notes into). It involved a diff -r over NFS=0A=
>showing differences (files missing) and then a=0A=
>later diff finding matches for the same files,=0A=
>no file system changes made on either machine.=0A=
>I'm unable to reproduce the oddity on demand.=0A=
>=0A=
>Note: A larger scope diff -r originally returned the=0A=
>below as well, but doing the narrower diff -r did=0A=
>repeat the result and that is what I show. (I=0A=
>make no use of devel/ice .)=0A=
>=0A=
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more=0A=
>Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD=0A=
>Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-Makefile=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-allTests.py=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-config-Make.rules=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-include-Ice-FactoryTableInit=
.h=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-Config.h=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-ScannerConfi=
g.h=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-src-Glacier2CryptPermissions=
Verifier-CryptPermissionsVerifierI.cpp=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-src-Ice-ProxyFactory.cpp=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-PluginFacadeI.cp=
p=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-RegistryI.cpp=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceSSL-Makefile=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-ssl-Server.cpp=
=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-staticFilterin=
g-run.py=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-info-AllTests.cpp=
=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-metrics-AllTests.cp=
p=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-objects-Makefile=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-properties-run.py=
=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-admin-run.py=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-AllTes=
ts.cpp=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-Makefi=
le=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-applic=
ation.xml=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-Al=
lTests.cpp=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-ap=
plication.xml=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-ru=
n.py=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-session-run.py=
=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-Al=
lTests.cpp=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-ru=
n.py=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-headers-run.py=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-unicodePaths-run.=
py=0A=
>Only in /usr/ports/devel/ice/files: patch-cpp-test-include-TestCommon.h=0A=
>Only in /usr/ports/devel/ice/files: patch-php-Makefile=0A=
>Only in /usr/ports/devel/ice/files: patch-php-config-Make.rules.php=0A=
>Only in /usr/ports/devel/ice/files: patch-php-lib-Makefile=0A=
>Only in /usr/ports/devel/ice/files: patch-python-Makefile=0A=
>Only in /usr/ports/devel/ice/files: patch-python-config-Make.rules=0A=
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.cpp=
=0A=
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.h=0A=
>Only in /usr/ports/devel/ice/files: patch-python-python-Makefile=0A=
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-info-AllTests.py=
=0A=
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-properties-run.p=
y=0A=
>Only in /usr/ports/devel/ice/files: patch-python-test-Slice-unicodePaths-r=
un.py=0A=
>Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py=0A=
>Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py=0A=
>Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py=0A=
>=0A=
>Note: The above was not expected. So I tried:=0A=
>=0A=
># ls -Tld /mnt/devel/ice/files/*=0A=
>-rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
Make.rules.FreeBSD=0A=
>-rw-r--r-- 1 root wheel 1542 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-config-Make.common.rules=0A=
>-rw-r--r-- 1 root wheel 388 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-Makefile=0A=
>-rw-r--r-- 1 root wheel 1695 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-allTests.py=0A=
>-rw-r--r-- 1 root wheel 1112 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-config-Make.rules=0A=
>-rw-r--r-- 1 root wheel 1512 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-include-Ice-FactoryTableInit.h=0A=
>-rw-r--r-- 1 root wheel 1496 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-include-IceUtil-Config.h=0A=
>-rw-r--r-- 1 root wheel 447 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-include-IceUtil-ScannerConfig.h=0A=
>-rw-r--r-- 1 root wheel 588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src->Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.c=
pp=0A=
>-rw-r--r-- 1 root wheel 384 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-Ice-ProxyFactory.cpp=0A=
>-rw-r--r-- 1 root wheel 691 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-IceGrid-PluginFacadeI.cpp=0A=
>-rw-r--r-- 1 root wheel 440 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-IceGrid-RegistryI.cpp=0A=
>-rw-r--r-- 1 root wheel 543 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-IceSSL-Makefile=0A=
>-rw-r--r-- 1 root wheel 990 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Glacier2-ssl-Server.cpp=0A=
>-rw-r--r-- 1 root wheel 1280 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Glacier2-staticFiltering-run.py=0A=
>-rw-r--r-- 1 root wheel 867 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-info-AllTests.cpp=0A=
>-rw-r--r-- 1 root wheel 3337 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-metrics-AllTests.cpp=0A=
>-rw-r--r-- 1 root wheel 752 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-objects-Makefile=0A=
>-rw-r--r-- 1 root wheel 730 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-properties-run.py=0A=
>-rw-r--r-- 1 root wheel 718 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-admin-run.py=0A=
>-rw-r--r-- 1 root wheel 8800 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-deployer-AllTests.cpp=0A=
>-rw-r--r-- 1 root wheel 438 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-deployer-Makefile=0A=
>-rw-r--r-- 1 root wheel 787 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-deployer-application.xml=0A=
>-rw-r--r-- 1 root wheel 997 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-distribution-AllTests.cpp=0A=
>-rw-r--r-- 1 root wheel 434 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-distribution-application.xml=0A=
>-rw-r--r-- 1 root wheel 459 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-distribution-run.py=0A=
>-rw-r--r-- 1 root wheel 741 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-session-run.py=0A=
>-rw-r--r-- 1 root wheel 1179 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceSSL-configuration-AllTests.cpp=0A=
>-rw-r--r-- 1 root wheel 1038 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceSSL-configuration-run.py=0A=
>-rw-r--r-- 1 root wheel 3490 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Slice-headers-run.py=0A=
>-rw-r--r-- 1 root wheel 484 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Slice-unicodePaths-run.py=0A=
>-rw-r--r-- 1 root wheel 608 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-include-TestCommon.h=0A=
>-rw-r--r-- 1 root wheel 341 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-php-Makefile=0A=
>-rw-r--r-- 1 root wheel 951 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-php-config-Make.rules.php=0A=
>-rw-r--r-- 1 root wheel 434 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-php-lib-Makefile=0A=
>-rw-r--r-- 1 root wheel 322 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-Makefile=0A=
>-rw-r--r-- 1 root wheel 2629 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-config-Make.rules=0A=
>-rw-r--r-- 1 root wheel 3394 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-modules-IcePy-Types.cpp=0A=
>-rw-r--r-- 1 root wheel 5092 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-modules-IcePy-Types.h=0A=
>-rw-r--r-- 1 root wheel 668 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-python-Makefile=0A=
>-rw-r--r-- 1 root wheel 2640 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-test-Ice-info-AllTests.py=0A=
>-rw-r--r-- 1 root wheel 736 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-test-Ice-properties-run.py=0A=
>-rw-r--r-- 1 root wheel 490 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-test-Slice-unicodePaths-run.py=0A=
>-rw-r--r-- 1 root wheel 381 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-scripts-Expect.py=0A=
>-rw-r--r-- 1 root wheel 793 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-scripts-IceGridAdmin.py=0A=
>-rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-scripts-TestUtil.py=0A=
>=0A=
>Note: So that indicated that the files were there on the=0A=
>machine that /mnt references. So attempting the original=0A=
>diff -r again:=0A=
>=0A=
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more=0A=
>#=0A=
>=0A=
>(Empty difference.)=0A=
>=0A=
>Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"=0A=
>the odd result of the diff -r no longer happened: no=0A=
>differences reported.=0A=
>=0A=
>=0A=
>=0A=
>For reference (both machines reported):=0A=
>=0A=
># uname -apKU=0A=
>FreeBSD CA72_16Gp_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n24=
4733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021 root@CA72_4c8G_Z=
FS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/=
GENERIC-NODBG-CA72 arm64 aarch64 1300139 1300139=0A=
>=0A=
># ~/fbsd-based-on-what-commit.sh=0A=
>branch: releng/13.0=0A=
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b=0A=
>merge-base: CommitDate: 2021-04-09 00:14:30 +0000=0A=
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.=
0) 13.0: update to RELEASE=0A=
>n244733 (--first-parent --count for merge-base)=0A=
>=0A=
># uname -apKU=0A=
>FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n24=
4733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021 root@CA72_4c8G_Z=
FS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/=
GENERIC-NODBG-CA72 arm64 aarch64 1300139 1300139=0A=
>=0A=
># ~/fbsd-based-on-what-commit.sh=0A=
>branch: releng/13.0=0A=
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b=0A=
>merge-base: CommitDate: 2021-04-09 00:14:30 +0000=0A=
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.=
0) 13.0: update to RELEASE=0A=
>n244733 (--first-parent --count for merge-base)=0A=
>=0A=
>From zfs list commands (one machine per line shown):=0A=
>=0A=
>zopt0/usr/ports 2.13G 236G 2.13G /usr/por=
ts=0A=
>zroot/usr/ports 2.13G 113G 2.13G /usr/por=
ts=0A=
>=0A=
>I've no clue if ZFS is important to the odditity=0A=
>or not.=0A=
>=0A=
>The original mount command was on CA72_16Gp_ZFS:=0A=
>=0A=
># mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/=0A=
The likely explanation for this is your use of a "soft" mount.=0A=
- If the NFS server is slow to respond or there is a temporary network issu=
e,=0A=
the RPC request can time out and then the=0A=
syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including th=
e=0A=
readdir(3) libc functions expect syscalls to fail this way...=0A=
Then the cached directory is messed up.=0A=
Doing the "ls" read the directory again and fixed the problem.=0A=
=0A=
Try to reproduce it for a mount without the "soft" option.=0A=
(If a mount point is hung, due to an unresponsive server "umount -N /mnt"=
=0A=
can usually get rid of it.)=0A=
Personally, I thought "soft" was a bad idea when Sun introduced it in NFS i=
n 1985=0A=
and I still feel that way.=0A=
--> If you can reproduce it without "soft" then I can't explain it.=0A=
To be honest, the directory reading/caching code in the NFSv3 client=
=0A=
hasn't changed significantly in literally decades, as far as I can re=
member.=0A=
=0A=
One additional thing to note is that cached directory contents are invalida=
ted=0A=
when the directory's ctime changes. I am not sure how/if/when ZFS changes a=
=0A=
directory's ctime. However, if it was badly broken, I'd hear about this a l=
ot.=0A=
(If the ZFS change to ZoL has changed its ctime handling, that might also e=
xplain it=0A=
and I'll be hearing a lot more soon as FreeBSD13 becomes adopted. I never =
use ZFS and,=0A=
as such, never test with it.)=0A=
--> For UFS, if you use mtime, directory caching does not work as well, whi=
ch is=0A=
why the client directory caching code uses ctime and not mtime to de=
tect that=0A=
a directory has changed and cached directory blocks need to be inval=
idated.=0A=
=0A=
Jason Bacon did report a directory reading issue some months ago that never=
=0A=
quite got resolved, although I recall he said he couldn't reproduce it afte=
r a=0A=
system update, so he thought it was related to some local change he had mad=
e.=0A=
(I can't remember his email or I'd add him to the cc so he could remind me =
what=0A=
his case was. I do recall it being somewhat reproducible and happened for =
both=0A=
UFS and ZFS.)=0A=
The network is just a local EtherNet.=0A=
=0A=
=3D=3D=3D=0A=
Mark Millard=0A=
marklmi at yahoo.com=0A=
( dsl-only.net went=0A=
away in early 2018-Mar)=0A=
=0A=
_______________________________________________=0A=
freebsd-stable@freebsd.org mailing list=0A=
https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A=
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"=
=0A=
=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968B29934D7BD73FCA73907DD299>