Date: Fri, 21 May 2021 01:09:19 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>, Mark Millard <marklmi@yahoo.com> Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs (in a zfs file systems context) Message-ID: <YTOPR0101MB0970A1257E4DD37335D5B52EDD299@YTOPR0101MB0970.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <YQXPR0101MB0968B29934D7BD73FCA73907DD299@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> References: <623369D9-5EE5-4FEF-B9AD-56499E8F1C09.ref@yahoo.com>, <623369D9-5EE5-4FEF-B9AD-56499E8F1C09@yahoo.com>, <YQXPR0101MB0968B29934D7BD73FCA73907DD299@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM>
next in thread | previous in thread | raw e-mail | index | archive | help
Oh, one additional thing that I'll dare to top post...
r367492 broke the TCP upcalls that the NFS server uses, such
that intermittent hangs of NFS mounts to FreeBSD13 servers can occur.
This has not yet been resolved in "main" etc and could explain
why an RPC could time out for a soft mount.
You can revert the patch in r367492 to avoid the problem.
Disabling TSO, LRO are also de-facto standard things to do when
you observe weird NFS behaviour, because they are often broken
in various network device drivers.
rick
________________________________________
From: owner-freebsd-stable@freebsd.org <owner-freebsd-stable@freebsd.org> o=
n behalf of Rick Macklem <rmacklem@uoguelph.ca>
Sent: Thursday, May 20, 2021 8:55 PM
To: FreeBSD-STABLE Mailing List; Mark Millard
Subject: Re: releng/13 release/13.0.0 : odd/incorrect diff result over nfs =
(in a zfs file systems context)
Mark Millard wrote:
>[I warn that I'm a fairly minimal user of NFS
>mounts, not knowing all that much. I'm mostly
>reporting this in case it ends up as evidence
>via eventually matching up with others observing
>possibly related oddities.]
>
>I got the following odd sequence (that I've
>mixed notes into). It involved a diff -r over NFS
>showing differences (files missing) and then a
>later diff finding matches for the same files,
>no file system changes made on either machine.
>I'm unable to reproduce the oddity on demand.
>
>Note: A larger scope diff -r originally returned the
>below as well, but doing the narrower diff -r did
>repeat the result and that is what I show. (I
>make no use of devel/ice .)
>
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>Only in /usr/ports/devel/ice/files: Make.rules.FreeBSD
>Only in /usr/ports/devel/ice/files: patch-config-Make.common.rules
>Only in /usr/ports/devel/ice/files: patch-cpp-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-allTests.py
>Only in /usr/ports/devel/ice/files: patch-cpp-config-Make.rules
>Only in /usr/ports/devel/ice/files: patch-cpp-include-Ice-FactoryTableInit=
.h
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-Config.h
>Only in /usr/ports/devel/ice/files: patch-cpp-include-IceUtil-ScannerConfi=
g.h
>Only in /usr/ports/devel/ice/files: patch-cpp-src-Glacier2CryptPermissions=
Verifier-CryptPermissionsVerifierI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-Ice-ProxyFactory.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-PluginFacadeI.cp=
p
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceGrid-RegistryI.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-src-IceSSL-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-ssl-Server.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Glacier2-staticFilterin=
g-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-info-AllTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-metrics-AllTests.cp=
p
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-objects-Makefile
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Ice-properties-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-admin-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-AllTes=
ts.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-Makefi=
le
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-deployer-applic=
ation.xml
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-Al=
lTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-ap=
plication.xml
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-distribution-ru=
n.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceGrid-session-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-Al=
lTests.cpp
>Only in /usr/ports/devel/ice/files: patch-cpp-test-IceSSL-configuration-ru=
n.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-headers-run.py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-Slice-unicodePaths-run.=
py
>Only in /usr/ports/devel/ice/files: patch-cpp-test-include-TestCommon.h
>Only in /usr/ports/devel/ice/files: patch-php-Makefile
>Only in /usr/ports/devel/ice/files: patch-php-config-Make.rules.php
>Only in /usr/ports/devel/ice/files: patch-php-lib-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-config-Make.rules
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.cpp
>Only in /usr/ports/devel/ice/files: patch-python-modules-IcePy-Types.h
>Only in /usr/ports/devel/ice/files: patch-python-python-Makefile
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-info-AllTests.py
>Only in /usr/ports/devel/ice/files: patch-python-test-Ice-properties-run.p=
y
>Only in /usr/ports/devel/ice/files: patch-python-test-Slice-unicodePaths-r=
un.py
>Only in /usr/ports/devel/ice/files: patch-scripts-Expect.py
>Only in /usr/ports/devel/ice/files: patch-scripts-IceGridAdmin.py
>Only in /usr/ports/devel/ice/files: patch-scripts-TestUtil.py
>
>Note: The above was not expected. So I tried:
>
># ls -Tld /mnt/devel/ice/files/*
>-rw-r--r-- 1 root wheel 755 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
Make.rules.FreeBSD
>-rw-r--r-- 1 root wheel 1542 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-config-Make.common.rules
>-rw-r--r-- 1 root wheel 388 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-Makefile
>-rw-r--r-- 1 root wheel 1695 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-allTests.py
>-rw-r--r-- 1 root wheel 1112 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-config-Make.rules
>-rw-r--r-- 1 root wheel 1512 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-include-Ice-FactoryTableInit.h
>-rw-r--r-- 1 root wheel 1496 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-include-IceUtil-Config.h
>-rw-r--r-- 1 root wheel 447 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-include-IceUtil-ScannerConfig.h
>-rw-r--r-- 1 root wheel 588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src->Glacier2CryptPermissionsVerifier-CryptPermissionsVerifierI.c=
pp
>-rw-r--r-- 1 root wheel 384 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-Ice-ProxyFactory.cpp
>-rw-r--r-- 1 root wheel 691 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-IceGrid-PluginFacadeI.cpp
>-rw-r--r-- 1 root wheel 440 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-IceGrid-RegistryI.cpp
>-rw-r--r-- 1 root wheel 543 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-src-IceSSL-Makefile
>-rw-r--r-- 1 root wheel 990 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Glacier2-ssl-Server.cpp
>-rw-r--r-- 1 root wheel 1280 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Glacier2-staticFiltering-run.py
>-rw-r--r-- 1 root wheel 867 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-info-AllTests.cpp
>-rw-r--r-- 1 root wheel 3337 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-metrics-AllTests.cpp
>-rw-r--r-- 1 root wheel 752 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-objects-Makefile
>-rw-r--r-- 1 root wheel 730 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Ice-properties-run.py
>-rw-r--r-- 1 root wheel 718 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-admin-run.py
>-rw-r--r-- 1 root wheel 8800 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-deployer-AllTests.cpp
>-rw-r--r-- 1 root wheel 438 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-deployer-Makefile
>-rw-r--r-- 1 root wheel 787 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-deployer-application.xml
>-rw-r--r-- 1 root wheel 997 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-distribution-AllTests.cpp
>-rw-r--r-- 1 root wheel 434 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-distribution-application.xml
>-rw-r--r-- 1 root wheel 459 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-distribution-run.py
>-rw-r--r-- 1 root wheel 741 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceGrid-session-run.py
>-rw-r--r-- 1 root wheel 1179 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceSSL-configuration-AllTests.cpp
>-rw-r--r-- 1 root wheel 1038 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-IceSSL-configuration-run.py
>-rw-r--r-- 1 root wheel 3490 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Slice-headers-run.py
>-rw-r--r-- 1 root wheel 484 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-Slice-unicodePaths-run.py
>-rw-r--r-- 1 root wheel 608 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-cpp-test-include-TestCommon.h
>-rw-r--r-- 1 root wheel 341 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-php-Makefile
>-rw-r--r-- 1 root wheel 951 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-php-config-Make.rules.php
>-rw-r--r-- 1 root wheel 434 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-php-lib-Makefile
>-rw-r--r-- 1 root wheel 322 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-Makefile
>-rw-r--r-- 1 root wheel 2629 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-config-Make.rules
>-rw-r--r-- 1 root wheel 3394 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-modules-IcePy-Types.cpp
>-rw-r--r-- 1 root wheel 5092 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-modules-IcePy-Types.h
>-rw-r--r-- 1 root wheel 668 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-python-Makefile
>-rw-r--r-- 1 root wheel 2640 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-test-Ice-info-AllTests.py
>-rw-r--r-- 1 root wheel 736 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-test-Ice-properties-run.py
>-rw-r--r-- 1 root wheel 490 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-python-test-Slice-unicodePaths-run.py
>-rw-r--r-- 1 root wheel 381 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-scripts-Expect.py
>-rw-r--r-- 1 root wheel 793 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-scripts-IceGridAdmin.py
>-rw-r--r-- 1 root wheel 2588 Apr 21 21:07:54 2021 /mnt/devel/ice/files/=
patch-scripts-TestUtil.py
>
>Note: So that indicated that the files were there on the
>machine that /mnt references. So attempting the original
>diff -r again:
>
># diff -r /usr/ports/devel/ice/files /mnt/devel/ice/files | more
>#
>
>(Empty difference.)
>
>Note: So after the explicit "ls -Tld /mnt/devel/ice/files/*"
>the odd result of the diff -r no longer happened: no
>differences reported.
>
>
>
>For reference (both machines reported):
>
># uname -apKU
>FreeBSD CA72_16Gp_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n24=
4733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021 root@CA72_4c8G_Z=
FS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/=
GENERIC-NODBG-CA72 arm64 aarch64 1300139 1300139
>
># ~/fbsd-based-on-what-commit.sh
>branch: releng/13.0
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
>merge-base: CommitDate: 2021-04-09 00:14:30 +0000
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.=
0) 13.0: update to RELEASE
>n244733 (--first-parent --count for merge-base)
>
># uname -apKU
>FreeBSD CA72_4c8G_ZFS 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n24=
4733-ea31abc261ff-dirty: Thu Apr 29 >21:53:20 PDT 2021 root@CA72_4c8G_Z=
FS:/usr/obj/BUILDs/13_0R-CA72-nodbg-clang/usr/13_0R->src/arm64.aarch64/sys/=
GENERIC-NODBG-CA72 arm64 aarch64 1300139 1300139
>
># ~/fbsd-based-on-what-commit.sh
>branch: releng/13.0
>merge-base: ea31abc261ffc01b6ff5671bffb15cf910a07f4b
>merge-base: CommitDate: 2021-04-09 00:14:30 +0000
>ea31abc261ff (HEAD -> releng/13.0, tag: release/13.0.0, freebsd/releng/13.=
0) 13.0: update to RELEASE
>n244733 (--first-parent --count for merge-base)
>
>From zfs list commands (one machine per line shown):
>
>zopt0/usr/ports 2.13G 236G 2.13G /usr/por=
ts
>zroot/usr/ports 2.13G 113G 2.13G /usr/por=
ts
>
>I've no clue if ZFS is important to the odditity
>or not.
>
>The original mount command was on CA72_16Gp_ZFS:
>
># mount -onoatime,soft 192.168.1.170:/usr/ports/ /mnt/
The likely explanation for this is your use of a "soft" mount.
- If the NFS server is slow to respond or there is a temporary network issu=
e,
the RPC request can time out and then the
syscall can fail with EINT/ETIMEDOUT. Since almost nothing, including th=
e
readdir(3) libc functions expect syscalls to fail this way...
Then the cached directory is messed up.
Doing the "ls" read the directory again and fixed the problem.
Try to reproduce it for a mount without the "soft" option.
(If a mount point is hung, due to an unresponsive server "umount -N /mnt"
can usually get rid of it.)
Personally, I thought "soft" was a bad idea when Sun introduced it in NFS i=
n 1985
and I still feel that way.
--> If you can reproduce it without "soft" then I can't explain it.
To be honest, the directory reading/caching code in the NFSv3 client
hasn't changed significantly in literally decades, as far as I can re=
member.
One additional thing to note is that cached directory contents are invalida=
ted
when the directory's ctime changes. I am not sure how/if/when ZFS changes a
directory's ctime. However, if it was badly broken, I'd hear about this a l=
ot.
(If the ZFS change to ZoL has changed its ctime handling, that might also e=
xplain it
and I'll be hearing a lot more soon as FreeBSD13 becomes adopted. I never =
use ZFS and,
as such, never test with it.)
--> For UFS, if you use mtime, directory caching does not work as well, whi=
ch is
why the client directory caching code uses ctime and not mtime to de=
tect that
a directory has changed and cached directory blocks need to be inval=
idated.
Jason Bacon did report a directory reading issue some months ago that never
quite got resolved, although I recall he said he couldn't reproduce it afte=
r a
system update, so he thought it was related to some local change he had mad=
e.
(I can't remember his email or I'd add him to the cc so he could remind me =
what
his case was. I do recall it being somewhat reproducible and happened for =
both
UFS and ZFS.)
The network is just a local EtherNet.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
_______________________________________________
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
_______________________________________________
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YTOPR0101MB0970A1257E4DD37335D5B52EDD299>