Date: Tue, 1 Apr 2025 12:39:54 +0200 (CEST) From: Ronald Klop <ronald-lists@klop.ws> To: Dave Cottlehuber <dch@FreeBSD.org> Cc: current@freebsd.org Subject: Re: nfs4: some shares have invisible files & directories Message-ID: <2001775390.7235.1743503994892@localhost> In-Reply-To: <7d155dfe-800c-4ba4-b59b-bf710ab45991@app.fastmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
Hi,
Do you have some local filesytem mounted over the NFS4 mount? Your remark about 'It was "resolved" by `zfs destroy zroot/usr/{ports,src}`' made me think in this direction. Did you do the zfs destroy on the NFS client or on the NFS server?
Regards,
Ronald.
Van: Dave Cottlehuber <dch@FreeBSD.org>
Datum: dinsdag, 1 april 2025 12:17
Aan: current@freebsd.org
Onderwerp: nfs4: some shares have invisible files & directories
>
> TLDR I have working nfs4 mounts but in some mounts, none of the
> files are visible to ls etc. But if you know the filename, you can
> still access it directly by name. Why?
>
> I should have a very simple setup for nfsv4. What I need is to
> have ro exports /usr/{ports,obj,src} available to clients, all
> FreeBSD.
>
> I can mount all exported filesystems, but only /usr/obj has visible
> files on the client, and I don't understand why. This behaviour is
> consistent across clients.
>
> during beinstall.sh from clients, they are unable to access files
> that definitely exist in /usr/obj/* such as /usr/bin/cc & /bin/rm.
>
> It was "resolved" by `zfs destroy zroot/usr/{ports,src}` & recreating
> them. I don't recall ever using getfacl/setfacl, and I can't think
> of anything else that might make the mount work but the files
> be invisible. Sadly I did not think to just rename the afflicted
> datasets.
>
> Does anybody know what might cause this? I did find a mention of
> this from SUSE linux, but in 2011, and I don't see any interesting
> commits related to readdir recently.
>
> https://forums.opensuse.org/t/11-4-strange-nfs-v4-problem-invisible-files/63107
>
> ## server
>
> - fast build box running nfsv4 only
> - 15-CURRENT, up to date
> - providing /usr/{src,obj,ports} to 3 clients
> - this is a zfs system, and each are on separate datasets
>
> ```
> # egrep -hr 'nfs|mount' /etc/rc.conf*
>
> mountd_enable=YES
> nfs_server_enable=YES
> nfsv4_server_enable=YES
> nfsv4_server_only=YES
>
> # cat /etc/exports
> V4: /usr -sec=sys
> /usr/src -ro
> /usr/obj -ro
> /usr/ports -ro
>
> # ls -AFGhld /usr/{src,obj,ports}
> drwxr-xr-x 3 dch wheel 3B Feb 28 23:27 /usr/obj/
> drwxr-xr-x 70 dch wheel 86B Apr 1 06:55 /usr/ports/
> drwxr-xr-x 27 dch wheel 47B Mar 31 13:26 /usr/src/
>
> # getfacl /usr/src
> # file: /usr/src
> # owner: dch
> # group: wheel
> owner@:rwxp--aARWcCos:-------:allow
> group@:rwxp--a-R-c--s:-------:allow
> everyone@:r-x---a-R-c--s:-------:allow
> # getfacl /usr/src/COPYRIGHT
> # file: /usr/src/COPYRIGHT
> # owner: dch
> # group: wheel
> owner@:rw-p--aARWcCos:-------:allow
> group@:rw-p--a-R-c--s:-------:allow
> everyone@:r-----a-R-c--s:-------:allow
> ```
>
> # clients
>
> - either 14.2-RELEASE or also 15-CURRENT
> - no daemons
>
> ```
> # mount_nfs -o vers=4,nolockd,retrycnt=0,noatime,ro 172.16.1.4:/obj /usr/obj
> # ls /usr/obj/usr/src/amd64.amd64/
> bin/ ... lots more files, very good
>
> # mount_nfs -o vers=4,nolockd,retrycnt=0,noatime,ro 172.16.1.4:/src /usr/src
> # mkdir /usr/src/foo
> mkdir: /usr/src/foo: Read-only file system # good its clearly mounted
>
> # ls -AFGhl /usr/src
> total 0
>
> # head /usr/src/COPYRIGHT
> The compilation of software known as FreeBSD is distributed under the
> following terms:
> .... what how is this file even here? ls shows nothing, tar also fails
> ```
>
> A+
> Dave
> ———
> O for a muse of fire, that would ascend the brightest heaven of invention!
>
>
>
>
[-- Attachment #2 --]
<html><head></head><body>Hi,<br>
<br>
Do you have some local filesytem mounted over the NFS4 mount? Your remark about 'It was "resolved" by `zfs destroy zroot/usr/{ports,src}`' made me think in this direction. Did you do the zfs destroy on the NFS client or on the NFS server?<br>
<br>
Regards,<br>
Ronald.<br>
<br>
<p><strong>Van:</strong> Dave Cottlehuber <dch@FreeBSD.org><br>
<strong>Datum:</strong> dinsdag, 1 april 2025 12:17<br>
<strong>Aan:</strong> current@freebsd.org<br>
<strong>Onderwerp:</strong> nfs4: some shares have invisible files & directories</p>
<blockquote style="padding-right: 0px; padding-left: 5px; margin-left: 5px; border-left: #000000 2px solid; margin-right: 0px">
<div class="MessageRFC822Viewer" id="P">
<div class="TextPlainViewer" id="P.P">TLDR I have working nfs4 mounts but in some mounts, none of the<br>
files are visible to ls etc. But if you know the filename, you can<br>
still access it directly by name. Why?<br>
<br>
I should have a very simple setup for nfsv4. What I need is to<br>
have ro exports /usr/{ports,obj,src} available to clients, all<br>
FreeBSD.<br>
<br>
I can mount all exported filesystems, but only /usr/obj has visible<br>
files on the client, and I don't understand why. This behaviour is<br>
consistent across clients.<br>
<br>
during beinstall.sh from clients, they are unable to access files<br>
that definitely exist in /usr/obj/* such as /usr/bin/cc & /bin/rm.<br>
<br>
It was "resolved" by `zfs destroy zroot/usr/{ports,src}` & recreating<br>
them. I don't recall ever using getfacl/setfacl, and I can't think<br>
of anything else that might make the mount work but the files<br>
be invisible. Sadly I did not think to just rename the afflicted<br>
datasets.<br>
<br>
Does anybody know what might cause this? I did find a mention of<br>
this from SUSE linux, but in 2011, and I don't see any interesting<br>
commits related to readdir recently.<br>
<br>
<a href="https://forums.opensuse.org/t/11-4-strange-nfs-v4-problem-invisible-files/63107">https://forums.opensuse.org/t/11-4-strange-nfs-v4-problem-invisible-files/63107</a><br>;
<br>
## server<br>
<br>
- fast build box running nfsv4 only<br>
- 15-CURRENT, up to date<br>
- providing /usr/{src,obj,ports} to 3 clients<br>
- this is a zfs system, and each are on separate datasets<br>
<br>
```<br>
# egrep -hr 'nfs|mount' /etc/rc.conf*<br>
<br>
mountd_enable=YES<br>
nfs_server_enable=YES<br>
nfsv4_server_enable=YES<br>
nfsv4_server_only=YES<br>
<br>
# cat /etc/exports<br>
V4: /usr -sec=sys<br>
/usr/src -ro<br>
/usr/obj -ro<br>
/usr/ports -ro<br>
<br>
# ls -AFGhld /usr/{src,obj,ports}<br>
drwxr-xr-x 3 dch wheel 3B Feb 28 23:27 /usr/obj/<br>
drwxr-xr-x 70 dch wheel 86B Apr 1 06:55 /usr/ports/<br>
drwxr-xr-x 27 dch wheel 47B Mar 31 13:26 /usr/src/<br>
<br>
# getfacl /usr/src<br>
# file: /usr/src<br>
# owner: dch<br>
# group: wheel<br>
owner@:rwxp--aARWcCos:-------:allow<br>
group@:rwxp--a-R-c--s:-------:allow<br>
everyone@:r-x---a-R-c--s:-------:allow<br>
# getfacl /usr/src/COPYRIGHT<br>
# file: /usr/src/COPYRIGHT<br>
# owner: dch<br>
# group: wheel<br>
owner@:rw-p--aARWcCos:-------:allow<br>
group@:rw-p--a-R-c--s:-------:allow<br>
everyone@:r-----a-R-c--s:-------:allow<br>
```<br>
<br>
# clients<br>
<br>
- either 14.2-RELEASE or also 15-CURRENT<br>
- no daemons<br>
<br>
```<br>
# mount_nfs -o vers=4,nolockd,retrycnt=0,noatime,ro 172.16.1.4:/obj /usr/obj<br>
# ls /usr/obj/usr/src/amd64.amd64/<br>
bin/ ... lots more files, very good<br>
<br>
# mount_nfs -o vers=4,nolockd,retrycnt=0,noatime,ro 172.16.1.4:/src /usr/src<br>
# mkdir /usr/src/foo<br>
mkdir: /usr/src/foo: Read-only file system # good its clearly mounted<br>
<br>
# ls -AFGhl /usr/src<br>
total 0<br>
<br>
# head /usr/src/COPYRIGHT<br>
The compilation of software known as FreeBSD is distributed under the<br>
following terms:<br>
.... what how is this file even here? ls shows nothing, tar also fails<br>
```<br>
<br>
A+<br>
Dave<br>
———<br>
O for a muse of fire, that would ascend the brightest heaven of invention!<br>
</div>
<hr></div>
</blockquote>
<br>
</body></html>
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2001775390.7235.1743503994892>