Date: Sat, 18 Nov 2023 13:58:20 -0800 From: Rick Macklem <rick.macklem@gmail.com> To: Mike Karels <mike@karels.net>, FreeBSD CURRENT <freebsd-current@freebsd.org> Cc: Rick Macklem <rmacklem@freebsd.org>, Garrett Wollman <wollman@bimajority.org>, Alexander Motin <mav@freebsd.org>, Martin Matuska <mm@freebsd.org> Subject: Re: NFS exports of ZFS snapshots broken Message-ID: <CAM5tNy6K6psrtem%2Bc3ta0pM-7fhy9LYci0DEeR0O-EF%2B-iGo6Q@mail.gmail.com> In-Reply-To: <CAM5tNy5LGWF424i9v_w2u%2B41wtzmW9kn6RZWwo6Pi0Y4wj92UA@mail.gmail.com> References: <25943.60056.880614.452966@hergotha.csail.mit.edu> <DDC3BBBB-561A-4779-B076-83EC054859C2@karels.net> <CAM5tNy6gcThf0nrcs4iC8r6J8cMFOHdUNcUjfJVDSpTW_tqXKQ@mail.gmail.com> <ACAFC3CD-0024-4076-8C69-CAA3B1550B41@karels.net> <AB4690A9-A8F4-490B-82AA-358CD8118DF0@karels.net> <CAM5tNy5LGWF424i9v_w2u%2B41wtzmW9kn6RZWwo6Pi0Y4wj92UA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000072469f060a745a22
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Sat, Nov 18, 2023 at 8:09=E2=80=AFAM Rick Macklem <rick.macklem@gmail.co=
m> wrote:
>
> On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels <mike@karels.net> wro=
te:
> >
> > On 17 Nov 2023, at 22:14, Mike Karels wrote:
> >
> > > On 17 Nov 2023, at 21:24, Rick Macklem wrote:
> > >
> > >> Most of the changes in stable/13 that are not in releng/13.2
> > >> are the "make it work in a jail" stuff. Unfortunately, they are
> > >> a large # of changes (mostly trivial edits adding vnet macros),
> > >> but it also includes export check changes.
> > >>
> > >> I have attached a trivial patch that I think disables the export
> > >> checks for jails. If either of you can try it and see if it fixes
> > >> the problem, that would be great.
> > >> (Note that this is only for testing, although it probably does not
> > >> matter unless you are running nfsd(8) in vnet jails.)
> > >
> > > Yes, I can see snapshots with the patch. This system is just a test
> > > system that doesn't normally run ZFS or NFS, so no problem messing
> > > with permissions. It's a bhyve VM, so I just added a small disk and
> > > enabled ZFS for testing.
> >
> > btw, you might try to get mm@ or maybe mav@ to help out from the ZFS
> > side. It must be doing something differently inside a snapshot than
> > outside, maybe with file handles or something like that.
> Yes. I've added freebsd-current@ (although Garrett is not on it, he is
> cc'd) and these guys specifically...
>
> So, here's what appears to be the problem...
> Commit 88175af (in main and stable/13, but not 13.2) added checks for
> nfsd(8) running in jails by filling in mnt_exjail with a reference to the=
cred
> used when the file system is exported.
> When mnt_exjail is found NULL, the current nfsd code assumes that there
> is no access allowed for the mount.
>
> My vague understanding is that when a ZFS snapshot is accessed, it is
> "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that
> mnt_exjail is NULL as a result.
> Since I do not know the ZFS code and don't even have an easy way to
> test this (thankfully Mike can test easily), I do not know what to do fro=
m
> here?
>
> Is there a "struct mount" constructed for this pseudo mount
> (or it actually appears to be the lookup of ".." that fails, so it
> might be the parent of the snapshot subdir?)?
>
> One thought is that I can check to see if the mount pointer is in the
> mountlist (I don't think the snapshot's mount is in the mountlist) and
> avoid the jail test for this case. This would assume that snapshots are
> always within the file system(s) exported via that jail (which includes
> the case of prison0, of course), so that they do not need a separate
> jail check.
>
> If this doesn't work, there will need to be some sort of messing about
> in ZFS to set mnt_exjail for these.
Ok, so now onto the hard part...
Thanks to Mike and others, I did create a snapshot under .zfs and I can
see the problem. It is that mnt_exjail =3D=3D NULL.
Now, is there a way that this "struct mount" can be recognized as "special"
for snapshots, so I can avoid the mnt_exjail =3D=3D NULL test?
(I had hoped that "mp->mnt_list.tqe_prev" would be NULL, but that is not
the case.)
Do I need to search mountlist for it?
rick
ps: The hack patch attached should fix the problem, but can only be
safely used if mountd/nfsd are not run in any jails.
>
> I will try and get a test setup going here, which leads me to..
> how do I create a ZFS snapshot? (I do have a simple ZFS pool running
> on a test machine, but I've never done a snapshot.)
>
> Although this problem is not in 13.2, it will have shipped in 14.0.
>
> Any help with be appreciated, rick
>
> >
> > Mike
> > >
> > >> rick
> > >>
> > >> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels <mike@karels.net=
> wrote:
> > >>>
> > >>> CAUTION: This email originated from outside of the University of Gu=
elph. Do not click links or open attachments unless you recognize the sende=
r and know the content is safe. If in doubt, forward suspicious emails to I=
Thelp@uoguelph.ca.
> > >>>
> > >>>
> > >>> Rick, have you been following this thread on freebsd-stable? I hav=
e been able
> > >>> to reproduce this using a 13-stable server from Oct 7 and a 15-curr=
ent system
> > >>> that is up to date using NFSv3. I did not reproduce with a 13.2 se=
rver. The
> > >>> client was running 13.2. Any ideas? A full bisect seems fairly pa=
inful, but
> > >>> maybe you have an idea of points to try. Fortunately, these are al=
l test
> > >>> systems that I can reboot at will.
> > >>>
> > >>> Mike
> > >>>
> > >>> Forwarded message:
> > >>>
> > >>>> From: Garrett Wollman <wollman@bimajority.org>
> > >>>> To: Mike Karels <mike@karels.net>
> > >>>> Cc: freebsd-stable@freebsd.org
> > >>>> Subject: Re: NFS exports of ZFS snapshots broken
> > >>>> Date: Fri, 17 Nov 2023 17:35:04 -0500
> > >>>>
> > >>>> <<On Fri, 17 Nov 2023 15:57:42 -0600, Mike Karels <mike@karels.net=
> said:
> > >>>>
> > >>>>> I have not run into this, so I tried it just now. I had no probl=
em.
> > >>>>> The server is 13.2, fully patched, the client is up-to-date -curr=
ent,
> > >>>>> and the mount is v4.
> > >>>>
> > >>>> On my 13.2 client and 13-stable server, I see:
> > >>>>
> > >>>> 25034 ls CALL open(0x237d32f9a000,0x120004<O_RDONLY|O_NONB=
LOCK|O_DIRECTORY|O_CLOEXEC>)
> > >>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45"
> > >>>> 25034 ls RET open 4
> > >>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0)
> > >>>> 25034 ls RET fcntl 0
> > >>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0x23=
7d32fa7028)
> > >>>> 25034 ls RET getdirentries -1 errno 5 Input/output error
> > >>>> 25034 ls CALL close(0x4)
> > >>>> 25034 ls RET close 0
> > >>>> 25034 ls CALL exit(0)
> > >>>>
> > >>>> Certainly a libc bug here that getdirentries(2) returning [EIO]
> > >>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is
> > >>>> consistent across both FreeBSD and Linux clients.
> > >>>>
> > >>>> Looking at this from the RPC side:
> > >>>>
> > >>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR)
> > >>>> [NFS4_OK for all ops]
> > >>>> (PUTFH, GETATTR)
> > >>>> [NFS4_OK, NFS4_OK]
> > >>>> (PUTFH, ACCESS(0x3f), GETATTR)
> > >>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK]
> > >>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR)
> > >>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE]
> > >>>>
> > >>>> and at this point the [EIO] is returned.
> > >>>>
> > >>>> It seems that clients always do a LOOKUPP before calling READDIR, =
and
> > >>>> this is failing when the subject file handle is the snapshot. The
> > >>>> client is perfectly able to *traverse into* the snapshot: if I try=
to
> > >>>> list a subdirectory I know exists in the snapshot, the client is a=
ble to
> > >>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with
> > >>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*.
> > >>>>
> > >>>> -GAWollman
> > >>>
--00000000000072469f060a745a22
Content-Type: application/octet-stream; name="zfsjail.patch"
Content-Disposition: attachment; filename="zfsjail.patch"
Content-Transfer-Encoding: base64
Content-ID: <f_lp4lbu7d0>
X-Attachment-Id: f_lp4lbu7d0
LS0tIHN5cy9mcy9uZnNzZXJ2ZXIvbmZzX25mc2Rwb3J0LmMuamFpbAkyMDIzLTExLTE3IDE5OjEw
OjEyLjg3MTYxOTAwMCAtMDgwMAorKysgc3lzL2ZzL25mc3NlcnZlci9uZnNfbmZzZHBvcnQuYwky
MDIzLTExLTE3IDE5OjExOjM4Ljk0MjY3NDAwMCAtMDgwMApAQCAtMzI4MiwxMSArMzI4MiwxMyBA
QCBuZnN2bm9fY2hlY2tleHAoc3RydWN0IG1vdW50ICptcCwgc3RydWN0IHNvY2thZGRyICpuYW0K
IAogCWVycm9yID0gMDsKIAkqY3JlZHAgPSBOVUxMOworI2lmZGVmIG5vdG5vdwogCU1OVF9JTE9D
SyhtcCk7CiAJaWYgKG1wLT5tbnRfZXhqYWlsID09IE5VTEwgfHwKIAkgICAgbXAtPm1udF9leGph
aWwtPmNyX3ByaXNvbiAhPSBjdXJ0aHJlYWQtPnRkX3VjcmVkLT5jcl9wcmlzb24pCiAJCWVycm9y
ID0gRUFDQ0VTOwogCU1OVF9JVU5MT0NLKG1wKTsKKyNlbmRpZgogCWlmIChlcnJvciA9PSAwKQog
CQllcnJvciA9IFZGU19DSEVDS0VYUChtcCwgbmFtLCAmZXhwLT5uZXNfZXhmbGFnLCBjcmVkcCwK
IAkJICAgICZleHAtPm5lc19udW1zZWNmbGF2b3IsIGV4cC0+bmVzX3NlY2ZsYXZvcnMpOwpAQCAt
MzMyMywxMSArMzMyNSwxMyBAQCBuZnN2bm9fZmh0b3ZwKHN0cnVjdCBtb3VudCAqbXAsIGZoYW5k
bGVfdCAqZmhwLCBzdHJ1Y3QKIAkJLyogTWFrZSBzdXJlIHRoZSBzZXJ2ZXIgcmVwbGllcyBFU1RB
TEUgdG8gdGhlIGNsaWVudC4gKi8KIAkJZXJyb3IgPSBFU1RBTEU7CiAJaWYgKG5hbSAmJiAhZXJy
b3IpIHsKKyNpZmRlZiBub3Rub3cKIAkJTU5UX0lMT0NLKG1wKTsKIAkJaWYgKG1wLT5tbnRfZXhq
YWlsID09IE5VTEwgfHwKIAkJICAgIG1wLT5tbnRfZXhqYWlsLT5jcl9wcmlzb24gIT0gY3VydGhy
ZWFkLT50ZF91Y3JlZC0+Y3JfcHJpc29uKQogCQkJZXJyb3IgPSBFQUNDRVM7CiAJCU1OVF9JVU5M
T0NLKG1wKTsKKyNlbmRpZgogCQlpZiAoZXJyb3IgPT0gMCkKIAkJCWVycm9yID0gVkZTX0NIRUNL
RVhQKG1wLCBuYW0sICZleHAtPm5lc19leGZsYWcsIGNyZWRwLAogCQkJICAgICZleHAtPm5lc19u
dW1zZWNmbGF2b3IsIGV4cC0+bmVzX3NlY2ZsYXZvcnMpOwo=
--00000000000072469f060a745a22--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAM5tNy6K6psrtem%2Bc3ta0pM-7fhy9LYci0DEeR0O-EF%2B-iGo6Q>