From nobody Sat Nov 18 16:09:34 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4SXdyM4NcGz51J0c for ; Sat, 18 Nov 2023 16:09:47 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4SXdyL3SjDz3MCN; Sat, 18 Nov 2023 16:09:46 +0000 (UTC) (envelope-from rick.macklem@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=GCgNzbV3; spf=pass (mx1.freebsd.org: domain of rick.macklem@gmail.com designates 2607:f8b0:4864:20::1033 as permitted sender) smtp.mailfrom=rick.macklem@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-x1033.google.com with SMTP id 98e67ed59e1d1-2802e5ae23bso2613125a91.2; Sat, 18 Nov 2023 08:09:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700323785; x=1700928585; darn=freebsd.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=4opcVmtRK/B77y0nsISDw1smlbvyYKU4ccDLJgR+TXw=; b=GCgNzbV33aY8aJpo0yocB0ISg9ovL1uMVdBvpomuQJ4oBldO0jfw9TEfkhVNB0xtil SlgeaPKiyJp9pdwOD0ejjaPPgxjYN6HcGPhWNbaFhxa03jWZwvHwkvpy5WpCo4xrt97f IsV73faP63/VfhwVCNNYlMkFTxnRq4dIPvKrDJXK+Z7gOsGKyG9+2jGRcnQsheyISkxl V8vVudgY5FC/4neP9pVSumRoczRJulSz6YcrkqiZKNVI/MzYLGDCYI+YUk6DNtgkBxKj kVP/G+OHhnvaUjkrgXMp8GI0RommdFR3Nq7iWQOJ6osMzS6B1soIECt9NtGB54eDLTEx CxkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700323785; x=1700928585; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4opcVmtRK/B77y0nsISDw1smlbvyYKU4ccDLJgR+TXw=; b=cddRDcaVOrIPvnd3xar55hOy8cDSeqCAOJ7akCz4ueVJax1o23iVxfF/CucfCZywBv IFzz8uNBw54ZeaQyboFr2d/NmDmviYvA/suZ1xdSHapPLInMWJorpPhj7Y3z4Hlc6pHl BCfX5QXtmPRGSeFerlEBS3v0grJmFTlt6Z0Xw9KUo4FIsBd283qJy0jaVWVSCcCBspHw 1fbJDgkkZj8oqIRKDoEPK2G4LnzVuSN+YbxkkF+ndUeJFLn5myqC0bGiMADWILAaUq8W VmaL0SQT4wlSA8h7W3CT4KT+MTVmSDdX3c5hRCKy2deZ4vCW17tULbnHc/Ev69WiB1jP J1DQ== X-Gm-Message-State: AOJu0YzifGxNtBdah7HtrEKoDySjXEBkaoJqi9ZZ/bw+pzURP0RCe231 3l35BBbIyiMv11RbQMB+OV1HbniHpNFWmiUDoIG/QtQ= X-Google-Smtp-Source: AGHT+IGXH4atTi4Ef+gVb7PekLMg9Euwr2M1XEqkjwORLAgsvUxxjxflYHvtE9hnJ5gQT0FZz9tP/acOCAgj//8Jywc= X-Received: by 2002:a17:90b:1c07:b0:280:235:19d with SMTP id oc7-20020a17090b1c0700b002800235019dmr2720118pjb.36.1700323784865; Sat, 18 Nov 2023 08:09:44 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <25943.60056.880614.452966@hergotha.csail.mit.edu> In-Reply-To: From: Rick Macklem Date: Sat, 18 Nov 2023 08:09:34 -0800 Message-ID: Subject: Re: NFS exports of ZFS snapshots broken To: Mike Karels , FreeBSD CURRENT Cc: Rick Macklem , Garrett Wollman , Alexander Motin , Martin Matuska Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1033:from]; DKIM_TRACE(0.00)[gmail.com:+]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4SXdyL3SjDz3MCN X-Spamd-Bar: --- On Fri, Nov 17, 2023 at 8:19=E2=80=AFPM Mike Karels wrote= : > > On 17 Nov 2023, at 22:14, Mike Karels wrote: > > > On 17 Nov 2023, at 21:24, Rick Macklem wrote: > > > >> Most of the changes in stable/13 that are not in releng/13.2 > >> are the "make it work in a jail" stuff. Unfortunately, they are > >> a large # of changes (mostly trivial edits adding vnet macros), > >> but it also includes export check changes. > >> > >> I have attached a trivial patch that I think disables the export > >> checks for jails. If either of you can try it and see if it fixes > >> the problem, that would be great. > >> (Note that this is only for testing, although it probably does not > >> matter unless you are running nfsd(8) in vnet jails.) > > > > Yes, I can see snapshots with the patch. This system is just a test > > system that doesn't normally run ZFS or NFS, so no problem messing > > with permissions. It's a bhyve VM, so I just added a small disk and > > enabled ZFS for testing. > > btw, you might try to get mm@ or maybe mav@ to help out from the ZFS > side. It must be doing something differently inside a snapshot than > outside, maybe with file handles or something like that. Yes. I've added freebsd-current@ (although Garrett is not on it, he is cc'd) and these guys specifically... So, here's what appears to be the problem... Commit 88175af (in main and stable/13, but not 13.2) added checks for nfsd(8) running in jails by filling in mnt_exjail with a reference to the c= red used when the file system is exported. When mnt_exjail is found NULL, the current nfsd code assumes that there is no access allowed for the mount. My vague understanding is that when a ZFS snapshot is accessed, it is "pseudo-mounted" by zfsctl_snapdir_lookup() and I am guessing that mnt_exjail is NULL as a result. Since I do not know the ZFS code and don't even have an easy way to test this (thankfully Mike can test easily), I do not know what to do from here? Is there a "struct mount" constructed for this pseudo mount (or it actually appears to be the lookup of ".." that fails, so it might be the parent of the snapshot subdir?)? One thought is that I can check to see if the mount pointer is in the mountlist (I don't think the snapshot's mount is in the mountlist) and avoid the jail test for this case. This would assume that snapshots are always within the file system(s) exported via that jail (which includes the case of prison0, of course), so that they do not need a separate jail check. If this doesn't work, there will need to be some sort of messing about in ZFS to set mnt_exjail for these. I will try and get a test setup going here, which leads me to.. how do I create a ZFS snapshot? (I do have a simple ZFS pool running on a test machine, but I've never done a snapshot.) Although this problem is not in 13.2, it will have shipped in 14.0. Any help with be appreciated, rick > > Mike > > > >> rick > >> > >> On Fri, Nov 17, 2023 at 6:14=E2=80=AFPM Mike Karels = wrote: > >>> > >>> CAUTION: This email originated from outside of the University of Guel= ph. Do not click links or open attachments unless you recognize the sender = and know the content is safe. If in doubt, forward suspicious emails to ITh= elp@uoguelph.ca. > >>> > >>> > >>> Rick, have you been following this thread on freebsd-stable? I have = been able > >>> to reproduce this using a 13-stable server from Oct 7 and a 15-curren= t system > >>> that is up to date using NFSv3. I did not reproduce with a 13.2 serv= er. The > >>> client was running 13.2. Any ideas? A full bisect seems fairly pain= ful, but > >>> maybe you have an idea of points to try. Fortunately, these are all = test > >>> systems that I can reboot at will. > >>> > >>> Mike > >>> > >>> Forwarded message: > >>> > >>>> From: Garrett Wollman > >>>> To: Mike Karels > >>>> Cc: freebsd-stable@freebsd.org > >>>> Subject: Re: NFS exports of ZFS snapshots broken > >>>> Date: Fri, 17 Nov 2023 17:35:04 -0500 > >>>> > >>>> < = said: > >>>> > >>>>> I have not run into this, so I tried it just now. I had no problem= . > >>>>> The server is 13.2, fully patched, the client is up-to-date -curren= t, > >>>>> and the mount is v4. > >>>> > >>>> On my 13.2 client and 13-stable server, I see: > >>>> > >>>> 25034 ls CALL open(0x237d32f9a000,0x120004) > >>>> 25034 ls NAMI "/mnt/tools/.zfs/snapshot/weekly-2023-45" > >>>> 25034 ls RET open 4 > >>>> 25034 ls CALL fcntl(0x4,F_ISUNIONSTACK,0x0) > >>>> 25034 ls RET fcntl 0 > >>>> 25034 ls CALL getdirentries(0x4,0x237d32faa000,0x1000,0x237d= 32fa7028) > >>>> 25034 ls RET getdirentries -1 errno 5 Input/output error > >>>> 25034 ls CALL close(0x4) > >>>> 25034 ls RET close 0 > >>>> 25034 ls CALL exit(0) > >>>> > >>>> Certainly a libc bug here that getdirentries(2) returning [EIO] > >>>> results in ls(1) returning EXIT_SUCCESS, but the [EIO] error is > >>>> consistent across both FreeBSD and Linux clients. > >>>> > >>>> Looking at this from the RPC side: > >>>> > >>>> (PUTFH, GETATTR, LOOKUP(snapshotname), GETFH, GETATTR) > >>>> [NFS4_OK for all ops] > >>>> (PUTFH, GETATTR) > >>>> [NFS4_OK, NFS4_OK] > >>>> (PUTFH, ACCESS(0x3f), GETATTR) > >>>> [NFS4_OK, NFS4_OK, rights =3D 0x03, NFS4_OK] > >>>> (PUTFH, GETATTR, LOOKUPP, GETFH, GETATTR) > >>>> [NFS4_OK, NFS4_OK, NFS4ERR_NOFILEHANDLE] > >>>> > >>>> and at this point the [EIO] is returned. > >>>> > >>>> It seems that clients always do a LOOKUPP before calling READDIR, an= d > >>>> this is failing when the subject file handle is the snapshot. The > >>>> client is perfectly able to *traverse into* the snapshot: if I try t= o > >>>> list a subdirectory I know exists in the snapshot, the client is abl= e to > >>>> LOOKUP(dirname) just fine, but LOOKUPP still fails with > >>>> NFS4ERR_NOFILEHANDLE *on the subndirectory*. > >>>> > >>>> -GAWollman > >>>