From nobody Thu Jun 25 18:57:30 2026 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gmSjk5Tk2z6jBk6 for ; Thu, 25 Jun 2026 18:57:46 +0000 (UTC) (envelope-from rlibby@gmail.com) Received: from mail-yw1-f174.google.com (mail-yw1-f174.google.com [209.85.128.174]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gmSjj02Gyz3VHF for ; Thu, 25 Jun 2026 18:57:45 +0000 (UTC) (envelope-from rlibby@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=freebsd.org (policy=none); spf=pass (mx1.freebsd.org: domain of rlibby@gmail.com designates 209.85.128.174 as permitted sender) smtp.mailfrom=rlibby@gmail.com Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-7ff05e5d009so2679667b3.1 for ; Thu, 25 Jun 2026 11:57:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782413864; x=1783018664; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=+Y7pe2Kcugc6+EbC9/hol4NhTTirxcQqtjr0pl/v59o=; b=Vg089oAIPT06X1ys84fspDzQ2ADNPO1sK4FwkgFaoBmWbcF09JDVmtb1v3GNxnqUPZ PBXNGuPBTxBbacnU9W0d2mZjUCYFhqRUlqqALZD2UA2JaVc8XRx/W0MZZp6xxHsJhXmo lNIVu6xipVDlHvLO3ASClkG464d7XOJlyeD6uGVTUG9RqXLfHHzLuykozkZtmAJb1p2g b0AeHHCt6yCa1LAumK7FygAmquQIKET9xWLvTZMaWjfSS4tbT2oCqg1USjdrnz38lcYu XLMfxdWXEmSccuG8hfEpQqfAwTheHxK2ucjw0GvWenN8B7RP4RNrBBY4f7Uz1cBiqOBm t3pg== X-Forwarded-Encrypted: i=1; AHgh+RoARuj+udbCu7WbsASUAn0mRPVqwaDjFw7Rv14vmhU/96BQ34Sv5tgWGMg+2AzbrikGsjNU1hZaI3KJDTFxeMNuB6cn@freebsd.org X-Gm-Message-State: AOJu0Yws3Qiov9w+spLH0S/i6gMXB08Fg7KJhUXrn/AVZ35RnZeu3JEB tmS0LdQgWNC1kSTtopatieuN7Akqo9Hknf1rvD5k9XkegBX96UHkQGhUtSLU2g== X-Gm-Gg: AfdE7cmBu6Odcq0aNLAu2WN3leQoAvmDD1S5+U4RbUjDcyb2aNsdnj/HhPziRql+T3q fLSOwxiz2TDm0Ey9k8z9gkjHdrsrqKfVlnKO4hGBgKj8TTbYu8jMP9e5abtqafda2fJOaRrOkxo cMQ7ms9Xd1Avj8U3Z6LBL5PnLlX7AYP0r8r3DpZ1hvlu5hyS7Z9K7myaBUhAEiW24+DTRG7NCJc oJyJ+FXBILSl+5hfOxIac7SksrwGLlfNyUFlF6+xs6OeORZ4w7iY46+o88YZLd5LRV3NKDQhppb 98FOHquaDr7n7X32AxQy8Zi7ZhA+akWGuKauXKlwT9aNLonwhwP55JKF8ld1sNyNYNGANQNLjDl 8WpKVwBbmjBMMNRw540gOzY03TMPjgJwvS4PCAp4qYuxJ6mS2FB5NBMdACt+IU4NmQCZAlAs8yj tGwP1hn2oFZqedPDyvIYoN4nuH8ZG5l309TEbpLUX+APgcXik= X-Received: by 2002:a05:690c:9689:b0:7ff:1996:df95 with SMTP id 00721157ae682-80a695b7862mr41235077b3.20.1782413863119; Thu, 25 Jun 2026 11:57:43 -0700 (PDT) Received: from mail-yx1-f48.google.com (mail-yx1-f48.google.com. [74.125.224.48]) by smtp.gmail.com with ESMTPSA id 00721157ae682-8025ffc700bsm75328647b3.36.2026.06.25.11.57.42 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 25 Jun 2026 11:57:42 -0700 (PDT) Received: by mail-yx1-f48.google.com with SMTP id 956f58d0204a3-664345f8645so259104d50.3 for ; Thu, 25 Jun 2026 11:57:42 -0700 (PDT) X-Forwarded-Encrypted: i=1; AHgh+RqM9k9Tjxqi++3llxxerkU35yFZ2I+SBlWSCwhC8owpMv34afcHrGZa0FeZbWPsOH/M2aMIoibgWeEIlLMcKeKBm8BM@freebsd.org X-Received: by 2002:a05:690e:148b:b0:651:bd70:1622 with SMTP id 956f58d0204a3-66487ade410mr3096422d50.10.1782413861762; Thu, 25 Jun 2026 11:57:41 -0700 (PDT) List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 References: <6a36dfd8.30369.18992bfe@gitrepo.freebsd.org> In-Reply-To: <6a36dfd8.30369.18992bfe@gitrepo.freebsd.org> From: Ryan Libby Date: Thu, 25 Jun 2026 11:57:30 -0700 X-Gmail-Original-Message-ID: X-Gm-Features: AVVi8CeTQ9ei9kutBBAUKAmzoRMvvuQ9GTrf8rmi42jm1dBR050kTJdyoz3gQC4 Message-ID: Subject: Re: git: d8e8f9251975 - main - pNFSd: Add a directory of newly created files for the pNFSd To: Rick Macklem Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-2.18 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FORGED_SENDER(0.30)[rlibby@freebsd.org,rlibby@gmail.com]; NEURAL_HAM_SHORT(-0.28)[-0.276]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[text/plain]; DMARC_POLICY_SOFTFAIL(0.10)[freebsd.org : SPF not aligned (relaxed), No valid DKIM,none]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[dev-commits-src-all@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[rlibby@freebsd.org,rlibby@gmail.com]; FREEFALL_USER(0.00)[rlibby]; MLMMJ_DEST(0.00)[dev-commits-src-all@freebsd.org]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.128.174:from]; R_DKIM_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[209.85.128.174:from,74.125.224.48:received] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4gmSjj02Gyz3VHF On Sat, Jun 20, 2026 at 11:45=E2=80=AFAM Rick Macklem wrote: > > The branch main has been updated by rmacklem: > > URL: https://cgit.FreeBSD.org/src/commit/?id=3Dd8e8f9251975b68fe9c73e3907= 4e0f211ec2145e > > commit d8e8f9251975b68fe9c73e39074e0f211ec2145e > Author: Rick Macklem > AuthorDate: 2026-06-20 18:43:45 +0000 > Commit: Rick Macklem > CommitDate: 2026-06-20 18:43:45 +0000 > > pNFSd: Add a directory of newly created files for the pNFSd > > When an NFSv4.1/4.2 server is configured as a pNFS server, > new file creation (via Open/Create from an NFS client) is > slow, due to the fact that the NFS server (MDS) must do > RPCs against the DS(s). > > This patch precreates files in a directory called ".pnfshide/numfiles= ", > so that the NFS server can just rename them for the Open/Create. > A kernel process called a "replenisher" creates more files in > ".pnfshide/numfiles" as required. (At this point, the MDS must > still do Setattr RPC(s) on the DS(s), but that will change when > the pNFS server is converted to the loosely coupled configuration. > > This patch only affects the pNFS server and only if the directory > .pnfshide/numfiles exists in the exported file system. > > Reviewed by: kib > Discussed with: markj > Differential Revision: https://reviews.freebsd.org/D57554 > --- > sys/fs/nfsserver/nfs_nfsdport.c | 621 ++++++++++++++++++++++++++++++++++= +++--- > sys/kern/vfs_mount.c | 34 +++ > 2 files changed, 623 insertions(+), 32 deletions(-) > > diff --git a/sys/fs/nfsserver/nfs_nfsdport.c b/sys/fs/nfsserver/nfs_nfsdp= ort.c > index 5927146c78d6..ca8c9575bf59 100644 > --- a/sys/fs/nfsserver/nfs_nfsdport.c > +++ b/sys/fs/nfsserver/nfs_nfsdport.c > @@ -47,6 +47,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -114,6 +115,8 @@ VNET_DEFINE_STATIC(bool, nfsrv_mntinited) =3D false; > static int nfssvc_srvcall(struct thread *, struct nfssvc_args *, > struct ucred *); > static void nfsvno_updateds(struct vnode *, struct ucred *, struct threa= d *); > +static void nfsvno_pnfsreplenish(void *); > +static int nfsvno_pnfsusenumfile(struct nameidata *, struct vattr *); > > int nfsrv_enable_crossmntpt =3D 1; > static int nfs_commit_blks; > @@ -197,6 +200,19 @@ VNET_DECLARE(int, nfsd_enable_stringtouid); > SYSCTL_INT(_vfs_nfsd, OID_AUTO, enable_stringtouid, > CTLFLAG_VNET | CTLFLAG_RW, &VNET_NAME(nfsd_enable_stringtouid), > 0, "Enable nfsd to accept numeric owner_names"); > +/* > + * vfs.nfsd.pnfsswitchforw and vfs.nfsd.pnfsnumfilemiss are writable so = that > + * statistics can be reset. > + */ > +static uint64_t nfsrv_pnfsswitchforw =3D 0; > +SYSCTL_U64(_vfs_nfsd, OID_AUTO, pnfsswitchforw, CTLFLAG_RW, > + &nfsrv_pnfsswitchforw, 0, "Number of times replenish switches to for= ward"); > +static uint64_t nfsrv_pnfsnumfilemiss =3D 0; > +SYSCTL_U64(_vfs_nfsd, OID_AUTO, pnfsnumfilemiss, CTLFLAG_RW, > + &nfsrv_pnfsnumfilemiss, 0, "Number of numfile misses"); > +static u_int nfsrv_pnfsforwcnt =3D 5; > +SYSCTL_UINT(_vfs_nfsd, OID_AUTO, pnfsreplenishforwcnt, CTLFLAG_RW, > + &nfsrv_pnfsforwcnt, 0, "Forward replenish cnt before switch to back"= ); > static int nfsrv_pnfsgetdsattr =3D 1; > SYSCTL_INT(_vfs_nfsd, OID_AUTO, pnfsgetdsattr, CTLFLAG_RW, > &nfsrv_pnfsgetdsattr, 0, "When set getattr gets DS attributes via RP= C"); > @@ -231,6 +247,34 @@ SYSCTL_PROC(_vfs_nfsd, OID_AUTO, dsdirsize, > CTLTYPE_UINT | CTLFLAG_MPSAFE | CTLFLAG_RW, 0, sizeof(nfsrv_dsdirsiz= e), > sysctl_dsdirsize, "IU", "Number of dsN subdirs on the DS servers"); > > +/* > + * nfsrv_pnfsmaxnumfiles can only be decreased when the nfsd is not > + * running. It can be increased when the nfsd is running, but the > + * additional numfiles should have been precreated in .pnfshide/numfiles > + * for all file systems before it is increased. > + */ > +static u_int nfsrv_pnfsmaxnumfiles =3D 1000; > +static int > +sysctl_pnfsmaxnumfiles(SYSCTL_HANDLER_ARGS) > +{ > + int error, new_maxnumfiles; > + > + new_maxnumfiles =3D nfsrv_pnfsmaxnumfiles; > + error =3D sysctl_handle_int(oidp, &new_maxnumfiles, 0, req); > + if (error !=3D 0 || req->newptr =3D=3D NULL) > + return (error); > + if (new_maxnumfiles < nfsrv_pnfsmaxnumfiles && newnfs_numnfsd != =3D 0) > + return (EBUSY); > + if (new_maxnumfiles > 10000 || new_maxnumfiles < 100) > + return (EINVAL); > + nfsrv_pnfsmaxnumfiles =3D new_maxnumfiles; > + return (0); > +} > +SYSCTL_PROC(_vfs_nfsd, OID_AUTO, pnfsmaxnumfiles, > + CTLTYPE_UINT | CTLFLAG_MPSAFE | CTLFLAG_RW, 0, > + sizeof(nfsrv_pnfsmaxnumfiles), sysctl_pnfsmaxnumfiles, > + "IU", "Maximum number of entries in .pnfshide/numfiles"); > + > /* > * nfs_srvmaxio can only be increased and only when the nfsd threads are > * not running. The setting must be a power of 2, with the current limi= t of > @@ -1285,20 +1329,32 @@ nfsvno_createsub(struct nfsrv_descript *nd, struc= t nameidata *ndp, > error =3D nd->nd_repstat; > if (!error && ndp->ni_vp =3D=3D NULL) { > if (nvap->na_type =3D=3D VREG || nvap->na_type =3D=3D VSO= CK) { > - error =3D VOP_CREATE(ndp->ni_dvp, > - &ndp->ni_vp, &ndp->ni_cnd, &nvap->na_vattr); > - /* For a pNFS server, create the data file on a D= S. */ > - if (error =3D=3D 0 && nvap->na_type =3D=3D VREG) = { > + error =3D ENOENT; > + if (nvap->na_type =3D=3D VREG && > + !TAILQ_EMPTY(&nfsrv_devidhead)) > + error =3D nfsvno_pnfsusenumfile(ndp, > + &nvap->na_vattr); > + if (error =3D=3D ENOENT) { > + error =3D VOP_CREATE(ndp->ni_dvp, > + &ndp->ni_vp, &ndp->ni_cnd, &nvap->na_= vattr); > /* > - * Create a data file on a DS for a pNFS = server. > - * This function just returns if not > - * running a pNFS DS or the creation fail= s. > + * For a pNFS server, create the data fil= e > + * on a DS. > */ > - nfsrv_pnfscreate(ndp->ni_vp, &nvap->na_va= ttr, > - nd->nd_cred, p); > + if (error =3D=3D 0 && nvap->na_type =3D= =3D VREG) { > + /* > + * Create a data file on a DS for= a > + * pNFS server. > + * This function just returns if = not > + * running a pNFS DS or the creat= ion > + * fails. > + */ > + nfsrv_pnfscreate(ndp->ni_vp, > + &nvap->na_vattr, nd->nd_cred,= p); > + } > + VOP_VPUT_PAIR(ndp->ni_dvp, error =3D=3D 0= ? > + &ndp->ni_vp : NULL, false); > } > - VOP_VPUT_PAIR(ndp->ni_dvp, error =3D=3D 0 ? &ndp-= >ni_vp : > - NULL, false); > nfsvno_relpathbuf(ndp); > if (!error) { > if (*exclusive_flagp) { > @@ -2030,6 +2086,486 @@ nfsvno_statfs(struct vnode *vp, struct statfs *sf= ) > return (error); > } > > +/* > + * Replenish the numfiles in .pnfshide/numfiles directory. > + * These files are used for the pNFS server when an Open/Create needs a > + * new regular file. By creating them here asynchronously, we can avoid > + * the delay of doing do for Open/Create, since creation requires RPCs t= o > + * the DSs be done. > + * (A) - When the sleep times out, work backwards creating > + * one new numfile for each cycle. > + * (Use a timeout of 10msec for now.) > + * (B) - When the sleep returns 0, this indicates that a > + * nfsd thread didn't find a numfiles. For this case > + * be more agressive and create numfiles going forward. > + * (Use a timeout of 1msec for now.) > + * Runs as a kernel process. > + */ > +static char pnfshide_name[] =3D ".pnfshide"; > +static char numfiles_name[] =3D "numfiles"; > + > +static void > +nfsvno_pnfsreplenish(void *arg) > +{ > + struct componentname cn; > + struct vattr va; > + char name[11]; > + struct timespec ts; > + struct mount *mp =3D (struct mount *)arg, *temp_mp; > + struct ucred *cred; > + struct vnode *numfiledvp, *vp; > + struct netexport *nep; > + uint64_t prevcnt; > + time_t prevsec; > + u_int cnt, last_back, next_back, next_forw, prevrate[4]; > + int averate, error, i, timo; > + bool back, use_same_num; > + > + cred =3D curthread->td_ucred; > + if (cred->cr_uid !=3D 0) > + printf("nfsvno_pnfsreplenish: not root\n"); > + > + /* > + * Do a lookup for ".pnfshide" in the root dir > + * of the file system. > + */ > + cn.cn_nameiop =3D LOOKUP; > + cn.cn_lkflags =3D LK_SHARED; > + cn.cn_flags =3D ISLASTCN | NOFOLLOW | LOCKLEAF | NOCROSSMOUNT; > + cn.cn_cred =3D cred; > + cn.cn_nameptr =3D pnfshide_name; > + cn.cn_namelen =3D sizeof(pnfshide_name) - 1; > + vp =3D NULL; > + numfiledvp =3D NULL; > + error =3D vn_lock(mp->mnt_rootvnode, LK_SHARED); > + if (error =3D=3D 0) { > + error =3D VOP_LOOKUP(mp->mnt_rootvnode, &vp, &cn); > + VOP_UNLOCK(mp->mnt_rootvnode); > + } > + > + /* > + * Do a lookup for "numfiles" in the ".pnfshide" dir > + * of the file system. > + */ > + if (error =3D=3D 0) { > + cn.cn_nameiop =3D LOOKUP; > + cn.cn_lkflags =3D LK_SHARED; > + cn.cn_flags =3D ISLASTCN | NOFOLLOW | LOCKLEAF | NOCROSSM= OUNT; > + cn.cn_cred =3D cred; > + cn.cn_nameptr =3D numfiles_name; > + cn.cn_namelen =3D sizeof(numfiles_name) - 1; > + error =3D VOP_LOOKUP(vp, &numfiledvp, &cn); > + vput(vp); > + if (error =3D=3D 0) > + VOP_UNLOCK(numfiledvp); > + } > + lockmgr(&mp->mnt_explock, LK_SHARED, NULL); > + nep =3D mp->mnt_export; > + if (nep !=3D NULL) > + (void)vfs_netexport_acquire(nep); > + lockmgr(&mp->mnt_explock, LK_RELEASE, NULL); > + > + /* > + * The states for ne_pnfsnumfile are as follows: > + * NULL - Initial state for normal operation. > + * PNFSD_START - Transition state when the > + * replenisher kernel process is starting up. > + * non-NULL valid pointer - Points to the directory vnode for the > + * "numfiles" directory. > + * PNFSD_STOP - Transition state when the > + * replenisher kernel process is shutting down. > + * PNFSD_STOPPED - Replenisher kernel process has stopped and > + * vrele()'d the "numfiles" directory vnode. > + */ > + if (error !=3D 0 || nep =3D=3D NULL) { > + if (error =3D=3D 0) > + error =3D ENOENT; > + goto out; > + } > + MNTEXP_LOCK(nep); > + KASSERT(nep->ne_pnfsnumfile =3D=3D PNFSD_START | > + nep->ne_pnfsnumfile =3D=3D PNFSD_STOP, > + ("nfsvno_pnfsreplenish: ne_pnfsnumfile not PNFSD_START/STOP")= ); > + if (nep->ne_pnfsnumfile =3D=3D PNFSD_START) { > + nep->ne_pnfsnumfile =3D numfiledvp; > + wakeup(&nep->ne_pnfsnumfile); > + } > + MNTEXP_UNLOCK(nep); > + > + VATTR_NULL(&va); > + va.va_mode =3D 0644; > + va.va_type =3D VREG; > + timo =3D hz / 1000; > + if (timo =3D=3D 0) > + timo =3D 1; > + cnt =3D 5; > + back =3D false; > + use_same_num =3D false; > + prevcnt =3D 0; > + for (i =3D 0; i < 4; i++) > + prevrate[i] =3D 1; > + prevsec =3D 0; > + averate =3D 100; > + last_back =3D next_back =3D next_forw =3D UINT_MAX; > + > + /* Loop around sleeping and then doing (A) or (B) */ > + for (;;) { > + /* Sample replenish rate once/sec. */ > + getnanouptime(&ts); > + if (ts.tv_sec !=3D prevsec) { > + /* Calculate a moving ave. of creates/sec. */ > + prevsec =3D ts.tv_sec; > + for (i =3D 0; i < 3; i++) > + prevrate[i + 1] =3D prevrate[i]; > + prevrate[0] =3D atomic_load_int(&nep->ne_pnfsnumc= nt) - > + prevcnt; > + if (prevrate[0] < 1) > + prevrate[0] =3D 1; > + prevcnt =3D atomic_load_int(&nep->ne_pnfsnumcnt); > + averate =3D prevrate[0] * 4 / 10 + prevrate[1] * = 3 / 10 + > + prevrate[2] * 2 / 10 + prevrate[3] / 10; > + if (averate < 1) > + averate =3D 1; > + averate *=3D 2; > + } > + > + if (cnt =3D=3D 0) { > + error =3D tsleep(&mp->mnt_export, PVFS, "pnfsrpl"= , timo); > + if (error =3D=3D ETIMEDOUT || error =3D=3D EAGAIN= ) { > + if (!back) > + next_back =3D last_back; > + back =3D true; > + timo =3D hz / averate; > + if (timo =3D=3D 0) > + timo =3D 1; > + } else { > + if (back) { > + next_forw =3D UINT_MAX; > + last_back =3D next_back; > + } > + back =3D false; > + cnt =3D nfsrv_pnfsforwcnt; > + timo =3D hz / 1000; > + if (timo =3D=3D 0) > + timo =3D 1; > + nfsrv_pnfsswitchforw++; > + } > + } > + > + /* Check for exports having gone away. */ > + if (mp->mnt_export =3D=3D NULL) > + break; > + /* And check for replenisher being stopped. */ > + MNTEXP_LOCK(nep); > + if (nep->ne_pnfsnumfile !=3D PNFSD_START && > + nep->ne_pnfsnumfile !=3D PNFSD_STOP) { > + KASSERT(numfiledvp =3D=3D nep->ne_pnfsnumfile, > + ("nfsvno_pnfsreplenish: numfiledvp changed"))= ; > + MNTEXP_UNLOCK(nep); > + } else { > + MNTEXP_UNLOCK(nep); > + break; > + } > + > + if (back) { > + /* This is (A) in this function's comment above. = */ > + error =3D vn_start_write(numfiledvp, &temp_mp, V_= NOWAIT); > + if (error =3D=3D 0) > + error =3D vn_lock(numfiledvp, LK_EXCLUSIV= E | > + LK_NOWAIT); > + if (error !=3D 0 && temp_mp !=3D NULL) > + vn_finished_write(temp_mp); > + if (error =3D=3D EBUSY || error =3D=3D EWOULDBLOC= K) > + continue; > + if (error !=3D 0) > + break; > + if (next_back =3D=3D UINT_MAX) { > + if (nep->ne_pnfsnextfile =3D=3D 0) > + next_back =3D nfsrv_pnfsmaxnumfil= es - 1; > + else > + next_back =3D nep->ne_pnfsnextfil= e - 1; > + } else if (!use_same_num) { > + if (next_back =3D=3D 0) > + next_back =3D nfsrv_pnfsmaxnumfil= es - 1; > + else > + next_back--; > + } > + snprintf(name, sizeof(name), "%d", next_back); > + } else { > + /* This is (B) in this function's comment, above.= */ > + vn_start_write(numfiledvp, &temp_mp, V_WAIT); > + error =3D vn_lock(numfiledvp, LK_EXCLUSIVE); > + if (error !=3D 0 && temp_mp !=3D NULL) > + vn_finished_write(temp_mp); > + if (error !=3D 0) > + break; > + if (next_forw =3D=3D UINT_MAX) > + next_forw =3D nep->ne_pnfsnextfile; > + else if (!use_same_num) > + next_forw =3D (next_forw + 1) % > + nfsrv_pnfsmaxnumfiles; > + snprintf(name, sizeof(name), "%d", next_forw); > + } > + use_same_num =3D false; > + > + /* Do a lookup for the file. */ > + cn.cn_nameiop =3D CREATE; > + cn.cn_lkflags =3D LK_EXCLUSIVE; > + cn.cn_flags =3D ISLASTCN | NOFOLLOW | LOCKLEAF | LOCKPARE= NT | > + NOCROSSMOUNT | MAKEENTRY; > + cn.cn_cred =3D cred; > + cn.cn_nameptr =3D name; > + cn.cn_namelen =3D strlen(name); > + vref(numfiledvp); > + error =3D VOP_LOOKUP(numfiledvp, &vp, &cn); > + if (error =3D=3D 0) { > + VOP_VPUT_PAIR(numfiledvp, &vp, true); > + if (temp_mp !=3D NULL) > + vn_finished_write(temp_mp); > + if (back) { > + last_back =3D next_back =3D UINT_MAX; > + timo =3D hz / 10; > + if (timo =3D=3D 0) > + timo =3D 1; > + } else { > + timo =3D hz / 100; > + if (timo =3D=3D 0) > + timo =3D 1; > + } > + cnt =3D 0; > + continue; > + } else if (error !=3D ENOENT && error !=3D EJUSTRETURN) { > + VOP_VPUT_PAIR(numfiledvp, NULL, true); > + if (temp_mp !=3D NULL) > + vn_finished_write(temp_mp); > + if (error =3D=3D ERELOOKUP) { > + use_same_num =3D true; > + continue; > + } > + printf("nfsvno_pnfsreplenish: lookup failed %d\n"= , > + error); > + break; > + } > + > + /* Create the numfile and its DS file(s). */ > + error =3D VOP_CREATE(numfiledvp, &vp, &cn, &va); > + if (error =3D=3D 0) { > + /* > + * Create a data file on a DS for a pNFS > + * server. This function just returns if > + * not running a pNFS DS or the creation > + * fails. > + */ > + nfsrv_pnfscreate(vp, &va, cred, curthread); > + } else > + printf("nfsvno_pnfsreplenish: vop_create failed %= d\n", > + error); > + VOP_VPUT_PAIR(numfiledvp, error =3D=3D 0 ? &vp : NULL, tr= ue); > + if (temp_mp !=3D NULL) > + vn_finished_write(temp_mp); > + cnt =3D cnt > 0 ? cnt - 1 : 0; > + } > +out: > + if (numfiledvp !=3D NULL) > + vrele(numfiledvp); > + if (nep !=3D NULL) { > + MNTEXP_LOCK(nep); > + nep->ne_pnfsnumfile =3D PNFSD_STOPPED; > + wakeup(&mp->mnt_explock); > + MNTEXP_UNLOCK(nep); > + vfs_netexport_release(nep); > + } else > + wakeup(&mp->mnt_explock); > + kproc_exit(0); > +} > + > +/* > + * Do a lookup of a file in the .numfiles directory. > + * If successful, use VOP_SETATTR() to set the uid/gid/mode and > + * then VOP_LINK()/VOP_REMOVE() the num file. > + * Return ENOENT to indicate that nfsvno_open() should fall back to > + * doing VOP_CREATE(), other errors for failure. > + * XXX This code probably is not correct for a stacked file > + * system, but should never be used for that case. > + */ > +static int > +nfsvno_pnfsusenumfile(struct nameidata *ndp, struct vattr *vap) > +{ > + struct componentname cn; > + struct vattr va; > + char name[11]; > + gid_t gid; > + struct ucred *cred, *savcred; > + struct vnode *numfiledvp; > + struct mount *mp; > + struct netexport *nep; > + u_int nextf; > + int error; > + > + cred =3D newnfs_getcred(); > + /* > + * Not sure if this is necessary. If all VOP calls use > + * cn_cred, it is not. > + */ > + savcred =3D curthread->td_ucred; > + curthread->td_ucred =3D cred; > + > + /* > + * If the replenish kernel process is not yet running, > + * start it up now. > + */ > + numfiledvp =3D NULL; > + ndp->ni_vp =3D NULL; > + mp =3D ndp->ni_dvp->v_mount; > + lockmgr(&mp->mnt_explock, LK_SHARED, NULL); > + nep =3D mp->mnt_export; > + if (nep =3D=3D NULL) { > + error =3D ENOENT; > + lockmgr(&mp->mnt_explock, LK_RELEASE, NULL); > + goto out; > + } > + (void)vfs_netexport_acquire(nep); > + lockmgr(&mp->mnt_explock, LK_RELEASE, NULL); > + MNTEXP_LOCK(nep); > + if (nep->ne_pnfsnumfile =3D=3D NULL) { > + /* Mark kernel process startup in-progress. */ > + nep->ne_pnfsnumfile =3D PNFSD_START; > + MNTEXP_UNLOCK(nep); > + > + /* Create the replenish kernel process. */ > + error =3D kproc_create(nfsvno_pnfsreplenish, mp, NULL, RF= HIGHPID, > + 0, "pnfsreplenish"); > + if (error !=3D 0) { > + printf("nfsvno_pnfsusenumfile: replenish won't st= art" > + " %d\n", error); > + error =3D ENOENT; > + goto out; > + } > + > + /* And wait for it to set up ne_pnfsnumfile. */ > + MNTEXP_LOCK(nep); > + (void)msleep(&nep->ne_pnfsnumfile, MNTEXP_MTX(nep), PVFS, > + "pnfsnumf", hz); > + } > + > + if (nep->ne_pnfsnumfile =3D=3D NULL || > + nep->ne_pnfsnumfile =3D=3D PNFSD_START || > + nep->ne_pnfsnumfile =3D=3D PNFSD_STOP) { > + MNTEXP_UNLOCK(nep); > + error =3D ENOENT; > + goto out; > + } else { > + numfiledvp =3D nep->ne_pnfsnumfile; > + MNTEXP_UNLOCK(nep); > + /* > + * Check to ensure the new file is not in ".pnfshide/numf= iles". > + */ > + if (numfiledvp =3D=3D ndp->ni_dvp) { > + error =3D ENOENT; > + numfiledvp =3D NULL; > + goto out; > + } > + } > + > + error =3D vn_lock(numfiledvp, LK_EXCLUSIVE); > + if (error !=3D 0) { > + error =3D ENOENT; > + numfiledvp =3D NULL; > + goto out; > + } > + vref(numfiledvp); > + > + /* Get the next filenum. */ > + nextf =3D nep->ne_pnfsnextfile; > + snprintf(name, sizeof(name), "%d", nextf); > + > + /* Now, look up the numbered file. */ > + cn.cn_nameiop =3D DELETE; > + cn.cn_lkflags =3D LK_EXCLUSIVE; > + cn.cn_flags =3D ISLASTCN | NOFOLLOW | LOCKLEAF | LOCKPARENT | > + NOCROSSMOUNT; > + cn.cn_cred =3D cred; > + cn.cn_nameptr =3D name; > + cn.cn_namelen =3D strlen(name); > + error =3D VOP_LOOKUP(numfiledvp, &ndp->ni_vp, &cn); > + if (error !=3D 0) { > + nfsrv_pnfsnumfilemiss++; > + VOP_UNLOCK(numfiledvp); > + ndp->ni_vp =3D NULL; > + if (error =3D=3D ENOENT || error =3D=3D EJUSTRETURN) > + wakeup(&mp->mnt_export); > + else > + VOP_VPUT_PAIR(ndp->ni_dvp, NULL, true); > + goto out; > + } > + > + /* > + * Set the new file's attributes to what VOP_CREATE() would > + * have set them to. > + */ > + gid =3D GID_NOGROUP; > + if (vap->va_gid =3D=3D VNOVAL && > + VOP_GETATTR(ndp->ni_dvp, &va, cred) =3D=3D 0) > + gid =3D va.va_gid; > + VATTR_NULL(&va); > + va.va_gid =3D gid; > + va.va_uid =3D ndp->ni_cnd.cn_cred->cr_uid; > + va.va_mode =3D vap->va_mode; > + error =3D VOP_SETATTR(ndp->ni_vp, &va, cred); > + if (error !=3D 0) { > + VOP_UNLOCK(numfiledvp); > + VOP_VPUT_PAIR(ndp->ni_dvp, &ndp->ni_vp, true); > + ndp->ni_vp =3D NULL; > + printf("nfsvno_pnfsusenumfile: setattr failed %d\n", > + error); > + if (error =3D=3D ENOENT) > + error =3D ENXIO; > + goto out; > + } > + > + /* > + * Link the numbered file to the name VOP_CREATE() would have > + * created in the correct directory and then VOP_REMOVE() the > + * numbered file. > + * Use VOP_LINK()/VOP_REMOVE() so that the numbered file > + * directory can remain locked. > + */ > + error =3D VOP_LINK(ndp->ni_dvp, ndp->ni_vp, &ndp->ni_cnd); > + /* Remove the file in .numfiles. */ > + if (error =3D=3D 0) { > + nep->ne_pnfsnextfile =3D (nextf + 1) % > + nfsrv_pnfsmaxnumfiles; > + error =3D VOP_REMOVE(numfiledvp, ndp->ni_vp, &cn); > + if (error !=3D 0) { > + /* Shut down the numfiles stuff. */ > + MNTEXP_LOCK(nep); > + nep->ne_pnfsnumfile =3D PNFSD_STOP; > + MNTEXP_UNLOCK(nep); > + printf("nfsvno_pnfsusenumfile: remove failed " > + "%d %s\n", error, name); > + } > + } > + VOP_UNLOCK(numfiledvp); > + if (error !=3D 0) { > + VOP_VPUT_PAIR(ndp->ni_dvp, &ndp->ni_vp, true); > + ndp->ni_vp =3D NULL; > + } else { > + atomic_add_int(&nep->ne_pnfsnumcnt, 1); > + VOP_VPUT_PAIR(ndp->ni_dvp, &ndp->ni_vp, false); > + } > + if (error =3D=3D ENOENT) > + error =3D ENXIO; > + > +out: > + if (numfiledvp !=3D NULL) > + vrele(numfiledvp); > + if (nep !=3D NULL) > + vfs_netexport_release(nep); > + curthread->td_ucred =3D savcred; /* Reset the thread's c= red. */ > + NFSFREECRED(cred); > + return (error); > +} > + > /* > * Do the vnode op stuff for Open. Similar to nfsvno_createsub(), but > * must handle nfsrv_opencheck() calls after any other access checks. > @@ -2048,6 +2584,7 @@ nfsvno_open(struct nfsrv_descript *nd, struct namei= data *ndp, > struct thread *p =3D curthread; > uint32_t oldrepstat; > u_long savflags; > + int error; > > if (ndp->ni_vp =3D=3D NULL) { > /* > @@ -2062,30 +2599,50 @@ nfsvno_open(struct nfsrv_descript *nd, struct nam= eidata *ndp, > } > if (!nd->nd_repstat) { > if (ndp->ni_vp =3D=3D NULL) { > - /* > - * Most file systems ignore va_flags for > - * VOP_CREATE(), however setting va_flags > - * for VOP_CREATE() causes problems for ZFS. > - * So disable them and let nfsrv_fixattr() > - * do them, as required. > - */ > - savflags =3D nvap->na_flags; > - nvap->na_flags =3D VNOVAL; > - nd->nd_repstat =3D VOP_CREATE(ndp->ni_dvp, > - &ndp->ni_vp, &ndp->ni_cnd, &nvap->na_vattr); > - /* For a pNFS server, create the data file on a D= S. */ > - if (nd->nd_repstat =3D=3D 0) { > + struct sockaddr_in *sin; > + struct sockaddr_in6 *sin6; > + bool try_pnfs; > + > + sin =3D (struct sockaddr_in *)nd->nd_nam; > + sin6 =3D (struct sockaddr_in6 *)nd->nd_nam; > + error =3D ENOENT; > + try_pnfs =3D !TAILQ_EMPTY(&nfsrv_devidhead); > + > + if (try_pnfs && !(sin->sin_family =3D=3D AF_INET = && > + IN_LOOPBACK(ntohl(sin->sin_addr.s_addr))) && > + !(sin6->sin6_family =3D=3D AF_INET6 && > + IN6_IS_ADDR_LOOPBACK(&sin6->sin6_addr))) { > + error =3D nfsvno_pnfsusenumfile(ndp, > + &nvap->na_vattr); > + if (error !=3D ENOENT) > + nd->nd_repstat =3D error; > + } > + if (error =3D=3D ENOENT) { > /* > - * Create a data file on a DS for a pNFS = server. > - * This function just returns if not > - * running a pNFS DS or the creation fail= s. > + * Most file systems ignore va_flags for > + * VOP_CREATE(), however setting va_flags > + * for VOP_CREATE() causes problems for Z= FS. > + * So disable them and let nfsrv_fixattr(= ) > + * do them, as required. > */ > - nfsrv_pnfscreate(ndp->ni_vp, &nvap->na_va= ttr, > - cred, p); > + savflags =3D nvap->na_flags; > + nvap->na_flags =3D VNOVAL; > + nd->nd_repstat =3D VOP_CREATE(ndp->ni_dvp= , > + &ndp->ni_vp, &ndp->ni_cnd, &nvap->na_= vattr); > + if (try_pnfs && nd->nd_repstat =3D=3D 0) = { > + /* > + * Create a data file on a DS for= a pNFS > + * server. This function just ret= urns if > + * not running a pNFS DS or the c= reation > + * fails. > + */ > + nfsrv_pnfscreate(ndp->ni_vp, > + &nvap->na_vattr, cred, p); > + } > + VOP_VPUT_PAIR(ndp->ni_dvp, nd->nd_repstat= =3D=3D 0 ? > + &ndp->ni_vp : NULL, false); > + nvap->na_flags =3D savflags; > } > - nvap->na_flags =3D savflags; > - VOP_VPUT_PAIR(ndp->ni_dvp, nd->nd_repstat =3D=3D = 0 ? > - &ndp->ni_vp : NULL, false); > nfsvno_relpathbuf(ndp); > if (!nd->nd_repstat) { > if (*exclusive_flagp !=3D NFSV4_EXCLUSIVE= _NONE) { > diff --git a/sys/kern/vfs_mount.c b/sys/kern/vfs_mount.c > index 4d724a0f1c38..f123c59981c4 100644 > --- a/sys/kern/vfs_mount.c > +++ b/sys/kern/vfs_mount.c > @@ -66,6 +66,10 @@ > #include > #include > > +#include > +#include > +#include > + > #include > > #include > @@ -77,6 +81,7 @@ static int vfs_domount(struct thread *td, const char= *fstype, char *fspath, > uint64_t fsflags, bool only_export, bool jail_export, > struct vfsoptlist **optlist); > static void free_mntarg(struct mntarg *ma); > +static void pnfsd_waitreplenish(struct mount *mp); > > static int usermount =3D 0; > SYSCTL_INT(_vfs, OID_AUTO, usermount, CTLFLAG_RW, &usermount, 0, > @@ -2386,6 +2391,10 @@ dounmount(struct mount *mp, uint64_t flags, struct= thread *td) > mp->mnt_flag &=3D ~MNT_ASYNC; > mp->mnt_kern_flag &=3D ~MNTK_ASYNC; > MNT_IUNLOCK(mp); > + > + /* Wait for any replenish kernel process to terminate. */ > + pnfsd_waitreplenish(mp); > + > vfs_deallocate_syncvnode(mp); > error =3D VFS_UNMOUNT(mp, flags); > vn_finished_write(mp); > @@ -3240,3 +3249,28 @@ resume_all_fs(void) > } > mtx_unlock(&mountlist_mtx); > } > + > +static void > +pnfsd_waitreplenish(struct mount *mp) > +{ > + struct netexport *nep; > + > + lockmgr(&mp->mnt_explock, LK_SHARED, NULL); > + nep =3D mp->mnt_export; > + if (nep !=3D NULL) { > + refcount_acquire(&nep->ne_ref); > + lockmgr(&mp->mnt_explock, LK_RELEASE, NULL); > + MNTEXP_LOCK(nep); > + if (nep->ne_pnfsnumfile !=3D NULL && > + nep->ne_pnfsnumfile !=3D PNFSD_STOPPED) { > + nep->ne_pnfsnumfile =3D PNFSD_STOP; > + wakeup(&mp->mnt_export); > + while (nep->ne_pnfsnumfile !=3D PNFSD_STOPPED) > + (void)msleep(&mp->mnt_explock, MNTEXP_MTX= (nep), > + PVFS, "pnfsw", hz); > + } > + MNTEXP_UNLOCK(nep); > + vfs_netexport_release(nep); > + } else > + lockmgr(&mp->mnt_explock, LK_RELEASE, NULL); > +} > The bitwise rather than logical OR here + KASSERT(nep->ne_pnfsnumfile =3D=3D PNFSD_START | + nep->ne_pnfsnumfile =3D=3D PNFSD_STOP, + ("nfsvno_pnfsreplenish: ne_pnfsnumfile not PNFSD_START/STOP")); causes a build failure for gcc due to -Werror=3Dparentheses. https://ci.freebsd.org/job/FreeBSD-main-amd64-gcc14_build/4006/consoleFull /workspace/src/sys/fs/nfsserver/nfs_nfsdport.c:2186:37: error: suggest parentheses around comparison in operand of '|' [-Werror=3Dparentheses] Ryan