From owner-dev-commits-src-all@freebsd.org Wed Jun 9 15:04:10 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9B45464EE55; Wed, 9 Jun 2021 15:04:10 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4G0VkL3xRrz4mqn; Wed, 9 Jun 2021 15:04:10 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 6D9BF1E536; Wed, 9 Jun 2021 15:04:10 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 159F4AXH051672; Wed, 9 Jun 2021 15:04:10 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 159F4AdN051671; Wed, 9 Jun 2021 15:04:10 GMT (envelope-from git) Date: Wed, 9 Jun 2021 15:04:10 GMT Message-Id: <202106091504.159F4AdN051671@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Rick Macklem Subject: git: 5e5ca4c8fc53 - main - nfscl: Add a "has acquired a delegation" flag for delegations MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: rmacklem X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 5e5ca4c8fc53d31bf71e182e08c4ccd8290428c7 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Jun 2021 15:04:10 -0000 The branch main has been updated by rmacklem: URL: https://cgit.FreeBSD.org/src/commit/?id=5e5ca4c8fc53d31bf71e182e08c4ccd8290428c7 commit 5e5ca4c8fc53d31bf71e182e08c4ccd8290428c7 Author: Rick Macklem AuthorDate: 2021-06-09 15:00:43 +0000 Commit: Rick Macklem CommitDate: 2021-06-09 15:00:43 +0000 nfscl: Add a "has acquired a delegation" flag for delegations A problem was reported via email, where a large (130000+) accumulation of NFSv4 opens on an NFSv4 mount caused significant lock contention on the mutex used to protect the client mount's open/lock state. Although the root cause for the accumulation of opens was not resolved, it is obvious that the NFSv4 client is not designed to handle 100000+ opens efficiently. For a common case where delegations are not being issued by the NFSv4 server, the code acquires the mutex lock for open/lock state, finds the delegation list empty and just unlocks the mutex and returns. This patch adds an NFS mount point flag that is set when a delegation is issued for the mount. Then the patched code checks for this flag before acquiring the open/lock mutex, avoiding the need to acquire the lock for the case where delegations are not being issued by the NFSv4 server. This change appears to be performance neutral for a small number of opens, but should reduce lock contention for a large number of opens for the common case where server is not issuing delegations. This commit should not affect the high level semantics of delegation handling. MFC after: 2 weeks --- sys/fs/nfsclient/nfs_clstate.c | 44 +++++++++++++++++++++++++++++++++++++++--- sys/fs/nfsclient/nfsmount.h | 1 + 2 files changed, 42 insertions(+), 3 deletions(-) diff --git a/sys/fs/nfsclient/nfs_clstate.c b/sys/fs/nfsclient/nfs_clstate.c index 5e4ac2ae9d4d..0659e77289e9 100644 --- a/sys/fs/nfsclient/nfs_clstate.c +++ b/sys/fs/nfsclient/nfs_clstate.c @@ -438,20 +438,32 @@ nfscl_deleg(mount_t mp, struct nfsclclient *clp, u_int8_t *nfhp, int fhlen, struct ucred *cred, NFSPROC_T *p, struct nfscldeleg **dpp) { struct nfscldeleg *dp = *dpp, *tdp; + struct nfsmount *nmp; + KASSERT(mp != NULL, ("nfscl_deleg: mp NULL")); + nmp = VFSTONFS(mp); /* * First, if we have received a Read delegation for a file on a * read/write file system, just return it, because they aren't * useful, imho. */ - if (mp != NULL && dp != NULL && !NFSMNT_RDONLY(mp) && + if (dp != NULL && !NFSMNT_RDONLY(mp) && (dp->nfsdl_flags & NFSCLDL_READ)) { - (void) nfscl_trydelegreturn(dp, cred, VFSTONFS(mp), p); + nfscl_trydelegreturn(dp, cred, nmp, p); free(dp, M_NFSCLDELEG); *dpp = NULL; return (0); } + /* + * Since a delegation might be added to the mount, + * set NFSMNTP_DELEGISSUED now. If a delegation already + * exagain ists, setting this flag is harmless. + */ + NFSLOCKMNT(nmp); + nmp->nm_privflag |= NFSMNTP_DELEGISSUED; + NFSUNLOCKMNT(nmp); + /* Look for the correct deleg, based upon FH */ NFSLOCKCLSTATE(); tdp = nfscl_finddeleg(clp, nfhp, fhlen); @@ -3356,12 +3368,20 @@ nfscl_delegreturnvp(vnode_t vp, NFSPROC_T *p) struct nfscldeleg *dp; struct ucred *cred; struct nfsnode *np; + struct nfsmount *nmp; + nmp = VFSTONFS(vp->v_mount); + NFSLOCKMNT(nmp); + if ((nmp->nm_privflag & NFSMNTP_DELEGISSUED) == 0) { + NFSUNLOCKMNT(nmp); + return; + } + NFSUNLOCKMNT(nmp); np = VTONFS(vp); cred = newnfs_getcred(); dp = NULL; NFSLOCKCLSTATE(); - clp = VFSTONFS(vp->v_mount)->nm_clp; + clp = nmp->nm_clp; if (clp != NULL) dp = nfscl_finddeleg(clp, np->n_fhp->nfh_fh, np->n_fhp->nfh_len); @@ -4500,6 +4520,12 @@ nfscl_nodeleg(vnode_t vp, int writedeleg) nmp = VFSTONFS(vp->v_mount); if (!NFSHASNFSV4(nmp)) return (1); + NFSLOCKMNT(nmp); + if ((nmp->nm_privflag & NFSMNTP_DELEGISSUED) == 0) { + NFSUNLOCKMNT(nmp); + return (1); + } + NFSUNLOCKMNT(nmp); NFSLOCKCLSTATE(); clp = nfscl_findcl(nmp); if (clp == NULL) { @@ -4856,6 +4882,12 @@ nfscl_delegmodtime(vnode_t vp) nmp = VFSTONFS(vp->v_mount); if (!NFSHASNFSV4(nmp)) return; + NFSLOCKMNT(nmp); + if ((nmp->nm_privflag & NFSMNTP_DELEGISSUED) == 0) { + NFSUNLOCKMNT(nmp); + return; + } + NFSUNLOCKMNT(nmp); NFSLOCKCLSTATE(); clp = nfscl_findcl(nmp); if (clp == NULL) { @@ -4885,6 +4917,12 @@ nfscl_deleggetmodtime(vnode_t vp, struct timespec *mtime) nmp = VFSTONFS(vp->v_mount); if (!NFSHASNFSV4(nmp)) return; + NFSLOCKMNT(nmp); + if ((nmp->nm_privflag & NFSMNTP_DELEGISSUED) == 0) { + NFSUNLOCKMNT(nmp); + return; + } + NFSUNLOCKMNT(nmp); NFSLOCKCLSTATE(); clp = nfscl_findcl(nmp); if (clp == NULL) { diff --git a/sys/fs/nfsclient/nfsmount.h b/sys/fs/nfsclient/nfsmount.h index 57adcd8f2fca..f8ea8c9a1418 100644 --- a/sys/fs/nfsclient/nfsmount.h +++ b/sys/fs/nfsclient/nfsmount.h @@ -115,6 +115,7 @@ struct nfsmount { #define NFSMNTP_NOXATTR 0x00000080 #define NFSMNTP_NOADVISE 0x00000100 #define NFSMNTP_NOALLOCATE 0x00000200 +#define NFSMNTP_DELEGISSUED 0x00000400 /* New mount flags only used by the kernel via nmount(2). */ #define NFSMNT_TLS 0x00000001