From nobody Mon Oct 18 22:09:57 2021
X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 39772180AC81;
	Mon, 18 Oct 2021 22:09:58 +0000 (UTC)
	(envelope-from git@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (4096 bits) client-digest SHA256)
	(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4HY9z96z4Hz4fbl;
	Mon, 18 Oct 2021 22:09:57 +0000 (UTC)
	(envelope-from git@FreeBSD.org)
Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
	(Client did not present a certificate)
	by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CB7CD5B1;
	Mon, 18 Oct 2021 22:09:57 +0000 (UTC)
	(envelope-from git@FreeBSD.org)
Received: from gitrepo.freebsd.org ([127.0.1.44])
	by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 19IM9vm7009649;
	Mon, 18 Oct 2021 22:09:57 GMT
	(envelope-from git@gitrepo.freebsd.org)
Received: (from git@localhost)
	by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 19IM9vUl009648;
	Mon, 18 Oct 2021 22:09:57 GMT
	(envelope-from git)
Date: Mon, 18 Oct 2021 22:09:57 GMT
Message-Id: <202110182209.19IM9vUl009648@gitrepo.freebsd.org>
To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org,
        dev-commits-src-main@FreeBSD.org
From: Rick Macklem <rmacklem@FreeBSD.org>
Subject: git: 52dee2bc0355 - main - nfscl: Handle NFSv4.1/4.2 Close RPC NFSERR_DELAY replies better
List-Id: Commit messages for the main branch of the src repository <dev-commits-src-main.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main
List-Help: <mailto:dev-commits-src-main+help@freebsd.org>
List-Post: <mailto:dev-commits-src-main@freebsd.org>
List-Subscribe: <mailto:dev-commits-src-main+subscribe@freebsd.org>
List-Unsubscribe: <mailto:dev-commits-src-main+unsubscribe@freebsd.org>
Sender: owner-dev-commits-src-main@freebsd.org
X-BeenThere: dev-commits-src-main@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Git-Committer: rmacklem
X-Git-Repository: src
X-Git-Refname: refs/heads/main
X-Git-Reftype: branch
X-Git-Commit: 52dee2bc035545f7ae2b838d8a0449f65043cd8a
Auto-Submitted: auto-generated
X-ThisMailContainsUnwantedMimeParts: N

The branch main has been updated by rmacklem:

URL: https://cgit.FreeBSD.org/src/commit/?id=52dee2bc035545f7ae2b838d8a0449f65043cd8a

commit 52dee2bc035545f7ae2b838d8a0449f65043cd8a
Author:     Rick Macklem <rmacklem@FreeBSD.org>
AuthorDate: 2021-10-18 22:02:21 +0000
Commit:     Rick Macklem <rmacklem@FreeBSD.org>
CommitDate: 2021-10-18 22:05:34 +0000

    nfscl: Handle NFSv4.1/4.2 Close RPC NFSERR_DELAY replies better
    
    Without this patch, if a NFSv4.1/4.2 server replies NFSERR_DELAY to
    a Close operation, the client loops retrying the Close while holding
    a shared lock on the clientID.  This shared lock blocks returns of
    delegations, even though the server has issued a CB_RECALL to request
    the delegation return.
    
    This patch delays doing a retry of a Close that received a reply of
    NFSERR_DELAY until after the shared lock on the clientID is released,
    for NFSv4.1/4.2.  To fix this for NFSv4.0 would be very difficult and
    since the only known NFSv4 server to reply NFSERR_DELAY to Close only
    does NFSv4.1/4.2, this fix is hoped to be sufficient.
    
    This problem was detected during a recent IETF working group NFSv4
    testing event.
    
    MFC after:      2 week
---
 sys/fs/nfs/nfs_var.h            |  3 ++-
 sys/fs/nfsclient/nfs_clrpcops.c | 11 +++++++----
 sys/fs/nfsclient/nfs_clstate.c  | 23 +++++++++++++++++++++--
 3 files changed, 30 insertions(+), 7 deletions(-)

diff --git a/sys/fs/nfs/nfs_var.h b/sys/fs/nfs/nfs_var.h
index 3f0fbc535166..7f0ca990540d 100644
--- a/sys/fs/nfs/nfs_var.h
+++ b/sys/fs/nfs/nfs_var.h
@@ -604,7 +604,8 @@ void nfscl_dumpstate(struct nfsmount *, int, int, int, int);
 void nfscl_dupopen(vnode_t, int);
 int nfscl_getclose(vnode_t, struct nfsclclient **);
 int nfscl_doclose(vnode_t, struct nfsclclient **, NFSPROC_T *);
-void nfsrpc_doclose(struct nfsmount *, struct nfsclopen *, NFSPROC_T *);
+int nfsrpc_doclose(struct nfsmount *, struct nfsclopen *, NFSPROC_T *, bool,
+    bool);
 int nfscl_deleg(mount_t, struct nfsclclient *, u_int8_t *, int,
     struct ucred *, NFSPROC_T *, struct nfscldeleg **);
 void nfscl_lockinit(struct nfsv4lock *);
diff --git a/sys/fs/nfsclient/nfs_clrpcops.c b/sys/fs/nfsclient/nfs_clrpcops.c
index 6b6fdc03441f..fa0df1c37261 100644
--- a/sys/fs/nfsclient/nfs_clrpcops.c
+++ b/sys/fs/nfsclient/nfs_clrpcops.c
@@ -760,8 +760,9 @@ nfsrpc_close(vnode_t vp, int doclose, NFSPROC_T *p)
 /*
  * Close the open.
  */
-void
-nfsrpc_doclose(struct nfsmount *nmp, struct nfsclopen *op, NFSPROC_T *p)
+int
+nfsrpc_doclose(struct nfsmount *nmp, struct nfsclopen *op, NFSPROC_T *p,
+    bool loop_on_delayed, bool freeop)
 {
 	struct nfsrv_descript nfsd, *nd = &nfsd;
 	struct nfscllockowner *lp, *nlp;
@@ -840,7 +841,7 @@ nfsrpc_doclose(struct nfsmount *nmp, struct nfsclopen *op, NFSPROC_T *p)
 	nfscl_lockexcl(&op->nfso_own->nfsow_rwlock, NFSCLSTATEMUTEXPTR);
 	NFSUNLOCKCLSTATE();
 	do {
-		error = nfscl_tryclose(op, tcred, nmp, p, true);
+		error = nfscl_tryclose(op, tcred, nmp, p, loop_on_delayed);
 		if (error == NFSERR_GRACE)
 			(void) nfs_catnap(PZERO, error, "nfs_close");
 	} while (error == NFSERR_GRACE);
@@ -849,9 +850,11 @@ nfsrpc_doclose(struct nfsmount *nmp, struct nfsclopen *op, NFSPROC_T *p)
 
 	LIST_FOREACH_SAFE(lp, &op->nfso_lock, nfsl_list, nlp)
 		nfscl_freelockowner(lp, 0);
-	nfscl_freeopen(op, 0, true);
+	if (freeop && error != NFSERR_DELAY)
+		nfscl_freeopen(op, 0, true);
 	NFSUNLOCKCLSTATE();
 	NFSFREECRED(tcred);
+	return (error);
 }
 
 /*
diff --git a/sys/fs/nfsclient/nfs_clstate.c b/sys/fs/nfsclient/nfs_clstate.c
index 7ad0f1c1a364..36f8bbd6b51f 100644
--- a/sys/fs/nfsclient/nfs_clstate.c
+++ b/sys/fs/nfsclient/nfs_clstate.c
@@ -3327,8 +3327,10 @@ int
 nfscl_doclose(vnode_t vp, struct nfsclclient **clpp, NFSPROC_T *p)
 {
 	struct nfsclclient *clp;
+	struct nfsmount *nmp;
 	struct nfsclowner *owp, *nowp;
-	struct nfsclopen *op;
+	struct nfsclopen *op, *nop;
+	struct nfsclopenhead delayed;
 	struct nfscldeleg *dp;
 	struct nfsfh *nfhp;
 	struct nfsclrecalllayout *recallp;
@@ -3339,6 +3341,7 @@ nfscl_doclose(vnode_t vp, struct nfsclclient **clpp, NFSPROC_T *p)
 		return (error);
 	*clpp = clp;
 
+	nmp = VFSTONFS(vp->v_mount);
 	nfhp = VTONFS(vp)->n_fhp;
 	recallp = malloc(sizeof(*recallp), M_NFSLAYRECALL, M_WAITOK);
 	NFSLOCKCLSTATE();
@@ -3363,6 +3366,7 @@ nfscl_doclose(vnode_t vp, struct nfsclclient **clpp, NFSPROC_T *p)
 	nfscl_retoncloselayout(vp, clp, nfhp->nfh_fh, nfhp->nfh_len, &recallp);
 
 	/* Now process the opens against the server. */
+	LIST_INIT(&delayed);
 lookformore:
 	LIST_FOREACH(op, NFSCLOPENHASH(clp, nfhp->nfh_fh, nfhp->nfh_len),
 	    nfso_hash) {
@@ -3376,8 +3380,16 @@ lookformore:
 			     op->nfso_opencnt));
 #endif
 			NFSUNLOCKCLSTATE();
-			nfsrpc_doclose(VFSTONFS(vp->v_mount), op, p);
+			if (NFSHASNFSV4N(nmp))
+				error = nfsrpc_doclose(nmp, op, p, false, true);
+			else
+				error = nfsrpc_doclose(nmp, op, p, true, true);
 			NFSLOCKCLSTATE();
+			if (error == NFSERR_DELAY) {
+				nfscl_unlinkopen(op);
+				op->nfso_own = NULL;
+				LIST_INSERT_HEAD(&delayed, op, nfso_list);
+			}
 			goto lookformore;
 		}
 	}
@@ -3388,6 +3400,13 @@ lookformore:
 	 * used by the function, but calling free() with a NULL pointer is ok.
 	 */
 	free(recallp, M_NFSLAYRECALL);
+
+	/* Now, loop retrying the delayed closes. */
+	LIST_FOREACH_SAFE(op, &delayed, nfso_list, nop) {
+		nfsrpc_doclose(nmp, op, p, true, false);
+		LIST_REMOVE(op, nfso_list);
+		nfscl_freeopen(op, 0, false);
+	}
 	return (0);
 }