From owner-svn-src-head@freebsd.org Sat Jun 16 19:45:07 2018 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48B571020AB0; Sat, 16 Jun 2018 19:45:07 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F323F6D25F; Sat, 16 Jun 2018 19:45:06 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BAE202CC2; Sat, 16 Jun 2018 19:45:06 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w5GJj60o058167; Sat, 16 Jun 2018 19:45:06 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w5GJj6V4058166; Sat, 16 Jun 2018 19:45:06 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201806161945.w5GJj6V4058166@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 16 Jun 2018 19:45:06 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r335263 - head/sys/fs/nfs X-SVN-Group: head X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: head/sys/fs/nfs X-SVN-Commit-Revision: 335263 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Jun 2018 19:45:07 -0000 Author: rmacklem Date: Sat Jun 16 19:45:06 2018 New Revision: 335263 URL: https://svnweb.freebsd.org/changeset/base/335263 Log: Fix NFSv4.1 client side handling of "soft,retrans=2" mounts. Normally "soft,retrans=2" cannot be safely used on NFSv4 mounts, since the RPC can fail and leave the open/lock state in an undefined state. Doing I/O on a pNFS DS is an exception to this, since no open/lock state is maintained on the DS server. It is useful to do "soft,retrans=2" connections to a DS when it is mirrored, so that the client can detect failure of the DS. As such, mounts from the MDS to the DSs should use these mount options when mirroring is enabled. However, the NFSv4.1 client still leaves the session in an undefined state when this happens. This patch fixes the problem by setting the session defunct, so it will no longer be used. The patch also sets "retries=2" on the connections done by the client to a DS, which is the internal equivalent of "soft,retrans=2". The client does not know if the server implements mirroring at connection time, but always doing this should be safe, since it will fall back on doing I/O via the MDS as a proxy when there is a failure doing an I/O RPC to the DS. This patch should not affect non-pNFS client mounts. MFC after: 2 weeks Modified: head/sys/fs/nfs/nfs_commonkrpc.c Modified: head/sys/fs/nfs/nfs_commonkrpc.c ============================================================================== --- head/sys/fs/nfs/nfs_commonkrpc.c Sat Jun 16 19:21:09 2018 (r335262) +++ head/sys/fs/nfs/nfs_commonkrpc.c Sat Jun 16 19:45:06 2018 (r335263) @@ -157,6 +157,9 @@ static int nfsv2_procid[NFS_V3NPROCS] = { /* * Initialize sockets and congestion for a new NFS connection. * We do not free the sockaddr if error. + * Which arguments are set to NULL indicate what kind of call it is. + * cred == NULL --> a call to connect to a pNFS DS + * nmp == NULL --> indicates an upcall to userland or a NFSv4 callback */ int newnfs_connect(struct nfsmount *nmp, struct nfssockreq *nrp, @@ -293,24 +296,38 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq retries = nmp->nm_retry; } else retries = INT_MAX; - /* cred == NULL for DS connects. */ - if (NFSHASNFSV4N(nmp) && cred != NULL) { - /* - * Make sure the nfscbd_pool doesn't get destroyed - * while doing this. - */ - NFSD_LOCK(); - if (nfs_numnfscbd > 0) { - nfs_numnfscbd++; - NFSD_UNLOCK(); - xprt = svc_vc_create_backchannel(nfscbd_pool); - CLNT_CONTROL(client, CLSET_BACKCHANNEL, xprt); + if (NFSHASNFSV4N(nmp)) { + if (cred != NULL) { + /* + * Make sure the nfscbd_pool doesn't get + * destroyed while doing this. + */ NFSD_LOCK(); - nfs_numnfscbd--; - if (nfs_numnfscbd == 0) - wakeup(&nfs_numnfscbd); + if (nfs_numnfscbd > 0) { + nfs_numnfscbd++; + NFSD_UNLOCK(); + xprt = svc_vc_create_backchannel( + nfscbd_pool); + CLNT_CONTROL(client, CLSET_BACKCHANNEL, + xprt); + NFSD_LOCK(); + nfs_numnfscbd--; + if (nfs_numnfscbd == 0) + wakeup(&nfs_numnfscbd); + } + NFSD_UNLOCK(); + } else { + /* + * cred == NULL for a DS connect. + * For connects to a DS, set a retry limit + * so that failed DSs will be detected. + * This is ok for NFSv4.1, since a DS does + * not maintain open/lock state and is the + * only case where using a "soft" mount is + * recommended for NFSv4. + */ + retries = 2; } - NFSD_UNLOCK(); } } else { /* @@ -762,6 +779,7 @@ tryagain: else stat = CLNT_CALL_MBUF(nrp->nr_client, &ext, procnum, nd->nd_mreq, &nd->nd_mrep, timo); + NFSCL_DEBUG(2, "clnt call=%d\n", stat); if (rep != NULL) { /* @@ -789,6 +807,60 @@ tryagain: error = EPROTONOSUPPORT; } else if (stat == RPC_INTR) { error = EINTR; + } else if (stat == RPC_CANTSEND || stat == RPC_CANTRECV || + stat == RPC_SYSTEMERROR) { + if ((nd->nd_flag & ND_NFSV41) != 0 && nmp != NULL && + nd->nd_procnum != NFSPROC_NULL) { + /* + * The nfsess_defunct field is protected by + * the NFSLOCKMNT()/nm_mtx lock and not the + * nfsess_mtx lock to simplify its handling, + * for the MDS session. This lock is also + * sufficient for nfsess_sessionid, since it + * never changes in the structure. + */ + NFSLOCKCLSTATE(); + NFSLOCKMNT(nmp); + /* The session must be marked defunct. */ + if (dssep == NULL) { + /* + * This is either an MDS proxy operation or + * a client mount with "soft,retrans=N" options. + * Mark the MDS session defunct and initiate + * recovery, as required. + */ + NFSCL_DEBUG(1, "Failed soft proxy RPC\n"); + sep = NFSMNT_MDSSESSION(nmp); + if (bcmp(sep->nfsess_sessionid, nd->nd_sequence, + NFSX_V4SESSIONID) == 0) { + /* Initiate recovery. */ + sep->nfsess_defunct = 1; + NFSCL_DEBUG(1, "Marked defunct\n"); + if (nmp->nm_clp != NULL) { + nmp->nm_clp->nfsc_flags |= + NFSCLFLAGS_RECOVER; + wakeup(nmp->nm_clp); + } + } + } else { + /* + * This is a client side DS RPC. Just mark + * the session defunct. A subsequent LayoutGet + * should get a new session. + */ + NFSCL_DEBUG(1, "Failed client DS RPC\n"); + if (bcmp(dssep->nfsess_sessionid, + nd->nd_sequence, NFSX_V4SESSIONID) == 0) { + /* Mark it defunct. */ + dssep->nfsess_defunct = 1; + NFSCL_DEBUG(1, "Marked defunct\n"); + } + } + NFSUNLOCKMNT(nmp); + NFSUNLOCKCLSTATE(); + } + NFSINCRGLOBAL(nfsstatsv1.rpcinvalid); + error = ENXIO; } else { NFSINCRGLOBAL(nfsstatsv1.rpcinvalid); error = EACCES;