From owner-svn-src-projects@freebsd.org Mon Jul 9 00:38:40 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F57110269A0 for ; Mon, 9 Jul 2018 00:38:40 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F42B75D1C; Mon, 9 Jul 2018 00:38:40 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 308255BC8; Mon, 9 Jul 2018 00:38:40 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w690ceLW014976; Mon, 9 Jul 2018 00:38:40 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w690cd2W014972; Mon, 9 Jul 2018 00:38:39 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807090038.w690cd2W014972@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Mon, 9 Jul 2018 00:38:39 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336116 - in projects/pnfs-planb-server/sys/fs: nfs nfsserver X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/pnfs-planb-server/sys/fs: nfs nfsserver X-SVN-Commit-Revision: 336116 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jul 2018 00:38:41 -0000 Author: rmacklem Date: Mon Jul 9 00:38:39 2018 New Revision: 336116 URL: https://svnweb.freebsd.org/changeset/base/336116 Log: Modify the pNFS server kernel code so that a "force" option can be implemented for pnfsdskill(8). pnfsdskill(8) will fail if there isn't a mirror for the DS that is being disabled. However, if the DS has failed, it may be necessary to do a pnfsdskill on the DS even when there is no valid mirror, so that the nfsd threads can be killed. This patch modifies the kernel to add a PNFSDOP_FORCEDELDS option that pnfsdkill(8) can use to force the disable. Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs.h projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdport.c projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdstate.c Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs.h ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfs/nfs.h Sun Jul 8 23:22:04 2018 (r336115) +++ projects/pnfs-planb-server/sys/fs/nfs/nfs.h Mon Jul 9 00:38:39 2018 (r336116) @@ -207,6 +207,7 @@ struct nfsd_pnfsd_args { #define PNFSDOP_DELDSSERVER 1 #define PNFSDOP_COPYMR 2 +#define PNFSDOP_FORCEDELDS 3 /* Old version. */ struct nfsd_nfsd_oargs { Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h Sun Jul 8 23:22:04 2018 (r336115) +++ projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h Mon Jul 9 00:38:39 2018 (r336116) @@ -159,8 +159,8 @@ int nfsrv_getdevinfo(char *, int, uint32_t *, uint32_t void nfsrv_freeonedevid(struct nfsdevice *); void nfsrv_freealllayoutsanddevids(void); void nfsrv_freefilelayouts(fhandle_t *); -int nfsrv_deldsserver(char *, NFSPROC_T *); -struct nfsdevice *nfsrv_deldsnmp(struct nfsmount *, NFSPROC_T *); +int nfsrv_deldsserver(int, char *, NFSPROC_T *); +struct nfsdevice *nfsrv_deldsnmp(int, struct nfsmount *, NFSPROC_T *); int nfsrv_createdevids(struct nfsd_nfsd_args *, NFSPROC_T *); int nfsrv_checkdsattr(struct nfsrv_descript *, vnode_t, NFSPROC_T *); int nfsrv_copymr(vnode_t, vnode_t, vnode_t, struct nfsdevice *, Modified: projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdport.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdport.c Sun Jul 8 23:22:04 2018 (r336115) +++ projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdport.c Mon Jul 9 00:38:39 2018 (r336116) @@ -3433,12 +3433,13 @@ nfssvc_nfsd(struct thread *td, struct nfssvc_args *uap free(nfsdarg.mdspath, M_TEMP); } else if (uap->flag & NFSSVC_PNFSDS) { error = copyin(uap->argp, &pnfsdarg, sizeof(pnfsdarg)); - if (error == 0 && pnfsdarg.op == PNFSDOP_DELDSSERVER) { + if (error == 0 && (pnfsdarg.op == PNFSDOP_DELDSSERVER || + pnfsdarg.op == PNFSDOP_FORCEDELDS)) { cp = malloc(PATH_MAX + 1, M_TEMP, M_WAITOK); error = copyinstr(pnfsdarg.dspath, cp, PATH_MAX + 1, NULL); if (error == 0) - error = nfsrv_deldsserver(cp, td); + error = nfsrv_deldsserver(pnfsdarg.op, cp, td); free(cp, M_TEMP); } else if (error == 0 && pnfsdarg.op == PNFSDOP_COPYMR) { cp = malloc(PATH_MAX + 1, M_TEMP, M_WAITOK); @@ -4014,7 +4015,7 @@ nfsrv_pnfscreate(struct vnode *vp, struct vattr *vap, NFSMNTP_CANCELRPCS)) == 0) { nmp->nm_privflag |= NFSMNTP_CANCELRPCS; NFSUNLOCKMNT(nmp); - ds = nfsrv_deldsnmp(nmp, p); + ds = nfsrv_deldsnmp(PNFSDOP_DELDSSERVER, nmp, p); NFSD_DEBUG(4, "dscreatfail fail=%d ds=%p\n", failpos, ds); if (ds != NULL) @@ -4248,7 +4249,7 @@ nfsrv_pnfsremove(struct vnode **dvp, int mirrorcnt, ch NFSMNTP_CANCELRPCS)) == 0) { nmp->nm_privflag |= NFSMNTP_CANCELRPCS; NFSUNLOCKMNT(nmp); - ds = nfsrv_deldsnmp(nmp, p); + ds = nfsrv_deldsnmp(PNFSDOP_DELDSSERVER, nmp, p); NFSD_DEBUG(4, "dsremovefail fail=%d ds=%p\n", failpos, ds); if (ds != NULL) @@ -4463,9 +4464,10 @@ tryagain: NFSMNTP_CANCELRPCS)) == 0) { failnmp->nm_privflag |= NFSMNTP_CANCELRPCS; NFSUNLOCKMNT(failnmp); - ds = nfsrv_deldsnmp(failnmp, p); - NFSD_DEBUG(4, "dsldsnmp fail=%d ds=%p\n", - failpos, ds); + ds = nfsrv_deldsnmp(PNFSDOP_DELDSSERVER, + failnmp, p); + NFSD_DEBUG(4, "nfsrv_proxyds: deldsnmp fail=%d " + "ds=%p\n", failpos, ds); if (ds != NULL) nfsrv_killrpcs(failnmp); NFSLOCKMNT(failnmp); Modified: projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdstate.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdstate.c Sun Jul 8 23:22:04 2018 (r336115) +++ projects/pnfs-planb-server/sys/fs/nfsserver/nfs_nfsdstate.c Mon Jul 9 00:38:39 2018 (r336116) @@ -7482,7 +7482,7 @@ out: * Look up the mount path for the DS server and delete it. */ int -nfsrv_deldsserver(char *dspathp, NFSPROC_T *p) +nfsrv_deldsserver(int op, char *dspathp, NFSPROC_T *p) { struct mount *mp; struct nfsmount *nmp; @@ -7524,7 +7524,7 @@ nfsrv_deldsserver(char *dspathp, NFSPROC_T *p) mtx_unlock(&mountlist_mtx); if (nmp != NULL) { - ds = nfsrv_deldsnmp(nmp, p); + ds = nfsrv_deldsnmp(op, nmp, p); NFSD_DEBUG(4, "deldsnmp=%p\n", ds); if (ds != NULL) { nfsrv_killrpcs(nmp); @@ -7544,20 +7544,26 @@ nfsrv_deldsserver(char *dspathp, NFSPROC_T *p) * Search for and remove a DS entry which matches the "nmp" argument. * The nfsdevice structure pointer is returned so that the caller can * free it via nfsrv_freeonedevid(). + * For the forced case, do not try to do LayoutRecalls, since the server + * must be shut down now anyhow. */ struct nfsdevice * -nfsrv_deldsnmp(struct nfsmount *nmp, NFSPROC_T *p) +nfsrv_deldsnmp(int op, struct nfsmount *nmp, NFSPROC_T *p) { struct nfsdevice *fndds; NFSD_DEBUG(4, "deldsdvp\n"); NFSDDSLOCK(); - fndds = nfsrv_findmirroredds(nmp); + if (op == PNFSDOP_FORCEDELDS) + fndds = nfsv4_findmirror(nmp); + else + fndds = nfsrv_findmirroredds(nmp); if (fndds != NULL) nfsrv_deleteds(fndds); NFSDDSUNLOCK(); if (fndds != NULL) { - nfsrv_flexmirrordel(fndds->nfsdev_deviceid, p); + if (op != PNFSDOP_FORCEDELDS) + nfsrv_flexmirrordel(fndds->nfsdev_deviceid, p); printf("pNFS server: mirror %s failed\n", fndds->nfsdev_host); } return (fndds); From owner-svn-src-projects@freebsd.org Mon Jul 9 00:47:37 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59D121027A95 for ; Mon, 9 Jul 2018 00:47:37 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A45B76332; Mon, 9 Jul 2018 00:47:37 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DA8755D58; Mon, 9 Jul 2018 00:47:36 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w690laFL020029; Mon, 9 Jul 2018 00:47:36 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w690laLX020028; Mon, 9 Jul 2018 00:47:36 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807090047.w690laLX020028@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Mon, 9 Jul 2018 00:47:36 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336117 - projects/pnfs-planb-server/usr.bin/pnfsdskill X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/pnfs-planb-server/usr.bin/pnfsdskill X-SVN-Commit-Revision: 336117 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jul 2018 00:47:37 -0000 Author: rmacklem Date: Mon Jul 9 00:47:36 2018 New Revision: 336117 URL: https://svnweb.freebsd.org/changeset/base/336117 Log: Add a -f/--force option to pnfsdskill, so that it disable a DS even if there is no valid mirror for it. This should only be used to facilitate shutting down of the nfsd when DSs have failed. Modified: projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.c Modified: projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.c ============================================================================== --- projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.c Mon Jul 9 00:38:39 2018 (r336116) +++ projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.c Mon Jul 9 00:47:36 2018 (r336117) @@ -30,6 +30,7 @@ __FBSDID("$FreeBSD$"); #include +#include #include #include #include @@ -44,6 +45,11 @@ __FBSDID("$FreeBSD$"); static void usage(void); +static struct option longopts[] = { + { "force", no_argument, NULL, 'f' }, + { NULL, 0, NULL, 0 } +}; + /* * This program disables use of a DS mirror. The "dspath" command line * argument must be an exact match for the mounted-on path of the DS. @@ -54,23 +60,39 @@ int main(int argc, char *argv[]) { struct nfsd_pnfsd_args pnfsdarg; + int ch, force; - if (argc != 2) - usage(); if (geteuid() != 0) errx(1, "Must be run as root/su"); + force = 0; + while ((ch = getopt_long(argc, argv, "f", longopts, NULL)) != -1) { + switch (ch) { + case 'f': + force = 1; + break; + default: + usage(); + } + } + argc -= optind; + argv += optind; + if (argc != 1) + usage(); - pnfsdarg.op = PNFSDOP_DELDSSERVER; - pnfsdarg.dspath = argv[1]; + if (force != 0) + pnfsdarg.op = PNFSDOP_FORCEDELDS; + else + pnfsdarg.op = PNFSDOP_DELDSSERVER; + pnfsdarg.dspath = *argv; if (nfssvc(NFSSVC_PNFSDS, &pnfsdarg) < 0) - err(1, "Can't kill %s", argv[1]); + err(1, "Can't kill %s", *argv); } static void usage(void) { - fprintf(stderr, "pnfsdsfile [filepath]\n"); + fprintf(stderr, "pnfsdsfile [-f] mounted-on-DS-dir\n"); exit(1); } From owner-svn-src-projects@freebsd.org Mon Jul 9 21:29:24 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4191A1032552 for ; Mon, 9 Jul 2018 21:29:24 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E68FF88AD4; Mon, 9 Jul 2018 21:29:23 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id C2F7A1A754; Mon, 9 Jul 2018 21:29:23 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w69LTNMg071535; Mon, 9 Jul 2018 21:29:23 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w69LTN93071534; Mon, 9 Jul 2018 21:29:23 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807092129.w69LTN93071534@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Mon, 9 Jul 2018 21:29:23 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336147 - projects/pnfs-planb-server/usr.bin/pnfsdskill X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/pnfs-planb-server/usr.bin/pnfsdskill X-SVN-Commit-Revision: 336147 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jul 2018 21:29:24 -0000 Author: rmacklem Date: Mon Jul 9 21:29:23 2018 New Revision: 336147 URL: https://svnweb.freebsd.org/changeset/base/336147 Log: Document the "-f" flag for pnfsdskill(1). Modified: projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.1 Modified: projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.1 ============================================================================== --- projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.1 Mon Jul 9 21:13:21 2018 (r336146) +++ projects/pnfs-planb-server/usr.bin/pnfsdskill/pnfsdskill.1 Mon Jul 9 21:29:23 2018 (r336147) @@ -23,7 +23,7 @@ .\" .\" $FreeBSD$ .\" -.Dd March 9, 2018 +.Dd July 9, 2018 .Dt PNFSDSKILL 1 .Os .Sh NAME @@ -32,6 +32,7 @@ disables a pNFS data storage server (DS) .Sh SYNOPSIS .Nm +.Op Fl f .Ar mounted-on-DS-dir .Sh DESCRIPTION The @@ -42,7 +43,19 @@ may use this command on the metadata server (MDS) to d This command must be used on the MDS and the .Ar mounted-on-DS-dir must be the exact pathname used when mounting the DS on the MDS. +Normally this command will fail if there is no valid mirror for the DS +available. .Pp +The following option is available: +.Bl -tag -width Ds +.It Fl f +This option forces the DS to be disabled even if there is no valid mirror +for the DS available. +It should only be used to allow the +.Xr nfsd 8 +threads to be terminated. +.El +.Pp If this command fails with .Dq Device not configured (ENXIO), it probably @@ -61,4 +74,5 @@ operational DSs is at least as many as the level of mi .Sh HISTORY The .Nm -command appeared in FreeBSD12. +command first appeared in +.Fx 12.0 . From owner-svn-src-projects@freebsd.org Thu Jul 12 20:28:54 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71D981041F3E for ; Thu, 12 Jul 2018 20:28:54 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2511A7E329; Thu, 12 Jul 2018 20:28:54 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0585D26440; Thu, 12 Jul 2018 20:28:54 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6CKSrQ8064468; Thu, 12 Jul 2018 20:28:53 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6CKSrhM064463; Thu, 12 Jul 2018 20:28:53 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807122028.w6CKSrhM064463@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Thu, 12 Jul 2018 20:28:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336228 - in projects/pnfs-planb-server/sys/fs: nfs nfsclient X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/pnfs-planb-server/sys/fs: nfs nfsclient X-SVN-Commit-Revision: 336228 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jul 2018 20:28:54 -0000 Author: rmacklem Date: Thu Jul 12 20:28:52 2018 New Revision: 336228 URL: https://svnweb.freebsd.org/changeset/base/336228 Log: Fix the NFSv4.1 pNFS client for the case of mirrored DSs on separate machines. My test environment has all the DSs colocated on the same machine, so I did not spot this bug until recently. The client was using the TCP connection for the first DS when doing writes/commits to subsequent mirrored DSs. This worked fine for my test setup, but would have failed when the DSs are on separate machines. This patch fixes the code so that it uses the correct TCP connection for the writes/commits to each of the mirrored DSs. Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h projects/pnfs-planb-server/sys/fs/nfs/nfsclstate.h projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h Thu Jul 12 19:02:59 2018 (r336227) +++ projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h Thu Jul 12 20:28:52 2018 (r336228) @@ -607,7 +607,7 @@ void nfscl_rellayout(struct nfscllayout *, int); struct nfscldevinfo *nfscl_getdevinfo(struct nfsclclient *, uint8_t *, struct nfscldevinfo *); void nfscl_reldevinfo(struct nfscldevinfo *); -int nfscl_adddevinfo(struct nfsmount *, struct nfscldevinfo *, +int nfscl_adddevinfo(struct nfsmount *, struct nfscldevinfo *, int, struct nfsclflayout *); void nfscl_freelayout(struct nfscllayout *); void nfscl_freeflayout(struct nfsclflayout *); Modified: projects/pnfs-planb-server/sys/fs/nfs/nfsclstate.h ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfs/nfsclstate.h Thu Jul 12 19:02:59 2018 (r336227) +++ projects/pnfs-planb-server/sys/fs/nfs/nfsclstate.h Thu Jul 12 20:28:52 2018 (r336228) @@ -269,6 +269,7 @@ struct nfscllayout { */ struct nfsffm { nfsv4stateid_t st; + struct nfscldevinfo *devp; char dev[NFSX_V4DEVICEID]; uint32_t eff; uid_t user; @@ -290,7 +291,6 @@ struct nfsclflayout { uint64_t nfsfl_off; uint64_t nfsfl_end; uint32_t nfsfl_iomode; - struct nfscldevinfo *nfsfl_devp; uint16_t nfsfl_flags; union { struct { @@ -299,6 +299,7 @@ struct nfsclflayout { uint32_t stripe1; uint8_t dev[NFSX_V4DEVICEID]; uint16_t fhcnt; + struct nfscldevinfo *devp; } fl; struct { uint64_t stripeunit; @@ -317,6 +318,7 @@ struct nfsclflayout { #define nfsfl_stripe1 nfsfl_un.fl.stripe1 #define nfsfl_dev nfsfl_un.fl.dev #define nfsfl_fhcnt nfsfl_un.fl.fhcnt +#define nfsfl_devp nfsfl_un.fl.devp #define nfsfl_stripeunit nfsfl_un.ff.stripeunit #define nfsfl_fflags nfsfl_un.ff.fflags #define nfsfl_statshint nfsfl_un.ff.statshint Modified: projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c Thu Jul 12 19:02:59 2018 (r336227) +++ projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c Thu Jul 12 20:28:52 2018 (r336228) @@ -5722,12 +5722,15 @@ nfscl_doiods(vnode_t vp, struct uio *uiop, int *iomode } } for (i = firstmirror; i < mirrorcnt && error == 0; i++){ - if ((layp->nfsly_flags & NFSLY_FLEXFILE) != 0) + if ((layp->nfsly_flags & NFSLY_FLEXFILE) != 0) { dev = rflp->nfsfl_ffm[i].dev; - else + dip = nfscl_getdevinfo(nmp->nm_clp, dev, + rflp->nfsfl_ffm[i].devp); + } else { dev = rflp->nfsfl_dev; - dip = nfscl_getdevinfo(nmp->nm_clp, dev, - rflp->nfsfl_devp); + dip = nfscl_getdevinfo(nmp->nm_clp, dev, + rflp->nfsfl_devp); + } if (dip != NULL) { if ((rflp->nfsfl_flags & NFSFL_FLEXFILE) != 0) @@ -6828,7 +6831,8 @@ nfsrv_parselayoutget(struct nfsrv_descript *nd, nfsv4s sizeof(struct nfsffm), M_NFSFLAYOUT, M_WAITOK); flp->nfsfl_flags = NFSFL_FLEXFILE; flp->nfsfl_mirrorcnt = mirrorcnt; - flp->nfsfl_devp = NULL; + for (j = 0; j < mirrorcnt; j++) + flp->nfsfl_ffm[j].devp = NULL; flp->nfsfl_off = off; if (flp->nfsfl_off + retlen < flp->nfsfl_off) flp->nfsfl_end = UINT64_MAX - flp->nfsfl_off; @@ -7588,6 +7592,7 @@ nfsrpc_layoutgetres(struct nfsmount *nmp, vnode_t vp, struct nfsclflayout *tflp; struct nfscldevinfo *dip; uint8_t *dev; + int i, mirrorcnt; if (laystat == NFSERR_UNKNLAYOUTTYPE) { NFSLOCKMNT(nmp); @@ -7604,25 +7609,35 @@ nfsrpc_layoutgetres(struct nfsmount *nmp, vnode_t vp, if (laystat == 0) { NFSCL_DEBUG(4, "nfsrpc_layoutgetres at FOREACH\n"); LIST_FOREACH(tflp, flhp, nfsfl_list) { - laystat = nfscl_adddevinfo(nmp, NULL, tflp); - NFSCL_DEBUG(4, "aft adddev=%d\n", laystat); - if (laystat != 0) { - if (layouttype == NFSLAYOUT_FLEXFILE) - dev = tflp->nfsfl_ffm[0].dev; - else - dev = tflp->nfsfl_dev; - laystat = nfsrpc_getdeviceinfo(nmp, dev, - layouttype, notifybit, &dip, cred, p); - NFSCL_DEBUG(4, "aft nfsrpc_gdi=%d\n", - laystat); - if (laystat != 0) - break; - laystat = nfscl_adddevinfo(nmp, dip, tflp); - if (laystat != 0) - printf("getlayout: cannot add\n"); + if (layouttype == NFSLAYOUT_FLEXFILE) + mirrorcnt = tflp->nfsfl_mirrorcnt; + else + mirrorcnt = 1; + for (i = 0; i < mirrorcnt; i++) { + laystat = nfscl_adddevinfo(nmp, NULL, i, tflp); + NFSCL_DEBUG(4, "aft adddev=%d\n", laystat); + if (laystat != 0) { + if (layouttype == NFSLAYOUT_FLEXFILE) + dev = tflp->nfsfl_ffm[i].dev; + else + dev = tflp->nfsfl_dev; + laystat = nfsrpc_getdeviceinfo(nmp, dev, + layouttype, notifybit, &dip, cred, + p); + NFSCL_DEBUG(4, "aft nfsrpc_gdi=%d\n", + laystat); + if (laystat != 0) + goto out; + laystat = nfscl_adddevinfo(nmp, dip, i, + tflp); + if (laystat != 0) + printf("nfsrpc_layoutgetresout" + ": cannot add\n"); + } } } } +out: if (laystat == 0) { /* * nfscl_layout() always returns with the nfsly_lock Modified: projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c Thu Jul 12 19:02:59 2018 (r336227) +++ projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c Thu Jul 12 20:28:52 2018 (r336228) @@ -5026,8 +5026,8 @@ nfscl_cancelreqs(struct nfsclds *dsp) NFSUNLOCKDS(dsp); CLNT_CLOSE(cl); /* - * This 1sec sleep is done to give the MDS time to recall - * layouts before this client attempts further I/O. + * This 1sec sleep is done to reduce the number of reconnect + * attempts made on the DS while it has failed. */ tsleep(&non_event, PVFS, "ndscls", hz); return; @@ -5152,7 +5152,7 @@ nfscl_mergeflayouts(struct nfsclflayouthead *fhlp, * This function consumes the structure pointed at by dip, if not NULL. */ APPLESTATIC int -nfscl_adddevinfo(struct nfsmount *nmp, struct nfscldevinfo *dip, +nfscl_adddevinfo(struct nfsmount *nmp, struct nfscldevinfo *dip, int ind, struct nfsclflayout *flp) { struct nfsclclient *clp; @@ -5170,11 +5170,14 @@ nfscl_adddevinfo(struct nfsmount *nmp, struct nfscldev if ((flp->nfsfl_flags & NFSFL_FILE) != 0) dev = flp->nfsfl_dev; else - dev = flp->nfsfl_ffm[0].dev; + dev = flp->nfsfl_ffm[ind].dev; tdip = nfscl_finddevinfo(clp, dev); if (tdip != NULL) { tdip->nfsdi_layoutrefs++; - flp->nfsfl_devp = tdip; + if ((flp->nfsfl_flags & NFSFL_FILE) != 0) + flp->nfsfl_devp = tdip; + else + flp->nfsfl_ffm[ind].devp = tdip; nfscl_reldevinfo_locked(tdip); NFSUNLOCKCLSTATE(); if (dip != NULL) @@ -5184,7 +5187,10 @@ nfscl_adddevinfo(struct nfsmount *nmp, struct nfscldev if (dip != NULL) { LIST_INSERT_HEAD(&clp->nfsc_devinfo, dip, nfsdi_list); dip->nfsdi_layoutrefs = 1; - flp->nfsfl_devp = dip; + if ((flp->nfsfl_flags & NFSFL_FILE) != 0) + flp->nfsfl_devp = dip; + else + flp->nfsfl_ffm[ind].devp = dip; } NFSUNLOCKCLSTATE(); if (dip == NULL) @@ -5225,15 +5231,19 @@ nfscl_freeflayout(struct nfsclflayout *flp) { int i, j; - if ((flp->nfsfl_flags & NFSFL_FILE) != 0) + if ((flp->nfsfl_flags & NFSFL_FILE) != 0) { for (i = 0; i < flp->nfsfl_fhcnt; i++) free(flp->nfsfl_fh[i], M_NFSFH); + if (flp->nfsfl_devp != NULL) + flp->nfsfl_devp->nfsdi_layoutrefs--; + } if ((flp->nfsfl_flags & NFSFL_FLEXFILE) != 0) - for (i = 0; i < flp->nfsfl_mirrorcnt; i++) + for (i = 0; i < flp->nfsfl_mirrorcnt; i++) { for (j = 0; j < flp->nfsfl_ffm[i].fhcnt; j++) free(flp->nfsfl_ffm[i].fh[j], M_NFSFH); - if (flp->nfsfl_devp != NULL) - flp->nfsfl_devp->nfsdi_layoutrefs--; + if (flp->nfsfl_ffm[i].devp != NULL) + flp->nfsfl_ffm[i].devp->nfsdi_layoutrefs--; + } free(flp, M_NFSFLAYOUT); } From owner-svn-src-projects@freebsd.org Sat Jul 14 13:39:23 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F1AB7102B5A3 for ; Sat, 14 Jul 2018 13:39:22 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A11E07A1C3; Sat, 14 Jul 2018 13:39:22 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 82B8E1FAE1; Sat, 14 Jul 2018 13:39:22 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6EDdMq8034722; Sat, 14 Jul 2018 13:39:22 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6EDdMD0034720; Sat, 14 Jul 2018 13:39:22 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807141339.w6EDdMD0034720@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 14 Jul 2018 13:39:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336276 - projects/pnfs-planb-server/sys/rpc X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/pnfs-planb-server/sys/rpc X-SVN-Commit-Revision: 336276 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jul 2018 13:39:23 -0000 Author: rmacklem Date: Sat Jul 14 13:39:21 2018 New Revision: 336276 URL: https://svnweb.freebsd.org/changeset/base/336276 Log: Add a timeout to the msleep() waiting for a socket connection for the pNFS client. Without this patch, the client TCP connection code can wait a long time for a connection attempt to fail when a DS is not operational. This patch adds a timeout for this msleep() if the CLSET_TIMEOUT has been set for the connection. This is only set for TCP connections to DSs. Modified: projects/pnfs-planb-server/sys/rpc/clnt_rc.c projects/pnfs-planb-server/sys/rpc/clnt_vc.c Modified: projects/pnfs-planb-server/sys/rpc/clnt_rc.c ============================================================================== --- projects/pnfs-planb-server/sys/rpc/clnt_rc.c Sat Jul 14 10:14:59 2018 (r336275) +++ projects/pnfs-planb-server/sys/rpc/clnt_rc.c Sat Jul 14 13:39:21 2018 (r336276) @@ -175,7 +175,7 @@ clnt_reconnect_connect(CLIENT *cl) (struct sockaddr *) &rc->rc_addr, rc->rc_prog, rc->rc_vers, rc->rc_sendsz, rc->rc_recvsz); else { - if (rc->rc_timeout.tv_sec > 0 && rc->rc_timeout.tv_usec != -1) { + if (rc->rc_timeout.tv_sec > 0 && rc->rc_timeout.tv_usec >= 0) { error = so_setsockopt(so, SOL_SOCKET, SO_SNDTIMEO, &rc->rc_timeout, sizeof(struct timeval)); if (error != 0) { Modified: projects/pnfs-planb-server/sys/rpc/clnt_vc.c ============================================================================== --- projects/pnfs-planb-server/sys/rpc/clnt_vc.c Sat Jul 14 10:14:59 2018 (r336275) +++ projects/pnfs-planb-server/sys/rpc/clnt_vc.c Sat Jul 14 13:39:21 2018 (r336276) @@ -142,6 +142,8 @@ clnt_vc_create( XDR xdrs; int error, interrupted, one = 1, sleep_flag; struct sockopt sopt; + struct timeval tv; + int timo; if (disrupt == 0) disrupt = (uint32_t)(long)raddr; @@ -156,6 +158,18 @@ clnt_vc_create( ct->ct_upcallrefs = 0; if ((so->so_state & (SS_ISCONNECTED|SS_ISCONFIRMING)) == 0) { + /* Check for a timeout on the soconnect(). */ + bzero(&sopt, sizeof(sopt)); + sopt.sopt_dir = SOPT_GET; + sopt.sopt_level = SOL_SOCKET; + sopt.sopt_name = SO_SNDTIMEO; + sopt.sopt_val = &tv; + sopt.sopt_valsize = sizeof(tv); + error = sogetopt(so, &sopt); + timo = 0; + if (error == 0 && tv.tv_sec > 0 && tv.tv_sec <= 600 && + tv.tv_usec >= 0) + timo = hz * tv.tv_sec; error = soconnect(so, raddr, curthread); SOCK_LOCK(so); interrupted = 0; @@ -165,9 +179,10 @@ clnt_vc_create( while ((so->so_state & SS_ISCONNECTING) && so->so_error == 0) { error = msleep(&so->so_timeo, SOCK_MTX(so), - sleep_flag, "connec", 0); + sleep_flag, "connec", timo); if (error) { - if (error == EINTR || error == ERESTART) + if (error == EINTR || error == ERESTART || + error == EAGAIN) interrupted = 1; break; } From owner-svn-src-projects@freebsd.org Sat Jul 14 13:45:32 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75952102B980 for ; Sat, 14 Jul 2018 13:45:32 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2730B7A5AF; Sat, 14 Jul 2018 13:45:32 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 087061FC73; Sat, 14 Jul 2018 13:45:32 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6EDjV6Q039647; Sat, 14 Jul 2018 13:45:31 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6EDjVKq039646; Sat, 14 Jul 2018 13:45:31 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807141345.w6EDjVKq039646@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 14 Jul 2018 13:45:31 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336277 - projects/pnfs-planb-server/sys/fs/nfs X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/pnfs-planb-server/sys/fs/nfs X-SVN-Commit-Revision: 336277 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jul 2018 13:45:32 -0000 Author: rmacklem Date: Sat Jul 14 13:45:31 2018 New Revision: 336277 URL: https://svnweb.freebsd.org/changeset/base/336277 Log: Set CLSET_TIMEOUT() on a TCP connection to a DS to lease_duration/4. Setting the value this way results in the DS being disabled after approximately the lease duration, due to retries within the krpc. At least this is what I am seeing during testing and making it a fraction of the lease duration seemed better than just picking an arbitrary timeout. Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c Sat Jul 14 13:39:21 2018 (r336276) +++ projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c Sat Jul 14 13:45:31 2018 (r336277) @@ -98,6 +98,7 @@ extern int nfscl_ticks; extern void (*ncl_call_invalcaches)(struct vnode *); extern int nfs_numnfscbd; extern int nfscl_debuglevel; +extern int nfsrv_lease; SVCPOOL *nfscbd_pool; static int nfsrv_gsscallbackson = 0; @@ -195,8 +196,10 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq */ if (nrp->nr_cred != NULL) td->td_ucred = nrp->nr_cred; - else if (cred != NULL) + else { + KASSERT(cred != NULL, ("newnfs_request: null cred")); td->td_ucred = cred; + } saddr = nrp->nr_nam; if (saddr->sa_family == AF_INET) @@ -300,7 +303,9 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq if (cred != NULL) { if (NFSHASSOFT(nmp)) { /* This should be a DS mount. */ - timo.tv_sec = 15; + timo.tv_sec = nfsrv_lease / 4; + if (timo.tv_sec < 10) + timo.tv_sec = 10; timo.tv_usec = 0; CLNT_CONTROL(client, CLSET_TIMEOUT, &timo); @@ -333,7 +338,9 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq * only case where using a "soft" mount is * recommended for NFSv4. */ - timo.tv_sec = 15; + timo.tv_sec = nfsrv_lease / 4; + if (timo.tv_sec < 10) + timo.tv_sec = 10; timo.tv_usec = 0; CLNT_CONTROL(client, CLSET_TIMEOUT, &timo); retries = 2; From owner-svn-src-projects@freebsd.org Sat Jul 14 20:39:11 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 905B5103D3FE for ; Sat, 14 Jul 2018 20:39:11 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 319E088A52; Sat, 14 Jul 2018 20:39:11 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 0D1BA23F11; Sat, 14 Jul 2018 20:39:11 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6EKdAoN049000; Sat, 14 Jul 2018 20:39:10 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6EKdA9i048998; Sat, 14 Jul 2018 20:39:10 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807142039.w6EKdA9i048998@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 14 Jul 2018 20:39:10 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336295 - in projects/pnfs-planb-server/sys/fs: nfs nfsclient X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: in projects/pnfs-planb-server/sys/fs: nfs nfsclient X-SVN-Commit-Revision: 336295 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jul 2018 20:39:11 -0000 Author: rmacklem Date: Sat Jul 14 20:39:10 2018 New Revision: 336295 URL: https://svnweb.freebsd.org/changeset/base/336295 Log: Add code to the pNFS client so that it shuts down a TCP connection to a DS when a renew operation (actually just a Sequence Op) fails due to ENXIO. Not particularily critical, but without this patch the client just does a TCP reconnect attempt every 30sec. Also, make the call to nfscl_cancelreqs() unconditional, since it checks for NFSCLDS_SAMECONN and it does so with the lock held. Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h Sat Jul 14 20:14:00 2018 (r336294) +++ projects/pnfs-planb-server/sys/fs/nfs/nfs_var.h Sat Jul 14 20:39:10 2018 (r336295) @@ -603,6 +603,7 @@ struct nfscllayout *nfscl_getlayout(struct nfsclclient uint64_t, struct nfsclflayout **, int *); void nfscl_dserr(uint32_t, uint32_t, struct nfscldevinfo *, struct nfscllayout *, struct nfsclds *); +void nfscl_cancelreqs(struct nfsclds *); void nfscl_rellayout(struct nfscllayout *, int); struct nfscldevinfo *nfscl_getdevinfo(struct nfsclclient *, uint8_t *, struct nfscldevinfo *); Modified: projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c Sat Jul 14 20:14:00 2018 (r336294) +++ projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clrpcops.c Sat Jul 14 20:39:10 2018 (r336295) @@ -4408,9 +4408,12 @@ nfsrpc_renew(struct nfsclclient *clp, struct nfsclds * if (dsp == NULL) error = newnfs_request(nd, nmp, NULL, nrp, NULL, p, cred, NFS_PROG, NFS_VER4, NULL, 1, NULL, NULL); - else + else { error = newnfs_request(nd, nmp, NULL, nrp, NULL, p, cred, NFS_PROG, NFS_VER4, NULL, 1, NULL, &dsp->nfsclds_sess); + if (error == ENXIO) + nfscl_cancelreqs(dsp); + } if (error) return (error); error = nd->nd_repstat; Modified: projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c Sat Jul 14 20:14:00 2018 (r336294) +++ projects/pnfs-planb-server/sys/fs/nfsclient/nfs_clstate.c Sat Jul 14 20:39:10 2018 (r336295) @@ -125,7 +125,6 @@ static struct nfscldeleg *nfscl_finddeleg(struct nfscl static void nfscl_retoncloselayout(vnode_t, struct nfsclclient *, uint8_t *, int, struct nfsclrecalllayout **); static void nfscl_reldevinfo_locked(struct nfscldevinfo *); -static void nfscl_cancelreqs(struct nfsclds *); static struct nfscllayout *nfscl_findlayout(struct nfsclclient *, u_int8_t *, int); static struct nfscldevinfo *nfscl_finddevinfo(struct nfsclclient *, uint8_t *); @@ -5001,16 +5000,17 @@ nfscl_dserr(uint32_t op, uint32_t stat, struct nfsclde free(recallp, M_NFSLAYRECALL); } - /* If the connection isn't used for other DSs, we can shut it down. */ - if ((dsp->nfsclds_flags & NFSCLDS_SAMECONN) == 0) - nfscl_cancelreqs(dsp); + /* And shut the TCP connection down. */ + nfscl_cancelreqs(dsp); } /* * Cancel all RPCs for this "dsp" by closing the connection. * Also, mark the session as defunct. + * If NFSCLDS_SAMECONN is set, the connection is shared with other DSs and + * cannot be shut down. */ -static void +APPLESTATIC void nfscl_cancelreqs(struct nfsclds *dsp) { struct __rpc_client *cl; From owner-svn-src-projects@freebsd.org Sat Jul 14 20:44:26 2018 Return-Path: Delivered-To: svn-src-projects@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58F2A103D813 for ; Sat, 14 Jul 2018 20:44:26 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id F26EB89092; Sat, 14 Jul 2018 20:44:25 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id CDFFE240A3; Sat, 14 Jul 2018 20:44:25 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w6EKiPBK054002; Sat, 14 Jul 2018 20:44:25 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w6EKiPfS054001; Sat, 14 Jul 2018 20:44:25 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <201807142044.w6EKiPfS054001@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Sat, 14 Jul 2018 20:44:25 +0000 (UTC) To: src-committers@freebsd.org, svn-src-projects@freebsd.org Subject: svn commit: r336296 - projects/pnfs-planb-server/sys/fs/nfs X-SVN-Group: projects X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: projects/pnfs-planb-server/sys/fs/nfs X-SVN-Commit-Revision: 336296 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-projects@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "SVN commit messages for the src " projects" tree" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jul 2018 20:44:26 -0000 Author: rmacklem Date: Sat Jul 14 20:44:25 2018 New Revision: 336296 URL: https://svnweb.freebsd.org/changeset/base/336296 Log: Add a comment explaining why the timeout for DS connections is set to lease_duration/4. Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c Modified: projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c ============================================================================== --- projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c Sat Jul 14 20:39:10 2018 (r336295) +++ projects/pnfs-planb-server/sys/fs/nfs/nfs_commonkrpc.c Sat Jul 14 20:44:25 2018 (r336296) @@ -302,7 +302,16 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq if (NFSHASNFSV4N(nmp)) { if (cred != NULL) { if (NFSHASSOFT(nmp)) { - /* This should be a DS mount. */ + /* + * This should be a DS mount. If the + * timeout is set to 1/4th of the lease + * duration, the DS should be disabled + * at approximately 1/2 lease duration, + * due to a retry count of 2. + * This will hopefully cause the client + * to continue without the disabled + * DS in less than the lease duration. + */ timo.tv_sec = nfsrv_lease / 4; if (timo.tv_sec < 10) timo.tv_sec = 10; @@ -337,6 +346,7 @@ newnfs_connect(struct nfsmount *nmp, struct nfssockreq * not maintain open/lock state and is the * only case where using a "soft" mount is * recommended for NFSv4. + * See the comment above w.r.t. timeout. */ timo.tv_sec = nfsrv_lease / 4; if (timo.tv_sec < 10)