From owner-svn-src-head@freebsd.org Wed Jul 22 23:33:38 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D0A88363C64; Wed, 22 Jul 2020 23:33:38 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BBsGp4y2Bz4tvq; Wed, 22 Jul 2020 23:33:38 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8D571F726; Wed, 22 Jul 2020 23:33:38 +0000 (UTC) (envelope-from rmacklem@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 06MNXcFu056341; Wed, 22 Jul 2020 23:33:38 GMT (envelope-from rmacklem@FreeBSD.org) Received: (from rmacklem@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 06MNXcpE056339; Wed, 22 Jul 2020 23:33:38 GMT (envelope-from rmacklem@FreeBSD.org) Message-Id: <202007222333.06MNXcpE056339@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: rmacklem set sender to rmacklem@FreeBSD.org using -f From: Rick Macklem Date: Wed, 22 Jul 2020 23:33:38 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r363437 - head/sys/fs/nfsclient X-SVN-Group: head X-SVN-Commit-Author: rmacklem X-SVN-Commit-Paths: head/sys/fs/nfsclient X-SVN-Commit-Revision: 363437 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jul 2020 23:33:38 -0000 Author: rmacklem Date: Wed Jul 22 23:33:37 2020 New Revision: 363437 URL: https://svnweb.freebsd.org/changeset/base/363437 Log: Modify writing to mirrored pNFS DSs to prepare for use of ext_pgs mbufs. This patch modifies writing to mirrored pNFS DSs slightly so that there is only one m_copym() call for a mirrored pair instead of two of them. This call replaces the custom nfsm_copym() call, which is no longer needed and deleted by this patch. The patch does introduce a new nfsm_split() function that only calls m_split() for the non-ext_pgs case. The semantics of nfsm_uiombuflist() is changed to include code that nul pads the generated mbuf list. This was done by nfsm_copym() prior to this patch. The main reason for this change is that it allows the data to be a list of ext_pgs mbufs, since the m_copym() is for the entire mbuf list. This support will be added in a future commit. This patch only affects writing to mirrored flexible file layout pNFS servers. Modified: head/sys/fs/nfsclient/nfs_clcomsubs.c head/sys/fs/nfsclient/nfs_clrpcops.c Modified: head/sys/fs/nfsclient/nfs_clcomsubs.c ============================================================================== --- head/sys/fs/nfsclient/nfs_clcomsubs.c Wed Jul 22 22:51:14 2020 (r363436) +++ head/sys/fs/nfsclient/nfs_clcomsubs.c Wed Jul 22 23:33:37 2020 (r363437) @@ -164,9 +164,9 @@ nfsm_uiombuflist(struct uio *uiop, int siz, struct mbu { char *uiocp; struct mbuf *mp, *mp2, *firstmp; - int xfer, left, mlen; + int i, left, mlen, rem, xfer; int uiosiz, clflg; - char *tcp; + char *mcp, *tcp; KASSERT(uiop->uio_iovcnt == 1, ("nfsm_uiotombuf: iovcnt != 1")); @@ -179,7 +179,9 @@ nfsm_uiombuflist(struct uio *uiop, int siz, struct mbu else NFSMGET(mp); mp->m_len = 0; + mcp = mtod(mp, char *); firstmp = mp2 = mp; + rem = NFSM_RNDUP(siz) - siz; while (siz > 0) { left = uiop->uio_iov->iov_len; uiocp = uiop->uio_iov->iov_base; @@ -194,18 +196,18 @@ nfsm_uiombuflist(struct uio *uiop, int siz, struct mbu else NFSMGET(mp); mp->m_len = 0; + mcp = mtod(mp, char *); mp2->m_next = mp; mp2 = mp; mlen = M_TRAILINGSPACE(mp); } xfer = (left > mlen) ? mlen : left; if (uiop->uio_segflg == UIO_SYSSPACE) - NFSBCOPY(uiocp, mtod(mp, caddr_t) + - mp->m_len, xfer); + NFSBCOPY(uiocp, mcp, xfer); else - copyin(uiocp, mtod(mp, caddr_t) + - mp->m_len, xfer); + copyin(uiocp, mcp, xfer); mp->m_len += xfer; + mcp += xfer; left -= xfer; uiocp += xfer; uiop->uio_offset += xfer; @@ -216,6 +218,13 @@ nfsm_uiombuflist(struct uio *uiop, int siz, struct mbu uiop->uio_iov->iov_base = (void *)tcp; uiop->uio_iov->iov_len -= uiosiz; siz -= uiosiz; + } + if (rem > 0) { + KASSERT(rem <= M_TRAILINGSPACE(mp), + ("nfsm_uiombuflist: no space for padding")); + for (i = 0; i < rem; i++) + *mcp++ = '\0'; + mp->m_len += rem; } if (cpp != NULL) *cpp = mtod(mp, caddr_t) + mp->m_len; Modified: head/sys/fs/nfsclient/nfs_clrpcops.c ============================================================================== --- head/sys/fs/nfsclient/nfs_clrpcops.c Wed Jul 22 22:51:14 2020 (r363436) +++ head/sys/fs/nfsclient/nfs_clrpcops.c Wed Jul 22 23:33:37 2020 (r363437) @@ -158,7 +158,6 @@ static int nfscl_dofflayoutio(vnode_t, struct uio *, i nfsv4stateid_t *, int, struct nfscldevinfo *, struct nfscllayout *, struct nfsclflayout *, uint64_t, uint64_t, int, int, struct mbuf *, struct nfsclwritedsdorpc *, struct ucred *, NFSPROC_T *); -static struct mbuf *nfsm_copym(struct mbuf *, int, int); static int nfsrpc_readds(vnode_t, struct uio *, nfsv4stateid_t *, int *, struct nfsclds *, uint64_t, int, struct nfsfh *, int, int, int, struct ucred *, NFSPROC_T *); @@ -220,6 +219,7 @@ static int nfsrpc_copyrpc(vnode_t, off_t, vnode_t, off struct nfsvattr *, int *, bool, int *, struct ucred *, NFSPROC_T *); static int nfsrpc_seekrpc(vnode_t, off_t *, nfsv4stateid_t *, bool *, int, struct nfsvattr *, int *, struct ucred *); +static struct mbuf *nfsm_split(struct mbuf *, uint64_t); int nfs_pnfsio(task_fn_t *, void *); @@ -5756,7 +5756,7 @@ nfscl_doiods(vnode_t vp, struct uio *uiop, int *iomode struct nfscllayout *layp; struct nfscldevinfo *dip; struct nfsclflayout *rflp; - struct mbuf *m; + struct mbuf *m, *m2; struct nfsclwritedsdorpc *drpc, *tdrpc; nfsv4stateid_t stateid; struct ucred *newcred; @@ -5870,6 +5870,13 @@ nfscl_doiods(vnode_t vp, struct uio *uiop, int *iomode } } for (i = firstmirror; i < mirrorcnt && error == 0; i++){ + m2 = NULL; + if (m != NULL && i < mirrorcnt - 1) + m2 = m_copym(m, 0, M_COPYALL, M_WAITOK); + else { + m2 = m; + m = NULL; + } if ((layp->nfsly_flags & NFSLY_FLEXFILE) != 0) { dev = rflp->nfsfl_ffm[i].dev; dip = nfscl_getdevinfo(nmp->nm_clp, dev, @@ -5886,7 +5893,7 @@ nfscl_doiods(vnode_t vp, struct uio *uiop, int *iomode uiop, iomode, must_commit, &eof, &stateid, rwaccess, dip, layp, rflp, off, xfer, - i, docommit, m, tdrpc, + i, docommit, m2, tdrpc, newcred, p); else error = nfscl_doflayoutio(vp, @@ -5895,8 +5902,11 @@ nfscl_doiods(vnode_t vp, struct uio *uiop, int *iomode dip, layp, rflp, off, xfer, docommit, newcred, p); nfscl_reldevinfo(dip); - } else + } else { + if (m2 != NULL) + m_freem(m2); error = EIO; + } tdrpc++; } if (m != NULL) @@ -5962,38 +5972,6 @@ nfscl_doiods(vnode_t vp, struct uio *uiop, int *iomode } /* - * Make a copy of the mbuf chain and add an mbuf for null padding, as required. - */ -static struct mbuf * -nfsm_copym(struct mbuf *m, int off, int xfer) -{ - struct mbuf *m2, *m3, *m4; - uint32_t *tl; - int rem; - - m2 = m_copym(m, off, xfer, M_WAITOK); - rem = NFSM_RNDUP(xfer) - xfer; - if (rem > 0) { - /* - * The zero padding to a multiple of 4 bytes is required by - * the XDR. So that the mbufs copied by reference aren't - * modified, add an mbuf with the zero'd bytes to the list. - * rem will be a maximum of 3, so one zero'd uint32_t is - * sufficient. - */ - m3 = m2; - while (m3->m_next != NULL) - m3 = m3->m_next; - NFSMGET(m4); - tl = mtod(m4, uint32_t *); - *tl = 0; - m4->m_len = rem; - m3->m_next = m4; - } - return (m2); -} - -/* * Find a file layout that will handle the first bytes of the requested * range and return the information from it needed to the I/O operation. */ @@ -6148,17 +6126,17 @@ nfscl_dofflayoutio(vnode_t vp, struct uio *uiop, int * uint64_t len, int mirror, int docommit, struct mbuf *mp, struct nfsclwritedsdorpc *drpc, struct ucred *cred, NFSPROC_T *p) { - uint64_t transfer, xfer; - int error, rel_off; + uint64_t xfer; + int error; struct nfsnode *np; struct nfsfh *fhp; struct nfsclds **dspp; struct ucred *tcred; - struct mbuf *m; + struct mbuf *m, *m2; + uint32_t copylen; np = VTONFS(vp); error = 0; - rel_off = 0; NFSCL_DEBUG(4, "nfscl_dofflayoutio: off=%ju len=%ju\n", (uintmax_t)off, (uintmax_t)len); /* Loop around, doing I/O for each stripe unit. */ @@ -6176,14 +6154,31 @@ nfscl_dofflayoutio(vnode_t vp, struct uio *uiop, int * } else tcred = cred; if (rwflag == NFSV4OPEN_ACCESSREAD) - transfer = dp->nfsdi_rsize; - else - transfer = dp->nfsdi_wsize; + copylen = dp->nfsdi_rsize; + else { + copylen = dp->nfsdi_wsize; + if (len > copylen && mp != NULL) { + /* + * When a mirrored configuration needs to do + * multiple writes to each mirror, all writes + * except the last one must be a multiple of + * 4 bytes. This is required so that the XDR + * does not need padding. + * If possible, clip the size to an exact + * multiple of the mbuf length, so that the + * split will be on an mbuf boundary. + */ + copylen &= 0xfffffffc; + if (copylen > mp->m_len) + copylen = copylen / mp->m_len * + mp->m_len; + } + } NFSLOCKNODE(np); np->n_flag |= NDSCOMMIT; NFSUNLOCKNODE(np); - if (len > transfer && docommit == 0) - xfer = transfer; + if (len > copylen && docommit == 0) + xfer = copylen; else xfer = len; if (docommit != 0) { @@ -6244,31 +6239,41 @@ nfscl_dofflayoutio(vnode_t vp, struct uio *uiop, int * NFSUNLOCKCLSTATE(); } } else { - m = nfsm_copym(mp, rel_off, xfer); - NFSCL_DEBUG(4, "mcopy reloff=%d xfer=%jd\n", - rel_off, (uintmax_t)xfer); + m = mp; + if (xfer < len) { + /* The mbuf list must be split. */ + m2 = nfsm_split(mp, xfer); + if (m2 != NULL) + mp = m2; + else { + m_freem(mp); + error = EIO; + } + } + NFSCL_DEBUG(4, "mcopy len=%jd xfer=%jd\n", + (uintmax_t)len, (uintmax_t)xfer); /* - * Do the writes after the first loop iteration - * and the write for the last mirror via this + * Do last write to a mirrored DS with this * thread. - * This loop only iterates for small values - * of nfsdi_wsize, which may never occur in - * practice. However, the drpc is completely - * used by the first iteration and, as such, - * cannot be used after that. */ - if (mirror < flp->nfsfl_mirrorcnt - 1 && - rel_off == 0) - error = nfsio_writedsmir(vp, iomode, - must_commit, stateidp, *dspp, off, - xfer, fhp, m, dp->nfsdi_vers, - dp->nfsdi_minorvers, drpc, tcred, - p); - else - error = nfsrpc_writedsmir(vp, iomode, - must_commit, stateidp, *dspp, off, - xfer, fhp, m, dp->nfsdi_vers, - dp->nfsdi_minorvers, tcred, p); + if (error == 0) { + if (mirror < flp->nfsfl_mirrorcnt - 1) + error = nfsio_writedsmir(vp, + iomode, must_commit, + stateidp, *dspp, off, + xfer, fhp, m, + dp->nfsdi_vers, + dp->nfsdi_minorvers, drpc, + tcred, p); + else + error = nfsrpc_writedsmir(vp, + iomode, must_commit, + stateidp, *dspp, off, + xfer, fhp, m, + dp->nfsdi_vers, + dp->nfsdi_minorvers, tcred, + p); + } NFSCL_DEBUG(4, "nfsio_writedsmir=%d\n", error); if (error != 0 && error != EACCES && error != ESTALE) { @@ -6283,7 +6288,6 @@ nfscl_dofflayoutio(vnode_t vp, struct uio *uiop, int * if (error == 0) { len -= xfer; off += xfer; - rel_off += xfer; } if ((dp->nfsdi_flags & NFSDI_TIGHTCOUPLED) == 0) NFSFREECRED(tcred); @@ -8615,3 +8619,14 @@ nfsmout: return (error); } +/* + * Split an mbuf list. For non-M_EXTPG mbufs, just use m_split(). + */ +static struct mbuf * +nfsm_split(struct mbuf *mp, uint64_t xfer) +{ + struct mbuf *m; + + m = m_split(mp, xfer, M_WAITOK); + return (m); +}