From nobody Tue Jul 7 00:11:40 2026 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4gvM8r6Qdzz6kSnH for ; Tue, 07 Jul 2026 00:11:40 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "YR1" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4gvM8r5xWwz3Hjr for ; Tue, 07 Jul 2026 00:11:40 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1783383100; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=b40vPXBb+vZgCkXMwsNqTNIIXE/BqWupaxMtylA/gdk=; b=pHzsKX/i9Ug2bjmYUvIUfZ85B1cbwkkPAqq9Jj9+Gu38O5aVy5N2jixxBz9o8NJ8v+7geL AcUIVtdbxw3AwlXcgQ5uJytzbs1m3JXhNdtzw0wXLJInLMcO1vyEaWwIoyLizm1vI3nkre 9m7sOgaH4mOuwp0eXY7EPx7tw7FL61fgwxgLCpYnqkvPJaZ8lebqNCKD41dV4ZggcSggy3 S6uC/qavcYiBppEoaB7GXg8UtMW9hp9rR7K9pZD2ee93jekhuil8B79+5lRfFQ43czvpgc hISYmKyu6biMhE05OMhVgHBf1+BC0R6iyMC62QMA9l4KpYRZS/qA/0oHQX4fWA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1783383100; a=rsa-sha256; cv=none; b=vKc8s/CvnE3ozEect+pbPcGrA5L7RVnE/MEcbt87fo8SRnQOeBSDT5ENLbcFdYlbvIGDOj oS1dcjpu/Az0FtCfzm5ERQ4413of2DtbdM7E3RdxTKI3ssdmPFlpCrSmp+7gjF3L5ABcaI d9W+hdAk4r3mv3pVAzoV8GJV3+/+DGtHzFrmJzYKgZCuH4L0CxaiuuB8/oiZfAi+mc86Zy EfZH0KNCKVjF1pg0Mi+0E2FAuybnCOWzLN1wgMKRM6Jzsbo6HIqOa30f8CDwbqfHvu1WnZ 79jbC0PLvMODAMDYVRVmASDDiMoQAyr7AlYLb4MN9VUk0D/IomXNglumWEFZKw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1783383100; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=b40vPXBb+vZgCkXMwsNqTNIIXE/BqWupaxMtylA/gdk=; b=YlCVj0Jp+wkPjNgJbNfy9fJWZ8/87QrUdE38e9RjZo/317QVV9ur7/gGBcdmOVDQ7ZDktU IgVSVnxDV6z0rhPi+kSn2Ujo592bYtp9dlnMMApW9gfPpZg0AvsVp6cJrJJ/EK90RPEJHa wlf0//p/vAqpRbOBDnQd5KYQwXUld9Knd3xhEL8zBWKZRhZn5nxncD8AACf7zeWSKJROZ0 MQcraQBqA5vBd8Ol9hKjtuIYc1s6K0uUAT12LLEWVCCU6wFOMhLqkb7PCG0uW5HXcCUXLF 8NYtPDodLf43CY2ruwLZs10Wz8bV4L77mEVOuZTjRAFY7g3/5Eyz70DxVvWkaw== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) by mxrelay.nyi.freebsd.org (Postfix) with ESMTP id 4gvM8r4pHQz1Phr for ; Tue, 07 Jul 2026 00:11:40 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from git (uid 1279) (envelope-from git@FreeBSD.org) id 34446 by gitrepo.freebsd.org (DragonFly Mail Agent v0.13+ on gitrepo.freebsd.org); Tue, 07 Jul 2026 00:11:40 +0000 To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Rick Macklem Subject: git: d516e52373e1 - main - nfsd: Optionally enable use of M_EXTPG mbufs for read replies List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: rmacklem X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: d516e52373e1768ea84bf1ca220671a44f413abe Auto-Submitted: auto-generated Date: Tue, 07 Jul 2026 00:11:40 +0000 Message-Id: <6a4c443c.34446.24e9c94e@gitrepo.freebsd.org> The branch main has been updated by rmacklem: URL: https://cgit.FreeBSD.org/src/commit/?id=d516e52373e1768ea84bf1ca220671a44f413abe commit d516e52373e1768ea84bf1ca220671a44f413abe Author: Rick Macklem AuthorDate: 2026-07-07 00:10:11 +0000 Commit: Rick Macklem CommitDate: 2026-07-07 00:10:11 +0000 nfsd: Optionally enable use of M_EXTPG mbufs for read replies A test site determined that, for a Mellanox NIC which can handle M_EXTPG mbufs, an improvement of 5-15% for read rate could be achieved if the read reply was in M_EXTPG mbufs. A patch that tried to determine if the outbound NIC supported M_EXTPG mbufs (IFCAP_MEXTPG) did not pass review. However, it does appear that this can be useful for NFS-over-RDMA. (Which just happen to use NICs that do support M_EXTPG mbufs.) As such, this patch enables them is xp_extpg is set to true, which is never for now, but might be set true for RDMA or when vfs.nfsd.enable_mextpg is set non-zero. (It is 0 by default, so this is never enabled by default at this time.) Tested by: Greg Becker MFC after: 2 weeks --- sys/fs/nfs/nfs.h | 1 + sys/fs/nfsserver/nfs_nfsdkrpc.c | 18 +++++++++++++++++- sys/fs/nfsserver/nfs_nfsdport.c | 16 ++++++++++------ sys/fs/nfsserver/nfs_nfsdserv.c | 7 +++++-- sys/rpc/svc.h | 1 + 5 files changed, 34 insertions(+), 9 deletions(-) diff --git a/sys/fs/nfs/nfs.h b/sys/fs/nfs/nfs.h index 1056595e8cde..4e79d3afbd2f 100644 --- a/sys/fs/nfs/nfs.h +++ b/sys/fs/nfs/nfs.h @@ -746,6 +746,7 @@ struct nfsrv_descript { #define ND_EXTLSCERTUSER 0x20000000000 #define ND_ERELOOKUP 0x40000000000 #define ND_MACHCRED 0x80000000000 +#define ND_CANEXTPG 0x100000000000 /* * ND_GSS should be the "or" of all GSS type authentications. diff --git a/sys/fs/nfsserver/nfs_nfsdkrpc.c b/sys/fs/nfsserver/nfs_nfsdkrpc.c index 8c557143efbd..16c21b4e1607 100644 --- a/sys/fs/nfsserver/nfs_nfsdkrpc.c +++ b/sys/fs/nfsserver/nfs_nfsdkrpc.c @@ -97,6 +97,10 @@ SYSCTL_INT(_vfs_nfsd, OID_AUTO, server_max_nfsvers, CTLFLAG_VNET | CTLFLAG_RWTUN, &VNET_NAME(nfs_maxvers), 0, "The highest version of NFS handled by the server"); +static bool nfsrv_mextpg = false; +SYSCTL_BOOL(_vfs_nfsd, OID_AUTO, enable_mextpg, CTLFLAG_RW, + &nfsrv_mextpg, 0, "Enable use of M_EXTPG mbufs"); + static int nfs_proc(struct nfsrv_descript *, u_int32_t, SVCXPRT *xprt, struct nfsrvcache **); @@ -315,7 +319,19 @@ nfssvc_program(struct svc_req *rqst, SVCXPRT *xprt) if ((xprt->xp_tls & RPCTLS_FLAGS_CERTUSER) != 0) nd.nd_flag |= ND_TLSCERTUSER; } - nd.nd_maxextsiz = 16384; + nd.nd_maxextsiz = MBUF_PEXT_MAX_PGS * PAGE_SIZE; + /* + * If the NIC can handle M_EXTPG mbufs, they can be used + * only if the reply will not be copied into the DRC. + * This implies NFSv3 over TCP and NFSv4.n, but not NFSv4.0. + * (NFSv4.n will set ND_SAVEREPLY if the reply is going + * to be copied into the session slot.) + * Check for TCP transport (UDP uses the DRC) and a direct + * map. + */ + if ((nfsrv_mextpg || xprt->xp_extpg) && nd.nd_nam2 == NULL && + PMAP_HAS_DMAP != 0) + nd.nd_flag |= ND_CANEXTPG; #ifdef MAC mac_cred_associate_nfsd(nd.nd_cred); #endif diff --git a/sys/fs/nfsserver/nfs_nfsdport.c b/sys/fs/nfsserver/nfs_nfsdport.c index 543c43295b20..cfc0cea2195e 100644 --- a/sys/fs/nfsserver/nfs_nfsdport.c +++ b/sys/fs/nfsserver/nfs_nfsdport.c @@ -3018,12 +3018,13 @@ again: /* * If cnt > MCLBYTES and the reply will not be saved, use - * ext_pgs mbufs for TLS. + * ext_pgs mbufs for TLS or if enabled via vfs.nfsd.enable_mextpg. * For NFSv4.0, we do not know for sure if the reply will * be saved, so do not use ext_pgs mbufs for NFSv4.0. */ if (cnt > MCLBYTES && siz > MCLBYTES && - (nd->nd_flag & (ND_TLS | ND_EXTPG | ND_SAVEREPLY)) == ND_TLS && + ((nd->nd_flag & (ND_TLS | ND_SAVEREPLY)) == ND_TLS || + (nd->nd_flag & (ND_CANEXTPG | ND_SAVEREPLY)) == ND_CANEXTPG) && (nd->nd_flag & (ND_NFSV4 | ND_NFSV41)) != ND_NFSV4) nd->nd_flag |= ND_EXTPG; @@ -3364,7 +3365,8 @@ ateof: /* * If the reply is likely to exceed MCLBYTES and the reply will - * not be saved, use ext_pgs mbufs for TLS. + * not be saved, use ext_pgs mbufs for TLS or if enabled via + * vfs.nfsd.enable_mextpg. * It is difficult to predict how large each entry will be and * how many entries have been read, so just assume the directory * entries grow by a factor of 4 when attributes are included. @@ -3372,7 +3374,8 @@ ateof: * be saved, so do not use ext_pgs mbufs for NFSv4.0. */ if (cnt > MCLBYTES && siz > MCLBYTES / 4 && - (nd->nd_flag & (ND_TLS | ND_EXTPG | ND_SAVEREPLY)) == ND_TLS && + ((nd->nd_flag & (ND_TLS | ND_SAVEREPLY)) == ND_TLS || + (nd->nd_flag & (ND_CANEXTPG | ND_SAVEREPLY)) == ND_CANEXTPG) && (nd->nd_flag & (ND_NFSV4 | ND_NFSV41)) != ND_NFSV4) nd->nd_flag |= ND_EXTPG; @@ -8213,13 +8216,14 @@ nfsvno_getxattr(struct vnode *vp, char *name, uint32_t maxresp, if (tlen > 0) { /* * If cnt > MCLBYTES and the reply will not be saved, use - * ext_pgs mbufs for TLS. + * ext_pgs mbufs for TLS or enabled via vfs.nfsd.enable_mextpg. * For NFSv4.0, we do not know for sure if the reply will * be saved, so do not use ext_pgs mbufs for NFSv4.0. * Always use ext_pgs mbufs if ND_EXTPG is set. */ if ((flag & ND_EXTPG) != 0 || (tlen > MCLBYTES && - (flag & (ND_TLS | ND_SAVEREPLY)) == ND_TLS && + ((flag & (ND_TLS | ND_SAVEREPLY)) == ND_TLS || + (flag & (ND_CANEXTPG | ND_SAVEREPLY)) == ND_CANEXTPG) && (flag & (ND_NFSV4 | ND_NFSV41)) != ND_NFSV4)) uiop->uio_iovcnt = nfsrv_createiovec_extpgs(tlen, maxextsiz, &m, &m2, &iv); diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c b/sys/fs/nfsserver/nfs_nfsdserv.c index fe7f0e217a74..704667bd27b6 100644 --- a/sys/fs/nfsserver/nfs_nfsdserv.c +++ b/sys/fs/nfsserver/nfs_nfsdserv.c @@ -1028,13 +1028,16 @@ nfsrvd_read(struct nfsrv_descript *nd, __unused int isdgram, if (cnt > 0) { /* * If cnt > MCLBYTES and the reply will not be saved, use - * ext_pgs mbufs for TLS. + * ext_pgs mbufs for TLS of if enabled via + * vfs.nfsd.enable_mextpg. * For NFSv4.0, we do not know for sure if the reply will * be saved, so do not use ext_pgs mbufs for NFSv4.0. * Always use ext_pgs mbufs if ND_EXTPG is set. */ if ((nd->nd_flag & ND_EXTPG) != 0 || (cnt > MCLBYTES && - (nd->nd_flag & (ND_TLS | ND_SAVEREPLY)) == ND_TLS && + ((nd->nd_flag & (ND_TLS | ND_SAVEREPLY)) == ND_TLS || + (nd->nd_flag & (ND_CANEXTPG | ND_SAVEREPLY)) == + ND_CANEXTPG) && (nd->nd_flag & (ND_NFSV4 | ND_NFSV41)) != ND_NFSV4)) nd->nd_repstat = nfsvno_read(vp, off, cnt, nd->nd_cred, nd->nd_maxextsiz, p, &m3, &m2); diff --git a/sys/rpc/svc.h b/sys/rpc/svc.h index b0fc78d4d044..e73e61e5ac93 100644 --- a/sys/rpc/svc.h +++ b/sys/rpc/svc.h @@ -155,6 +155,7 @@ typedef struct __rpc_svcxprt { uid_t xp_uid; gid_t *xp_gidp; int xp_doneddp; + bool_t xp_extpg; /* Can use M_EXTPG mbufs. */ } SVCXPRT; /*