From owner-svn-src-all@freebsd.org Mon Mar 30 23:30:03 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7661226A6DA; Mon, 30 Mar 2020 23:30:03 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48rpbD4w7pz3KvZ; Mon, 30 Mar 2020 23:30:00 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 2A8F0266BB; Mon, 30 Mar 2020 23:29:54 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 02UNTrY3092546; Mon, 30 Mar 2020 23:29:53 GMT (envelope-from gallatin@FreeBSD.org) Received: (from gallatin@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 02UNTrWa092545; Mon, 30 Mar 2020 23:29:53 GMT (envelope-from gallatin@FreeBSD.org) Message-Id: <202003302329.02UNTrWa092545@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gallatin set sender to gallatin@FreeBSD.org using -f From: Andrew Gallatin Date: Mon, 30 Mar 2020 23:29:53 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r359474 - head/sys/kern X-SVN-Group: head X-SVN-Commit-Author: gallatin X-SVN-Commit-Paths: head/sys/kern X-SVN-Commit-Revision: 359474 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2020 23:30:03 -0000 Author: gallatin Date: Mon Mar 30 23:29:53 2020 New Revision: 359474 URL: https://svnweb.freebsd.org/changeset/base/359474 Log: KTLS: Coalesce adjacent TLS trailers & headers to improve PCIe bus efficiency KTLS uses the embedded header and trailer fields of unmapped mbufs. This can lead to "silly" buffer lengths, where we have an mbuf chain that will create a scatter/gather lists with a regular pattern of 13 bytes followed by 16 bytes between each adjacent TLS record. For software ktls we typically wind up with a pattern where we have several TLS records encrypted, and made ready at once. When these records are made ready, we can coalesce these silly buffers in sbready_compress by copying 13b TLS header of the next record into the 16b TLS trailer of the current record. After doing so, we now have a small 29 byte chunk between each TLS record. This marginally increases PCIe bus efficiency. We've seen an almost 1Gb/s increase in peak throughput on Broadwell based Xeons running a 100% software TLS workload with Mellanox ConnectX-4 NICs. Note that this change is ifdef'ed for KTLS, as KTLS is currently the only user of the hdr/trailer feature of unmapped mbufs, and peeking into them is expensive, since the ext_pgs struct lives in separately allocated memory, and may be cold in cache. This optimization is not applicable to HW ("NIC") TLS, as that depends on having the entire TLS record described by a single unmapped mbuf, so we cannot shift parts of the record between mbufs for HW TLS. Reviewed by: jhb, hselasky, scottl Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D24204 Modified: head/sys/kern/uipc_sockbuf.c Modified: head/sys/kern/uipc_sockbuf.c ============================================================================== --- head/sys/kern/uipc_sockbuf.c Mon Mar 30 22:13:32 2020 (r359473) +++ head/sys/kern/uipc_sockbuf.c Mon Mar 30 23:29:53 2020 (r359474) @@ -112,7 +112,42 @@ sbready_compress(struct sockbuf *sb, struct mbuf *m0, for (m = m0; m != end; m = m->m_next) { MPASS((m->m_flags & M_NOTREADY) == 0); + /* + * NB: In sbcompress(), 'n' is the last mbuf in the + * socket buffer and 'm' is the new mbuf being copied + * into the trailing space of 'n'. Here, the roles + * are reversed and 'n' is the next mbuf after 'm' + * that is being copied into the trailing space of + * 'm'. + */ + n = m->m_next; +#ifdef KERN_TLS + /* Try to coalesce adjacent ktls mbuf hdr/trailers. */ + if ((n != NULL) && (n != end) && (m->m_flags & M_EOR) == 0 && + (m->m_flags & M_NOMAP) && + (n->m_flags & M_NOMAP) && + !mbuf_has_tls_session(m) && + !mbuf_has_tls_session(n)) { + struct mbuf_ext_pgs *mpgs, *npgs; + int hdr_len, trail_len; + mpgs = m->m_ext.ext_pgs; + npgs = n->m_ext.ext_pgs; + hdr_len = npgs->hdr_len; + trail_len = mpgs->trail_len; + if (trail_len != 0 && hdr_len != 0 && + trail_len + hdr_len <= MBUF_PEXT_TRAIL_LEN) { + /* copy n's header to m's trailer */ + memcpy(&mpgs->trail[trail_len], npgs->hdr, + hdr_len); + mpgs->trail_len += hdr_len; + m->m_len += hdr_len; + npgs->hdr_len = 0; + n->m_len -= hdr_len; + } + } +#endif + /* Compress small unmapped mbufs into plain mbufs. */ if ((m->m_flags & M_NOMAP) && m->m_len <= MLEN && !mbuf_has_tls_session(m)) { @@ -124,15 +159,6 @@ sbready_compress(struct sockbuf *sb, struct mbuf *m0, } } - /* - * NB: In sbcompress(), 'n' is the last mbuf in the - * socket buffer and 'm' is the new mbuf being copied - * into the trailing space of 'n'. Here, the roles - * are reversed and 'n' is the next mbuf after 'm' - * that is being copied into the trailing space of - * 'm'. - */ - n = m->m_next; while ((n != NULL) && (n != end) && (m->m_flags & M_EOR) == 0 && M_WRITABLE(m) && (m->m_flags & M_NOMAP) == 0 &&