From owner-svn-src-head@freebsd.org Fri Sep 27 20:08:20 2019 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AF9B912B8EE; Fri, 27 Sep 2019 20:08:20 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46g2sw4H62z4FyT; Fri, 27 Sep 2019 20:08:20 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 764481E1EF; Fri, 27 Sep 2019 20:08:20 +0000 (UTC) (envelope-from gallatin@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x8RK8KY3076944; Fri, 27 Sep 2019 20:08:20 GMT (envelope-from gallatin@FreeBSD.org) Received: (from gallatin@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x8RK8JNu076941; Fri, 27 Sep 2019 20:08:19 GMT (envelope-from gallatin@FreeBSD.org) Message-Id: <201909272008.x8RK8JNu076941@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: gallatin set sender to gallatin@FreeBSD.org using -f From: Andrew Gallatin Date: Fri, 27 Sep 2019 20:08:19 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r352816 - in head/sys: kern sys X-SVN-Group: head X-SVN-Commit-Author: gallatin X-SVN-Commit-Paths: in head/sys: kern sys X-SVN-Commit-Revision: 352816 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Sep 2019 20:08:20 -0000 Author: gallatin Date: Fri Sep 27 20:08:19 2019 New Revision: 352816 URL: https://svnweb.freebsd.org/changeset/base/352816 Log: kTLS: Fix a bug where we would not encrypt anon data inplace. Software Kernel TLS needs to allocate a new destination crypto buffer when encrypting data from the page cache, so as to avoid overwriting shared clear-text file data with encrypted data specific to a single socket. When the data is anonymous, eg, not tied to a file, then we can encrypt in place and avoid allocating a new page. This fixes a bug where the existing code always assumes the data is private, and never encrypts in place. This results in unneeded page allocations and potentially more memory bandwidth consumption when doing socket writes. When the code was written at Netflix, ktls_encrypt() looked at private sendfile flags to determine if the pages being encrypted where part of the page cache (coming from sendfile) or anonymous (coming from sosend). This was broken internally at Netflix when the sendfile flags were made private, and the M_WRITABLE() check was added. Unfortunately, M_WRITABLE() will always be false for M_NOMAP mbufs, since one cannot just mtod() them. This change introduces a new flags field to the mbuf_ext_pgs struct by stealing a byte from the tls hdr. Note that the current header is still 2 bytes larger than the largest header we support: AES-CBC with explicit IV. We set MBUF_PEXT_FLAG_ANON when creating an unmapped mbuf in m_uiotombuf_nomap() (which is the path that socket writes take), and we check for that flag in ktls_encrypt() when looking for anon pages. Reviewed by: jhb Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D21796 Modified: head/sys/kern/kern_mbuf.c head/sys/kern/uipc_ktls.c head/sys/kern/uipc_mbuf.c head/sys/sys/mbuf.h Modified: head/sys/kern/kern_mbuf.c ============================================================================== --- head/sys/kern/kern_mbuf.c Fri Sep 27 19:26:52 2019 (r352815) +++ head/sys/kern/kern_mbuf.c Fri Sep 27 20:08:19 2019 (r352816) @@ -1171,6 +1171,7 @@ mb_alloc_ext_pgs(int how, bool pkthdr, m_ext_free_t ex ext_pgs->nrdy = 0; ext_pgs->first_pg_off = 0; ext_pgs->last_pg_len = 0; + ext_pgs->flags = 0; ext_pgs->hdr_len = 0; ext_pgs->trail_len = 0; ext_pgs->tls = NULL; Modified: head/sys/kern/uipc_ktls.c ============================================================================== --- head/sys/kern/uipc_ktls.c Fri Sep 27 19:26:52 2019 (r352815) +++ head/sys/kern/uipc_ktls.c Fri Sep 27 20:08:19 2019 (r352816) @@ -1363,7 +1363,7 @@ ktls_encrypt(struct mbuf_ext_pgs *pgs) * (from sendfile), anonymous wired pages are * allocated and assigned to the destination iovec. */ - is_anon = M_WRITABLE(m); + is_anon = (pgs->flags & MBUF_PEXT_FLAG_ANON) != 0; off = pgs->first_pg_off; for (i = 0; i < pgs->npgs; i++, off = 0) { @@ -1416,6 +1416,9 @@ retry_page: /* Use the basic free routine. */ m->m_ext.ext_free = mb_free_mext_pgs; + + /* Pages are now writable. */ + pgs->flags |= MBUF_PEXT_FLAG_ANON; } /* Modified: head/sys/kern/uipc_mbuf.c ============================================================================== --- head/sys/kern/uipc_mbuf.c Fri Sep 27 19:26:52 2019 (r352815) +++ head/sys/kern/uipc_mbuf.c Fri Sep 27 20:08:19 2019 (r352816) @@ -1664,6 +1664,7 @@ m_uiotombuf_nomap(struct uio *uio, int how, int len, i prev->m_next = mb; prev = mb; pgs = mb->m_ext.ext_pgs; + pgs->flags = MBUF_PEXT_FLAG_ANON; needed = length = MIN(maxseg, total); for (i = 0; needed > 0; i++, needed -= PAGE_SIZE) { retry_page: Modified: head/sys/sys/mbuf.h ============================================================================== --- head/sys/sys/mbuf.h Fri Sep 27 19:26:52 2019 (r352815) +++ head/sys/sys/mbuf.h Fri Sep 27 20:08:19 2019 (r352816) @@ -312,7 +312,7 @@ struct socket; * - 21 (AES-CBC with explicit IV) * - 13 (AES-GCM with 8 byte explicit IV) */ -#define MBUF_PEXT_HDR_LEN 24 +#define MBUF_PEXT_HDR_LEN 23 /* * TLS records for TLS 1.0-1.2 can have the following maximum trailer @@ -333,6 +333,8 @@ struct socket; #define MBUF_PEXT_MAX_BYTES \ (MBUF_PEXT_MAX_PGS * PAGE_SIZE + MBUF_PEXT_HDR_LEN + MBUF_PEXT_TRAIL_LEN) +#define MBUF_PEXT_FLAG_ANON 1 /* Data can be encrypted in place. */ + /* * This struct is 256 bytes in size and is arranged so that the most * common case (accessing the first 4 pages of a 16KB TLS record) will @@ -347,6 +349,7 @@ struct mbuf_ext_pgs { uint16_t last_pg_len; /* Length of last page */ vm_paddr_t pa[MBUF_PEXT_MAX_PGS]; /* phys addrs of pages */ char hdr[MBUF_PEXT_HDR_LEN]; /* TLS header */ + uint8_t flags; /* Flags */ struct ktls_session *tls; /* TLS session */ #if defined(__i386__) || \ (defined(__powerpc__) && !defined(__powerpc64__) && defined(BOOKE))