From owner-freebsd-arch@freebsd.org Mon Jan 25 18:45:46 2021 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1CA4E4EC2DF for ; Mon, 25 Jan 2021 18:45:46 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [IPv6:2001:470:1:474::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPf2K5pRqz4qqQ; Mon, 25 Jan 2021 18:45:45 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 363E71AF49; Mon, 25 Jan 2021 18:45:39 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 363E71AF49 Subject: Re: Should we enable KERN_TLS on amd64 for FreeBSD 13? To: Andrew Gallatin , freebsd-arch@FreeBSD.org, Ed Maste , John Baldwin References: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> From: Allan Jude Message-ID: Date: Mon, 25 Jan 2021 13:45:38 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4DPf2K5pRqz4qqQ X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 18:45:46 -0000 On 2021-01-08 12:26, Andrew Gallatin wrote: > > Kernel TLS (KTLS) support was added roughly a year ago, and provides > an efficient software or hardware accelerated path to have the kernel > (or the NIC) handle TLS crypto.  This is quite useful for web and > NFS servers, and provides a huge (2x -> 5x) efficiency gain by > avoiding data copies into userspace for crypto, and potentially > offloading the crypto to hardware. > > > KTLS is well tested on amd64, having been used in production at Netflix > for nearly 4 years.   The vast majority of Netflix video has been served > via KTLS for the last few years.  Its what has allowed us to serve > 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve > nearly 400Gb/s on AMD servers with NICs which support crypto offload. > > I have received a few requests to enable it by default in GENERIC, and > I'd like to get some opinions. > > There are essentially 3 options > > 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and > flipping kern.ipc.tls.enable=1 > > The advantage of this is that it "just works" out of the box for users, > and for reviewers. > > The drawback is that new code is thrust on unsuspecting users, > potentially exposing them to bugs that we have not found in our > somewhat limited web serving workload. > > 2) Enable KTLS in GENERIC, but leave it turned off by default. > > This option allows users to enable ktls without a rebuild of GENERIC, > but does not enable it by default. So they can enable it if they > know about it, but are protected from bugs. > > The disadvantages of this are that it increases the kernel size > by ~20K, starts up one thread per core on every amd64 machine, > and it adds more required tuning to get good performance from FreeBSD. > > > 3) Continue along with KTLS disabled in GENERIC > > This is the lowest risk, but adds a higher bar for users wanting > to use ktls. > > > > Note that the discussion is focused on amd64 only, as KTLS will > only work on 64-bit platforms which use a direct map.  It has > not been tested at all on ppc64, and currently causes a > panic-at-boot on arm64 due to what are suspected to be problems > in the arm64 PCB setup. See: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945 > > Drew > Just before this went in, Ed cleaned up the arm64 GENERIC to get it closer to the amd64 one. Can we enable KERN_TLS in arm64 GENERIC as well? -- Allan Jude