From owner-freebsd-arch@freebsd.org Tue Jan 26 15:30:23 2021 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B56104F31A7 for ; Tue, 26 Jan 2021 15:30:23 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DQ9fR4lplz4Xpj; Tue, 26 Jan 2021 15:30:23 +0000 (UTC) (envelope-from gbe@freebsd.org) Received: from localhost (p200300d5d70b55db8499059115e9bb88.dip0.t-ipconnect.de [IPv6:2003:d5:d70b:55db:8499:591:15e9:bb88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: gbe) by smtp.freebsd.org (Postfix) with ESMTPSA id 3D02EABF1; Tue, 26 Jan 2021 15:30:23 +0000 (UTC) (envelope-from gbe@freebsd.org) Date: Tue, 26 Jan 2021 16:30:22 +0100 From: Gordon Bergling To: John Baldwin Cc: Allan Jude , Andrew Gallatin , freebsd-arch@freebsd.org, Ed Maste Subject: Re: Should we enable KERN_TLS on amd64 for FreeBSD 13? Message-ID: References: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Url: X-Operating-System: FreeBSD 12.2-STABLE amd64 X-Host-Uptime: 4:18PM up 13 days, 4:43, 4 users, load averages: 0.48, 0.33, 0.27 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2021 15:30:23 -0000 On Mon, Jan 25, 2021 at 11:59:55AM -0800, John Baldwin wrote: > On 1/25/21 10:45 AM, Allan Jude wrote: > > On 2021-01-08 12:26, Andrew Gallatin wrote: > >> > >> Kernel TLS (KTLS) support was added roughly a year ago, and provides > >> an efficient software or hardware accelerated path to have the kernel > >> (or the NIC) handle TLS crypto.  This is quite useful for web and > >> NFS servers, and provides a huge (2x -> 5x) efficiency gain by > >> avoiding data copies into userspace for crypto, and potentially > >> offloading the crypto to hardware. > >> > >> > >> KTLS is well tested on amd64, having been used in production at Netflix > >> for nearly 4 years.   The vast majority of Netflix video has been served > >> via KTLS for the last few years.  Its what has allowed us to serve > >> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve > >> nearly 400Gb/s on AMD servers with NICs which support crypto offload. > >> > >> I have received a few requests to enable it by default in GENERIC, and > >> I'd like to get some opinions. > >> > >> There are essentially 3 options > >> > >> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and > >> flipping kern.ipc.tls.enable=1 > >> > >> The advantage of this is that it "just works" out of the box for users, > >> and for reviewers. > >> > >> The drawback is that new code is thrust on unsuspecting users, > >> potentially exposing them to bugs that we have not found in our > >> somewhat limited web serving workload. > >> > >> 2) Enable KTLS in GENERIC, but leave it turned off by default. > >> > >> This option allows users to enable ktls without a rebuild of GENERIC, > >> but does not enable it by default. So they can enable it if they > >> know about it, but are protected from bugs. > >> > >> The disadvantages of this are that it increases the kernel size > >> by ~20K, starts up one thread per core on every amd64 machine, > >> and it adds more required tuning to get good performance from FreeBSD. > >> > >> > >> 3) Continue along with KTLS disabled in GENERIC > >> > >> This is the lowest risk, but adds a higher bar for users wanting > >> to use ktls. > >> > >> > >> > >> Note that the discussion is focused on amd64 only, as KTLS will > >> only work on 64-bit platforms which use a direct map.  It has > >> not been tested at all on ppc64, and currently causes a > >> panic-at-boot on arm64 due to what are suspected to be problems > >> in the arm64 PCB setup. See: > >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945 > >> > >> Drew > >> > > > > Just before this went in, Ed cleaned up the arm64 GENERIC to get it > > closer to the amd64 one. Can we enable KERN_TLS in arm64 GENERIC as well? > > Well, I also fixed a bug KERN_TLS exposed on arm64 that was gating for > this (247945). I would not be opposed to enabling it on arm64, but I > have not personally tested it on arm64. If someone can verify it works > ok on arm64 I'd be happy for it to be enabled there. I am the author of the mentioned PR and have beeing running the respective patch since a few weeks on a RPi4B without seeing any problems. The KTLS thread is present and the sysctl 'kern.ipc.tls.enable' is set to 1. I haven't done any real workload tests using encrypted NFSv4 traffic or nginx based HTTPS traffic. Maybe [1] should be integrated first to enable ATF based tests with the shipped OpenSSL version. Besides of this two points, I think it should be safe to enable it for arm64, as it was okay to enable it for amd64. --Gordon [1] https://reviews.freebsd.org/D28273