Date: Mon, 25 Jan 2021 16:30:20 -0500 From: Andrew Gallatin <gallatin@cs.duke.edu> To: Allan Jude <allanjude@freebsd.org>, John Baldwin <jhb@FreeBSD.org>, freebsd-arch@FreeBSD.org, Ed Maste <emaste@FreeBSD.org> Subject: Re: Should we enable KERN_TLS on amd64 for FreeBSD 13? Message-ID: <e6d93e1a-7c0b-5ffd-61c0-e9bee3c84fc2@cs.duke.edu> In-Reply-To: <545e9227-a4a2-8c77-1400-c4371b654f36@freebsd.org> References: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> <d3505804-e55f-dfad-7027-060d05675d4a@freebsd.org> <ecc65818-9426-e366-11b4-300e8608f99e@FreeBSD.org> <7c8f5dfa-3ae5-5620-2505-2324d41deaca@cs.duke.edu> <545e9227-a4a2-8c77-1400-c4371b654f36@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/25/21 4:21 PM, Allan Jude wrote: > On 2021-01-25 15:33, Andrew Gallatin wrote: >> On 1/25/21 2:59 PM, John Baldwin wrote: >>> On 1/25/21 10:45 AM, Allan Jude wrote: >>>> On 2021-01-08 12:26, Andrew Gallatin wrote: >>>>> >>>>> Kernel TLS (KTLS) support was added roughly a year ago, and provides >>>>> an efficient software or hardware accelerated path to have the kernel >>>>> (or the NIC) handle TLS crypto. This is quite useful for web and >>>>> NFS servers, and provides a huge (2x -> 5x) efficiency gain by >>>>> avoiding data copies into userspace for crypto, and potentially >>>>> offloading the crypto to hardware. >>>>> >>>>> >>>>> KTLS is well tested on amd64, having been used in production at Netflix >>>>> for nearly 4 years. The vast majority of Netflix video has been >>>>> served >>>>> via KTLS for the last few years. Its what has allowed us to serve >>>>> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve >>>>> nearly 400Gb/s on AMD servers with NICs which support crypto offload. >>>>> >>>>> I have received a few requests to enable it by default in GENERIC, and >>>>> I'd like to get some opinions. >>>>> >>>>> There are essentially 3 options >>>>> >>>>> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and >>>>> flipping kern.ipc.tls.enable=1 >>>>> >>>>> The advantage of this is that it "just works" out of the box for users, >>>>> and for reviewers. >>>>> >>>>> The drawback is that new code is thrust on unsuspecting users, >>>>> potentially exposing them to bugs that we have not found in our >>>>> somewhat limited web serving workload. >>>>> >>>>> 2) Enable KTLS in GENERIC, but leave it turned off by default. >>>>> >>>>> This option allows users to enable ktls without a rebuild of GENERIC, >>>>> but does not enable it by default. So they can enable it if they >>>>> know about it, but are protected from bugs. >>>>> >>>>> The disadvantages of this are that it increases the kernel size >>>>> by ~20K, starts up one thread per core on every amd64 machine, >>>>> and it adds more required tuning to get good performance from FreeBSD. >>>>> >>>>> >>>>> 3) Continue along with KTLS disabled in GENERIC >>>>> >>>>> This is the lowest risk, but adds a higher bar for users wanting >>>>> to use ktls. >>>>> >>>>> >>>>> >>>>> Note that the discussion is focused on amd64 only, as KTLS will >>>>> only work on 64-bit platforms which use a direct map. It has >>>>> not been tested at all on ppc64, and currently causes a >>>>> panic-at-boot on arm64 due to what are suspected to be problems >>>>> in the arm64 PCB setup. See: >>>>> https://urldefense.com/v3/__https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945__;!!OToaGQ!7pQUcHPbxA12vEdKTCp5jkyVxDCqYEJ-BI38kgHqGgweT7yYYG1BVhbDek0_Jc7mqA$ >>>>> >>>>> Drew >>>>> >>>> >>>> Just before this went in, Ed cleaned up the arm64 GENERIC to get it >>>> closer to the amd64 one. Can we enable KERN_TLS in arm64 GENERIC as >>>> well? >>> >>> Well, I also fixed a bug KERN_TLS exposed on arm64 that was gating for >>> this (247945). I would not be opposed to enabling it on arm64, but I >>> have not personally tested it on arm64. If someone can verify it works >>> ok on arm64 I'd be happy for it to be enabled there. >>> >> >> Yeah, that's the thing, I have much less confidence in ktls on arm64 >> because we have not run it in production recently. So I'm personally >> much less confident in enabling it on arm64. >> >> Drew > > Klara has tested it on arm64 fairly heavily, and only found an issue > with OpenSSL, but not found any issues with KERN_TLS itself. > I have no objection if you want to enable it. But since I don't currently have an arm64 test machine, I don't want to be the one to enable it, if that makes sense.. Drew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e6d93e1a-7c0b-5ffd-61c0-e9bee3c84fc2>