Date: Mon, 25 Jan 2021 15:33:19 -0500 From: Andrew Gallatin <gallatin@cs.duke.edu> To: John Baldwin <jhb@FreeBSD.org>, Allan Jude <allanjude@freebsd.org>, freebsd-arch@FreeBSD.org, Ed Maste <emaste@FreeBSD.org> Subject: Re: Should we enable KERN_TLS on amd64 for FreeBSD 13? Message-ID: <7c8f5dfa-3ae5-5620-2505-2324d41deaca@cs.duke.edu> In-Reply-To: <ecc65818-9426-e366-11b4-300e8608f99e@FreeBSD.org> References: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> <d3505804-e55f-dfad-7027-060d05675d4a@freebsd.org> <ecc65818-9426-e366-11b4-300e8608f99e@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 1/25/21 2:59 PM, John Baldwin wrote: > On 1/25/21 10:45 AM, Allan Jude wrote: >> On 2021-01-08 12:26, Andrew Gallatin wrote: >>> >>> Kernel TLS (KTLS) support was added roughly a year ago, and provides >>> an efficient software or hardware accelerated path to have the kernel >>> (or the NIC) handle TLS crypto. This is quite useful for web and >>> NFS servers, and provides a huge (2x -> 5x) efficiency gain by >>> avoiding data copies into userspace for crypto, and potentially >>> offloading the crypto to hardware. >>> >>> >>> KTLS is well tested on amd64, having been used in production at Netflix >>> for nearly 4 years. The vast majority of Netflix video has been served >>> via KTLS for the last few years. Its what has allowed us to serve >>> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve >>> nearly 400Gb/s on AMD servers with NICs which support crypto offload. >>> >>> I have received a few requests to enable it by default in GENERIC, and >>> I'd like to get some opinions. >>> >>> There are essentially 3 options >>> >>> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and >>> flipping kern.ipc.tls.enable=1 >>> >>> The advantage of this is that it "just works" out of the box for users, >>> and for reviewers. >>> >>> The drawback is that new code is thrust on unsuspecting users, >>> potentially exposing them to bugs that we have not found in our >>> somewhat limited web serving workload. >>> >>> 2) Enable KTLS in GENERIC, but leave it turned off by default. >>> >>> This option allows users to enable ktls without a rebuild of GENERIC, >>> but does not enable it by default. So they can enable it if they >>> know about it, but are protected from bugs. >>> >>> The disadvantages of this are that it increases the kernel size >>> by ~20K, starts up one thread per core on every amd64 machine, >>> and it adds more required tuning to get good performance from FreeBSD. >>> >>> >>> 3) Continue along with KTLS disabled in GENERIC >>> >>> This is the lowest risk, but adds a higher bar for users wanting >>> to use ktls. >>> >>> >>> >>> Note that the discussion is focused on amd64 only, as KTLS will >>> only work on 64-bit platforms which use a direct map. It has >>> not been tested at all on ppc64, and currently causes a >>> panic-at-boot on arm64 due to what are suspected to be problems >>> in the arm64 PCB setup. See: >>> https://urldefense.com/v3/__https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945__;!!OToaGQ!7pQUcHPbxA12vEdKTCp5jkyVxDCqYEJ-BI38kgHqGgweT7yYYG1BVhbDek0_Jc7mqA$ >>> >>> Drew >>> >> >> Just before this went in, Ed cleaned up the arm64 GENERIC to get it >> closer to the amd64 one. Can we enable KERN_TLS in arm64 GENERIC as well? > > Well, I also fixed a bug KERN_TLS exposed on arm64 that was gating for > this (247945). I would not be opposed to enabling it on arm64, but I > have not personally tested it on arm64. If someone can verify it works > ok on arm64 I'd be happy for it to be enabled there. > Yeah, that's the thing, I have much less confidence in ktls on arm64 because we have not run it in production recently. So I'm personally much less confident in enabling it on arm64. Drew
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7c8f5dfa-3ae5-5620-2505-2324d41deaca>