From owner-freebsd-arch@freebsd.org Mon Jan 25 20:33:22 2021 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9532D4F0E53 for ; Mon, 25 Jan 2021 20:33:22 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPhQV3TxQz3JsM; Mon, 25 Jan 2021 20:33:22 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.1.2] (pool-74-110-137-7.rcmdva.fios.verizon.net [74.110.137.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id B334427008D5; Mon, 25 Jan 2021 15:33:20 -0500 (EST) DMARC-Filter: OpenDMARC Filter v1.3.1 duke.cs.duke.edu B334427008D5 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail0816; t=1611606801; bh=B2qvrFzIVyj8rXi4oOaonoArCCgHKp25vEUOAzrlD2c=; h=Subject:To:From:Date:From; b=pgG6mAZK/U5mOGvmbl9nqULh8QIXILzGJetLDRYEvFtFJKVEbRlp7gfLztwK+4pxZ /6M2yNJEZryNDzsWxdfPl0FbBXpU+arWYcCKgyOf5ujava+vDVRXQi6cDBORVery76 M9ai/9OIIjMq/0BIcBwx1PR4vzLvcLjNODWMLFPpIGY1I41faonJPD8H3hX3GT0xf5 jOZLYSrmJJzGc4Ad2MH/exBbPdEkx3wFZ6RqanKT8MqDYXGk06AG4RsnFKuKL4bKyX oz1Rs07su5xPrHjP+h8phylmFGWyWhFWfjyM/pkmPfyESjKTsAkNv+X91i5XiHVYSD XhS5ZHLaP2Rng== Subject: Re: Should we enable KERN_TLS on amd64 for FreeBSD 13? To: John Baldwin , Allan Jude , freebsd-arch@FreeBSD.org, Ed Maste References: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> From: Andrew Gallatin Message-ID: <7c8f5dfa-3ae5-5620-2505-2324d41deaca@cs.duke.edu> Date: Mon, 25 Jan 2021 15:33:19 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4DPhQV3TxQz3JsM X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Mailman-Approved-At: Tue, 26 Jan 2021 10:50:30 +0000 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 20:33:22 -0000 On 1/25/21 2:59 PM, John Baldwin wrote: > On 1/25/21 10:45 AM, Allan Jude wrote: >> On 2021-01-08 12:26, Andrew Gallatin wrote: >>> >>> Kernel TLS (KTLS) support was added roughly a year ago, and provides >>> an efficient software or hardware accelerated path to have the kernel >>> (or the NIC) handle TLS crypto.  This is quite useful for web and >>> NFS servers, and provides a huge (2x -> 5x) efficiency gain by >>> avoiding data copies into userspace for crypto, and potentially >>> offloading the crypto to hardware. >>> >>> >>> KTLS is well tested on amd64, having been used in production at Netflix >>> for nearly 4 years.   The vast majority of Netflix video has been served >>> via KTLS for the last few years.  Its what has allowed us to serve >>> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve >>> nearly 400Gb/s on AMD servers with NICs which support crypto offload. >>> >>> I have received a few requests to enable it by default in GENERIC, and >>> I'd like to get some opinions. >>> >>> There are essentially 3 options >>> >>> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and >>> flipping kern.ipc.tls.enable=1 >>> >>> The advantage of this is that it "just works" out of the box for users, >>> and for reviewers. >>> >>> The drawback is that new code is thrust on unsuspecting users, >>> potentially exposing them to bugs that we have not found in our >>> somewhat limited web serving workload. >>> >>> 2) Enable KTLS in GENERIC, but leave it turned off by default. >>> >>> This option allows users to enable ktls without a rebuild of GENERIC, >>> but does not enable it by default. So they can enable it if they >>> know about it, but are protected from bugs. >>> >>> The disadvantages of this are that it increases the kernel size >>> by ~20K, starts up one thread per core on every amd64 machine, >>> and it adds more required tuning to get good performance from FreeBSD. >>> >>> >>> 3) Continue along with KTLS disabled in GENERIC >>> >>> This is the lowest risk, but adds a higher bar for users wanting >>> to use ktls. >>> >>> >>> >>> Note that the discussion is focused on amd64 only, as KTLS will >>> only work on 64-bit platforms which use a direct map.  It has >>> not been tested at all on ppc64, and currently causes a >>> panic-at-boot on arm64 due to what are suspected to be problems >>> in the arm64 PCB setup. See: >>> https://urldefense.com/v3/__https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945__;!!OToaGQ!7pQUcHPbxA12vEdKTCp5jkyVxDCqYEJ-BI38kgHqGgweT7yYYG1BVhbDek0_Jc7mqA$ >>> >>> Drew >>> >> >> Just before this went in, Ed cleaned up the arm64 GENERIC to get it >> closer to the amd64 one. Can we enable KERN_TLS in arm64 GENERIC as well? > > Well, I also fixed a bug KERN_TLS exposed on arm64 that was gating for > this (247945).  I would not be opposed to enabling it on arm64, but I > have not personally tested it on arm64.  If someone can verify it works > ok on arm64 I'd be happy for it to be enabled there. > Yeah, that's the thing, I have much less confidence in ktls on arm64 because we have not run it in production recently. So I'm personally much less confident in enabling it on arm64. Drew