From owner-freebsd-arch@freebsd.org Sat Jan 9 00:46:14 2021 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D5714DB72C for ; Sat, 9 Jan 2021 00:46:14 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DCLr62Nfyz3ldd; Sat, 9 Jan 2021 00:46:13 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.1.2] (pool-74-110-137-7.rcmdva.fios.verizon.net [74.110.137.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id E53862700131; Fri, 8 Jan 2021 19:46:10 -0500 (EST) DMARC-Filter: OpenDMARC Filter v1.3.1 duke.cs.duke.edu E53862700131 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail0816; t=1610153171; bh=7JZbwKHQ7v15omYe37aqXodqzux57v4mOrDyWY/Z/Fk=; h=Subject:To:From:Date:From; b=cZOaPCuk0lWFK9TZTgMInKiudYgXeQTiMwByYVHUbkkRkzDx9bfRVoGaBN32TNF0a roqpy4D0cdRFpe0TxtxPisOTBUZxmY7LAwIDnbefX7n1sGRYyF325+/hV1J1f5FPXL Adw20GQYIWLsRiD7Pmyr40nG8bGQRjviJ4fNchkB9uXZ0eQc2ubyf02xz5kxQKXlOO JlsAKu0rZEh6wCXgl7wGXTnp5RrQ5Fd3Xr3ShchF+X1qapMwszSZZPXo8J0DltnT4G S6h4+RIlqgV4+uBMOsvoV2z72GryYr5xdUdypWQN4IMfz8hr2u2jndogOGunIY07kn R6Dr1QEq8XXxA== Subject: Re: Should we enable KERN_TLS on amd64 for FreeBSD 13? To: Steve Kargl Cc: freebsd-arch@freebsd.org, Rick Macklem , Allan Jude References: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> <20210108202256.GA7669@troutmask.apl.washington.edu> From: Andrew Gallatin Message-ID: <10fb8ede-b8cf-645c-ceee-a9cb3f9fe39f@cs.duke.edu> Date: Fri, 8 Jan 2021 19:46:10 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 MIME-Version: 1.0 In-Reply-To: <20210108202256.GA7669@troutmask.apl.washington.edu> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DCLr62Nfyz3ldd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Mailman-Approved-At: Sat, 09 Jan 2021 07:52:37 +0000 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jan 2021 00:46:14 -0000 On 1/8/21 3:22 PM, Steve Kargl wrote: > On Fri, Jan 08, 2021 at 12:26:38PM -0500, Andrew Gallatin wrote: >> >> Kernel TLS (KTLS) support was added roughly a year ago, and provides >> an efficient software or hardware accelerated path to have the kernel >> (or the NIC) handle TLS crypto. This is quite useful for web and >> NFS servers, and provides a huge (2x -> 5x) efficiency gain by >> avoiding data copies into userspace for crypto, and potentially >> offloading the crypto to hardware. >> >> KTLS is well tested on amd64, having been used in production at Netflix >> for nearly 4 years. The vast majority of Netflix video has been served >> via KTLS for the last few years. Its what has allowed us to serve >> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve >> nearly 400Gb/s on AMD servers with NICs which support crypto offload. >> >> I have received a few requests to enable it by default in GENERIC, and >> I'd like to get some opinions. >> >> There are essentially 3 options >> >> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and >> flipping kern.ipc.tls.enable=1 >> >> The advantage of this is that it "just works" out of the box for users, >> and for reviewers. >> >> The drawback is that new code is thrust on unsuspecting users, >> potentially exposing them to bugs that we have not found in our >> somewhat limited web serving workload. >> >> 2) Enable KTLS in GENERIC, but leave it turned off by default. >> >> This option allows users to enable ktls without a rebuild of GENERIC, >> but does not enable it by default. So they can enable it if they >> know about it, but are protected from bugs. >> >> The disadvantages of this are that it increases the kernel size >> by ~20K, starts up one thread per core on every amd64 machine, >> and it adds more required tuning to get good performance from FreeBSD. >> >> >> 3) Continue along with KTLS disabled in GENERIC >> >> This is the lowest risk, but adds a higher bar for users wanting >> to use ktls. >> > > Drew, > > For those that use a custom kernel configuration, would we need > to add 'options KERN_TLS' to our config files, or can a module > be loaded from the boot loader (ie. via /boot/loader.conf)? > > I have no preference between 1 or 2, either seems acceptable to > me for those running the bleeding edge. > Its not as simple as just loading a module, you'd need to have options KERN_TLS in your kernel config. There are a few places in the kernel with ifdefs for KERN_TLS (sendfile, and sockets, for example). Thank you for the feedback! Drew