From owner-freebsd-arch@freebsd.org Fri Jan 8 17:26:44 2021 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 03F414D0041 for ; Fri, 8 Jan 2021 17:26:44 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DC94w1M6Yz4ggH; Fri, 8 Jan 2021 17:26:39 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.1.2] (pool-74-110-137-7.rcmdva.fios.verizon.net [74.110.137.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id CB87227000B7; Fri, 8 Jan 2021 12:26:38 -0500 (EST) DMARC-Filter: OpenDMARC Filter v1.3.1 duke.cs.duke.edu CB87227000B7 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail0816; t=1610126799; bh=7CvA/t8L/5N6Cmk1gmapepnHMIHaSAUA9Vwp6wFQNwQ=; h=To:From:Subject:Date:From; b=sE7kLDfYJwtyjClQQsHWNesATI+saMr7O026/4Sd7D2JBiR1fI6a1rQ35bbYlppMX s65Em1sgkuGpaFXv2ZGtHlMhzLPVGUKUww6nJQufz+ArsZmfu4H9NpNLcHUwCq8W3S KnVe5i7Pm6rhtJg3OwZurNsFVUXHbPjPc627cuqzK3Ik8Fxo+JiB4vJ1UqZTRdvcIE DOEi/6bICpUoZCyRGvt/oVxkWXiTWb8ffS5uNktpGihQPOV0iPwkE1itx5Te0aRdfp lvSB8NPFkLKQ7qYC/n0L1Hd/LiYVk6p+qw4QVtGO+WJzoP9lySVPEIKRz4bRRoR7UU UWHyDIFDirZmw== To: freebsd-arch@FreeBSD.org From: Andrew Gallatin Subject: Should we enable KERN_TLS on amd64 for FreeBSD 13? Cc: Allan Jude , John Baldwin , Rick Macklem Message-ID: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> Date: Fri, 8 Jan 2021 12:26:38 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4DC94w1M6Yz4ggH X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.duke.edu header.s=mail0816 header.b=sE7kLDfY; dmarc=pass (policy=none) header.from=cs.duke.edu; spf=pass (mx1.freebsd.org: domain of gallatin@cs.duke.edu designates 152.3.140.1 as permitted sender) smtp.mailfrom=gallatin@cs.duke.edu X-Spamd-Result: default: False [-2.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:152.3.140.0/23]; DKIM_TRACE(0.00)[cs.duke.edu:+]; DMARC_POLICY_ALLOW(-0.50)[cs.duke.edu,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[74.110.137.7:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[152.3.140.1:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:13371, ipnet:152.3.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[cs.duke.edu:s=mail0816]; FREEFALL_USER(0.00)[gallatin]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; DWL_DNSWL_LOW(-1.00)[duke.edu:dkim]; RCVD_IN_DNSWL_LOW(-0.10)[152.3.140.1:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; SPAMHAUS_ZRD(0.00)[152.3.140.1:from:127.0.2.255]; NEURAL_SPAM_LONG(1.00)[1.000]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-arch] X-Mailman-Approved-At: Fri, 08 Jan 2021 20:10:32 +0000 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jan 2021 17:26:44 -0000 Kernel TLS (KTLS) support was added roughly a year ago, and provides an efficient software or hardware accelerated path to have the kernel (or the NIC) handle TLS crypto. This is quite useful for web and NFS servers, and provides a huge (2x -> 5x) efficiency gain by avoiding data copies into userspace for crypto, and potentially offloading the crypto to hardware. KTLS is well tested on amd64, having been used in production at Netflix for nearly 4 years. The vast majority of Netflix video has been served via KTLS for the last few years. Its what has allowed us to serve 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve nearly 400Gb/s on AMD servers with NICs which support crypto offload. I have received a few requests to enable it by default in GENERIC, and I'd like to get some opinions. There are essentially 3 options 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and flipping kern.ipc.tls.enable=1 The advantage of this is that it "just works" out of the box for users, and for reviewers. The drawback is that new code is thrust on unsuspecting users, potentially exposing them to bugs that we have not found in our somewhat limited web serving workload. 2) Enable KTLS in GENERIC, but leave it turned off by default. This option allows users to enable ktls without a rebuild of GENERIC, but does not enable it by default. So they can enable it if they know about it, but are protected from bugs. The disadvantages of this are that it increases the kernel size by ~20K, starts up one thread per core on every amd64 machine, and it adds more required tuning to get good performance from FreeBSD. 3) Continue along with KTLS disabled in GENERIC This is the lowest risk, but adds a higher bar for users wanting to use ktls. Note that the discussion is focused on amd64 only, as KTLS will only work on 64-bit platforms which use a direct map. It has not been tested at all on ppc64, and currently causes a panic-at-boot on arm64 due to what are suspected to be problems in the arm64 PCB setup. See: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945 Drew