From owner-freebsd-arch@freebsd.org Mon Jan 25 21:30:22 2021 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D3C044F20F7 for ; Mon, 25 Jan 2021 21:30:22 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from duke.cs.duke.edu (duke.cs.duke.edu [152.3.140.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DPjhF60xDz3N5s; Mon, 25 Jan 2021 21:30:21 +0000 (UTC) (envelope-from gallatin@cs.duke.edu) Received: from [192.168.1.2] (pool-74-110-137-7.rcmdva.fios.verizon.net [74.110.137.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: gallatin) by duke.cs.duke.edu (Postfix) with ESMTPSA id EC6AE2700318; Mon, 25 Jan 2021 16:30:20 -0500 (EST) DMARC-Filter: OpenDMARC Filter v1.3.1 duke.cs.duke.edu EC6AE2700318 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=cs.duke.edu; s=mail0816; t=1611610221; bh=LOKOZat1H3CgMvlx/xHifqZOGHgQ+Nk5AmIvChronNM=; h=Subject:To:From:Date:From; b=n9rxTaRAS4RguKLrnwJ4bvaAprITCmsECvEFZqV5GCxNlv4DiKIOR1m1h0qakLrxv OfSDINl1Ih3Za0mPNp76KFHtzL6JFebEJj+6wONYHZhI64hvQM33Q8ihr4kOEqjjJn Re/Jao0JvZErAS4F0mbtKj+PWCealgZvDpaEZJOlCA/Ylj5NIak32HBBReKm8wzYTm yRBcd/tdVR+SA3SrCSmkAfNvIF72Z9lrtMbSLsPw1hEDT+RsMWarT25s7Fl4UeDPmW 3pBPRcrvwM1yYl90KsdXB+4F/8fjrBRfZ5cE8U9htXqKV0iPQxEwDAPE1gbIVn1ame 6dILrho/+ZxTA== Subject: Re: Should we enable KERN_TLS on amd64 for FreeBSD 13? To: Allan Jude , John Baldwin , freebsd-arch@FreeBSD.org, Ed Maste References: <8eff83e5-49bc-d410-626e-603c03877b80@cs.duke.edu> <7c8f5dfa-3ae5-5620-2505-2324d41deaca@cs.duke.edu> <545e9227-a4a2-8c77-1400-c4371b654f36@freebsd.org> From: Andrew Gallatin Message-ID: Date: Mon, 25 Jan 2021 16:30:20 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.6.1 MIME-Version: 1.0 In-Reply-To: <545e9227-a4a2-8c77-1400-c4371b654f36@freebsd.org> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4DPjhF60xDz3N5s X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.duke.edu header.s=mail0816 header.b=n9rxTaRA; dmarc=pass (policy=none) header.from=cs.duke.edu; spf=pass (mx1.freebsd.org: domain of gallatin@cs.duke.edu designates 152.3.140.1 as permitted sender) smtp.mailfrom=gallatin@cs.duke.edu X-Spamd-Result: default: False [-4.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:152.3.140.0/23]; DKIM_TRACE(0.00)[cs.duke.edu:+]; DMARC_POLICY_ALLOW(-0.50)[cs.duke.edu,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[74.110.137.7:received]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[152.3.140.1:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:13371, ipnet:152.3.128.0/17, country:US]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[cs.duke.edu:s=mail0816]; FREEFALL_USER(0.00)[gallatin]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SUBJECT_ENDS_QUESTION(1.00)[]; SPAMHAUS_ZRD(0.00)[152.3.140.1:from:127.0.2.255]; RCVD_IN_DNSWL_LOW(-0.10)[152.3.140.1:from]; DWL_DNSWL_LOW(-1.00)[duke.edu:dkim]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-arch] X-Mailman-Approved-At: Tue, 26 Jan 2021 10:50:50 +0000 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Jan 2021 21:30:22 -0000 On 1/25/21 4:21 PM, Allan Jude wrote: > On 2021-01-25 15:33, Andrew Gallatin wrote: >> On 1/25/21 2:59 PM, John Baldwin wrote: >>> On 1/25/21 10:45 AM, Allan Jude wrote: >>>> On 2021-01-08 12:26, Andrew Gallatin wrote: >>>>> >>>>> Kernel TLS (KTLS) support was added roughly a year ago, and provides >>>>> an efficient software or hardware accelerated path to have the kernel >>>>> (or the NIC) handle TLS crypto.  This is quite useful for web and >>>>> NFS servers, and provides a huge (2x -> 5x) efficiency gain by >>>>> avoiding data copies into userspace for crypto, and potentially >>>>> offloading the crypto to hardware. >>>>> >>>>> >>>>> KTLS is well tested on amd64, having been used in production at Netflix >>>>> for nearly 4 years.   The vast majority of Netflix video has been >>>>> served >>>>> via KTLS for the last few years.  Its what has allowed us to serve >>>>> 100Gb/s on Xeon 2697A cpus for years, and what allows us to serve >>>>> nearly 400Gb/s on AMD servers with NICs which support crypto offload. >>>>> >>>>> I have received a few requests to enable it by default in GENERIC, and >>>>> I'd like to get some opinions. >>>>> >>>>> There are essentially 3 options >>>>> >>>>> 1) Fully enable KTLS by adding 'options KERN_TLS' to GENERIC, and >>>>> flipping kern.ipc.tls.enable=1 >>>>> >>>>> The advantage of this is that it "just works" out of the box for users, >>>>> and for reviewers. >>>>> >>>>> The drawback is that new code is thrust on unsuspecting users, >>>>> potentially exposing them to bugs that we have not found in our >>>>> somewhat limited web serving workload. >>>>> >>>>> 2) Enable KTLS in GENERIC, but leave it turned off by default. >>>>> >>>>> This option allows users to enable ktls without a rebuild of GENERIC, >>>>> but does not enable it by default. So they can enable it if they >>>>> know about it, but are protected from bugs. >>>>> >>>>> The disadvantages of this are that it increases the kernel size >>>>> by ~20K, starts up one thread per core on every amd64 machine, >>>>> and it adds more required tuning to get good performance from FreeBSD. >>>>> >>>>> >>>>> 3) Continue along with KTLS disabled in GENERIC >>>>> >>>>> This is the lowest risk, but adds a higher bar for users wanting >>>>> to use ktls. >>>>> >>>>> >>>>> >>>>> Note that the discussion is focused on amd64 only, as KTLS will >>>>> only work on 64-bit platforms which use a direct map.  It has >>>>> not been tested at all on ppc64, and currently causes a >>>>> panic-at-boot on arm64 due to what are suspected to be problems >>>>> in the arm64 PCB setup. See: >>>>> https://urldefense.com/v3/__https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=247945__;!!OToaGQ!7pQUcHPbxA12vEdKTCp5jkyVxDCqYEJ-BI38kgHqGgweT7yYYG1BVhbDek0_Jc7mqA$ >>>>> >>>>> Drew >>>>> >>>> >>>> Just before this went in, Ed cleaned up the arm64 GENERIC to get it >>>> closer to the amd64 one. Can we enable KERN_TLS in arm64 GENERIC as >>>> well? >>> >>> Well, I also fixed a bug KERN_TLS exposed on arm64 that was gating for >>> this (247945).  I would not be opposed to enabling it on arm64, but I >>> have not personally tested it on arm64.  If someone can verify it works >>> ok on arm64 I'd be happy for it to be enabled there. >>> >> >> Yeah, that's the thing, I have much less confidence in ktls on arm64 >> because we have not run it in production recently.  So I'm personally >> much less confident in enabling it on arm64. >> >> Drew > > Klara has tested it on arm64 fairly heavily, and only found an issue > with OpenSSL, but not found any issues with KERN_TLS itself. > I have no objection if you want to enable it. But since I don't currently have an arm64 test machine, I don't want to be the one to enable it, if that makes sense.. Drew