From owner-freebsd-current@freebsd.org Thu Mar 11 03:15:10 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 42E8C5AD4EB for ; Thu, 11 Mar 2021 03:15:10 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DwvFm3gsBz3jCm; Thu, 11 Mar 2021 03:15:08 +0000 (UTC) (envelope-from kaduk@mit.edu) Received: from kduck.mit.edu ([24.16.140.251]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 12B3F1JK023312 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 10 Mar 2021 22:15:06 -0500 Date: Wed, 10 Mar 2021 19:15:01 -0800 From: Benjamin Kaduk To: Alan Somers Cc: FreeBSD CURRENT Subject: Re: Getting started with ktls Message-ID: <20210311031501.GP56617@kduck.mit.edu> References: <20210311003136.GM56617@kduck.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: 4DwvFm3gsBz3jCm X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of kaduk@mit.edu designates 18.9.28.11 as permitted sender) smtp.mailfrom=kaduk@mit.edu X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_VERYGOOD(0.00)[18.9.28.11:from]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:18.9.28.0/24]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[mit.edu]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[18.9.28.11:from]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3, ipnet:18.9.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-current]; RECEIVED_SPAMHAUS_PBL(0.00)[24.16.140.251:received] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2021 03:15:10 -0000 On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote: > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > > I'm trying to make ktls work with "zfs send/recv" to substantially reduce > > > the CPU utilization of applications like zrepl. But I have a few > > questions: > > > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled by a > > > successful set of the TCP_TXTLS_ENABLE socket option", but the "Supported > > > Libraries" section says "Applications using a supported library should > > > generally work with ktls without any changes". These sentences seem to > > be > > > contradictory. I think it means that the TCP_TXTLS_ENABLE option is > > > necessary, but OpenSSL sets it automatically? > > > > Yes, OpenSSL sets it automatically for the builtin socket and connection > > BIO classes. Applications using other BIO classes will need to do things > > manually (or implement the appropriate _ctrl() parameters for their BIO > > class). > > > > > * When using OpenSSL, the library will automatically call setsockopt(_, > > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > > application to tell if ktls is enabled on a particular socket or OpenSSL > > > session? > > > > IIRC the lack of answer for this is part of why upstream OpenSSL doesn't > > have specific KTLS tests enabled in the automated test suite. > > > > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reason > why it's not implemented? That might be the easiest way to check for the > ktls status of an individual socket. I think that's probably more of a question for jhb than me. I don't know of a reason why not, but I do know that there is some desire to keep the functionality that openssl exposes roughly compatible between linux and FreeBSD KTLS. I don't know whether Linux has something similar. > > > > > > * From experiment, I can see that OpenSSL attempts to set > > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why not? > > > From reading ktls_start and ossl_statem_server_post_work, it looks like > > > maybe a single socket cannot have ktls enabled for both sending and > > > receiving at the same time. Is that true? > > > > No. They just get enabled separately, since change_cipher_state() is > > called separately for read and write transitions. > > > > Apologies if I'm too ignorant, but what is a transition in SSL-speak? This > is my first attempt at any kind of SSL programming. What I know from > ktrace is that TCP_RXTLS_ENABLE never gets set. Sorry! I'm pretty conversant with this stuff (I'm the security area director that is responsible for the IETF TLS working group) and don't always target the right level. Basically, for a decent encrypting protocol you want different encrytion keys for the read and write direction (whichever peer you are), and the TLS (1.3) handshake has a multi-stage key hierarchy to try to encrypt as much of it as possible. So, for example, the client will need to update it's encryption key for reading once it reads the ServerHello (and before reading the Encrypted Extensions) message, even though the keys the client uses for writing don't change at that time. Internally, OpenSSL implements this "transition" of key material with a change_cipher_state() abstraction, that takes a flags argument (`which`). The flags indicate which set of keys to update, and which direction (read or write). So, by my read of the code, what's *supposed* to happen is that we call: if (BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE)) And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() with an `is_txt` value that evaluates to true; otherwise, `is_txt` is false, which corresponds to the RX case that you're failing to see happen. Just to get the boring stuff out of the way: what version of openssl are you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not defined when ktls_start() is being compiled (so that the setsockopt(fd, IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)? Thanks, Ben