From owner-freebsd-current@freebsd.org Thu Mar 11 15:42:58 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 681F257996A for ; Thu, 11 Mar 2021 15:42:58 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660071.outbound.protection.outlook.com [40.107.66.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DxCrd3V8kz3DYP; Thu, 11 Mar 2021 15:42:57 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CrPnGx0ea0v0CdWUcQIBZd6Xjlhtd5SghcRAKDXN0XGaAWxaNMRExlIW+ToKTXoLZisV0qXP/S2u/T3K/Q41G7xXJfeEo+FK2wRYgs4rFb+I5nEK80CLwoaMi1Xgqq0iouzDYaFJL43l7OHpwYl0KRcZmSK+4I333n5ZNXThKqq7xyK3a9naTSRzMuraujKmLY367RnjAZB8v7/NotpABav4Q0WRtqM88/1yDuJP/ex7DiwgHWIxoSH5AwOEP36I3ier4+t9LFyLPTlcGIbA4YLoUWZxUupjk7wy5RaCC1my1DodegBrSPCRjlt5WLihUBqA3NNAXJi4f1eOQ1BbSA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ksvkI4oT2BuxxofPB7eO+kYnBsEhhRCWPKgrMB6biO8=; b=mAHt15SmEupCb6ThaDPsyz1mADKycap+2Dlx8Zv9LXmAVMh1yfbIK/cr8NliOF24b9gzG95KLn8WLqN+B8gdefqlbMJ2VTyYCrrxP43BzQ8z71cGFkQBkng/jtxx9N+I7e3ziOiL7Yu+PWB1LDl6R/ZakzqrWoGNUHX85SoAE1E6/FejK3qdH/puduiUesCXsiL2CfH2bpX7GEy1x7U589U1BpFUcJ3vdLtb2ReBfJFJA79E2utqoI9k714K/xoNpDr91RXbhCmj62nz9JqJZ4uYTwmY5hcUPT13pzrAmp1uT21ZTSwNfOwHwmxXTMLxWhF26MDKVyWjGI6nYmEqbA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ksvkI4oT2BuxxofPB7eO+kYnBsEhhRCWPKgrMB6biO8=; b=Qu1W9LEayl/XJsB+wpozst/E3fUlb5v36nPFo5xiFt+fD4MWXTE9xDo2YfFK2Fyn3PkE6lMsfloQyRuNwlqHoqZfEUo1NAMaQqIHvVp2mVtxyhxDGmUOFTBC2DHDg1NpRb0IW7gGB2nTLTjb7EV77h5os6mfjC/MI24hVUdI3Yw5XPWUwJckjUFd7+PcICreuKulrX7Eo6tLEiwCBDObQWfa6JoAZaCZzji3OB9QG3jvvF6AabZ8HleUtfeQ8aIBi111vo0nxH3nTvi3grHC8INm8Q9YSjg/YUTsLarFjUKwIozSD0Gvgqj6cmAf1Dc73AhAQIvIOq2fF6+WdUbo2A== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQXPR01MB3688.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:4e::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3868.31; Thu, 11 Mar 2021 15:42:55 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3890.041; Thu, 11 Mar 2021 15:42:55 +0000 From: Rick Macklem To: Alan Somers , Benjamin Kaduk CC: FreeBSD CURRENT Subject: Re: Getting started with ktls Thread-Topic: Getting started with ktls Thread-Index: AQHXFgwYWcBrnpJjzEOOvEeMKEi/Wap977EAgAAM4QCAACDHgIAACzeAgADCPHs= Date: Thu, 11 Mar 2021 15:42:55 +0000 Message-ID: References: <20210311003136.GM56617@kduck.mit.edu> <20210311031501.GP56617@kduck.mit.edu>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: cf876a6c-39c3-4642-02da-08d8e4a456bb x-ms-traffictypediagnostic: YQXPR01MB3688: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: xcpK8zctCSlM7wAF/YgPpi2FvG2Qmh1Y//G2A5i04Pv99Ex1fR6hI2KL3rdOpGNfJzHwW9tBrKq4wEAgrXY0eayJCB55zL0PlnBqFNzPg/Eda7aihmFzwD0b04edI7H2QE06R0UM+WbyFtsjB1eo5ROCcug0b8oHtZAnvsHHzYzzSxoMAWPq5+ekQw3Arj0e/brlEmjwlujWpzebLmK+m+kd2QVbh0kvvemIPtexFGhN2Eo8IQU3WeHufwPYKKZHkmNrncaVTnSmcSC6RNbKc2yYTEmacgm9NnfoW//gqbb2u4u5anXD2+ivyGBRX0qw+GgswyLmNDIwtyOCJa1bDmopHdSH+a3dIrFotA1SqSJ2VRkEGa4AOGKAFYkCrG/mF2Xiw/d0CKHK+OoJQeq7I4nrd35nEChdPmTppvto0A4egPb1Z1fBJWPmYU7M0FslPei0DXOBPvTnKJx77lOZulzZ+L7Wu1VeilVA1GghmcqiRI2/WYNJWGPmWJoeQuF4PpvvhXFGFCKRaTrcCV0LFUHBLeiGjxzOdFVm1zV+c3V2KvPGwpSACIpR4BTrqTBDuCGY5Rxyp+aVbBnmzbpNjcAe+YXmAuE2J/MY4VaEiQ4= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(376002)(346002)(396003)(366004)(39860400002)(136003)(9686003)(5660300002)(55016002)(316002)(786003)(8936002)(91956017)(76116006)(66476007)(8676002)(66446008)(3480700007)(478600001)(4326008)(52536014)(966005)(66946007)(7696005)(110136005)(64756008)(66556008)(2906002)(83380400001)(186003)(86362001)(53546011)(6506007)(33656002)(71200400001); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: =?us-ascii?Q?MhP7E6+2audbhwVktFmXLYFkR6Za+lafMhBwhuBZWQuKhR/Gkgg/1AUOSHTF?= =?us-ascii?Q?sgu6xEjDPMi7W0zS3lwilTpfwwIfRkxa+RgeCDjSJEGh+oVIUwVdkPAGrJ9H?= =?us-ascii?Q?LhB9dJsdig/f4L8z/s1fuPrwbkclwOYEPBgQw5/fCMEGSyb7VsPHU1zplsY9?= =?us-ascii?Q?8eN5Xk6MXHCTRJ4/4kcF7XsQ+qWgs8e7DJZCCltQru99WyZ7cE04ALCWSyav?= =?us-ascii?Q?zaQPVfvpjd3oOcr58PCGzHJ9EEwfEtYcUVUYc55lUIJ+bnV1+kIsoWxYLN40?= =?us-ascii?Q?gLwDL3Qvor5G6VrpfA0qpHr+ODkOcdxsTW7sUZxh5vEyfl8oKpxmwxXzzWNn?= =?us-ascii?Q?7LDTaJu1KrC3rovIg9jlwac6vwtflKKLzLu6iMTuH59SQiraRe98Ci8eZasf?= =?us-ascii?Q?x2kN/UIEIKzy/LPop27EvTJmcvjnk/mQmakmXhjixVZlx/hsSs2JQ/vK0sY9?= =?us-ascii?Q?aHl+NGFd/tM+g1gRoPB27bkDnOymZ0VkCucCHpl+9H6s+NXKlVMl8Xke3q5C?= =?us-ascii?Q?kcIfJ71QKpOEv7tNbaWLJ9UN94CCg6MOWRnXg6AJxWK6Fyw9JTRMlLTqkW8m?= =?us-ascii?Q?WU8I/auG+s8to6ydugxw9UufUWudl9sHqWmiy0uwUPl5rA/JXEkbxD3EPwq9?= =?us-ascii?Q?yxvCOSkI/SRMFKzmhL9Y/SmulHr7lUC0xlKvGwqE5aXQO3t9ahu9qeQ/R7Dm?= =?us-ascii?Q?ZQMxk3Bl2uX0S99OmQ9mw7YcP7OMSRnOLZYnxnqEw1/LpJCkMmr4ItpUe+zu?= =?us-ascii?Q?7+5SCQH4Ptl5jXovZewQNhIdKhMkoZEDIsebNaJY7bIp03Tz2ivfEJqFU1HQ?= =?us-ascii?Q?XRbVSYBdZ8iKLCjqXXJTMGgcES6+T6S/1bncJfvW/lLx1glD3sdFOwGzM37E?= =?us-ascii?Q?60jCmEYT/mxMCcU7nnCfu8MNkPFOjCycuoZ94ks8+Sw4YH+PGK+b7mBg5/nA?= =?us-ascii?Q?rcMD0HMqxpOIZKw5xxk7LvTu01T7RK8WXYpS6yE0dBS21KTTGjH/tXOGHIbt?= =?us-ascii?Q?ntAv+vOp1Th6UCmfU8GqfzhV9ka73QLO0xY8LnuN04zxy2UQkO3vmS3mHrJZ?= =?us-ascii?Q?fnpiVBg3snhyzbFUmTXvpZvP4TtPgv/L04P0yXgs4DVVoKguDkSx8sqRXr0i?= =?us-ascii?Q?1qI2OhcmV7MRvdGqTyZW+yt0j6cH89AAKo4dzka0hgiovbvyYFBMQGWWiZUy?= =?us-ascii?Q?6XxDBoTDxIpeYGQ/xWT6VeWHdINYXJojXCMOHrayg24x1lYUWMN4Fc5LnF8A?= =?us-ascii?Q?+R7Nwgko/DKUOPO29cc29TG6N276i+PmY3j06WRFv2cyup35QTbT4d+PPod3?= =?us-ascii?Q?W0w4pMtugJNHn4A+IeTulj/ifSnCSDifUvWTyVcPRETA8jsI7X+VVxwqyZx0?= =?us-ascii?Q?3yVJ9pw=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: cf876a6c-39c3-4642-02da-08d8e4a456bb X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Mar 2021 15:42:55.2681 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: PZeLQeWFuoo94omGNIks/xcu7AgKLVlRTIg5a2NeNX7khwiutSDsZJpcnArpuTinC9B6uZEjrmoEwilCZF5jkA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3688 X-Rspamd-Queue-Id: 4DxCrd3V8kz3DYP X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=Qu1W9LEa; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.71 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.67 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[40.107.66.71:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[40.107.66.71:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.71:from]; NEURAL_HAM_SHORT(-0.67)[-0.672]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-current]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.71:from] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2021 15:42:58 -0000 I'm going to cheat and top post (the discussion looks pretty convoluted). - The kernel must be built with "options KERN_TLS" - OpenSSL must be built with KTLS enabled - These two sysctls need to be set to 1 kern.ipc.tls.enable kern.ipc.mb_use_ext_pgs then it all happens "behind the curtain" in the OpenSSL libraries. To find out if it is enabled, do the following after the handshake. (SSL_connect() or SSL_accept() or ??) ret =3D BIO_get_ktls_send(SSL_get_wbio(ssl)); if (ret !=3D 0) ret =3D BIO_get_ktls_recv(SSL_get_rbio(ssl)); if (ret !=3D 0) /* KTLS enabled for both send and recv. */ The calls return non-zero if it is enabled for that direction. You'll need it to use TLS 1.2 if you want both directions to work at this time. (The code is in usr.sbin/rpc.tlsclntd and usr.sbin/rpc.tlsservd. Much easier to look at than man pages imho.) --> Now you can sosend()/soreceive() on the socket in the kernel. If your data is already in mbufs, then they must be M_EXTPG mbufs. There is a function that copies an mbuf chain into M_EXTPG mbufs, but I can't remember what I called it. rick ________________________________________ From: owner-freebsd-current@freebsd.org = on behalf of Alan Somers Sent: Wednesday, March 10, 2021 10:55 PM To: Benjamin Kaduk Cc: FreeBSD CURRENT Subject: Re: Getting started with ktls CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo= guelph.ca On Wed, Mar 10, 2021 at 8:15 PM Benjamin Kaduk wrote: > On Wed, Mar 10, 2021 at 06:17:42PM -0700, Alan Somers wrote: > > On Wed, Mar 10, 2021 at 5:31 PM Benjamin Kaduk wrote: > > > > > On Wed, Mar 10, 2021 at 05:18:24PM -0700, Alan Somers wrote: > > > > I'm trying to make ktls work with "zfs send/recv" to substantially > reduce > > > > the CPU utilization of applications like zrepl. But I have a few > > > questions: > > > > > > > > * ktls(4)'s "Transmit" section says "Once TLS transmit is enabled b= y > a > > > > successful set of the TCP_TXTLS_ENABLE socket option", but the > "Supported > > > > Libraries" section says "Applications using a supported library > should > > > > generally work with ktls without any changes". These sentences see= m > to > > > be > > > > contradictory. I think it means that the TCP_TXTLS_ENABLE option i= s > > > > necessary, but OpenSSL sets it automatically? > > > > > > Yes, OpenSSL sets it automatically for the builtin socket and > connection > > > BIO classes. Applications using other BIO classes will need to do > things > > > manually (or implement the appropriate _ctrl() parameters for their B= IO > > > class). > > > > > > > * When using OpenSSL, the library will automatically call > setsockopt(_, > > > > TCP_TXTLS_ENABLE). But it swallows the error, if any. How is an > > > > application to tell if ktls is enabled on a particular socket or > OpenSSL > > > > session? > > > > > > IIRC the lack of answer for this is part of why upstream OpenSSL > doesn't > > > have specific KTLS tests enabled in the automated test suite. > > > > > > > getsockopt(_. TCP_TXTLS_ENABLE) returns ENOPROTOOPT. Is there any reas= on > > why it's not implemented? That might be the easiest way to check for t= he > > ktls status of an individual socket. > > I think that's probably more of a question for jhb than me. I don't know > of a reason why not, but I do know that there is some desire to keep the > functionality that openssl exposes roughly compatible between linux and > FreeBSD KTLS. I don't know whether Linux has something similar. > > > > > > > > > > * From experiment, I can see that OpenSSL attempts to set > > > > TCP_TXTLS_ENABLE. But it doesn't try to set TCP_RXTLS_ENABLE. Why > not? > > > > From reading ktls_start and ossl_statem_server_post_work, it looks > like > > > > maybe a single socket cannot have ktls enabled for both sending and > > > > receiving at the same time. Is that true? > > > > > > No. They just get enabled separately, since change_cipher_state() is > > > called separately for read and write transitions. > > > > > > > Apologies if I'm too ignorant, but what is a transition in SSL-speak? > This > > is my first attempt at any kind of SSL programming. What I know from > > ktrace is that TCP_RXTLS_ENABLE never gets set. > > Sorry! I'm pretty conversant with this stuff (I'm the security area > director that is responsible for the IETF TLS working group) and don't > always target the right level. Basically, for a decent encrypting protoc= ol > you want different encrytion keys for the read and write direction > (whichever peer you are), and the TLS (1.3) handshake has a multi-stage k= ey > hierarchy to try to encrypt as much of it as possible. So, for example, > the client will need to update it's encryption key for reading once it > reads the ServerHello (and before reading the Encrypted Extensions) > message, even though the keys the client uses for writing don't change at > that time. Internally, OpenSSL implements this "transition" of key > material with a change_cipher_state() abstraction, that takes a flags > argument (`which`). The flags indicate which set of keys to update, and > which direction (read or write). So, by my read of the code, what's > *supposed* to happen is that we call: > > if (BIO_set_ktls(bio, &crypto_info, which & SSL3_CC_WRITE)) > > And if SSL3_CC_WRITE is set, that translates to calling BIO_set_ktls() wi= th > an `is_txt` value that evaluates to true; otherwise, `is_txt` is false, > which corresponds to the RX case that you're failing to see happen. > > Just to get the boring stuff out of the way: what version of openssl are > you testing against, and did you verify that OPENSSL_NO_KTLS_RX is not > defined when ktls_start() is being compiled (so that the setsockopt(fd, > IPPROTO_TCP, TCP_RXTLS_ENABLE, .) is compiled in at all)? > > Thanks, > > Ben > I'm using the OpenSSL that's in base in 14.0-CURRENT: 1.1.1j-freebsd . I haven't recompiled the code to check whether OPENSSL_NO_KTLS_RX is defined, but it sure looks like it shouldn't be, based on my reading of the source. -Alan _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"