From owner-freebsd-current@freebsd.org Mon Jan 27 23:28:27 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 46DEE232413 for ; Mon, 27 Jan 2020 23:28:27 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0620.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::620]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4865XW0gtsz4LXy; Mon, 27 Jan 2020 23:28:26 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bre4rz2E8JkvpTwjVtV+pfGtgmqhSp9w+5kOwwaPVN8oH1eHwA1/9I6imhlvfnIKYA9PC4sKTj1cztpB0O/8ibgyHeBQkjR//a5TUPhUTie4orz3Y25r9PdCNlTvZgaJGKxU+bi6+wPThbb25HB9KB7zcxadhC5f122is4h3bffsJJNlKrk/U+LWdnd8pCrLCHRV0wBeBIw2K3NXBENS+75B32msjyTu1BvoLVK3OekfpKjxLdbBN+8yYBz8KopqfHTimdZFBoKU9VLIygx+Ys0C8cp3eO7UGUCT3qcuox36JtHuxnyWzRorZHPCQRiiHamWcze8z1mQv6ppAqJrDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XjIwH66pSMyCcEu7+7N/aowVthFe1gl31qgJYUoBk7Q=; b=atx3Qo3bgiPhGmYz3hQFci45DCNAB5pxt+OQaeYy/3+nh89IjzwHsMtRHbct5v6DFebXqpeSjpnKUQNN7nZoHpXDXbprleNL/fc0gZRu/sVmr7Z181VDyNQBtEEQxrjKhX96O7IijE02/yzjeAE1JiMgwBYImiztBSIMv66sCvm/r4Uc/i4uNNSztudBrGi1lw4Au8P7P0U/37gVjZuWBX7KU88ZyU6Yh3w4bN4t7QDT301tQP8KC6XA2uglsePdJz919QbXqhVEGAVdPHb3ucitLh0GMIHWbdt3XS3OpAmT1DtmZwVAgy1iUTQltww0TW9eqbbe+90ydn3gcARMTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB2033.CANPRD01.PROD.OUTLOOK.COM (52.132.71.161) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2665.23; Mon, 27 Jan 2020 23:28:24 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::6588:45c3:4892:f98]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::6588:45c3:4892:f98%7]) with mapi id 15.20.2665.025; Mon, 27 Jan 2020 23:28:24 +0000 From: Rick Macklem To: John Baldwin , "freebsd-current@FreeBSD.org" Subject: Re: how to use the ktls Thread-Topic: how to use the ktls Thread-Index: AQHVxa2HeRfmo36hWEyrGcMaBhE88KfhEeoAgBxnlpmAAOgvAIAA4IKR Date: Mon, 27 Jan 2020 23:28:24 +0000 Message-ID: References: <5be57c87-90fe-fcbe-ea37-bdb1bcff2da8@FreeBSD.org> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f8380221-8255-45ff-0dd1-08d7a3809b04 x-ms-traffictypediagnostic: YQBPR0101MB2033: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6790; x-forefront-prvs: 02951C14DC x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(376002)(346002)(136003)(39860400002)(366004)(199004)(189003)(5660300002)(55016002)(9686003)(110136005)(478600001)(8676002)(81156014)(81166006)(786003)(6506007)(186003)(8936002)(7696005)(316002)(26005)(2906002)(71200400001)(66476007)(66556008)(64756008)(66446008)(450100002)(86362001)(91956017)(76116006)(52536014)(66946007)(33656002); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB2033; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: iju2krU/IE5n9ginlN9mARJJ9qMBWTD4BH2vycM6JqSGpy/4ymX3ihyv8nx4lUlisHwg/nFo36L6Li89prBg3hJLml1SadrM13ZuDbuaPjAUfBu7omGArZXZWTURMxddHbMgUD/toh2UHRD8Y1vJ/LEvfcbc89YF8Gyu/08/Vtf004lCof3qJClT6Ivxv84WtSOkWkrkSL5sw9zoNKbOB+lDpb/5H/bxOuqhLXRyHSsoL7RoaKLWdiOX/K49evJAe/v0ta65wt643IS/gYa3YwM/plM/G/wpSm12WUdXpAg3mm71gFM8ujnMyUPxBOeA3z546aUwB/EcyH8Ho46kMei78HOgaJOSnIufx733nD/r/CGYn17s6nPqakOhKxc9X9QyKSxZ2FvS94EyQGWpmDmMhD34qL9kgHmITHbNMzcUhj/D28wIFqM8QH7dkw6V x-ms-exchange-antispam-messagedata: OSwp1cjb6I1/ZuJzAbGbptsVrcrgqLXbWDj5XUrC/K5I1EY7iOAtyrhuh6buDfp3i3gtezFM/R7eSb9lVjGBormhYQpirUfv1De+rLh46qkEsGi/lHVapXxeY5EniFVAdngxQpORsjLvhMmH5KME2w== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: f8380221-8255-45ff-0dd1-08d7a3809b04 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jan 2020 23:28:24.6810 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: W0OlR7SpsuD5pM2HbUNEt/iWCPezbQ+LvS/uB20HPKDLh+jkZZzyIbGpWzYjnJwXETxZik9Xj9HwC/HWNGi7Sg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB2033 X-Rspamd-Queue-Id: 4865XW0gtsz4LXy X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Jan 2020 23:28:27 -0000 John Baldwin wrote:=0A= >On 1/26/20 8:08 PM, Rick Macklem wrote:=0A= >> John Baldwin wrote:=0A= >> [stuff snipped]=0A= >>> Hmmm, this might be a fair bit of work indeed.=0A= >>>=0A= >>> Right now KTLS only works for transmit (though I have some WIP for rece= ive).=0A= >>>=0A= >>> KTLS does assumes that the initial handshake and key negotiation is han= dled by=0A= >>> OpenSSL. OpenSSL uses custom setockopt() calls to tell the kernel whic= h=0A= >>> session keys to use.=0A= >>>=0A= >>> I think what you would want to do is use something like OpenSSL_connect= () in=0A= >>> userspace, and then check to see if KTLS "worked". If it did, you can = tell=0A= >>> the kernel it can write to the socket directly, otherwise you will have= to=0A= >>> bounce data back out to userspace to run it through SSL_write() and hav= e=0A= >>> userspace do SSL_read() and then feed data into the kernel.=0A= >>>=0A= >>> The pseudo-code might look something like:=0A= >>>=0A= >>> SSL *s;=0A= >>>=0A= >>> s =3D SSL_new(...);=0A= >>>=0A= >>> /* fd is the existing TCP socket */=0A= >>> SSL_set_fd(s, fd);=0A= >>> OpenSSL_connect(s);=0A= >>> if (BIO_get_ktls_send(SSL_get_wbio(s)) {=0A= >>> /* Can use KTLS for transmit. */=0A= >>> }=0A= >>> if (BIO_get_ktls_recv(SSL_get_rbio(s)) {=0A= >>> /* Can use KTLS for receive. */=0A= >>> }=0A= >>=0A= >> So, I've been making some progress. The first stab at the daemons that d= o the=0A= >> handshake are now on svn in base/projects/nfs-over-tls/usr.sbin/rpctlscd= and=0A= >> rpctlssd.=0A= >>=0A= >> A couple of questions...=0A= >> 1 - I haven't found BIO_get_ktls_send() or BIO_get_ktls_recv(). Are they= in some=0A= >> different library?=0A= >=0A= >They only existing currently in OpenSSL master (which will be OpenSSL 3.0.= 0 when it=0A= >is released). I have some not-yet-tested WIP changes to backport those ch= anges into=0A= >the base OpenSSL, but it will also add overhead to future OpenSSL imports = perhaps,=0A= >so it is something I need to work with secteam@ on to decide if it's viabl= e once I=0A= >have a tested PoC.=0A= >=0A= >I will try to at least provide a patch to the security/openssl port to add= a KTLS=0A= >option "soon" that you could use for testing.=0A= John, I wouldn't worry much about this.=0A= The calls are currently #ifdef notnow in the daemon and I'm fine with that.= =0A= SSL_connect() has returned 1, so the daemon knows that the handshake is com= plete and=0A= the kernel code that did the upcall to the daemon can check for KERN_TLS su= pport.=0A= =0A= >> 2 - After a successful SSL_connect(), the receive queue for the socket h= as 478bytes=0A= >> of stuff in it. SSL_read() seems to know how to skip over it, but = I haven't=0A= >> figured out a good way to do this. (I currently just do a recv(..4= 78,0) on the=0A= >> socket.)=0A= >> Any idea what to do with this? (Or will the receive side of the kt= ls figure out=0A= >> how to skip over it?)=0A= >=0A= >I don't know yet. :-/ With the TOE-based TLS I had been testing with, thi= s doesn't=0A= >happen because the NIC blocks the data until it gets the key and then it's= always=0A= >available via KTLS. With software-based KTLS for RX (which I'm going to s= tart=0A= >working on soon), this won't be the case and you will potentially have som= e data=0A= >already ready by OpenSSL that needs to be drained from OpenSSL before you = can=0A= >depend on KTLS. It's probably only the first few messsages, but I will ne= ed to figure=0A= >out a way that you can tell how much pending data in userland you need to = read via=0A= >SSL_read() and then pass back into the kernel before relying on KTLS (it w= ould just=0A= >be a single chunk of data after SSL_connect you would have to do this for)= .=0A= Well, SSL_read() doesn't return these bytes. I think it just throws them aw= ay.=0A= =0A= I have a simple test client/server where the client sends "HELLO THERE" to = the=0A= server and the server replies "GOODBYE" after the SSL_connect()/SSL_accept(= )=0A= has been done.=0A= --> If the "HELLO THERE"/"GOODBYE" is done with SSL_write()/SSL_read() it w= orks.=0A= however=0A= --> If the above is done with send()/recv(), the server gets the "HELLO THE= RE", but=0A= the client gets 485bytes of data, where the last 7 are "GOODBYE".=0A= --> If I do a recv( ..475..) in the client right after SSL_connect() = it works ok.=0A= =0A= I do this for testing, since it can then do the NFS mount (unencrypted).=0A= =0A= Looking inside SSL_read() I found:=0A= *=0A= 1742 * If we are a client and haven't received the ServerHello etc the= n we=0A= 1743 * better do that=0A= 1744 */=0A= 1745 ossl_statem_check_finish_init(s, 0);=0A= =0A= but all ossl_statem_check_finish_init(s, 0); seems to do is set a variable = "in_init =3D 1".=0A= Then it calls the method->read() function, which I'd guess is in the crypto= code=0A= and it seems to get rid of these bytes. (I hate trying to figure out what c= alls what=0A= for object oriented code;-).=0A= =0A= Btw. SSL_is_finished_init() returns 1 right after the SSL_connect(), so it = seems to=0A= think the hadnshake is done, although it hasn't read these 478 bytes from t= he server.=0A= =0A= Anyhow, I'd guess the TOE engine knows how to do get rid of this stuff like= SSL_read()=0A= does?=0A= =0A= >> I'm currently testing with a kernel that doesn't have options KERN_TLS a= nd=0A= >> (so long as I get rid of the 478 bytes), it then just does unencrypted R= PCs.=0A= >>=0A= >> So, I guess the big question is.... can I get access to your WIP code fo= r KTLS=0A= >> receive? (I have no idea if I can make progress on it, but I can't do a = lot more=0A= >> before I have that.)=0A= >=0A= >The WIP only works right now if you have a Chelsio T6 NIC as it uses the T= 6's TCP=0A= >offload engine to do TLS. If you don't have that gear, ping me off-list. = It=0A= >would also let you not worry about the SSL_read case for now for initial t= esting.=0A= I may contact you in April about this.=0A= Since you've noted above that you haven't started the software version, I m= ight try=0A= coming up with something and will let you know if I do.=0A= =0A= Thanks, rick=0A=