Date: Mon, 27 Jan 2020 23:28:24 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: John Baldwin <jhb@FreeBSD.org>, "freebsd-current@FreeBSD.org" <freebsd-current@FreeBSD.org> Subject: Re: how to use the ktls Message-ID: <YQBPR0101MB1427789FF1B79EC532737D20DD0B0@YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <d29b4343-b8e0-88e7-9076-d2a94eb45386@FreeBSD.org> References: <YQBPR0101MB142760894682CA3663CB53BDDD3F0@YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM> <5be57c87-90fe-fcbe-ea37-bdb1bcff2da8@FreeBSD.org> <YQBPR0101MB1427F6950084C3CA30713A75DD080@YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM>, <d29b4343-b8e0-88e7-9076-d2a94eb45386@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
John Baldwin wrote:=0A=
>On 1/26/20 8:08 PM, Rick Macklem wrote:=0A=
>> John Baldwin wrote:=0A=
>> [stuff snipped]=0A=
>>> Hmmm, this might be a fair bit of work indeed.=0A=
>>>=0A=
>>> Right now KTLS only works for transmit (though I have some WIP for rece=
ive).=0A=
>>>=0A=
>>> KTLS does assumes that the initial handshake and key negotiation is han=
dled by=0A=
>>> OpenSSL. OpenSSL uses custom setockopt() calls to tell the kernel whic=
h=0A=
>>> session keys to use.=0A=
>>>=0A=
>>> I think what you would want to do is use something like OpenSSL_connect=
() in=0A=
>>> userspace, and then check to see if KTLS "worked". If it did, you can =
tell=0A=
>>> the kernel it can write to the socket directly, otherwise you will have=
to=0A=
>>> bounce data back out to userspace to run it through SSL_write() and hav=
e=0A=
>>> userspace do SSL_read() and then feed data into the kernel.=0A=
>>>=0A=
>>> The pseudo-code might look something like:=0A=
>>>=0A=
>>> SSL *s;=0A=
>>>=0A=
>>> s =3D SSL_new(...);=0A=
>>>=0A=
>>> /* fd is the existing TCP socket */=0A=
>>> SSL_set_fd(s, fd);=0A=
>>> OpenSSL_connect(s);=0A=
>>> if (BIO_get_ktls_send(SSL_get_wbio(s)) {=0A=
>>> /* Can use KTLS for transmit. */=0A=
>>> }=0A=
>>> if (BIO_get_ktls_recv(SSL_get_rbio(s)) {=0A=
>>> /* Can use KTLS for receive. */=0A=
>>> }=0A=
>>=0A=
>> So, I've been making some progress. The first stab at the daemons that d=
o the=0A=
>> handshake are now on svn in base/projects/nfs-over-tls/usr.sbin/rpctlscd=
and=0A=
>> rpctlssd.=0A=
>>=0A=
>> A couple of questions...=0A=
>> 1 - I haven't found BIO_get_ktls_send() or BIO_get_ktls_recv(). Are they=
in some=0A=
>> different library?=0A=
>=0A=
>They only existing currently in OpenSSL master (which will be OpenSSL 3.0.=
0 when it=0A=
>is released). I have some not-yet-tested WIP changes to backport those ch=
anges into=0A=
>the base OpenSSL, but it will also add overhead to future OpenSSL imports =
perhaps,=0A=
>so it is something I need to work with secteam@ on to decide if it's viabl=
e once I=0A=
>have a tested PoC.=0A=
>=0A=
>I will try to at least provide a patch to the security/openssl port to add=
a KTLS=0A=
>option "soon" that you could use for testing.=0A=
John, I wouldn't worry much about this.=0A=
The calls are currently #ifdef notnow in the daemon and I'm fine with that.=
=0A=
SSL_connect() has returned 1, so the daemon knows that the handshake is com=
plete and=0A=
the kernel code that did the upcall to the daemon can check for KERN_TLS su=
pport.=0A=
=0A=
>> 2 - After a successful SSL_connect(), the receive queue for the socket h=
as 478bytes=0A=
>> of stuff in it. SSL_read() seems to know how to skip over it, but =
I haven't=0A=
>> figured out a good way to do this. (I currently just do a recv(..4=
78,0) on the=0A=
>> socket.)=0A=
>> Any idea what to do with this? (Or will the receive side of the kt=
ls figure out=0A=
>> how to skip over it?)=0A=
>=0A=
>I don't know yet. :-/ With the TOE-based TLS I had been testing with, thi=
s doesn't=0A=
>happen because the NIC blocks the data until it gets the key and then it's=
always=0A=
>available via KTLS. With software-based KTLS for RX (which I'm going to s=
tart=0A=
>working on soon), this won't be the case and you will potentially have som=
e data=0A=
>already ready by OpenSSL that needs to be drained from OpenSSL before you =
can=0A=
>depend on KTLS. It's probably only the first few messsages, but I will ne=
ed to figure=0A=
>out a way that you can tell how much pending data in userland you need to =
read via=0A=
>SSL_read() and then pass back into the kernel before relying on KTLS (it w=
ould just=0A=
>be a single chunk of data after SSL_connect you would have to do this for)=
.=0A=
Well, SSL_read() doesn't return these bytes. I think it just throws them aw=
ay.=0A=
=0A=
I have a simple test client/server where the client sends "HELLO THERE" to =
the=0A=
server and the server replies "GOODBYE" after the SSL_connect()/SSL_accept(=
)=0A=
has been done.=0A=
--> If the "HELLO THERE"/"GOODBYE" is done with SSL_write()/SSL_read() it w=
orks.=0A=
however=0A=
--> If the above is done with send()/recv(), the server gets the "HELLO THE=
RE", but=0A=
the client gets 485bytes of data, where the last 7 are "GOODBYE".=0A=
--> If I do a recv( ..475..) in the client right after SSL_connect() =
it works ok.=0A=
=0A=
I do this for testing, since it can then do the NFS mount (unencrypted).=0A=
=0A=
Looking inside SSL_read() I found:=0A=
*=0A=
1742 * If we are a client and haven't received the ServerHello etc the=
n we=0A=
1743 * better do that=0A=
1744 */=0A=
1745 ossl_statem_check_finish_init(s, 0);=0A=
=0A=
but all ossl_statem_check_finish_init(s, 0); seems to do is set a variable =
"in_init =3D 1".=0A=
Then it calls the method->read() function, which I'd guess is in the crypto=
code=0A=
and it seems to get rid of these bytes. (I hate trying to figure out what c=
alls what=0A=
for object oriented code;-).=0A=
=0A=
Btw. SSL_is_finished_init() returns 1 right after the SSL_connect(), so it =
seems to=0A=
think the hadnshake is done, although it hasn't read these 478 bytes from t=
he server.=0A=
=0A=
Anyhow, I'd guess the TOE engine knows how to do get rid of this stuff like=
SSL_read()=0A=
does?=0A=
=0A=
>> I'm currently testing with a kernel that doesn't have options KERN_TLS a=
nd=0A=
>> (so long as I get rid of the 478 bytes), it then just does unencrypted R=
PCs.=0A=
>>=0A=
>> So, I guess the big question is.... can I get access to your WIP code fo=
r KTLS=0A=
>> receive? (I have no idea if I can make progress on it, but I can't do a =
lot more=0A=
>> before I have that.)=0A=
>=0A=
>The WIP only works right now if you have a Chelsio T6 NIC as it uses the T=
6's TCP=0A=
>offload engine to do TLS. If you don't have that gear, ping me off-list. =
It=0A=
>would also let you not worry about the SSL_read case for now for initial t=
esting.=0A=
I may contact you in April about this.=0A=
Since you've noted above that you haven't started the software version, I m=
ight try=0A=
coming up with something and will let you know if I do.=0A=
=0A=
Thanks, rick=0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQBPR0101MB1427789FF1B79EC532737D20DD0B0>