Date: Sat, 27 Feb 2021 00:11:09 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Warner Losh <imp@bsdimp.com>, "Rodney W. Grimes" <freebsd-rwg@gndrsh.dnsmgr.net> Cc: Alan Somers <asomers@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: KTLS with zfs recv Message-ID: <YQXPR0101MB0968396874186A746F2401BDDD9D9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CANCZdfqGUkz9SLNYs_kCk9JweMtDm84Wm755tMuebBTwcyNURw@mail.gmail.com> References: <CAOtMX2geavU-dvoT_WhifHh6ijmf%2BwSCkV52k89JWn8DgjdArA@mail.gmail.com> <202102261816.11QIGME9031366@gndrsh.dnsmgr.net>, <CANCZdfqGUkz9SLNYs_kCk9JweMtDm84Wm755tMuebBTwcyNURw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Warner Losh wrote: >On Fri, Feb 26, 2021 at 11:16 AM Rodney W. Grimes < >freebsd-rwg@gndrsh.dnsmgr.net> wrote: > >> > On Fri, Feb 26, 2021 at 9:24 AM Rodney W. Grimes < >> > freebsd-rwg@gndrsh.dnsmgr.net> wrote: >> > >> > > > My understanding is that KTLS works very well with OpenSSL for >> sending, >> > > but >> > > > not as well for receiving, because there's nothing like a recvfile >> > > > syscall. However, it works great for both send and receive with NFS, >> > > where >> > > > all the data remains in the kernel. What about zfs recv? A very >> common >> > > > pattern is for an application to read from an SSL socket and then >> pipe >> > > the >> > > > data to zfs recv. For example, zrepl does that. Could zfs recv >> instead >> > > > read directly from the KTLS socket, bypassing userspace? That could >> > > > potentially save a _lot_ of cycles for a _lot_ of people. >> > > >> > > I did some patches and a short presentation at BSDCan that basically >> > > shoves the whole zfs send and zfs recv process into the kernel, ie >> > > it opens the sockets up, makes the connections, then the socket >> > > is passed into the kernel(s) and it all runs in kernel mode. >> > > >> > > >> > > >> https://www.bsdcan.org/2018/schedule/attachments/479_BSDCan-2018-zfs-send.pdf >> > > >> > > A few things need fixed like reversing who does the listen for >> > > security reasons, but this feature is probably ready for prime >> > > time. >> > > >> > > > -Alan >> > > >> > > -- >> > > Rod Grimes >> > > rgrimes@freebsd.org >> > >> > >> > That looks potentially useful, but it doesn't use encryption. Would it >> > work if the socket had been opened by openssl with ktls? >> >> Yes, it should. Internally the zfs send and recv code just does reads >> and writes to the socket, so what ever you setup for "connected" sockets >> should work. >> > >Yea, KTLS generally wants userland to do the initial negotiation and share >the connection state before doing the bulk encryption in the kernel... Yes. KTLS only handles application data records. The TLS handshake must be done in userspace. Handling of non-application data records received after the initial handshake also needs to be handled in userspace. All the userspace program does is an SSL_read() with a len argument == 0. This will always return an error, but only after it has read the non-application data records off the head of the socket's receive queue. The kernel code needs to tell the userspace program to do this somehow. The nfs-over-tls code does this via an upcall RPC to the userspace daemon (rpc.tlsclntd or rpc.tlsservd). I added MSG_TLSAPPDATA which can be used as an argument to soreceive() to tell it to return an error when there is a non-application data record at the head of the socket's receive queue, to facilitate this. You do normal sosend()/soreceive() on the socket with unencrypted data, although each soreceive() will give you a ctrlmsg mbuf with TLS record info, if you care (when you use MSG_TLSAPPDATA, they're all just application data records). Each soreceive() gives you one application data record, but the code probably won't care. Finally, when done transferring data, userspace probably wants to do SSL_shutdown() to do the peer reset (although if/when you need to do peer resets seems to be a bit sketchy in the TLS game). rpc.tlsclntd.c and rpc.tlscommon.c probably gives you some useful hints. It sounds worthwhile to explore this. Good luck with it, rick Warner _______________________________________________ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968396874186A746F2401BDDD9D9>