Date: Sat, 27 Feb 2021 00:11:09 +0000 From: Rick Macklem <rmacklem@uoguelph.ca> To: Warner Losh <imp@bsdimp.com>, "Rodney W. Grimes" <freebsd-rwg@gndrsh.dnsmgr.net> Cc: Alan Somers <asomers@freebsd.org>, FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: KTLS with zfs recv Message-ID: <YQXPR0101MB0968396874186A746F2401BDDD9D9@YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM> In-Reply-To: <CANCZdfqGUkz9SLNYs_kCk9JweMtDm84Wm755tMuebBTwcyNURw@mail.gmail.com> References: <CAOtMX2geavU-dvoT_WhifHh6ijmf%2BwSCkV52k89JWn8DgjdArA@mail.gmail.com> <202102261816.11QIGME9031366@gndrsh.dnsmgr.net>, <CANCZdfqGUkz9SLNYs_kCk9JweMtDm84Wm755tMuebBTwcyNURw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Warner Losh wrote:=0A= >On Fri, Feb 26, 2021 at 11:16 AM Rodney W. Grimes <=0A= >freebsd-rwg@gndrsh.dnsmgr.net> wrote:=0A= >=0A= >> > On Fri, Feb 26, 2021 at 9:24 AM Rodney W. Grimes <=0A= >> > freebsd-rwg@gndrsh.dnsmgr.net> wrote:=0A= >> >=0A= >> > > > My understanding is that KTLS works very well with OpenSSL for=0A= >> sending,=0A= >> > > but=0A= >> > > > not as well for receiving, because there's nothing like a recvfile= =0A= >> > > > syscall. However, it works great for both send and receive with N= FS,=0A= >> > > where=0A= >> > > > all the data remains in the kernel. What about zfs recv? A very= =0A= >> common=0A= >> > > > pattern is for an application to read from an SSL socket and then= =0A= >> pipe=0A= >> > > the=0A= >> > > > data to zfs recv. For example, zrepl does that. Could zfs recv=0A= >> instead=0A= >> > > > read directly from the KTLS socket, bypassing userspace? That cou= ld=0A= >> > > > potentially save a _lot_ of cycles for a _lot_ of people.=0A= >> > >=0A= >> > > I did some patches and a short presentation at BSDCan that basically= =0A= >> > > shoves the whole zfs send and zfs recv process into the kernel, ie= =0A= >> > > it opens the sockets up, makes the connections, then the socket=0A= >> > > is passed into the kernel(s) and it all runs in kernel mode.=0A= >> > >=0A= >> > >=0A= >> > >=0A= >> https://www.bsdcan.org/2018/schedule/attachments/479_BSDCan-2018-zfs-sen= d.pdf=0A= >> > >=0A= >> > > A few things need fixed like reversing who does the listen for=0A= >> > > security reasons, but this feature is probably ready for prime=0A= >> > > time.=0A= >> > >=0A= >> > > > -Alan=0A= >> > >=0A= >> > > --=0A= >> > > Rod Grimes=0A= >> > > rgrimes@freebsd.org=0A= >> >=0A= >> >=0A= >> > That looks potentially useful, but it doesn't use encryption. Would i= t=0A= >> > work if the socket had been opened by openssl with ktls?=0A= >>=0A= >> Yes, it should. Internally the zfs send and recv code just does reads= =0A= >> and writes to the socket, so what ever you setup for "connected" sockets= =0A= >> should work.=0A= >>=0A= >=0A= >Yea, KTLS generally wants userland to do the initial negotiation and share= =0A= >the connection state before doing the bulk encryption in the kernel...=0A= Yes. KTLS only handles application data records. The TLS handshake must=0A= be done in userspace.=0A= =0A= Handling of non-application data records received after the initial handsha= ke=0A= also needs to be handled in userspace. All the userspace program does is an= =0A= SSL_read() with a len argument =3D=3D 0. This will always return an error, = but only=0A= after it has read the non-application data records off the head of the sock= et's=0A= receive queue. The kernel code needs to tell the userspace program to do th= is=0A= somehow. The nfs-over-tls code does this via an upcall RPC to the userspace= =0A= daemon (rpc.tlsclntd or rpc.tlsservd).=0A= =0A= I added MSG_TLSAPPDATA which can be used as an argument to soreceive()=0A= to tell it to return an error when there is a non-application data record a= t the=0A= head of the socket's receive queue, to facilitate this.=0A= =0A= You do normal sosend()/soreceive() on the socket with unencrypted data,=0A= although each soreceive() will give you a ctrlmsg mbuf with TLS record=0A= info, if you care (when you use MSG_TLSAPPDATA, they're all just applicatio= n=0A= data records). Each soreceive() gives you one application data record, but= =0A= the code probably won't care.=0A= =0A= Finally, when done transferring data, userspace probably wants to do=0A= SSL_shutdown() to do the peer reset (although if/when you need to do=0A= peer resets seems to be a bit sketchy in the TLS game).=0A= =0A= rpc.tlsclntd.c and rpc.tlscommon.c probably gives you some useful hints.=0A= =0A= It sounds worthwhile to explore this.=0A= =0A= Good luck with it, rick=0A= =0A= Warner=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YQXPR0101MB0968396874186A746F2401BDDD9D9>