From owner-freebsd-current@freebsd.org Sat Feb 27 00:11:12 2021 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1F2A4553DEC for ; Sat, 27 Feb 2021 00:11:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670087.outbound.protection.outlook.com [40.107.67.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DnRl36nrmz3N6g; Sat, 27 Feb 2021 00:11:11 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F0kus52YStqvFOBTaVDBgxZXW/I9Ss6UhZGU0L/oEY1hd53qp3BjiDoPRxJH74n164OOfND89v2B7c+sskpi7WaHc85L3zJaEayWBMqMhXZ5wysGRYBvpmeQ+3U8ZNipL6TmxUTGuPR1KBO3BJB0CgH+APHiUzm0J/nFX4rWQlGEsNa8/ocZdVruorAVbK5bpzhdusmSi5usbJ1c6ksKjAsaWHgDX/P663s3UU2AwiK2I75Jh0O7t3shgqqdQur1I+2dDEaD+3TGKJCGrMTNYQUe8jctlSJ+ateiOdf9X163pNMQoo7a395kD5yECu4HPO+ifKDUmu/+5HS5u8UnYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eI2/3aZFnX8PnZnKYq/VzUc5uTv65za6IzJSdFO1JWE=; b=OXjr2JzPc8YelNSnWm5zywBszXrc4T3LC0kOuF0nGmhJlLep/6vgV20mUt4ki2vAz1kn671BtsI35yFlArXsEjuqnUE/nGyMmUYUn4+HxaEuUgmcdazyQn4nyO7ILruSAcvhATYcaHlczq9NwCQoR11fkrWXm70Ks1IhZ9vpvKsZwsjWGmxRHq3R5xHMhlKjLS/XYWQVlZnk7TO7TvIi+LuRRuzpQTNzlJdQlKnSkpmOyGRJgGsG3t7KsOmLfFTKoZ9bLQr9ZO0Q3sAQ0UxN0fR4sO0DjMeKSjEyNQIjmNt2LPrW/hfeUue1YOsNRXPYfL9gJKXM0nd7J9F0dCo3wA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=eI2/3aZFnX8PnZnKYq/VzUc5uTv65za6IzJSdFO1JWE=; b=VklA4b+waFFkZpmpLuwinYDC3Ka6XPbGqEYSJb4VL8WibYHaLQH909pdBMGTtKyU8irHU/HT1oFoTb4niOEQFTM98jd6aThcuHNjOr8YkCVjDitikfj7b7S7veCXJ1R9rtG6xiAk8z3tuc8O12dWc5IIjQfVeca+Rz3OKB4JIXZzgflfP4RCee9pSddwffJ+0gbwPIAvMdCXkFcAq8WlPOgd27xM/XB4gT/jpRsVWEZYQqyoHZ/hUY4ISdMAw73kYNFSx75nhUXrYAANhjDVsaGUSkpHJE7YO9BizIkNuxBLmV+ZZWpOXBi6Ded1WlY4I0knqrutncD2P38G1A6LQg== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB2082.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:f::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3868.32; Sat, 27 Feb 2021 00:11:09 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::6073:6fc0:5ddf:dc8a%7]) with mapi id 15.20.3846.048; Sat, 27 Feb 2021 00:11:09 +0000 From: Rick Macklem To: Warner Losh , "Rodney W. Grimes" CC: Alan Somers , FreeBSD CURRENT Subject: Re: KTLS with zfs recv Thread-Topic: KTLS with zfs recv Thread-Index: AQHXC/8ayZIiwV83pUycfuNq2VZYvqpqn9CAgAAYm4CAAAaPAIAACkCAgABS0fk= Date: Sat, 27 Feb 2021 00:11:09 +0000 Message-ID: References: <202102261816.11QIGME9031366@gndrsh.dnsmgr.net>, In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b79cdfb2-3da7-4122-0ef9-08d8dab42f7b x-ms-traffictypediagnostic: YQBPR0101MB2082: x-ms-exchange-minimumurldomainage: bsdcan.org#6392 x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: eYyu6kP/AnuasvvjBoWlw57LdeNCV5JLCI52plQIPeNAjI4RyZ0v1OUGLZx8sDaqDa11Vu5P7Qw1KS0KDnGZLFI0DrM3huFdyTynKG9LekrocHqKDGt3RVtbaktA9H7Xyb5yCXxX6nb7973vRLVPfqBdqqHtOT0hif+TFbiHgD3J1LU0LsV3GSXNBP8wD2I/qEgEEmpVzuviCkJUYKXAezFwPYFLP0lKP112r8Ok03bBhXsyRL7Zx/3uc2N1G/DrWHVOey1sNAsE1oFYoJ4Pvc9hLIkKdYBs7+rp2jzesyfeEivwSkH+qciRXB/aRb/SyoLeYJfNct9PfPpKQOBFReGu/24t33bSO6tkwoHaE3Qxxy80h+QjTMnvhaIsyipreObnKtOi1o6R2FLJN6bOc2GTQ4drvFbnoxA83pHdF+PRIFjAMSj/j/bR7xnrbsrZqnfgF6tGizt8Als+yO49LToky341dRjBwZIZ4ZqrbqM5LKedQKOBK6LAoKuR2RVeZ55seqNWGS1EUnJbWwI3zmWhz4C4D7cLwYPHHme8/1Isn+GZA/vEV2qyp8mMOzkNqhYgQu9gIQPc82429YzEXzSYzmlfP/OOvvhsHU8b5cY= x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFS:(136003)(39860400002)(396003)(366004)(346002)(376002)(83380400001)(4326008)(53546011)(64756008)(66446008)(91956017)(66556008)(66476007)(76116006)(2906002)(86362001)(33656002)(5660300002)(66946007)(8936002)(186003)(8676002)(110136005)(54906003)(55016002)(7696005)(478600001)(786003)(966005)(316002)(71200400001)(9686003)(52536014)(6506007)(3480700007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: UqS1WkHorU731DecivEoOlvDP9ffpXuL1vzhiy4aegpfKVBiv/kzJ/XueKyZ5HJFMqJ8Ht3+UaFYDSIVLg/qqX899Q9Rfh7NrVc+5RDSgi2Bdru0IIuBzHJxKK+VZa+ViEYrd1HjvDuhryCwsB4wOndiZW6nlCEObd2qc+DL9qpjouvvOeUzxsO3hU0nPbPLNwIwrakUJ31uou5DCRjbWfkbwF9YvCOF8z8V6SAvLU0ydaSs8s/Hyixj3VI+QzAchgWNQBvVl2tPGnlW5vz8VULvRVYpSD8MVvoCQ6Iu6KZBY0h5gmguVU6cTIs3fym//9usnwly0HAmnyhHcSAzCJ7qsOpJagb6gATRlfHkypun9CktKPSU4Hsr4q6NwoNHjncL3TaxHJrZ1VvZvvUz0f0ENweBr6qIcaRCM3LIvefLNwY9wyznoVVNAgVGdly4ksMRxny8YaV9e0WzaenzF+YmJlCuVXm7XSZLqYA+q/GqYdbQNFhff2Q9pRrl+TUb2YgeRpf+i0pwnrfwcY01HY4Q3XetiSeTTKo5IZR0bCGJNNVwtd5yJuKr69PNA5YLANjpKSvoJmlf+rgaiURc1mMHqD7Nd6S5n3MFCHOr6PoSAliqfRnknqJnraB2oBf6OEnCg25z53he/3RPGu52K3NEEsVOQ+0TUuHTuHkn5rIArH9jqzqUWGU1rW74isyt41Yw8PBWliK/0J+ohUZLeR4lGf80O+Q4JF71R9Hv95xLKe9aLhCGMCG0LOij4ZWVzUJQYycBtm/QUT7VUR3tOIk5r3djot4+JpAfAxHGxGw= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: b79cdfb2-3da7-4122-0ef9-08d8dab42f7b X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Feb 2021 00:11:09.7488 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: RvldRAbnpG/qX3D8QlzCXoveh5zSSsilv53jj/zToQ+FOcYkczFfllmTiwq+3rPQZEPtjWBQEf2JDOIAs0Rqug== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB2082 X-Rspamd-Queue-Id: 4DnRl36nrmz3N6g X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Feb 2021 00:11:12 -0000 Warner Losh wrote:=0A= >On Fri, Feb 26, 2021 at 11:16 AM Rodney W. Grimes <=0A= >freebsd-rwg@gndrsh.dnsmgr.net> wrote:=0A= >=0A= >> > On Fri, Feb 26, 2021 at 9:24 AM Rodney W. Grimes <=0A= >> > freebsd-rwg@gndrsh.dnsmgr.net> wrote:=0A= >> >=0A= >> > > > My understanding is that KTLS works very well with OpenSSL for=0A= >> sending,=0A= >> > > but=0A= >> > > > not as well for receiving, because there's nothing like a recvfile= =0A= >> > > > syscall. However, it works great for both send and receive with N= FS,=0A= >> > > where=0A= >> > > > all the data remains in the kernel. What about zfs recv? A very= =0A= >> common=0A= >> > > > pattern is for an application to read from an SSL socket and then= =0A= >> pipe=0A= >> > > the=0A= >> > > > data to zfs recv. For example, zrepl does that. Could zfs recv=0A= >> instead=0A= >> > > > read directly from the KTLS socket, bypassing userspace? That cou= ld=0A= >> > > > potentially save a _lot_ of cycles for a _lot_ of people.=0A= >> > >=0A= >> > > I did some patches and a short presentation at BSDCan that basically= =0A= >> > > shoves the whole zfs send and zfs recv process into the kernel, ie= =0A= >> > > it opens the sockets up, makes the connections, then the socket=0A= >> > > is passed into the kernel(s) and it all runs in kernel mode.=0A= >> > >=0A= >> > >=0A= >> > >=0A= >> https://www.bsdcan.org/2018/schedule/attachments/479_BSDCan-2018-zfs-sen= d.pdf=0A= >> > >=0A= >> > > A few things need fixed like reversing who does the listen for=0A= >> > > security reasons, but this feature is probably ready for prime=0A= >> > > time.=0A= >> > >=0A= >> > > > -Alan=0A= >> > >=0A= >> > > --=0A= >> > > Rod Grimes=0A= >> > > rgrimes@freebsd.org=0A= >> >=0A= >> >=0A= >> > That looks potentially useful, but it doesn't use encryption. Would i= t=0A= >> > work if the socket had been opened by openssl with ktls?=0A= >>=0A= >> Yes, it should. Internally the zfs send and recv code just does reads= =0A= >> and writes to the socket, so what ever you setup for "connected" sockets= =0A= >> should work.=0A= >>=0A= >=0A= >Yea, KTLS generally wants userland to do the initial negotiation and share= =0A= >the connection state before doing the bulk encryption in the kernel...=0A= Yes. KTLS only handles application data records. The TLS handshake must=0A= be done in userspace.=0A= =0A= Handling of non-application data records received after the initial handsha= ke=0A= also needs to be handled in userspace. All the userspace program does is an= =0A= SSL_read() with a len argument =3D=3D 0. This will always return an error, = but only=0A= after it has read the non-application data records off the head of the sock= et's=0A= receive queue. The kernel code needs to tell the userspace program to do th= is=0A= somehow. The nfs-over-tls code does this via an upcall RPC to the userspace= =0A= daemon (rpc.tlsclntd or rpc.tlsservd).=0A= =0A= I added MSG_TLSAPPDATA which can be used as an argument to soreceive()=0A= to tell it to return an error when there is a non-application data record a= t the=0A= head of the socket's receive queue, to facilitate this.=0A= =0A= You do normal sosend()/soreceive() on the socket with unencrypted data,=0A= although each soreceive() will give you a ctrlmsg mbuf with TLS record=0A= info, if you care (when you use MSG_TLSAPPDATA, they're all just applicatio= n=0A= data records). Each soreceive() gives you one application data record, but= =0A= the code probably won't care.=0A= =0A= Finally, when done transferring data, userspace probably wants to do=0A= SSL_shutdown() to do the peer reset (although if/when you need to do=0A= peer resets seems to be a bit sketchy in the TLS game).=0A= =0A= rpc.tlsclntd.c and rpc.tlscommon.c probably gives you some useful hints.=0A= =0A= It sounds worthwhile to explore this.=0A= =0A= Good luck with it, rick=0A= =0A= Warner=0A= _______________________________________________=0A= freebsd-current@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-current=0A= To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"= =0A= =0A=