From owner-freebsd-current@freebsd.org Tue Jan 28 23:01:35 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A7772244E6F for ; Tue, 28 Jan 2020 23:01:35 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670058.outbound.protection.outlook.com [40.107.67.58]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 486hv23Lkfz3Hk9; Tue, 28 Jan 2020 23:01:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=beL5VH7X6KKIryWEux1+8zV+AXrPD6kYRqdOcnw963iOQC6OnCdCvJNbbd3fV5lhT5BZvz/FQnNkDS5OWy1FCeJpxlbTp2z6LzShk3ghOC8QzzzcBVpTvZhM9AEwznDViK/4KHDxc34kIp3CvEx7xPIYbsA0NwIvEQpPZx+2uQXYrvRVGudOQ7NYFLCY/KtuhM+YamI2Tl59SEwe4g1uMal+p7eqAF5eWDS3d16iSKbcU1Qwh5KYaw/DDzz6EX/Kmoqr9M38DUrNchhHB3LuFC3hG3lSxiakEQBH+h9YlJ4wMv3PmNaB96tx6k/sUJAM2KMFR36IdW50+o2LMk5eZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Aj4SZF5tNn3oS92r+zalkwYoenYOzNG1Qj1qJJXmJwA=; b=J0GzG29ga0YgII7emc33t/c2iWLMxRTWlCs+e6HleyLhLB3prXrMv3d+boW3gWdf2fzKQ3YtggJxMpUtT8oMIMPJ420eaRO6+esQ2pymT7HWVvPvc5xaWhJM8jPM6QAIN8niUeRmOLROv9SXRYPYwtMlv4oGiUVcVqtfhZHC9B1ASP7/ls3ZpVOmzIR2GsLJJaO052LyBkIyU1JEHyw5qoS1RmdEwyEHMMl2bWs8FCY8JJSpPTxQYcqdYa7fVwJHV9ASN2AQUr4AetPKJePX32oT4oAw3DYT8a8Kl1xSv/Ko1wqFO1YVE3O725h+r5svqUhhDKGAJs/yTPVlwWyAfA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB1363.CANPRD01.PROD.OUTLOOK.COM (52.132.70.144) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2665.23; Tue, 28 Jan 2020 23:01:32 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::6588:45c3:4892:f98]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::6588:45c3:4892:f98%7]) with mapi id 15.20.2665.026; Tue, 28 Jan 2020 23:01:32 +0000 From: Rick Macklem To: John Baldwin , "freebsd-current@FreeBSD.org" Subject: Re: how to use the ktls Thread-Topic: how to use the ktls Thread-Index: AQHVxa2HeRfmo36hWEyrGcMaBhE88KfhEeoAgBxnlpmAAOgvAIACbdpN Date: Tue, 28 Jan 2020 23:01:31 +0000 Message-ID: References: <5be57c87-90fe-fcbe-ea37-bdb1bcff2da8@FreeBSD.org> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 9664ef9f-f2c7-44e7-1254-08d7a4460426 x-ms-traffictypediagnostic: YQBPR0101MB1363: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 029651C7A1 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(346002)(376002)(396003)(366004)(136003)(199004)(189003)(2906002)(186003)(71200400001)(478600001)(26005)(7696005)(786003)(6506007)(110136005)(316002)(66556008)(66446008)(66476007)(64756008)(91956017)(76116006)(81166006)(81156014)(66946007)(5660300002)(450100002)(52536014)(86362001)(55016002)(9686003)(8936002)(33656002)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1363; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 23rxiow/4d61kEArog8608BoTgXnZg8rSr10F282xkQfK0x0oQvxdjY0QZV7bKCANkCL8SXaTpIlDBFWqY9KILu/P2OBRK08QfNx7ILGXKoOFg069jhGs3c6SCoJFECXAI37SSCXWAwdjTEdYUMn+IUuJ2fpWa5dumxIrSBYMHKAfIhtZhUr5fpJMAkwIjDOtROIeubIjPxG+ZqV4bXlkbP630n+Ny0sZVVYlnBMp10O0wOBIBaTMZa/3yllf2JwfqYq3hWk9p9+bQ4NY6vbNSRdH8Bl9pYwLOI5FBUS7HIOap0frRjLcxNzqZJcTVPSgyZUj9fVEgjAw3tXp9l/3WdSHQCbGIoP6XRdBc6Is8fkBb2yyzs+BLL0SEQenKop8yH7ahM0R0yLFCeEkGoto0LjUBzCC9QnkS2GTBnEx/SA8AoA37jnATYLXYjU5tms x-ms-exchange-antispam-messagedata: yrm+zLyjedIaFpSJs+jqwNiPtgqdmdn+vM0f08VsyZCiCAhSEU0BSFuc0lRGCdu+NWnInCKyIzhkK4COPEERgUz3IYdOwIlZE2EifNvKf+gJAt6QRKU/yDPI4z9itIXvi/sXFQ3fa99Q0TZXmTV3kA== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 9664ef9f-f2c7-44e7-1254-08d7a4460426 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Jan 2020 23:01:31.8995 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: sBGvPMkBNiCVSjWOHKDP1bU63KcreMlBf3bnhTxNrwsitBZ7XLGS4oscHzX1AnSP6scnW67Hxa5Ad/Ydaj4V/w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1363 X-Rspamd-Queue-Id: 486hv23Lkfz3Hk9 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.58 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.69 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-1.39)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-3.05), country: US(-0.05)]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[58.67.107.40.list.dnswl.org : 127.0.3.0]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jan 2020 23:01:35 -0000 John Baldwin wrote:=0A= [stuff snipped]=0A= >I don't know yet. :-/ With the TOE-based TLS I had been testing with, thi= s doesn't=0A= >happen because the NIC blocks the data until it gets the key and then it's= always=0A= >available via KTLS. With software-based KTLS for RX (which I'm going to s= tart=0A= >working on soon), this won't be the case and you will potentially have som= e data=0A= >already ready by OpenSSL that needs to be drained from OpenSSL before you = can=0A= >depend on KTLS. It's probably only the first few messsages, but I will ne= ed to figure=0A= >out a way that you can tell how much pending data in userland you need to = read via=0A= >SSL_read() and then pass back into the kernel before relying on KTLS (it w= ould just=0A= >be a single chunk of data after SSL_connect you would have to do this for)= .=0A= I think SSL_read() ends up calling ssl3_read_bytes(..APPLICATION..) and the= n it throws=0A= away non-application data records. (Not sure, ssl3_read_bytes() gets pretty= convoluted at=0A= a glance.;-)=0A= =0A= I've found another issue that should keep me amused for a while (this is be= coming an=0A= interesting little project;-).=0A= The KERN_TLS needs unmapped pages on the mbuf chain, but that isn't what NF= S=0A= generates.=0A= I think I'll have to implement some sort of copy function that creates mbuf= s with unmapped=0A= pages and then maps them into kernel space for long enough that the data ca= n be copied,=0A= called just before sosend(). Most NFS RPC messages will easily fit in one p= age.=0A= =0A= Someday, the biggies like server read reply may be able to do what sendfile= does and=0A= put the read data in unmapped page mbufs, avoiding the long list of mbuf cl= usters=0A= that VOP_READ() currently copies the data into.=0A= --> But that's longer term than getting this to work.;-)=0A= =0A= Thanks for all your help John, rick=0A= =0A= > I'm currently testing with a kernel that doesn't have options KERN_TLS an= d=0A= > (so long as I get rid of the 478 bytes), it then just does unencrypted RP= Cs.=0A= >=0A= > So, I guess the big question is.... can I get access to your WIP code for= KTLS=0A= > receive? (I have no idea if I can make progress on it, but I can't do a l= ot more=0A= > before I have that.)=0A= =0A= The WIP only works right now if you have a Chelsio T6 NIC as it uses the T6= 's TCP=0A= offload engine to do TLS. If you don't have that gear, ping me off-list. = It=0A= would also let you not worry about the SSL_read case for now for initial te= sting.=0A= =0A= --=0A= John Baldwin=0A=