From owner-freebsd-current@freebsd.org Sun Jul 19 23:34:36 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4776A36CCD0 for ; Sun, 19 Jul 2020 23:34:36 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660045.outbound.protection.outlook.com [40.107.66.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B91RG2NgMz4Bx0; Sun, 19 Jul 2020 23:34:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QdKyiEveEmVe/tZ3zz+6SaYinXBmVvB8/2rGmsdDq03mrocUgvBZk1hG4bTcbpsVHGs9tFIg4CwkCqQ6VR24BsCDA3neMHANBOBdnqBggAbuVGzrw6bgEQfAjW+vZGysjotDNItsQQA4W/ol/EKWdOPhfXKPG90hc4B5PFWF2FMFQ7mUcdH10olAhjgi4Q7lYIflToZmaOOCchYRWSsizytT78o0ZrpQyfkcv6ouFSr9o5cfa49ttHZ/O7jZljvU++vQSi8F2JZoU9W2W6MUHb/6nMPAD8lCbqpjpuIF02b3BI1DsSKHpolGAZnviKP+2WqjYvz64XynFzFge5m+Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qsv7M4SzoDBvhWytsfEr52wNquIgVfhSQrpCWHlkeF4=; b=njlzEl7+morRKbVjIvPTO9e99y4R2sUHo6y5Kq2AY8MMLiiWqSfltdyS5NlHglpPyF+1BNNq5vGlOwjfwXs+kyuD+PeluRe2j+E04qeunYuPSIyBsQQhh47Ij4uFIjavDMw9eo/TST0LZZL3iRlTHr1yFShzCQbmxTSp69ruV3FCcwI8XwlyTF4nGSr/6fwoB9hPzbmpbCYAMb4tyDoKC2zrntHi3SoUI01cTrYbMo6d2jxv0UocBuBrGlbXYJKGPp6Il1LfIpv1WVQR850jNZgg/9v8xNkMw2iwqG9G2GpyiisISbed+RRiGVT4H6B7dtLsNA00SnMADD4heyov0w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qsv7M4SzoDBvhWytsfEr52wNquIgVfhSQrpCWHlkeF4=; b=KsTdFcOVeXKN8N7VrnaOoOQuf/mSQg7XJk2obdpDQIzmC3Fg36I5M0oUxugt35/1Ao53CO9qqP4Wb+FfX0rHoLOV7VQErg7io+e2vlW2o++S6tG6sSUK2CHj1XDBx8mTVH5n9wNdIetRJbCy2Qpj3J211vTyjWiONZ8vFrvEnDhTr564oS/fZhYGbLz0rdIctUku86Wx8RhTQhjF6HZVU0GgOiSTsOHXvE32RGvO+JYV99GPrQrdrnPmf2Vj6SRfNrLBtDuMNyv8udtd28EkOGs3K75OjIJ1cfLLOr/x616ZvKcg7Qo4ga0Su3kS0AGADxlV1gEccwM03uGxlx1f3g== Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:38::14) by YQXPR01MB3205.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:43::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3195.25; Sun, 19 Jul 2020 23:34:32 +0000 Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::60f3:4ca2:8a4a:1e91]) by QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::60f3:4ca2:8a4a:1e91%7]) with mapi id 15.20.3195.025; Sun, 19 Jul 2020 23:34:24 +0000 From: Rick Macklem To: "freebsd-current@FreeBSD.org" CC: "jhb@FreeBSD.org" , "gallatin@freebsd.org" , Gleb Smirnoff Subject: RFC: ktls and krpc using M_EXTPG mbufs Thread-Topic: RFC: ktls and krpc using M_EXTPG mbufs Thread-Index: AQHWXiFLFxOPU111VUSvS2ovnXgBPw== Date: Sun, 19 Jul 2020 23:34:24 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 7a5fccc3-c70a-4562-7133-08d82c3c4578 x-ms-traffictypediagnostic: YQXPR01MB3205: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6790; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: O7nWW6g6PbLhe8yxbHSaMlnDxENXU/uenvC4gtziw4hCTpJjNFStwuHn0gdv418Yg9opjuWdRCw9FcJ8dDQ8mGUr2g4R7nu1IWc/yXWxmn3+G3BCOah9ZwX3XTOClsYm0HUJqPdn8WwT6hj3LQVeLJf0hKJwkpuOkyf2u21m+G8RvH1MtFdfThxyK8mYaAj12w6C/BqKn90lWtcazg32DFS4pL0ReBj9JMEWGG4dIjqFLmqX/v3bHFkucSc5+5Edt1LgHHXjmC4zRiOu2NYv1sK5stPwU8f/3vXEL3uZdl7l5MliYUH8yva1sduoCwJQEEHKkC11NLdpxVvuqg2eMQ== x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(39850400004)(396003)(376002)(366004)(346002)(136003)(9686003)(186003)(786003)(316002)(7696005)(83380400001)(55016002)(33656002)(6916009)(6506007)(450100002)(4326008)(64756008)(8676002)(2906002)(52536014)(8936002)(54906003)(71200400001)(5660300002)(86362001)(66556008)(91956017)(76116006)(66446008)(66946007)(478600001)(66476007); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: valzXIRInwUjAtPSaGnc2oMOiRmkaEI+jRsZRVEGQ3ZFFU/RBG8uhpALimKiua9UhsKxRq8lQWjCWepypyGzmwQeYlt3MSv9SrvghMesWCQNm6OqRh6GdqANP8xW1MMosjP2m3uQUYHX2X2z7BFLh/Pb94ylSZV7A0o970SdxeV9L0xsw57zUKw4F78kZrWRWs2ADKZo700dUoLRZRiEJ3R6MD7BzcmCdKGj+3m0Dec3AVCQAuqWWrfHq1njL3xFoKLkHKZM/9iGMzEporLuWV11PIOfx6Dd1J+QfwPoifiGl4Qdh5/jlmHYA5gdiZ8FigH/zajSkLDtk6+SiicAVVB6/L8lqEutarSFDiRr858OdXXN9/sdu1QaGqQ5UGzhFI6qdvLTEgVygAk1kjhwt7jeSZtIjn0eYLjMQmu9If3HWSZ7+cn5EK3Bm2028F0lxmHvvYm0IdrL6EmtV4KHb1fOW20l911JuOP2wSa8Eu2nbOJMXc2oUSHYOKbHCm16egvoRulyrCbq0BSWOj+zZWzAZwzjFK8j777cLCUTXeU= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 7a5fccc3-c70a-4562-7133-08d82c3c4578 X-MS-Exchange-CrossTenant-originalarrivaltime: 19 Jul 2020 23:34:24.7236 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: SJi85S08s6GkeOGsP3B1iRdKZ4KLEW3XuJeKVTzsOXvjuxdx8I3AWAt8vVCg+FV5zU2OHCkQZIjNs1YLsxYaig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQXPR01MB3205 X-Rspamd-Queue-Id: 4B91RG2NgMz4Bx0 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=KsTdFcOV; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.45 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.39 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.031]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; DMARC_NA(0.00)[uoguelph.ca]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-0.89)[-0.893]; RCVD_IN_DNSWL_NONE(0.00)[40.107.66.45:from]; NEURAL_HAM_LONG(-0.97)[-0.967]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.66.45:from] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Jul 2020 23:34:36 -0000 I spent a little time chasing a problem in the nfs-over-tls code, where it= =0A= would sometimes end up with corrupted data in the file(s) of a mirrored=0A= pNFS configuration.=0A= =0A= I think the problem was that the code filled the data to be written into=0A= anonymous page M_EXTPG mbufs, then did a m_copym() { copy by=0A= reference } and used the copies for the mirrored writes.=0A= --> In ktls_encrypt(), the encryption was done to the same pages and,=0A= sometimes, the encrypted data got encrypted again during the=0A= sosend() of the other copy.=0A= =0A= Although I haven't reproduced it, a regular kernel write RPC could suffer t= he=0A= same consequences if the RPC is retried (it keeps an m_copym() copy=0A= of the request in the krpc for an RPC retry).=0A= =0A= At this time, the code in projects/nfs-over-tls works correctly, since it= =0A= always fills the data to be written into mbuf clusters, m_copym()s those=0A= and then copies those { real copying using memcpy() } via=0A= mb_mapped_to_unmapped() just before calling sosend().=0A= --> This works, but it would be nice to avoid the mb_mapped_to_unmapped()= =0A= copying for all the data being written via an NFS over TLS connection= .=0A= =0A= For the TCP_TLS_MODE_SW case:=0A= --> The NFS code can fill the written data into anonymous pages on M_EXTPG= =0A= mbufs.=0A= Then, the ktls_encrypt() could be modified to=0A= allocate a new set of anonymous pages for the destination side of=0A= the encryption (it already does this for the sendfile case) and put those= =0A= in a new mbuf list.=0A= --> This would result in new anonymous pages and mbufs being allocated,=0A= but would not do memcpy()s.=0A= After encryption, it would just do a m_freem() on the unencrypted list.=0A= --> For the krpc client case, this call would only decrement the reference= =0A= count on the unencrypted list and it could be used for a retry by the= krpc=0A= and then be free'd { m_freem() call } after a reply is received.=0A= =0A= If doing this for all the sosend()s of anonymous page M_EXTPG mbufs seems= =0A= like unnecessary overhead, the above could be enabled via a setsockopt()=0A= on the socket.=0A= =0A= What do others think of this?=0A= =0A= For the hardware offload case:=0A= - Can I assume that the anonymous pages in M_EXTPG mbufs will remain=0A= unchanged?=0A= --> If so, and it won't change to TCP_TLS_MODE_SW, the NFS code could=0A= fill the data to be written into M_EXTPG mbufs safely.=0A= =0A= - And, if so, can I safely use the ktls_session mode field to decide if off= load=0A= is happening?=0A= I see the TCP_TXTLS_MODE socket opt which seems to=0A= switch the mode to TCP_TLS_MODE_SW.=0A= When does this happen? Or, can this happen to a session once in use?=0A= =0A= Thanks for any/all comments on this, rick=0A= =0A=