From owner-freebsd-current@freebsd.org Thu Aug 6 20:46:01 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D77853AA96D for ; Thu, 6 Aug 2020 20:46:01 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670076.outbound.protection.outlook.com [40.107.67.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BN0rS30YPz4ZLM for ; Thu, 6 Aug 2020 20:46:00 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SwUBsyPt6b1mtp7NGAKQRIf95Vp4dODUiPFZIEk+vRMdmFBY/481inDhP06RhaiwJRm9esxuhpksfd2B2csTJwf3tFyCb3YVzSD/BmtsSHgFAoJFyBMwb33CFrL7GF1n8WtHAwY5eJj0SS0lpqeulgalOsPEdG9poRg1ma3KDUNxVlJzxWgJdAAGi3WdXfpXGRaIcMXB6Cmv+25I2f1bkQTHzSLbjvwlIbuJpW5jFmDylH5WPU1SJwWLL5nvt/Xo4hjt7XoW5Javv05jCGdbuPGEhB9/yjH8shag2TqPRIQvCXNTn9jswfrNssgXusW/bfeA13NTX/82/iT9qT3QhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6HDy0PBT9KtyCFGLAE8M+VBrEMb5O+RHOXwjqXJmdwo=; b=ifQBswHY6g+iOOiuqqSM5rr1GGjr5pmd1Gwn/qgUHr9LGe/c1EIKd9bMpK/SVgcAOpeWu/LCVuNcjryIAMiYjwwMtvGV+uM2SH2X/Tojl3u/YEtHzS9yNaqruKjGpJwpj8nQjbTaicrVas4ffI2EDrBVIaJwhXfSNwn4r5GDd556sP9UnoLOh2WXHj8X8mcYkpd9LXLyLlC4JcW+fqlb1DlOIiQFwcyEh36iBg9IF10WCstRWdIT3J1o2gCY4UZC3/gwkbpuPvmpfand7jDBslCEE7WBIRoh+aOAaypE8lqeQ+f0p0VVkolg9yegaMc1ez7m75pk6XgtBcZEYv1rjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6HDy0PBT9KtyCFGLAE8M+VBrEMb5O+RHOXwjqXJmdwo=; b=kNNzxgShzGimIvdevnEY9PwWclIqxzBG0GF30xoh7N2YrsUjQDgkYtdmwcF3nkNz/qnFnskaahx/z148MvHSDFrO2TPSdZMDVhEtthMWKC55PD/O8pAnpP94kdAigF8gXuUviESjpGYlz6r+eQOmi07A6TjToT/p+dQqad3kKIEHeGrU3KWQyjXqb2EktzQ64Y1gwhMEEw9TXjH16CPIxEWMWQSWFLzRqOa9kKvdqSN5+FKAvcP9qY7KhrDW/Fj23awhi1gvPGQyP21fOX8/giCU8TRuGgqd5AwE6W21t4fvroosKPRtqTjJ0kuqaCrURu9z85NAozpohX9MHOpUCw== Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:38::14) by QB1PR01MB3060.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:3b::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3239.21; Thu, 6 Aug 2020 20:45:58 +0000 Received: from QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::e89a:a655:91ca:4e63]) by QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM ([fe80::e89a:a655:91ca:4e63%5]) with mapi id 15.20.3261.019; Thu, 6 Aug 2020 20:45:58 +0000 From: Rick Macklem To: "freebsd-current@FreeBSD.org" Subject: can buffer cache pages be used in ext_pgs mbufs? Thread-Topic: can buffer cache pages be used in ext_pgs mbufs? Thread-Index: AQHWbC7TP/nOu8hGHE+IVczIn5RP+g== Date: Thu, 6 Aug 2020 20:45:58 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6484fe06-8371-4cd3-cecd-08d83a49b932 x-ms-traffictypediagnostic: QB1PR01MB3060: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: reHhTM34dHqtNV2BQlSuIMYiutGZuSOyqzJDRkLwGn3Jb0d6wWnhLkVZpWg62MmtdSrO6ugDngFLjJ5ZLKmJ92C1wUGQqvWju/sVl+K4BJgy0GBtMEwVcxB+QrOrSx7KMeBDHOxvFi61/JwZLr0+kmeKlViluGhOElGQTCVeAJxuAQlOFQiL6AC5vd2d3iKmcbFTOrjsyG44VgZIq5vlbChTRw8ZQVNz9fOzZP3tyNYSJhilICTzoMuAar33CTzcPmoG+ylQ+2RFZioIptIQyrfte5dwwmKgXTOscywlXp0TaEk9eDU1qW6R/vvKIGA7 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(396003)(366004)(39860400002)(346002)(136003)(376002)(7696005)(83380400001)(6916009)(186003)(33656002)(52536014)(66946007)(5660300002)(64756008)(66556008)(316002)(86362001)(9686003)(786003)(55016002)(8676002)(91956017)(8936002)(76116006)(71200400001)(6506007)(478600001)(2906002)(66476007)(66446008); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: i2fKGPs43XpV0stZk0VTysMwdPfze3rlP9dDQFcQV15gYgKCb8DYiHZZmBYSosXo5+ffPsZy98HpG8W3Rq5DLLp5p4gUMj06LBT8+F64Tkc/YrIIqKDJQvOH/VFiLaV4lBRHwI178BIlxZqW0emapBH0Yq2tm/KBwb5hRQsvCGulu27xT2aQSwIbuRfrL8KPCkj6mQcP5itz9do7Zm4gOKEOvBzEf5s8KNaznZs5xo+dgcykl1zW2MgKTnFQJxHNxcEz5Fgp7F3JgQ1qa1Vlhx/zpJgCHrZQ/jPN6txFWG1O3J9vGKhP9Kmhk2LzTOvETs8hNPVhJHspgNwEyf5RS/019eiLk3AtxjHxa+lqL2OdWGwI1y4SaySCwa6Uu0lera6PZsEr1DmYtLbfglP3sd++jzHdp9SEeKuIX5uwM7Ik+1lIq2/QZziQXgoCe5MQmoJHSCkkzXFMBptXuS9acb3lphOA9vZEtABHOdViBMAH6/pLYkgArmlZuoeP/lTrD53ymMAi4aVkuslEofCfVm8jB5FQmQL2D5OYi82tQhbip9jBYxjZiMYqV6r/c9QSgYDgR5xxdjXJPxN+05q/oOiXB/yffR7nis96201hDFQ56gMEmiJQKCpGcv2wq/moQFPu3PLR79sav2v+YVTUTIU5kuQiAHhi6Zxc0DBG8oFBtGLxQ6q0MsQq3b3WHb83qN2eUHRGx/rWrMvehCulsQ== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: QB1PR01MB3364.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 6484fe06-8371-4cd3-cecd-08d83a49b932 X-MS-Exchange-CrossTenant-originalarrivaltime: 06 Aug 2020 20:45:58.6084 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 1gvZPidXqMsyOeYve/Nqb/F24BNjRHMpNBdrwbUOLXuvnC/KiS7UDq19wHmYM85xY22NQwtpcfPKtyrojFPcdQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB3060 X-Rspamd-Queue-Id: 4BN0rS30YPz4ZLM X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=kNNzxgSh; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.76 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.18 / 15.00]; NEURAL_HAM_MEDIUM(-1.05)[-1.049]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; NEURAL_HAM_LONG(-0.99)[-0.993]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[uoguelph.ca:+]; NEURAL_HAM_SHORT(-0.64)[-0.636]; RCVD_IN_DNSWL_NONE(0.00)[40.107.67.76:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.67.76:from] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Aug 2020 20:46:01 -0000 Hi,=0A= =0A= I've been at this game for a while and one of the axioms is...=0A= "Everything is harder than it at first looks."=0A= =0A= Currently, when the FreeBSD NFS client does a write, it does:=0A= - VOP_WRITE() copies the data into buffer cache block(s).=0A= --> An nfsiod thread (or sometimes the thread that called VOP_WRITE()),=0A= copies the data from the buffer cache block into a list of mbuf clusters= ,=0A= prepends the NFS and RPC headers, then passes it down to TCP via sosend(= ).=0A= =0A= After the RPC reply is received (or the RPC fails due to timeout):=0A= - m_freem() is called on the mbuf list.=0A= - bufdone()/brelse() is called for the buffer cache block.=0A= =0A= For TLS, the mbuf list passed into sosend() must be ext_pgs mbufs, so the= =0A= mbuf clusters get copied to ext_pgs mbufs with anonymous pages before=0A= the sosend() call.=0A= =0A= So, what if the pages associated with the buffer cache block (b_pages)=0A= were entered in the m_epg_pa[] array for the ext_pgs mbufs, instead of=0A= copying the data into mbuf clusters?=0A= - At a glance, this just seems like it would work.=0A= It looks like the buffer cache pages are wired down until bufdone()/brels= e(),=0A= which happens after m_freem() on the mbuf list.=0A= - There would need to be a custom m__ext.ext_free, but it looks like a no-o= p.=0A= (ie. does nothing, since the buffer cache code deals with the pages later= .)=0A= =0A= The only thing I can think of (and I don't understand the vm/memory cache= =0A= parts of FreeBSD) is that, since the buffer cache pages are written via cop= ying=0A= into their kva addresses and then read via the direct map of their physical= =0A= pages, there might be some sort of memory cache flush needed to ensure the= =0A= physical pages are up to date (no data still working its way through write-= back).=0A= - Is this a problem and how is it handled?=0A= =0A= In summary, what am I missing that makes this difficult/impossible to do?= =0A= =0A= If no one has an answer, I'll just code it up and see what happens.=0A= =0A= Thanks for any comments, rick=0A= =0A=