From owner-dev-commits-src-all@freebsd.org Sun Jun 6 20:45:13 2021
Return-Path: <owner-dev-commits-src-all@freebsd.org>
Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D36163DB9B;
Sun, 6 Jun 2021 20:45:13 +0000 (UTC) (envelope-from git@FreeBSD.org)
Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org
[IPv6:2610:1c1:1:606c::19:3])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
client-signature RSA-PSS (4096 bits) client-digest SHA256)
(Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK))
by mx1.freebsd.org (Postfix) with ESMTPS id 4FypRF1MbLz3v6C;
Sun, 6 Jun 2021 20:45:13 +0000 (UTC) (envelope-from git@FreeBSD.org)
Received: from gitrepo.freebsd.org (gitrepo.freebsd.org
[IPv6:2610:1c1:1:6068::e6a:5])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256)
(Client did not present a certificate)
by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 118EC209F;
Sun, 6 Jun 2021 20:45:13 +0000 (UTC) (envelope-from git@FreeBSD.org)
Received: from gitrepo.freebsd.org ([127.0.1.44])
by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 156KjC8e052006;
Sun, 6 Jun 2021 20:45:12 GMT (envelope-from git@gitrepo.freebsd.org)
Received: (from git@localhost)
by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 156KjC4Z052005;
Sun, 6 Jun 2021 20:45:12 GMT (envelope-from git)
Date: Sun, 6 Jun 2021 20:45:12 GMT
Message-Id: <202106062045.156KjC4Z052005@gitrepo.freebsd.org>
To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org,
dev-commits-src-main@FreeBSD.org
From: Mark Johnston <markj@FreeBSD.org>
Subject: git: c05748e028b8 - main - riscv: Handle hardware-managed dirty bit
updates in pmap_promote_l2()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
X-Git-Committer: markj
X-Git-Repository: src
X-Git-Refname: refs/heads/main
X-Git-Reftype: branch
X-Git-Commit: c05748e028b84c216d0161e70418f8cb09e074e4
Auto-Submitted: auto-generated
X-BeenThere: dev-commits-src-all@freebsd.org
X-Mailman-Version: 2.1.34
Precedence: list
List-Id: Commit messages for all branches of the src repository
<dev-commits-src-all.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/dev-commits-src-all>,
<mailto:dev-commits-src-all-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/dev-commits-src-all/>
List-Post: <mailto:dev-commits-src-all@freebsd.org>
List-Help: <mailto:dev-commits-src-all-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/dev-commits-src-all>,
<mailto:dev-commits-src-all-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 06 Jun 2021 20:45:13 -0000
The branch main has been updated by markj:
URL: https://cgit.FreeBSD.org/src/commit/?id=c05748e028b84c216d0161e70418f8cb09e074e4
commit c05748e028b84c216d0161e70418f8cb09e074e4
Author: Mark Johnston <markj@FreeBSD.org>
AuthorDate: 2021-06-06 20:41:35 +0000
Commit: Mark Johnston <markj@FreeBSD.org>
CommitDate: 2021-06-06 20:44:46 +0000
riscv: Handle hardware-managed dirty bit updates in pmap_promote_l2()
pmap_promote_l2() failed to handle implementations which set the
accessed and dirty flags. In particular, when comparing the attributes
of a run of 512 PTEs, we must handle the possibility that the hardware
will set PTE_D on a clean, writable mapping.
Following the example of amd64 and arm64, change riscv's
pmap_promote_l2() to downgrade clean, writable mappings to read-only, so
that updates are synchronized by the pmap lock.
Fixes: f6893f09d
Reported by: Nathaniel Filardo <nwf20@cl.cam.ac.uk>
Tested by: Nathaniel Filardo <nwf20@cl.cam.ac.uk>
Reviewed by: jrtc27, alc, Nathaniel Filardo
MFC after: 1 week
Sponsored by: The FreeBSD Foundation
Differential Revision: https://reviews.freebsd.org/D30644
---
sys/riscv/riscv/pmap.c | 41 ++++++++++++++++++++++++++++++++---------
1 file changed, 32 insertions(+), 9 deletions(-)
diff --git a/sys/riscv/riscv/pmap.c b/sys/riscv/riscv/pmap.c
index d0b51d5199e2..747e75b765b2 100644
--- a/sys/riscv/riscv/pmap.c
+++ b/sys/riscv/riscv/pmap.c
@@ -2540,7 +2540,7 @@ static void
pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va,
struct rwlock **lockp)
{
- pt_entry_t *firstl3, *l3;
+ pt_entry_t *firstl3, firstl3e, *l3, l3e;
vm_paddr_t pa;
vm_page_t ml3;
@@ -2551,7 +2551,8 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va,
("pmap_promote_l2: invalid l2 entry %p", l2));
firstl3 = (pt_entry_t *)PHYS_TO_DMAP(PTE_TO_PHYS(pmap_load(l2)));
- pa = PTE_TO_PHYS(pmap_load(firstl3));
+ firstl3e = pmap_load(firstl3);
+ pa = PTE_TO_PHYS(firstl3e);
if ((pa & L2_OFFSET) != 0) {
CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx pmap %p",
va, pmap);
@@ -2559,17 +2560,40 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va,
return;
}
+ /*
+ * Downgrade a clean, writable mapping to read-only to ensure that the
+ * hardware does not set PTE_D while we are comparing PTEs.
+ *
+ * Upon a write access to a clean mapping, the implementation will
+ * either atomically check protections and set PTE_D, or raise a page
+ * fault. In the latter case, the pmap lock provides atomicity. Thus,
+ * we do not issue an sfence.vma here and instead rely on pmap_fault()
+ * to do so lazily.
+ */
+ while ((firstl3e & (PTE_W | PTE_D)) == PTE_W) {
+ if (atomic_fcmpset_64(firstl3, &firstl3e, firstl3e & ~PTE_W)) {
+ firstl3e &= ~PTE_W;
+ break;
+ }
+ }
+
pa += PAGE_SIZE;
for (l3 = firstl3 + 1; l3 < firstl3 + Ln_ENTRIES; l3++) {
- if (PTE_TO_PHYS(pmap_load(l3)) != pa) {
+ l3e = pmap_load(l3);
+ if (PTE_TO_PHYS(l3e) != pa) {
CTR2(KTR_PMAP,
"pmap_promote_l2: failure for va %#lx pmap %p",
va, pmap);
atomic_add_long(&pmap_l2_p_failures, 1);
return;
}
- if ((pmap_load(l3) & PTE_PROMOTE) !=
- (pmap_load(firstl3) & PTE_PROMOTE)) {
+ while ((l3e & (PTE_W | PTE_D)) == PTE_W) {
+ if (atomic_fcmpset_64(l3, &l3e, l3e & ~PTE_W)) {
+ l3e &= ~PTE_W;
+ break;
+ }
+ }
+ if ((l3e & PTE_PROMOTE) != (firstl3e & PTE_PROMOTE)) {
CTR2(KTR_PMAP,
"pmap_promote_l2: failure for va %#lx pmap %p",
va, pmap);
@@ -2589,11 +2613,10 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va,
return;
}
- if ((pmap_load(firstl3) & PTE_SW_MANAGED) != 0)
- pmap_pv_promote_l2(pmap, va, PTE_TO_PHYS(pmap_load(firstl3)),
- lockp);
+ if ((firstl3e & PTE_SW_MANAGED) != 0)
+ pmap_pv_promote_l2(pmap, va, PTE_TO_PHYS(firstl3e), lockp);
- pmap_store(l2, pmap_load(firstl3));
+ pmap_store(l2, firstl3e);
atomic_add_long(&pmap_l2_promotions, 1);
CTR2(KTR_PMAP, "pmap_promote_l2: success for va %#lx in pmap %p", va,