From owner-dev-commits-src-all@freebsd.org Sun Jun 6 20:45:13 2021 Return-Path: Delivered-To: dev-commits-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D36163DB9B; Sun, 6 Jun 2021 20:45:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FypRF1MbLz3v6C; Sun, 6 Jun 2021 20:45:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 118EC209F; Sun, 6 Jun 2021 20:45:13 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 156KjC8e052006; Sun, 6 Jun 2021 20:45:12 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 156KjC4Z052005; Sun, 6 Jun 2021 20:45:12 GMT (envelope-from git) Date: Sun, 6 Jun 2021 20:45:12 GMT Message-Id: <202106062045.156KjC4Z052005@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: c05748e028b8 - main - riscv: Handle hardware-managed dirty bit updates in pmap_promote_l2() MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: c05748e028b84c216d0161e70418f8cb09e074e4 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-all@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for all branches of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jun 2021 20:45:13 -0000 The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=c05748e028b84c216d0161e70418f8cb09e074e4 commit c05748e028b84c216d0161e70418f8cb09e074e4 Author: Mark Johnston AuthorDate: 2021-06-06 20:41:35 +0000 Commit: Mark Johnston CommitDate: 2021-06-06 20:44:46 +0000 riscv: Handle hardware-managed dirty bit updates in pmap_promote_l2() pmap_promote_l2() failed to handle implementations which set the accessed and dirty flags. In particular, when comparing the attributes of a run of 512 PTEs, we must handle the possibility that the hardware will set PTE_D on a clean, writable mapping. Following the example of amd64 and arm64, change riscv's pmap_promote_l2() to downgrade clean, writable mappings to read-only, so that updates are synchronized by the pmap lock. Fixes: f6893f09d Reported by: Nathaniel Filardo Tested by: Nathaniel Filardo Reviewed by: jrtc27, alc, Nathaniel Filardo MFC after: 1 week Sponsored by: The FreeBSD Foundation Differential Revision: https://reviews.freebsd.org/D30644 --- sys/riscv/riscv/pmap.c | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/sys/riscv/riscv/pmap.c b/sys/riscv/riscv/pmap.c index d0b51d5199e2..747e75b765b2 100644 --- a/sys/riscv/riscv/pmap.c +++ b/sys/riscv/riscv/pmap.c @@ -2540,7 +2540,7 @@ static void pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, struct rwlock **lockp) { - pt_entry_t *firstl3, *l3; + pt_entry_t *firstl3, firstl3e, *l3, l3e; vm_paddr_t pa; vm_page_t ml3; @@ -2551,7 +2551,8 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, ("pmap_promote_l2: invalid l2 entry %p", l2)); firstl3 = (pt_entry_t *)PHYS_TO_DMAP(PTE_TO_PHYS(pmap_load(l2))); - pa = PTE_TO_PHYS(pmap_load(firstl3)); + firstl3e = pmap_load(firstl3); + pa = PTE_TO_PHYS(firstl3e); if ((pa & L2_OFFSET) != 0) { CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx pmap %p", va, pmap); @@ -2559,17 +2560,40 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, return; } + /* + * Downgrade a clean, writable mapping to read-only to ensure that the + * hardware does not set PTE_D while we are comparing PTEs. + * + * Upon a write access to a clean mapping, the implementation will + * either atomically check protections and set PTE_D, or raise a page + * fault. In the latter case, the pmap lock provides atomicity. Thus, + * we do not issue an sfence.vma here and instead rely on pmap_fault() + * to do so lazily. + */ + while ((firstl3e & (PTE_W | PTE_D)) == PTE_W) { + if (atomic_fcmpset_64(firstl3, &firstl3e, firstl3e & ~PTE_W)) { + firstl3e &= ~PTE_W; + break; + } + } + pa += PAGE_SIZE; for (l3 = firstl3 + 1; l3 < firstl3 + Ln_ENTRIES; l3++) { - if (PTE_TO_PHYS(pmap_load(l3)) != pa) { + l3e = pmap_load(l3); + if (PTE_TO_PHYS(l3e) != pa) { CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx pmap %p", va, pmap); atomic_add_long(&pmap_l2_p_failures, 1); return; } - if ((pmap_load(l3) & PTE_PROMOTE) != - (pmap_load(firstl3) & PTE_PROMOTE)) { + while ((l3e & (PTE_W | PTE_D)) == PTE_W) { + if (atomic_fcmpset_64(l3, &l3e, l3e & ~PTE_W)) { + l3e &= ~PTE_W; + break; + } + } + if ((l3e & PTE_PROMOTE) != (firstl3e & PTE_PROMOTE)) { CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx pmap %p", va, pmap); @@ -2589,11 +2613,10 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, return; } - if ((pmap_load(firstl3) & PTE_SW_MANAGED) != 0) - pmap_pv_promote_l2(pmap, va, PTE_TO_PHYS(pmap_load(firstl3)), - lockp); + if ((firstl3e & PTE_SW_MANAGED) != 0) + pmap_pv_promote_l2(pmap, va, PTE_TO_PHYS(firstl3e), lockp); - pmap_store(l2, pmap_load(firstl3)); + pmap_store(l2, firstl3e); atomic_add_long(&pmap_l2_promotions, 1); CTR2(KTR_PMAP, "pmap_promote_l2: success for va %#lx in pmap %p", va,