From nobody Wed Jun 28 17:53:06 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Qrq1b0MJ9z4kL4p; Wed, 28 Jun 2023 17:53:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Qrq1Z73n0z42Z9; Wed, 28 Jun 2023 17:53:06 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1687974787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=GnFy+qIDiJdSztPeGWNscfR9FOp/lkz/iw78piMMIoA=; b=FXRZQtKRftW7DyPDlON+SNeqEsYVElm0jDw9tsoZ73xuvLkR7l7LV5TTAKQ4dzkYSx6Vy4 x3UOTbCiJEFBMx8fMMPVAiu4+9SUWleiL6TyKo8Ng8DEu0vaQHd7TH+NSbouHYK6ESvrBk mQxwM8jndjBnwPxhfN56wvAXMXhJcZC0fMu1+oS/+NEBpteq8czxhRn26k8frSAS7njkvf BuZoE6a1yEG1OZjyd9ZnyKpF4CpjkmKDVRMWKeaehQbh4FGGhrq8bJv9vKPKsydbHtWA22 xMFVdilLyAf/Bwv7Ov6ZV3MWFP+QPKoyFzOpz3MPZ+P+2wIxPGLsuWGY2CZSgQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1687974787; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=GnFy+qIDiJdSztPeGWNscfR9FOp/lkz/iw78piMMIoA=; b=fwx69glpxR8bYr1K9LzlI0R/l/pADS4kIZLMu+putDiq4I4zQcKWZS1nGIhdBEp+HBVUJN lDbCMthlUUGrj34hQ6IHwBaAefp8scWWepWDCUhf7idPejCEAztHukPcKOWXJFm5fbyY7i LxezqmM4aFqK/+YqRXnenYiKAV8vLi2s8B5WTLM/JeadHpnMRi4YDvn0JlR10TeqM0MJPf PxCiAOLiF+5NmFCkZofTLEbBsHQtL1V8Sr9Ya4wDkVU6Wnfw7veHTolJ+EeYha9DfaAV7a wcgM024+Szw+6jP0a27HyEhjjR/Ha2bTfajh9z53f+D/vCsql+CROLslflYV4w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1687974787; a=rsa-sha256; cv=none; b=YgYbsXUHdHsLINcFWzEAmNtxWRS9oYwj+qSgp1gkPWt5DvcvqFuGIAHKGs/ASwpWYZqZ0U CijbfMaK2Nrpeg91/ndWVnzLm0/y9uJmwAy6W7WoZXU39ybTZUiWRGd3u+zB0cmg2T8hSt R5/jC9H4FCoCfG7cjTqbRoJbnVWIYPrG7cu/F1edthVLIfJiV3pnICT0OKzUJfsT2Xgfgx jH078E6ZBqxzylYxyhYvzHLj9TQdoK/S1t45y+/irMr8YBTtc9xn0m+85XlxYxsRuV3/J7 otu4lAg2Ar3KBdmZy1uYcIFrZDXnWFjcjZmkhJNlGqNt7XF0HirJoH7B6m2CkA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Qrq1Z5rlpzjxN; Wed, 28 Jun 2023 17:53:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 35SHr6FA074550; Wed, 28 Jun 2023 17:53:06 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 35SHr6CA074549; Wed, 28 Jun 2023 17:53:06 GMT (envelope-from git) Date: Wed, 28 Jun 2023 17:53:06 GMT Message-Id: <202306281753.35SHr6CA074549@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: 3767de839742 - main - arm64 pmap: Tidy up pmap_promote_l2() calls List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 3767de83974206e4267dabf7fbe66b151c1a0b14 Auto-Submitted: auto-generated X-ThisMailContainsUnwantedMimeParts: N The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=3767de83974206e4267dabf7fbe66b151c1a0b14 commit 3767de83974206e4267dabf7fbe66b151c1a0b14 Author: Alan Cox AuthorDate: 2023-06-28 07:08:02 +0000 Commit: Alan Cox CommitDate: 2023-06-28 17:46:15 +0000 arm64 pmap: Tidy up pmap_promote_l2() calls Since pmap_ps_enabled() is true by default, check it inside of pmap_promote_l2() instead of at every call site. Modify pmap_promote_l2() to return true if the promotion succeeded and false otherwise. (A similar change was applied to the amd64 pmap in 0d2f98c2f092.) Reviewed by: kib, markj Differential Revision: https://reviews.freebsd.org/D40781 --- sys/arm64/arm64/pmap.c | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index 3166b3d7959b..46520889728f 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -435,7 +435,7 @@ void (*pmap_stage2_invalidate_all)(uint64_t); #define TLBI_VA(addr) (((addr) >> TLBI_VA_SHIFT) & TLBI_VA_MASK) #define TLBI_VA_L3_INCR (L3_SIZE >> TLBI_VA_SHIFT) -static int superpages_enabled = 1; +static int __read_frequently superpages_enabled = 1; SYSCTL_INT(_vm_pmap, OID_AUTO, superpages_enabled, CTLFLAG_RDTUN | CTLFLAG_NOFETCH, &superpages_enabled, 0, "Are large page mappings enabled?"); @@ -4141,14 +4141,21 @@ pmap_pv_promote_l2(pmap_t pmap, vm_offset_t va, vm_paddr_t pa, * aligned, contiguous physical memory and (2) the 4KB page mappings must have * identical characteristics. */ -static void +static bool pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t mpte, struct rwlock **lockp) { pt_entry_t all_l3e_AF, *firstl3, *l3, newl2, oldl3, pa; PMAP_LOCK_ASSERT(pmap, MA_OWNED); - PMAP_ASSERT_STAGE1(pmap); + + /* + * Currently, this function only supports promotion on stage 1 pmaps + * because it tests stage 1 specific fields and performs a break- + * before-make sequence that is incorrect for stage 2 pmaps. + */ + if (pmap->pm_stage != PM_STAGE1 || !pmap_ps_enabled(pmap)) + return (false); /* * Examine the first L3E in the specified PTP. Abort if this L3E is @@ -4157,14 +4164,14 @@ pmap_promote_l2(pmap_t pmap, pd_entry_t *l2, vm_offset_t va, vm_page_t mpte, firstl3 = (pt_entry_t *)PHYS_TO_DMAP(PTE_TO_PHYS(pmap_load(l2))); newl2 = pmap_load(firstl3); if ((newl2 & ATTR_SW_NO_PROMOTE) != 0) - return; + return (false); /* ... is not the first physical page within an L2 block */ if ((PTE_TO_PHYS(newl2) & L2_OFFSET) != 0 || ((newl2 & ATTR_DESCR_MASK) != L3_PAGE)) { /* ... or is invalid */ atomic_add_long(&pmap_l2_p_failures, 1); CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx" " in pmap %p", va, pmap); - return; + return (false); } /* @@ -4212,7 +4219,7 @@ setl2: atomic_add_long(&pmap_l2_p_failures, 1); CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx" " in pmap %p", va, pmap); - return; + return (false); } setl3: if ((oldl3 & (ATTR_S1_AP_RW_BIT | ATTR_SW_DBM)) == @@ -4232,7 +4239,7 @@ setl3: atomic_add_long(&pmap_l2_p_failures, 1); CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx" " in pmap %p", va, pmap); - return; + return (false); } all_l3e_AF &= oldl3; pa -= PAGE_SIZE; @@ -4263,7 +4270,7 @@ setl3: CTR2(KTR_PMAP, "pmap_promote_l2: failure for va %#lx in pmap %p", va, pmap); - return; + return (false); } if ((newl2 & ATTR_SW_MANAGED) != 0) @@ -4277,6 +4284,7 @@ setl3: atomic_add_long(&pmap_l2_promotions, 1); CTR2(KTR_PMAP, "pmap_promote_l2: success for va %#lx in pmap %p", va, pmap); + return (true); } #endif /* VM_NRESERVLEVEL > 0 */ @@ -4681,17 +4689,13 @@ validate: #if VM_NRESERVLEVEL > 0 /* - * Try to promote from level 3 pages to a level 2 superpage. This - * currently only works on stage 1 pmaps as pmap_promote_l2 looks at - * stage 1 specific fields and performs a break-before-make sequence - * that is incorrect a stage 2 pmap. + * If both the page table page and the reservation are fully + * populated, then attempt promotion. */ if ((mpte == NULL || mpte->ref_count == NL3PG) && - pmap_ps_enabled(pmap) && pmap->pm_stage == PM_STAGE1 && (m->flags & PG_FICTITIOUS) == 0 && - vm_reserv_level_iffullpop(m) == 0) { - pmap_promote_l2(pmap, pde, va, mpte, &lock); - } + vm_reserv_level_iffullpop(m) == 0) + (void)pmap_promote_l2(pmap, pde, va, mpte, &lock); #endif rv = KERN_SUCCESS; @@ -5096,18 +5100,17 @@ pmap_enter_quick_locked(pmap_t pmap, vm_offset_t va, vm_page_t m, * attempt promotion. */ if ((mpte == NULL || mpte->ref_count == NL3PG) && - pmap_ps_enabled(pmap) && pmap->pm_stage == PM_STAGE1 && (m->flags & PG_FICTITIOUS) == 0 && vm_reserv_level_iffullpop(m) == 0) { if (l2 == NULL) l2 = pmap_pde(pmap, va, &lvl); - pmap_promote_l2(pmap, l2, va, mpte, lockp); /* * If promotion succeeds, then the next call to this function * should not be given the unmapped PTP as a hint. */ - mpte = NULL; + if (pmap_promote_l2(pmap, l2, va, mpte, lockp)) + mpte = NULL; } #endif