From nobody Wed Jul 26 05:37:52 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R9jNK1DhSz4pjRh; Wed, 26 Jul 2023 05:37:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R9jNJ5Tz1z46sD; Wed, 26 Jul 2023 05:37:52 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1690349872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=6r6II68ntkYIzweH/Zk9YJHQTZbCjQyyfy9Njw9PQwA=; b=AhyrsadML8irh6JrjwjdQInorh0OrY514fLulKdhfJspw+eSCrKQCqunsRzzOPGffMZrF1 Ljcq87IVV0EAE5fImZKkNVzTAp7I+lfAmUKZWKOJKWFIPkIN2hCqtZaoW9bLZGCDT8/e+/ k7mVyGT7DPl+HJXTBWnyM0YthfaSIBZpkHj/F58tK80RQS8B1Nw1lNoamQRiIH+KjjTz05 qp4rMzVgS8J2XJaniYMYJhXrndZHotFGmkP0lT0yMTVf/R5RoTrYVY3UsJTqhpYgHCI2SD AfBjZ/s//hoPnJLlRRVhLwR2bnl/RUte2e9uAnrBCL6S0e6RFf+kgHLPipjgKA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1690349872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=6r6II68ntkYIzweH/Zk9YJHQTZbCjQyyfy9Njw9PQwA=; b=Z4nipQXULpiQHCozc+xi7+OTLOqqi2wt7bMqMCgv2ef4o2OzcdqChG5e0+17lGbJMZL0sR 1RPhAcm1FOc4ljI/H37CBhtzrD5DjLkbQB/kSmicmHU14FYJZW24labBs7iwbLWQg6U5Yd QQPcz85MCPfkKfOA48iT7DzelFhmx5V77GpeOWeW8VOzVtXPRNtFIB8WQJaZeu5wZHpueJ texi210rv3vkg/lrgmctJb7Y7Ibn52XIYtKNwCz+AlWW3DhCqB6tZGCW7MS6fXLMLwJNo8 NwagAHWI/805+QNhEFwUV94oO4FnTUC9ytolzVtKjKggovSeE8pzpxXMOKyyxg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1690349872; a=rsa-sha256; cv=none; b=nnXRY7IGmMWlgO7lcLK9fiRJEmUJakZp3koZqZWUA5KGsJfPX5G/bYHt5wqXVljOc01onF vIx55MrPzjuZuI9uA1i7EtG43uwCUINqDIeJfXjpaT3y1VanRg5nCNEW76xbGKhe0t9cs6 UnKU/Bl2Z2B0xeO1+L7/pInu31+lMDcSrNX4vIkLih42GPCbgwUpOSThbqoCqiRMu0w3jX XL2/6rsZ1NbBaIfswL2mJDVZQos7ZJkv8gpksdJfpFHzxWPtdyrSHDE+kZZLWRLg1+wKHp acb24uaMmmIL/myNYcbdtIAINgoX30NF1kiUQHLmbT44fOBC+xlQ3g6IlhgAcQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4R9jNJ4XyFzW4v; Wed, 26 Jul 2023 05:37:52 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 36Q5bqfL043238; Wed, 26 Jul 2023 05:37:52 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 36Q5bqTh043237; Wed, 26 Jul 2023 05:37:52 GMT (envelope-from git) Date: Wed, 26 Jul 2023 05:37:52 GMT Message-Id: <202307260537.36Q5bqTh043237@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Alan Cox Subject: git: a98a0090b2ba - main - arm64 pmap: Eliminate unnecessary TLB invalidations List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: alc X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a98a0090b2ba64ff0bc3cf71a00fb5f9e31fc1d3 Auto-Submitted: auto-generated The branch main has been updated by alc: URL: https://cgit.FreeBSD.org/src/commit/?id=a98a0090b2ba64ff0bc3cf71a00fb5f9e31fc1d3 commit a98a0090b2ba64ff0bc3cf71a00fb5f9e31fc1d3 Author: Alan Cox AuthorDate: 2023-07-23 07:11:43 +0000 Commit: Alan Cox CommitDate: 2023-07-26 05:37:13 +0000 arm64 pmap: Eliminate unnecessary TLB invalidations Eliminate unnecessary TLB invalidations by pmap_kenter(), pmap_qenter(), and pmap_mapbios() when the old page table entries were invalid. While I'm here, correct some nearby whitespace issues. MFC after: 2 weeks --- sys/arm64/arm64/pmap.c | 49 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index 379296f375ae..fa09d2026550 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -1972,19 +1972,20 @@ void pmap_kenter(vm_offset_t sva, vm_size_t size, vm_paddr_t pa, int mode) { pd_entry_t *pde; - pt_entry_t *pte, attr; + pt_entry_t attr, old_l3e, *pte; vm_offset_t va; int lvl; KASSERT((pa & L3_OFFSET) == 0, - ("pmap_kenter: Invalid physical address")); + ("pmap_kenter: Invalid physical address")); KASSERT((sva & L3_OFFSET) == 0, - ("pmap_kenter: Invalid virtual address")); + ("pmap_kenter: Invalid virtual address")); KASSERT((size & PAGE_MASK) == 0, ("pmap_kenter: Mapping is not page-sized")); attr = ATTR_DEFAULT | ATTR_S1_AP(ATTR_S1_AP_RW) | ATTR_S1_XN | ATTR_S1_IDX(mode) | L3_PAGE; + old_l3e = 0; va = sva; while (size != 0) { pde = pmap_pde(kernel_pmap, va, &lvl); @@ -1993,13 +1994,21 @@ pmap_kenter(vm_offset_t sva, vm_size_t size, vm_paddr_t pa, int mode) KASSERT(lvl == 2, ("pmap_kenter: Invalid level %d", lvl)); pte = pmap_l2_to_l3(pde, va); - pmap_load_store(pte, PHYS_TO_PTE(pa) | attr); + old_l3e |= pmap_load_store(pte, PHYS_TO_PTE(pa) | attr); va += PAGE_SIZE; pa += PAGE_SIZE; size -= PAGE_SIZE; } - pmap_s1_invalidate_range(kernel_pmap, sva, va, true); + if ((old_l3e & ATTR_DESCR_VALID) != 0) + pmap_s1_invalidate_range(kernel_pmap, sva, va, true); + else { + /* + * Because the old entries were invalid and the new mappings + * are not executable, an isb is not required. + */ + dsb(ishst); + } } void @@ -2082,11 +2091,12 @@ void pmap_qenter(vm_offset_t sva, vm_page_t *ma, int count) { pd_entry_t *pde; - pt_entry_t *pte, pa, attr; + pt_entry_t attr, old_l3e, pa, *pte; vm_offset_t va; vm_page_t m; int i, lvl; + old_l3e = 0; va = sva; for (i = 0; i < count; i++) { pde = pmap_pde(kernel_pmap, va, &lvl); @@ -2100,11 +2110,19 @@ pmap_qenter(vm_offset_t sva, vm_page_t *ma, int count) attr = ATTR_DEFAULT | ATTR_S1_AP(ATTR_S1_AP_RW) | ATTR_S1_XN | ATTR_S1_IDX(m->md.pv_memattr) | L3_PAGE; pte = pmap_l2_to_l3(pde, va); - pmap_load_store(pte, PHYS_TO_PTE(pa) | attr); + old_l3e |= pmap_load_store(pte, PHYS_TO_PTE(pa) | attr); va += L3_SIZE; } - pmap_s1_invalidate_range(kernel_pmap, sva, va, true); + if ((old_l3e & ATTR_DESCR_VALID) != 0) + pmap_s1_invalidate_range(kernel_pmap, sva, va, true); + else { + /* + * Because the old entries were invalid and the new mappings + * are not executable, an isb is not required. + */ + dsb(ishst); + } } /* @@ -6441,7 +6459,7 @@ pmap_mapbios(vm_paddr_t pa, vm_size_t size) { struct pmap_preinit_mapping *ppim; vm_offset_t va, offset; - pd_entry_t *pde; + pd_entry_t old_l2e, *pde; pt_entry_t *l2; int i, lvl, l2_blocks, free_l2_count, start_idx; @@ -6501,6 +6519,7 @@ pmap_mapbios(vm_paddr_t pa, vm_size_t size) /* Map L2 blocks */ pa = rounddown2(pa, L2_SIZE); + old_l2e = 0; for (i = 0; i < l2_blocks; i++) { pde = pmap_pde(kernel_pmap, va, &lvl); KASSERT(pde != NULL, @@ -6511,14 +6530,22 @@ pmap_mapbios(vm_paddr_t pa, vm_size_t size) /* Insert L2_BLOCK */ l2 = pmap_l1_to_l2(pde, va); - pmap_load_store(l2, + old_l2e |= pmap_load_store(l2, PHYS_TO_PTE(pa) | ATTR_DEFAULT | ATTR_S1_XN | ATTR_S1_IDX(VM_MEMATTR_WRITE_BACK) | L2_BLOCK); va += L2_SIZE; pa += L2_SIZE; } - pmap_s1_invalidate_all(kernel_pmap); + if ((old_l2e & ATTR_DESCR_VALID) != 0) + pmap_s1_invalidate_all(kernel_pmap); + else { + /* + * Because the old entries were invalid and the new + * mappings are not executable, an isb is not required. + */ + dsb(ishst); + } va = preinit_map_va + (start_idx * L2_SIZE);