From nobody Thu Sep 4 17:41:44 2025 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cHmxh6mMMz66TN0; Thu, 04 Sep 2025 17:41:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cHmxh4b9hz3Cch; Thu, 04 Sep 2025 17:41:44 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1757007704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=sXUYPnrbUNifNzCGjbQ0jO2GfEh6l8UTVvBViI7pU/k=; b=MagOK/cocncKZNOFLf2/ULJ2S1BUi+lCzYDtyCYOUP8XuutG4Zcszb9JyonZ4qAeFCUfQ7 cMuEG+y5axhzXSM0/R/sOG+ts8YIDWiG2GZBJwEjUI4ChLZnCk0R4UnJlIq9rMRGzJSsD3 XAgSWM+dvz4aqSjIHMIb5ClGJ3UYS7aCU8UaUyAGkDEh28xnXIZKjDzZRtqSw7MsgkSUri snxiFROPQR+YOCDLMmmZsgtyD5e3L50cXAY+G8SfYh40Ze/aScjWioccw/O7DOIXjticiX FNyvPrESEVCu74lF9bCSCIYNBQx3hH5NeEm2lQwuCVGDFe3SwgmqyWsjNLoa4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1757007704; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=sXUYPnrbUNifNzCGjbQ0jO2GfEh6l8UTVvBViI7pU/k=; b=NqyNcbUYjM7tgaOmYqWKG2e7HL2HIszVnZVYN2BKwszDpOsU9+3z4pa6vcuJfbob2IGT/K SKu2OjoVZpgM+WnlSGofU5CRoZWCLB1HK1M/Pyc2BUmE6fNZC7JWcod5KNMdfHBx4Y3lXT CqofTWEfyd39nwinG7NbWV5/Jh6RbHn2HqUaF9phNoaHZzJDM1B5v/WumOcIqVwfROqkUJ 2oLqCgedBejqBP4z7uoCI5dz/rdoFR8dcOutSOcqvbRJxa+PdCClMUNeDAPPDOQlNEpijg hBlJn03rKGHWRH5zywcXwioEWVbIRAQ7xF7Hy+2FpvZTbfZY0F46wPPnOEuqyQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1757007704; a=rsa-sha256; cv=none; b=fVRz7liXLahb7HnH/Lup/RQQUm2P5GNqd49lbylfa+KGIsuouYqb08nDYzQfeL1ImnVPmk DziOF3dU0aOZ/2jrazPvvDa3WR8N8B7nnJ4y7hHvFPRhHI1W0k/s+rkTTFC5J+l263qlfK 18xdNxUU1bhmIGwA66yYN+fwZtvqxBlZ1g6RRsj9paBfhMjGx8LfyJU/7s81Xhguk0oM+V zEhdwe/oTRwoDnMZyKusHlBH/z0KWNjwe8tgwWjJ1l0iIV4Kayvpd8Sxkkq0f6+BJ/5Ag3 OxY35UJkVH8+jJaj6j+naQGrflRBYcSULG/B3wFs8ygs2iBEj7V1pcOvTAGL+g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4cHmxh2KbZz1BxC; Thu, 04 Sep 2025 17:41:44 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 584HfijS074155; Thu, 4 Sep 2025 17:41:44 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 584HfiVv074152; Thu, 4 Sep 2025 17:41:44 GMT (envelope-from git) Date: Thu, 4 Sep 2025 17:41:44 GMT Message-Id: <202509041741.584HfiVv074152@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Andrew Turner Subject: git: a884f699e4bf - main - arm64: Add a multiple TLBI workaround List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-main@freebsd.org Sender: owner-dev-commits-src-main@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: andrew X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: a884f699e4bfc1be4e721d3ec4fa93915be18a86 Auto-Submitted: auto-generated The branch main has been updated by andrew: URL: https://cgit.FreeBSD.org/src/commit/?id=a884f699e4bfc1be4e721d3ec4fa93915be18a86 commit a884f699e4bfc1be4e721d3ec4fa93915be18a86 Author: Andrew Turner AuthorDate: 2025-09-04 17:24:56 +0000 Commit: Andrew Turner CommitDate: 2025-09-04 17:25:25 +0000 arm64: Add a multiple TLBI workaround The Arm Cortex-A55, Cortex-A76, and Cortex-A510 CPUs have errata that require multiple TLBI, DSB instructions to workaround. Add support to pmap to implement these. As it appears that all affected TLBI calls are via pmap.c this should be sufficient. As all variants of this erratum are Category-B (rare) require the user to enable it at boot time. Reviewed by: alc Sponsored by: Arm Ltd Differential Revision: https://reviews.freebsd.org/D52190 --- sys/arm64/arm64/pmap.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/sys/arm64/arm64/pmap.c b/sys/arm64/arm64/pmap.c index 0c359c6eea01..8a4395aa1c89 100644 --- a/sys/arm64/arm64/pmap.c +++ b/sys/arm64/arm64/pmap.c @@ -190,6 +190,8 @@ pt_entry_t __read_mostly pmap_gp_attr; #define PMAP_SAN_PTE_BITS (ATTR_AF | ATTR_S1_XN | pmap_sh_attr | \ ATTR_KERN_GP | ATTR_S1_IDX(VM_MEMATTR_WRITE_BACK) | ATTR_S1_AP(ATTR_S1_AP_RW)) +static bool __read_mostly pmap_multiple_tlbi = false; + struct pmap_large_md_page { struct rwlock pv_lock; struct md_page pv_page; @@ -1723,6 +1725,51 @@ CPU_FEAT(feat_hafdbs, "Hardware management of the Access flag and dirty state", pmap_dbm_check, pmap_dbm_has_errata, pmap_dbm_enable, CPU_FEAT_AFTER_DEV | CPU_FEAT_PER_CPU); +static cpu_feat_en +pmap_multiple_tlbi_check(const struct cpu_feat *feat __unused, u_int midr) +{ + /* + * Cortex-A55 erratum 2441007 (Cat B rare) + * Present in all revisions + */ + if (CPU_IMPL(midr) == CPU_IMPL_ARM && + CPU_PART(midr) == CPU_PART_CORTEX_A55) + return (FEAT_DEFAULT_DISABLE); + + /* + * Cortex-A76 erratum 1286807 (Cat B rare) + * Present in r0p0 - r3p0 + * Fixed in r3p1 + */ + if (midr_check_var_part_range(midr, CPU_IMPL_ARM, CPU_PART_CORTEX_A76, + 0, 0, 3, 0)) + return (FEAT_DEFAULT_DISABLE); + + /* + * Cortex-A510 erratum 2441009 (Cat B rare) + * Present in r0p0 - r1p1 + * Fixed in r1p2 + */ + if (midr_check_var_part_range(midr, CPU_IMPL_ARM, CPU_PART_CORTEX_A510, + 0, 0, 1, 1)) + return (FEAT_DEFAULT_DISABLE); + + return (FEAT_ALWAYS_DISABLE); +} + +static bool +pmap_multiple_tlbi_enable(const struct cpu_feat *feat __unused, + cpu_feat_errata errata_status, u_int *errata_list __unused, + u_int errata_count __unused) +{ + pmap_multiple_tlbi = true; + return (true); +} + +CPU_FEAT(errata_multi_tlbi, "Multiple TLBI errata", + pmap_multiple_tlbi_check, NULL, pmap_multiple_tlbi_enable, + CPU_FEAT_EARLY_BOOT | CPU_FEAT_PER_CPU); + /* * Initialize the pmap module. * @@ -1876,9 +1923,17 @@ pmap_s1_invalidate_page(pmap_t pmap, vm_offset_t va, bool final_only) r = TLBI_VA(va); if (pmap == kernel_pmap) { pmap_s1_invalidate_kernel(r, final_only); + if (pmap_multiple_tlbi) { + dsb(ish); + pmap_s1_invalidate_kernel(r, final_only); + } } else { r |= ASID_TO_OPERAND(COOKIE_TO_ASID(pmap->pm_cookie)); pmap_s1_invalidate_user(r, final_only); + if (pmap_multiple_tlbi) { + dsb(ish); + pmap_s1_invalidate_user(r, final_only); + } } dsb(ish); isb(); @@ -1920,12 +1975,24 @@ pmap_s1_invalidate_strided(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, end = TLBI_VA(eva); for (r = start; r < end; r += TLBI_VA(stride)) pmap_s1_invalidate_kernel(r, final_only); + + if (pmap_multiple_tlbi) { + dsb(ish); + for (r = start; r < end; r += TLBI_VA(stride)) + pmap_s1_invalidate_kernel(r, final_only); + } } else { start = end = ASID_TO_OPERAND(COOKIE_TO_ASID(pmap->pm_cookie)); start |= TLBI_VA(sva); end |= TLBI_VA(eva); for (r = start; r < end; r += TLBI_VA(stride)) pmap_s1_invalidate_user(r, final_only); + + if (pmap_multiple_tlbi) { + dsb(ish); + for (r = start; r < end; r += TLBI_VA(stride)) + pmap_s1_invalidate_user(r, final_only); + } } dsb(ish); isb(); @@ -1967,6 +2034,10 @@ pmap_s1_invalidate_all_kernel(void) dsb(ishst); __asm __volatile("tlbi vmalle1is"); dsb(ish); + if (pmap_multiple_tlbi) { + __asm __volatile("tlbi vmalle1is"); + dsb(ish); + } isb(); } @@ -1984,9 +2055,17 @@ pmap_s1_invalidate_all(pmap_t pmap) dsb(ishst); if (pmap == kernel_pmap) { __asm __volatile("tlbi vmalle1is"); + if (pmap_multiple_tlbi) { + dsb(ish); + __asm __volatile("tlbi vmalle1is"); + } } else { r = ASID_TO_OPERAND(COOKIE_TO_ASID(pmap->pm_cookie)); __asm __volatile("tlbi aside1is, %0" : : "r" (r)); + if (pmap_multiple_tlbi) { + dsb(ish); + __asm __volatile("tlbi aside1is, %0" : : "r" (r)); + } } dsb(ish); isb();