From nobody Sun Jul 30 15:47:17 2023 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RDQjv1dG8z4p9Nd; Sun, 30 Jul 2023 15:47:20 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RDQjg3S9Rz4Kfc; Sun, 30 Jul 2023 15:47:19 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1690732039; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hjJ6nKOe1I8y1DeSFwb3ddzdb7TtdyoTIdc4/Ft3ApU=; b=AV6pnWObMtWegxQ3uEJOFmLfgUftlLbN0kmZ+3eEP/FbVVncUPeLO+cz0SGBkgtAwBIRe1 mS30Gork4G6HatNkEYMcVujjfk+E1D/UYsxKQF5nFOZpFATLWCHex0cDqhX9BEyHQzcQUB X3MLaq1PPPLFKoHgAeoUupaFuZQuF7nVQtU1lgKXtD6yjaYnDLhxKaNt1eDWKGDAiJGu2i htxIPRlsR9T9omZ90r8GVVJeoszZNOZi8E9Y/hGjBHhXFeutt+4S5/k/sby3OEy4L3vwjZ 4pn19ASAzg7JXRtT7isQ8Q0SAAOP7+HDhieK4NJg8QB+E5XsWPJ1t7kfFbJZZw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1690732039; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=hjJ6nKOe1I8y1DeSFwb3ddzdb7TtdyoTIdc4/Ft3ApU=; b=aSykVwm1gq875NV86gGQTSiOEqvCqtoZ77chi4ZLXbq+1+RtrW5wM8yD47q2axuBztPCL+ +mSjV10bG8NhKgbyau9jXHz1eiPsiA6rB2TJyItzy7hMq4FlLipo/fuKQBlU9lqOlhyZO/ zl/l8PmowJ4XEDF/Jhmw2qIfbf05q97W7AEE2ZpDmdm3bHBazWmwv5Dh4m3e362Awhog1/ rPmQBHHGpF9+oqOLmdbv17uin3MiFAuQfhG70s0Yt2vZ1Xv0MuYMsqiKH1m1/PfOe0RE/O bIsQEfDf20Cp/8I8iQ3j3Nuyq0BnxsErtOGJQ9O38d/PAWJFO5hh3Z+RhHKb5g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1690732039; a=rsa-sha256; cv=none; b=NOf7QBQ4imRnwCueQ+GVQUNogYTiq7rMi0p35FljVXSxBA6I27C5WZdw+8Hd7FVG8IDRFn C94hqHfhdzpGus+d1cOzWLwSa0joWNSocl4GPJCPHYZjnuAZINGqh7zi/bf4S2ZQ1g/jqZ sGIyVwPY83QEMPmj7nmuJnhg2WeV2oGJstth3wgENdBm7fyUJZIDTllsdHdT+kWkNluQP4 w4Bs4v5Lnman47dVwJTI4LajzpzpJLsD2cBNcdDlj4CxoDAMfWXc62YzuNHI956Akvkf0J cEY7JiRTNhDpzsCIkhkojlMHRALdZT3Y70I9lEzEOxgfU8C8iDpgqYp1O8LQGQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4RDQjg0X3HzKjd; Sun, 30 Jul 2023 15:47:18 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 36UFlH57089124; Sun, 30 Jul 2023 15:47:17 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 36UFlHLk089123; Sun, 30 Jul 2023 15:47:17 GMT (envelope-from git) Date: Sun, 30 Jul 2023 15:47:17 GMT Message-Id: <202307301547.36UFlHLk089123@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Mark Johnston Subject: git: 5ad29bc8d4d4 - main - amd64: Fix TLB invalidation routines in !SMP kernels List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: markj X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 5ad29bc8d4d436660954f6db507909965369fc32 Auto-Submitted: auto-generated The branch main has been updated by markj: URL: https://cgit.FreeBSD.org/src/commit/?id=5ad29bc8d4d436660954f6db507909965369fc32 commit 5ad29bc8d4d436660954f6db507909965369fc32 Author: Mark Johnston AuthorDate: 2023-07-30 15:12:35 +0000 Commit: Mark Johnston CommitDate: 2023-07-30 15:12:35 +0000 amd64: Fix TLB invalidation routines in !SMP kernels amd64 is special in that its implementation of zpcpu_offset_cpu() is not the identity transformation, even in !SMP kernels. Because the pm_pcidp array of amd64's struct pmap is allocated from a pcpu UMA zone, this means that accessing pm_pcidp directly, as is done in !SMP implementations of pmap_invalidate_*, does not work. Specifically, I see occasional unexplicable crashes in userspace when PCIDs are enabled. Apply a minimal patch to fix the problem. While it would also make sense to provide separate implementations of zpcpu_* for !SMP kernels, fixing it this way makes the SMP and !SMP implementations of pmap_invalidate_* more similar. Reviewed by: alc, kib MFC after: 1 week Sponsored by: Klara, Inc. Sponsored by: Juniper Networks, Inc. Differential Revision: https://reviews.freebsd.org/D41230 --- sys/amd64/amd64/pmap.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index c1968fc11844..5bc40bd00a32 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -3489,6 +3489,7 @@ void pmap_invalidate_page(pmap_t pmap, vm_offset_t va) { struct invpcid_descr d; + struct pmap_pcid *pcidp; uint64_t kcr3, ucr3; uint32_t pcid; @@ -3504,7 +3505,7 @@ pmap_invalidate_page(pmap_t pmap, vm_offset_t va) if (pmap == PCPU_GET(curpmap) && pmap_pcid_enabled && pmap->pm_ucr3 != PMAP_NO_CR3) { critical_enter(); - pcid = pmap->pm_pcidp->pm_pcid; + pcid = pmap_get_pcid(pmap); if (invpcid_works) { d.pcid = pcid | PMAP_PCID_USER_PT; d.pad = 0; @@ -3518,16 +3519,20 @@ pmap_invalidate_page(pmap_t pmap, vm_offset_t va) } critical_exit(); } - } else if (pmap_pcid_enabled) - pmap->pm_pcidp->pm_gen = 0; + } else if (pmap_pcid_enabled) { + pcidp = zpcpu_get(pmap->pm_pcidp); + pcidp->pm_gen = 0; + } } void pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) { struct invpcid_descr d; + struct pmap_pcid *pcidp; vm_offset_t addr; uint64_t kcr3, ucr3; + uint32_t pcid; if (pmap->pm_type == PT_RVI || pmap->pm_type == PT_EPT) { pmap->pm_eptgen++; @@ -3542,24 +3547,24 @@ pmap_invalidate_range(pmap_t pmap, vm_offset_t sva, vm_offset_t eva) if (pmap == PCPU_GET(curpmap) && pmap_pcid_enabled && pmap->pm_ucr3 != PMAP_NO_CR3) { critical_enter(); + pcid = pmap_get_pcid(pmap); if (invpcid_works) { - d.pcid = pmap->pm_pcidp->pm_pcid | - PMAP_PCID_USER_PT; + d.pcid = pcid | PMAP_PCID_USER_PT; d.pad = 0; d.addr = sva; for (; d.addr < eva; d.addr += PAGE_SIZE) invpcid(&d, INVPCID_ADDR); } else { - kcr3 = pmap->pm_cr3 | pmap->pm_pcidp-> - pm_pcid | CR3_PCID_SAVE; - ucr3 = pmap->pm_ucr3 | pmap->pm_pcidp-> - pm_pcid | PMAP_PCID_USER_PT | CR3_PCID_SAVE; + kcr3 = pmap->pm_cr3 | pcid | CR3_PCID_SAVE; + ucr3 = pmap->pm_ucr3 | pcid | + PMAP_PCID_USER_PT | CR3_PCID_SAVE; pmap_pti_pcid_invlrng(ucr3, kcr3, sva, eva); } critical_exit(); } } else if (pmap_pcid_enabled) { - pmap->pm_pcidp->pm_gen = 0; + pcidp = zpcpu_get(pmap->pm_pcidp); + pcidp->pm_gen = 0; } } @@ -3567,7 +3572,9 @@ void pmap_invalidate_all(pmap_t pmap) { struct invpcid_descr d; + struct pmap_pcid *pcidp; uint64_t kcr3, ucr3; + uint32_t pcid; if (pmap->pm_type == PT_RVI || pmap->pm_type == PT_EPT) { pmap->pm_eptgen++; @@ -3586,8 +3593,9 @@ pmap_invalidate_all(pmap_t pmap) } else if (pmap == PCPU_GET(curpmap)) { if (pmap_pcid_enabled) { critical_enter(); + pcid = pmap_get_pcid(pmap); if (invpcid_works) { - d.pcid = pmap->pm_pcidp->pm_pcid; + d.pcid = pcid; d.pad = 0; d.addr = 0; invpcid(&d, INVPCID_CTX); @@ -3596,10 +3604,10 @@ pmap_invalidate_all(pmap_t pmap) invpcid(&d, INVPCID_CTX); } } else { - kcr3 = pmap->pm_cr3 | pmap->pm_pcidp->pm_pcid; + kcr3 = pmap->pm_cr3 | pcid; if (pmap->pm_ucr3 != PMAP_NO_CR3) { - ucr3 = pmap->pm_ucr3 | pmap->pm_pcidp-> - pm_pcid | PMAP_PCID_USER_PT; + ucr3 = pmap->pm_ucr3 | pcid | + PMAP_PCID_USER_PT; pmap_pti_pcid_invalidate(ucr3, kcr3); } else load_cr3(kcr3); @@ -3609,7 +3617,8 @@ pmap_invalidate_all(pmap_t pmap) invltlb(); } } else if (pmap_pcid_enabled) { - pmap->pm_pcidp->pm_gen = 0; + pcidp = zpcpu_get(pmap->pm_pcidp); + pcidp->pm_gen = 0; } } @@ -3623,12 +3632,15 @@ pmap_invalidate_cache(void) static void pmap_update_pde(pmap_t pmap, vm_offset_t va, pd_entry_t *pde, pd_entry_t newpde) { + struct pmap_pcid *pcidp; pmap_update_pde_store(pmap, pde, newpde); if (pmap == kernel_pmap || pmap == PCPU_GET(curpmap)) pmap_update_pde_invalidate(pmap, va, newpde); - else - pmap->pm_pcidp->pm_gen = 0; + else { + pcidp = zpcpu_get(pmap->pm_pcidp); + pcidp->pm_gen = 0; + } } #endif /* !SMP */