From owner-dev-commits-src-main@freebsd.org Tue Jan 12 01:10:07 2021 Return-Path: Delivered-To: dev-commits-src-main@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 00FCB4EB82C; Tue, 12 Jan 2021 01:10:07 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DFCDG6XJdz3hPk; Tue, 12 Jan 2021 01:10:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id D2FD124614; Tue, 12 Jan 2021 01:10:06 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 10C1A6or069688; Tue, 12 Jan 2021 01:10:06 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 10C1A6mg069684; Tue, 12 Jan 2021 01:10:06 GMT (envelope-from git) Date: Tue, 12 Jan 2021 01:10:06 GMT Message-Id: <202101120110.10C1A6mg069684@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Andrew Gallatin Subject: git: 7eaea04a5bb1 - main - amd64: compare TLB shootdown target to all_cpus MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: gallatin X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 7eaea04a5bb1dc86c43ce046311e1c1a042994d3 Auto-Submitted: auto-generated X-BeenThere: dev-commits-src-main@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Commit messages for the main branch of the src repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Jan 2021 01:10:07 -0000 The branch main has been updated by gallatin: URL: https://cgit.FreeBSD.org/src/commit/?id=7eaea04a5bb1dc86c43ce046311e1c1a042994d3 commit 7eaea04a5bb1dc86c43ce046311e1c1a042994d3 Author: Andrew Gallatin AuthorDate: 2021-01-12 01:03:37 +0000 Commit: Andrew Gallatin CommitDate: 2021-01-12 01:09:32 +0000 amd64: compare TLB shootdown target to all_cpus On amd64, the pmap code passes all_cpus to smp_targeted_tlb_shootdown() when unmapping from the kernel pmap. This function has an optimized path to send IPIs to all but itself, which it intends to do when the target is all cpus. However, we need to compare the target cpu mask with all_cpus, rather than using CPU_ISFULLSET(). Comparing with CPU_ISFULLSET() will only work when we have MAXCPU cpus active in the system, otherwise, we'll be sending repeated IPIs, rather than a single IPI to all CPUs but ourself. Fixing this should reduce the time spent in native_lapic_ipi_wait() as we will be sending ipis in parallel, rather than one-by-one. This is confirmed by dtrace. Reviewed by: alc, jhb, kib, markj Sponsored by: Netflix Differential Revision: https://reviews.freebsd.org/D28102 --- sys/amd64/amd64/mp_machdep.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sys/amd64/amd64/mp_machdep.c b/sys/amd64/amd64/mp_machdep.c index 63777014e151..794a11bf1276 100644 --- a/sys/amd64/amd64/mp_machdep.c +++ b/sys/amd64/amd64/mp_machdep.c @@ -673,7 +673,7 @@ smp_targeted_tlb_shootdown(cpuset_t mask, pmap_t pmap, vm_offset_t addr1, /* * Check for other cpus. Return if none. */ - if (CPU_ISFULLSET(&mask)) { + if (!CPU_CMP(&mask, &all_cpus)) { if (mp_ncpus <= 1) goto local_cb; } else { @@ -719,7 +719,7 @@ smp_targeted_tlb_shootdown(cpuset_t mask, pmap_t pmap, vm_offset_t addr1, * (zeroing slot) and reading from it below (wait for * acknowledgment). */ - if (CPU_ISFULLSET(&mask)) { + if (!CPU_CMP(&mask, &all_cpus)) { ipi_all_but_self(IPI_INVLOP); other_cpus = all_cpus; CPU_CLR(PCPU_GET(cpuid), &other_cpus);