From owner-svn-src-all@freebsd.org Sun May 12 06:36:56 2019 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 47D6C158E139; Sun, 12 May 2019 06:36:56 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E1F608A68A; Sun, 12 May 2019 06:36:55 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id BE02826E63; Sun, 12 May 2019 06:36:55 +0000 (UTC) (envelope-from mjg@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x4C6atqa094486; Sun, 12 May 2019 06:36:55 GMT (envelope-from mjg@FreeBSD.org) Received: (from mjg@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x4C6asUq094481; Sun, 12 May 2019 06:36:54 GMT (envelope-from mjg@FreeBSD.org) Message-Id: <201905120636.x4C6asUq094481@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mjg set sender to mjg@FreeBSD.org using -f From: Mateusz Guzik Date: Sun, 12 May 2019 06:36:54 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r347502 - in head/sys: amd64/amd64 amd64/include i386/i386 i386/include x86/include x86/x86 X-SVN-Group: head X-SVN-Commit-Author: mjg X-SVN-Commit-Paths: in head/sys: amd64/amd64 amd64/include i386/i386 i386/include x86/include x86/x86 X-SVN-Commit-Revision: 347502 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: E1F608A68A X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.91 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; NEURAL_HAM_SHORT(-0.91)[-0.913,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 May 2019 06:36:56 -0000 Author: mjg Date: Sun May 12 06:36:54 2019 New Revision: 347502 URL: https://svnweb.freebsd.org/changeset/base/347502 Log: x86: store pending bitmapped IPIs in per-cpu areas This gets rid of the global cpu_ipi_pending array. While replace cmpset with fcmpset in the delivery code and opportunistically check if given IPI is already pending. Sponsored by: The FreeBSD Foundation Modified: head/sys/amd64/amd64/mp_machdep.c head/sys/amd64/include/pcpu.h head/sys/i386/i386/mp_machdep.c head/sys/i386/include/pcpu.h head/sys/x86/include/x86_smp.h head/sys/x86/x86/mp_x86.c Modified: head/sys/amd64/amd64/mp_machdep.c ============================================================================== --- head/sys/amd64/amd64/mp_machdep.c Sun May 12 06:34:58 2019 (r347501) +++ head/sys/amd64/amd64/mp_machdep.c Sun May 12 06:36:54 2019 (r347502) @@ -193,7 +193,6 @@ cpu_mp_start(void) /* Initialize the logical ID to APIC ID table. */ for (i = 0; i < MAXCPU; i++) { cpu_apic_ids[i] = -1; - cpu_ipi_pending[i] = 0; } /* Install an inter-CPU IPI for TLB invalidation */ Modified: head/sys/amd64/include/pcpu.h ============================================================================== --- head/sys/amd64/include/pcpu.h Sun May 12 06:34:58 2019 (r347501) +++ head/sys/amd64/include/pcpu.h Sun May 12 06:36:54 2019 (r347502) @@ -84,7 +84,8 @@ _Static_assert(sizeof(struct monitorbuf) == 128, "2x c uint32_t pc_pcid_gen; \ uint32_t pc_smp_tlb_done; /* TLB op acknowledgement */ \ uint32_t pc_ibpb_set; \ - char __pad[3288] /* pad to UMA_PCPU_ALLOC_SIZE */ + u_int pc_ipi_bitmap; \ + char __pad[3284] /* pad to UMA_PCPU_ALLOC_SIZE */ #define PC_DBREG_CMD_NONE 0 #define PC_DBREG_CMD_LOAD 1 Modified: head/sys/i386/i386/mp_machdep.c ============================================================================== --- head/sys/i386/i386/mp_machdep.c Sun May 12 06:34:58 2019 (r347501) +++ head/sys/i386/i386/mp_machdep.c Sun May 12 06:36:54 2019 (r347502) @@ -153,7 +153,6 @@ cpu_mp_start(void) /* Initialize the logical ID to APIC ID table. */ for (i = 0; i < MAXCPU; i++) { cpu_apic_ids[i] = -1; - cpu_ipi_pending[i] = 0; } /* Install an inter-CPU IPI for TLB invalidation */ Modified: head/sys/i386/include/pcpu.h ============================================================================== --- head/sys/i386/include/pcpu.h Sun May 12 06:34:58 2019 (r347501) +++ head/sys/i386/include/pcpu.h Sun May 12 06:36:54 2019 (r347502) @@ -87,7 +87,8 @@ _Static_assert(sizeof(struct monitorbuf) == 128, "2x c caddr_t pc_pmap_eh_ptep; \ uint32_t pc_smp_tlb_done; /* TLB op acknowledgement */ \ uint32_t pc_ibpb_set; \ - char __pad[3610] + u_int pc_ipi_bitmap; \ + char __pad[3606] #ifdef _KERNEL Modified: head/sys/x86/include/x86_smp.h ============================================================================== --- head/sys/x86/include/x86_smp.h Sun May 12 06:34:58 2019 (r347501) +++ head/sys/x86/include/x86_smp.h Sun May 12 06:36:54 2019 (r347502) @@ -34,7 +34,6 @@ extern char *bootSTK; extern void *bootstacks[]; extern unsigned int boot_address; extern unsigned int bootMP_size; -extern volatile u_int cpu_ipi_pending[]; extern volatile int aps_ready; extern struct mtx ap_boot_mtx; extern int cpu_logical; Modified: head/sys/x86/x86/mp_x86.c ============================================================================== --- head/sys/x86/x86/mp_x86.c Sun May 12 06:34:58 2019 (r347501) +++ head/sys/x86/x86/mp_x86.c Sun May 12 06:36:54 2019 (r347502) @@ -137,9 +137,6 @@ _Static_assert(MAXCPU <= MAX_APIC_ID, _Static_assert(xAPIC_MAX_APIC_ID <= MAX_APIC_ID, "xAPIC_MAX_APIC_ID cannot be larger that MAX_APIC_ID"); -/* Holds pending bitmap based IPIs per CPU */ -volatile u_int cpu_ipi_pending[MAXCPU]; - static void release_aps(void *dummy); static void cpustop_handler_post(u_int cpu); @@ -1224,19 +1221,24 @@ ipi_startup(int apic_id, int vector) void ipi_send_cpu(int cpu, u_int ipi) { - u_int bitmap, old_pending, new_pending; + u_int bitmap, old, new; + u_int *cpu_bitmap; KASSERT(cpu_apic_ids[cpu] != -1, ("IPI to non-existent CPU %d", cpu)); if (IPI_IS_BITMAPED(ipi)) { bitmap = 1 << ipi; ipi = IPI_BITMAP_VECTOR; - do { - old_pending = cpu_ipi_pending[cpu]; - new_pending = old_pending | bitmap; - } while (!atomic_cmpset_int(&cpu_ipi_pending[cpu], - old_pending, new_pending)); - if (old_pending) + cpu_bitmap = &cpuid_to_pcpu[cpu]->pc_ipi_bitmap; + old = *cpu_bitmap; + for (;;) { + if ((old & bitmap) == bitmap) + break; + new = old | bitmap; + if (atomic_fcmpset_int(cpu_bitmap, &old, new)) + break; + } + if (old) return; } lapic_ipi_vectored(ipi, cpu_apic_ids[cpu]); @@ -1255,7 +1257,7 @@ ipi_bitmap_handler(struct trapframe frame) td->td_intr_nesting_level++; oldframe = td->td_intr_frame; td->td_intr_frame = &frame; - ipi_bitmap = atomic_readandclear_int(&cpu_ipi_pending[cpu]); + ipi_bitmap = atomic_readandclear_int(&cpuid_to_pcpu[cpu]->pc_ipi_bitmap); if (ipi_bitmap & (1 << IPI_PREEMPT)) { #ifdef COUNT_IPIS (*ipi_preempt_counts[cpu])++;