From owner-svn-src-all@freebsd.org Fri Apr 24 13:49:52 2020 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D03582B57FF; Fri, 24 Apr 2020 13:49:52 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 497wXJ57shz4NfF; Fri, 24 Apr 2020 13:49:52 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id A73A51ECD1; Fri, 24 Apr 2020 13:49:52 +0000 (UTC) (envelope-from markj@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id 03ODnqL0088736; Fri, 24 Apr 2020 13:49:52 GMT (envelope-from markj@FreeBSD.org) Received: (from markj@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id 03ODnpwo088732; Fri, 24 Apr 2020 13:49:51 GMT (envelope-from markj@FreeBSD.org) Message-Id: <202004241349.03ODnpwo088732@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: markj set sender to markj@FreeBSD.org using -f From: Mark Johnston Date: Fri, 24 Apr 2020 13:49:51 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-stable@freebsd.org, svn-src-stable-12@freebsd.org Subject: svn commit: r360259 - in stable/12/sys: arm64/arm64 riscv/riscv X-SVN-Group: stable-12 X-SVN-Commit-Author: markj X-SVN-Commit-Paths: in stable/12/sys: arm64/arm64 riscv/riscv X-SVN-Commit-Revision: 360259 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Apr 2020 13:49:52 -0000 Author: markj Date: Fri Apr 24 13:49:51 2020 New Revision: 360259 URL: https://svnweb.freebsd.org/changeset/base/360259 Log: MFC r359280: Remove the secondary_stacks array in arm64 and riscv kernels. Modified: stable/12/sys/arm64/arm64/locore.S stable/12/sys/arm64/arm64/mp_machdep.c stable/12/sys/riscv/riscv/locore.S stable/12/sys/riscv/riscv/mp_machdep.c Directory Properties: stable/12/ (props changed) Modified: stable/12/sys/arm64/arm64/locore.S ============================================================================== --- stable/12/sys/arm64/arm64/locore.S Fri Apr 24 13:31:22 2020 (r360258) +++ stable/12/sys/arm64/arm64/locore.S Fri Apr 24 13:49:51 2020 (r360259) @@ -183,11 +183,10 @@ ENTRY(mpentry) br x15 mp_virtdone: - ldr x4, =secondary_stacks - mov x5, #(PAGE_SIZE * KSTACK_PAGES) - mul x5, x0, x5 - add sp, x4, x5 - + /* Start using the AP boot stack */ + ldr x4, =bootstack + ldr x4, [x4] + mov sp, x4 b init_secondary END(mpentry) #endif Modified: stable/12/sys/arm64/arm64/mp_machdep.c ============================================================================== --- stable/12/sys/arm64/arm64/mp_machdep.c Fri Apr 24 13:31:22 2020 (r360258) +++ stable/12/sys/arm64/arm64/mp_machdep.c Fri Apr 24 13:49:51 2020 (r360259) @@ -122,7 +122,6 @@ static void ipi_preempt(void *); static void ipi_rendezvous(void *); static void ipi_stop(void *); -struct mtx ap_boot_mtx; struct pcb stoppcbs[MAXCPU]; static device_t cpu_list[MAXCPU]; @@ -137,10 +136,18 @@ static int cpu0 = -1; void mpentry(unsigned long cpuid); void init_secondary(uint64_t); -uint8_t secondary_stacks[MAXCPU - 1][PAGE_SIZE * KSTACK_PAGES] __aligned(16); +/* Synchronize AP startup. */ +static struct mtx ap_boot_mtx; +/* Stacks for AP initialization, discarded once idle threads are started. */ +void *bootstack; +static void *bootstacks[MAXCPU]; + +/* Count of started APs, used to synchronize access to bootstack. */ +static volatile int aps_started; + /* Set to 1 once we're ready to let the APs out of the pen. */ -volatile int aps_ready = 0; +static volatile int aps_ready; /* Temporary variables for init_secondary() */ void *dpcpu[MAXCPU - 1]; @@ -277,14 +284,14 @@ init_secondary(uint64_t cpu) "mov x18, %0 \n" "msr tpidr_el1, %0" :: "r"(pcpup)); - /* Spin until the BSP releases the APs */ - while (!aps_ready) + /* Signal the BSP and spin until it has released all APs. */ + atomic_add_int(&aps_started, 1); + while (!atomic_load_int(&aps_ready)) __asm __volatile("wfe"); /* Initialize curthread */ KASSERT(PCPU_GET(idlethread) != NULL, ("no idle thread")); pcpup->pc_curthread = pcpup->pc_idlethread; - pcpup->pc_curpcb = pcpup->pc_idlethread->td_pcb; /* * Identify current CPU. This is necessary to setup @@ -314,6 +321,11 @@ init_secondary(uint64_t cpu) } mtx_unlock_spin(&ap_boot_mtx); + /* + * Assert that smp_after_idle_runnable condition is reasonable. + */ + MPASS(PCPU_GET(curpcb) == NULL); + /* Enter the scheduler */ sched_throw(NULL); @@ -321,6 +333,24 @@ init_secondary(uint64_t cpu) /* NOTREACHED */ } +static void +smp_after_idle_runnable(void *arg __unused) +{ + struct pcpu *pc; + int cpu; + + for (cpu = 1; cpu < mp_ncpus; cpu++) { + if (bootstacks[cpu] != NULL) { + pc = pcpu_find(cpu); + while (atomic_load_ptr(&pc->pc_curpcb) == NULL) + cpu_spinwait(); + kmem_free((vm_offset_t)bootstacks[cpu], PAGE_SIZE); + } + } +} +SYSINIT(smp_after_idle_runnable, SI_SUB_SMP, SI_ORDER_ANY, + smp_after_idle_runnable, NULL); + /* * Send IPI thru interrupt controller. */ @@ -451,7 +481,7 @@ start_cpu(u_int id, uint64_t target_cpu) struct pcpu *pcpup; vm_paddr_t pa; u_int cpuid; - int err; + int err, naps; /* Check we are able to start this cpu */ if (id > mp_maxid) @@ -465,7 +495,7 @@ start_cpu(u_int id, uint64_t target_cpu) /* * Rotate the CPU IDs to put the boot CPU as CPU 0. We keep the other - * CPUs ordered as the are likely grouped into clusters so it can be + * CPUs ordered as they are likely grouped into clusters so it can be * useful to keep that property, e.g. for the GICv3 driver to send * an IPI to all CPUs in the cluster. */ @@ -480,29 +510,41 @@ start_cpu(u_int id, uint64_t target_cpu) dpcpu[cpuid - 1] = (void *)kmem_malloc(DPCPU_SIZE, M_WAITOK | M_ZERO); dpcpu_init(dpcpu[cpuid - 1], cpuid); + bootstacks[cpuid] = (void *)kmem_malloc(PAGE_SIZE, M_WAITOK | M_ZERO); + + naps = atomic_load_int(&aps_started); + bootstack = (char *)bootstacks[cpuid] + PAGE_SIZE; + printf("Starting CPU %u (%lx)\n", cpuid, target_cpu); pa = pmap_extract(kernel_pmap, (vm_offset_t)mpentry); - err = psci_cpu_on(target_cpu, pa, cpuid); if (err != PSCI_RETVAL_SUCCESS) { /* * Panic here if INVARIANTS are enabled and PSCI failed to - * start the requested CPU. If psci_cpu_on returns PSCI_MISSING + * start the requested CPU. psci_cpu_on() returns PSCI_MISSING * to indicate we are unable to use it to start the given CPU. */ KASSERT(err == PSCI_MISSING || (mp_quirks & MP_QUIRK_CPULIST) == MP_QUIRK_CPULIST, - ("Failed to start CPU %u (%lx)\n", id, target_cpu)); + ("Failed to start CPU %u (%lx), error %d\n", + id, target_cpu, err)); pcpu_destroy(pcpup); kmem_free((vm_offset_t)dpcpu[cpuid - 1], DPCPU_SIZE); dpcpu[cpuid - 1] = NULL; + kmem_free((vm_offset_t)bootstacks[cpuid], PAGE_SIZE); + bootstacks[cpuid] = NULL; mp_ncpus--; /* Notify the user that the CPU failed to start */ - printf("Failed to start CPU %u (%lx)\n", id, target_cpu); - } else + printf("Failed to start CPU %u (%lx), error %d\n", + id, target_cpu, err); + } else { + /* Wait for the AP to switch to its boot stack. */ + while (atomic_load_int(&aps_started) < naps + 1) + cpu_spinwait(); CPU_SET(cpuid, &all_cpus); + } return (true); } Modified: stable/12/sys/riscv/riscv/locore.S ============================================================================== --- stable/12/sys/riscv/riscv/locore.S Fri Apr 24 13:31:22 2020 (r360258) +++ stable/12/sys/riscv/riscv/locore.S Fri Apr 24 13:49:51 2020 (r360259) @@ -296,14 +296,8 @@ ENTRY(mpentry) beqz t1, 1b /* Setup stack pointer */ - lla t0, secondary_stacks - li t1, (PAGE_SIZE * KSTACK_PAGES) - mulw t2, t1, a0 - add t0, t0, t2 - add t0, t0, t1 - sub t0, t0, s9 - li t1, KERNBASE - add sp, t0, t1 + lla t0, bootstack + ld sp, 0(t0) /* Setup supervisor trap vector */ lla t0, mpva Modified: stable/12/sys/riscv/riscv/mp_machdep.c ============================================================================== --- stable/12/sys/riscv/riscv/mp_machdep.c Fri Apr 24 13:31:22 2020 (r360258) +++ stable/12/sys/riscv/riscv/mp_machdep.c Fri Apr 24 13:49:51 2020 (r360259) @@ -86,7 +86,6 @@ static device_attach_t riscv64_cpu_attach; static int ipi_handler(void *); -struct mtx ap_boot_mtx; struct pcb stoppcbs[MAXCPU]; extern uint32_t boot_hart; @@ -97,13 +96,19 @@ static uint32_t cpu_reg[MAXCPU][2]; #endif static device_t cpu_list[MAXCPU]; -void mpentry(unsigned long cpuid); void init_secondary(uint64_t); -uint8_t secondary_stacks[MAXCPU][PAGE_SIZE * KSTACK_PAGES] __aligned(16); +static struct mtx ap_boot_mtx; +/* Stacks for AP initialization, discarded once idle threads are started. */ +void *bootstack; +static void *bootstacks[MAXCPU]; + +/* Count of started APs, used to synchronize access to bootstack. */ +static volatile int aps_started; + /* Set to 1 once we're ready to let the APs out of the pen. */ -volatile int aps_ready = 0; +static volatile int aps_ready; /* Temporary variables for init_secondary() */ void *dpcpu[MAXCPU - 1]; @@ -237,14 +242,14 @@ init_secondary(uint64_t hart) csr_set(sie, SIE_SSIE); csr_set(sip, SIE_SSIE); - /* Spin until the BSP releases the APs */ - while (!aps_ready) + /* Signal the BSP and spin until it has released all APs. */ + atomic_add_int(&aps_started, 1); + while (!atomic_load_int(&aps_ready)) __asm __volatile("wfi"); /* Initialize curthread */ KASSERT(PCPU_GET(idlethread) != NULL, ("no idle thread")); pcpup->pc_curthread = pcpup->pc_idlethread; - pcpup->pc_curpcb = pcpup->pc_idlethread->td_pcb; /* * Identify current CPU. This is necessary to setup @@ -278,6 +283,11 @@ init_secondary(uint64_t hart) mtx_unlock_spin(&ap_boot_mtx); + /* + * Assert that smp_after_idle_runnable condition is reasonable. + */ + MPASS(PCPU_GET(curpcb) == NULL); + /* Enter the scheduler */ sched_throw(NULL); @@ -285,6 +295,24 @@ init_secondary(uint64_t hart) /* NOTREACHED */ } +static void +smp_after_idle_runnable(void *arg __unused) +{ + struct pcpu *pc; + int cpu; + + for (cpu = 1; cpu < mp_ncpus; cpu++) { + if (bootstacks[cpu] != NULL) { + pc = pcpu_find(cpu); + while (atomic_load_ptr(&pc->pc_curpcb) == NULL) + cpu_spinwait(); + kmem_free((vm_offset_t)bootstacks[cpu], PAGE_SIZE); + } + } +} +SYSINIT(smp_after_idle_runnable, SI_SUB_SMP, SI_ORDER_ANY, + smp_after_idle_runnable, NULL); + static int ipi_handler(void *arg) { @@ -377,6 +405,7 @@ cpu_init_fdt(u_int id, phandle_t node, u_int addr_size struct pcpu *pcpup; uint64_t hart; u_int cpuid; + int naps; /* Check if this hart supports MMU. */ if (OF_getproplen(node, "mmu-type") < 0) @@ -423,8 +452,17 @@ cpu_init_fdt(u_int id, phandle_t node, u_int addr_size dpcpu[cpuid - 1] = (void *)kmem_malloc(DPCPU_SIZE, M_WAITOK | M_ZERO); dpcpu_init(dpcpu[cpuid - 1], cpuid); + bootstacks[cpuid] = (void *)kmem_malloc(PAGE_SIZE, M_WAITOK | M_ZERO); + + naps = atomic_load_int(&aps_started); + bootstack = (char *)bootstacks[cpuid] + PAGE_SIZE; + printf("Starting CPU %u (hart %lx)\n", cpuid, hart); - __riscv_boot_ap[hart] = 1; + atomic_store_32(&__riscv_boot_ap[hart], 1); + + /* Wait for the AP to switch to its boot stack. */ + while (atomic_load_int(&aps_started) < naps + 1) + cpu_spinwait(); CPU_SET(cpuid, &all_cpus); CPU_SET(hart, &all_harts);