From owner-svn-src-head@freebsd.org Fri Jan 4 19:07:31 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B1181435DA4; Fri, 4 Jan 2019 19:07:31 +0000 (UTC) (envelope-from freebsd@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B519576F78; Fri, 4 Jan 2019 19:07:30 +0000 (UTC) (envelope-from freebsd@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x04J7R7K084675; Fri, 4 Jan 2019 11:07:27 -0800 (PST) (envelope-from freebsd@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id x04J7Rse084674; Fri, 4 Jan 2019 11:07:27 -0800 (PST) (envelope-from freebsd) From: "Rodney W. Grimes" Message-Id: <201901041907.x04J7Rse084674@pdx.rh.CN85.dnsmgr.net> Subject: Re: svn commit: r342771 - in head: share/man/man4 sys/kern sys/powerpc/powernv sys/powerpc/powerpc sys/powerpc/ps3 sys/powerpc/pseries sys/sys sys/x86/x86 In-Reply-To: <201901041831.x04IVHa5070991@repo.freebsd.org> To: Conrad Meyer Date: Fri, 4 Jan 2019 11:07:27 -0800 (PST) CC: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Reply-To: rgrimes@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: B519576F78 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.97)[-0.970,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jan 2019 19:07:31 -0000 > Author: cem > Date: Fri Jan 4 18:31:17 2019 > New Revision: 342771 > URL: https://svnweb.freebsd.org/changeset/base/342771 > > Log: > Expose threads-per-core and physical core count information > > With new sysctls (to the best of our ability do detect them). Restructured > smp.4 slightly for clarity (keep relevant stuff closer to the top) while > documenting. > > Reviewed by: markj, jhibbits (ppc parts) > MFC after: 3 days > Sponsored by: Dell EMC Isilon > Differential Revision: https://reviews.freebsd.org/D18322 > > Modified: > head/share/man/man4/smp.4 > head/sys/kern/subr_smp.c > head/sys/powerpc/powernv/platform_powernv.c > head/sys/powerpc/powerpc/mp_machdep.c > head/sys/powerpc/ps3/platform_ps3.c > head/sys/powerpc/pseries/platform_chrp.c > head/sys/sys/smp.h > head/sys/x86/x86/mp_x86.c Can you please alter this to match how topology is expressed by most tools, Sockets, Cores per socket, Threads per core. ncpu = S * C * T This matches how bhyve, qemu, kvm and others express this information. Thanks, Rod > Modified: head/share/man/man4/smp.4 > ============================================================================== > --- head/share/man/man4/smp.4 Fri Jan 4 18:21:49 2019 (r342770) > +++ head/share/man/man4/smp.4 Fri Jan 4 18:31:17 2019 (r342771) > @@ -23,7 +23,7 @@ > .\" > .\" $FreeBSD$ > .\" > -.Dd January 6, 2018 > +.Dd January 4, 2019 > .Dt SMP 4 > .Os > .Sh NAME > @@ -35,27 +35,6 @@ > The > .Nm > kernel implements symmetric multi-processor support. > -.Sh COMPATIBILITY > -Support for multi-processor systems is present for all Tier-1 > -architectures on > -.Fx . > -Currently, this includes amd64, i386 and sparc64. > -Support is enabled using > -.Cd options SMP . > -It is permissible to use the SMP kernel configuration on non-SMP equipped > -motherboards. > -.Sh I386 NOTES > -For i386 systems, the > -.Nm > -kernel supports motherboards that follow the Intel MP specification, > -version 1.4. > -In addition to > -.Cd options SMP , > -i386 also requires > -.Cd device apic . > -The > -.Xr mptable 1 > -command may be used to view the status of multi-processor support. > .Pp > .Nm > support can be disabled by setting the loader tunable > @@ -66,6 +45,13 @@ The number of CPUs detected by the system is available > the read-only sysctl variable > .Va hw.ncpu . > .Pp > +The number of online threads per CPU core is available in the read-only sysctl > +variable > +.Va kern.smp.threads_per_core . > +The number of physical CPU cores detected by the system is available in the > +read-only sysctl variable > +.Va kern.smp.cores . > +.Pp > .Fx > allows specific CPUs on a multi-processor system to be disabled. > This can be done using the > @@ -74,6 +60,12 @@ tunable, where X is the APIC ID of a CPU. > Setting this tunable to 1 will result in the corresponding CPU being > disabled. > .Pp > +.Fx > +supports simultaneous multithreading on x86 and powerpc platforms. > +On x86, the logical CPUs can be disabled by setting the > +.Va machdep.hyperthreading_allowed > +tunable to zero. > +.Pp > The > .Xr sched_ule 4 > scheduler implements CPU topology detection and adjusts the scheduling > @@ -122,13 +114,26 @@ two quad-core processors is: > .Pp > This information is used internally by the kernel to schedule related > tasks on CPUs that are closely grouped together. > -.Pp > -.Fx > -supports hyperthreading on Intel CPU's on the i386 and AMD64 platforms. > -Because using logical CPUs can cause performance penalties under certain loads, > -the logical CPUs can be disabled by setting the > -.Va machdep.hyperthreading_allowed > -tunable to zero. > +.Sh COMPATIBILITY > +Support for multi-processor systems is present for all Tier-1 and Tier-2 > +architectures on > +.Fx . > +Currently, this includes x86, powerpc, arm, and sparc64. > +Support is enabled using > +.Cd options SMP . > +It is permissible to use the SMP kernel configuration on non-SMP hardware. > +.Sh I386 NOTES > +For i386 systems, the > +.Nm > +kernel supports motherboards that follow the Intel MP specification, > +version 1.4. > +In addition to > +.Cd options SMP , > +i386 also requires > +.Cd device apic . > +The > +.Xr mptable 1 > +command may be used to view the status of multi-processor support. > .Sh SEE ALSO > .Xr cpuset 1 , > .Xr mptable 1 , > @@ -166,3 +171,20 @@ in > also introduced support for SMP on the sparc64 architecture. > .Sh AUTHORS > .An Steve Passe Aq Mt fsmp@FreeBSD.org > +.Sh CAVEATS > +The > +.Va kern.smp.threads_per_core > +and > +.Va kern.smp.cores > +sysctl variables are provided as a best-effort guess. > +If an architecture or platform adds SMT and > +.Fx > +has not yet implemented detection, the reported values may be inaccurate. > +In this case, > +.Va kern.smp.threads_per_core > +will report > +.Dv 1 > +and > +.Va kern.smp.cores > +will report the same value as > +.Va hw.ncpu . > > Modified: head/sys/kern/subr_smp.c > ============================================================================== > --- head/sys/kern/subr_smp.c Fri Jan 4 18:21:49 2019 (r342770) > +++ head/sys/kern/subr_smp.c Fri Jan 4 18:31:17 2019 (r342771) > @@ -98,6 +98,14 @@ int smp_cpus = 1; /* how many cpu's running */ > SYSCTL_INT(_kern_smp, OID_AUTO, cpus, CTLFLAG_RD|CTLFLAG_CAPRD, &smp_cpus, 0, > "Number of CPUs online"); > > +int smp_threads_per_core = 1; /* how many SMT threads are running per core */ > +SYSCTL_INT(_kern_smp, OID_AUTO, threads_per_core, CTLFLAG_RD|CTLFLAG_CAPRD, > + &smp_threads_per_core, 0, "Number of SMT threads online per core"); > + > +int mp_ncores = -1; /* how many physical cores running */ > +SYSCTL_INT(_kern_smp, OID_AUTO, cores, CTLFLAG_RD|CTLFLAG_CAPRD, &mp_ncores, 0, > + "Number of CPUs online"); > + > int smp_topology = 0; /* Which topology we're using. */ > SYSCTL_INT(_kern_smp, OID_AUTO, topology, CTLFLAG_RDTUN, &smp_topology, 0, > "Topology override setting; 0 is default provided by hardware."); > @@ -154,6 +162,7 @@ mp_start(void *dummy) > > /* Probe for MP hardware. */ > if (smp_disabled != 0 || cpu_mp_probe() == 0) { > + mp_ncores = 1; > mp_ncpus = 1; > CPU_SETOF(PCPU_GET(cpuid), &all_cpus); > return; > @@ -162,6 +171,11 @@ mp_start(void *dummy) > cpu_mp_start(); > printf("FreeBSD/SMP: Multiprocessor System Detected: %d CPUs\n", > mp_ncpus); > + > + /* Provide a default for most architectures that don't have SMT/HTT. */ > + if (mp_ncores < 0) > + mp_ncores = mp_ncpus; > + > cpu_mp_announce(); > } > SYSINIT(cpu_mp, SI_SUB_CPU, SI_ORDER_THIRD, mp_start, NULL); > @@ -823,6 +837,7 @@ static void > mp_setvariables_for_up(void *dummy) > { > mp_ncpus = 1; > + mp_ncores = 1; > mp_maxid = PCPU_GET(cpuid); > CPU_SETOF(mp_maxid, &all_cpus); > KASSERT(PCPU_GET(cpuid) == 0, ("UP must have a CPU ID of zero")); > > Modified: head/sys/powerpc/powernv/platform_powernv.c > ============================================================================== > --- head/sys/powerpc/powernv/platform_powernv.c Fri Jan 4 18:21:49 2019 (r342770) > +++ head/sys/powerpc/powernv/platform_powernv.c Fri Jan 4 18:31:17 2019 (r342771) > @@ -435,12 +435,16 @@ powernv_smp_topo(platform_t plat) > break; > } > > + smp_threads_per_core = nthreads; > + > if (mp_ncpus % nthreads != 0) { > printf("WARNING: Irregular SMP topology. Performance may be " > "suboptimal (%d threads, %d on first core)\n", > mp_ncpus, nthreads); > return (smp_topo_none()); > } > + > + mp_ncores = mp_ncpus / nthreads; > > /* Don't do anything fancier for non-threaded SMP */ > if (nthreads == 1) > > Modified: head/sys/powerpc/powerpc/mp_machdep.c > ============================================================================== > --- head/sys/powerpc/powerpc/mp_machdep.c Fri Jan 4 18:21:49 2019 (r342770) > +++ head/sys/powerpc/powerpc/mp_machdep.c Fri Jan 4 18:31:17 2019 (r342771) > @@ -186,6 +186,11 @@ cpu_mp_start(void) > next: > error = platform_smp_next_cpu(&cpu); > } > + > +#ifdef SMP > + /* Probe mp_ncores and smp_threads_per_core as a side effect. */ > + (void)cpu_topo(); > +#endif > } > > void > > Modified: head/sys/powerpc/ps3/platform_ps3.c > ============================================================================== > --- head/sys/powerpc/ps3/platform_ps3.c Fri Jan 4 18:21:49 2019 (r342770) > +++ head/sys/powerpc/ps3/platform_ps3.c Fri Jan 4 18:31:17 2019 (r342771) > @@ -246,6 +246,8 @@ ps3_smp_start_cpu(platform_t plat, struct pcpu *pc) > static struct cpu_group * > ps3_smp_topo(platform_t plat) > { > + mp_ncores = 1; > + smp_threads_per_core = 2; > return (smp_topo_1level(CG_SHARE_L1, 2, CG_FLAG_SMT)); > } > #endif > > Modified: head/sys/powerpc/pseries/platform_chrp.c > ============================================================================== > --- head/sys/powerpc/pseries/platform_chrp.c Fri Jan 4 18:21:49 2019 (r342770) > +++ head/sys/powerpc/pseries/platform_chrp.c Fri Jan 4 18:31:17 2019 (r342771) > @@ -517,6 +517,8 @@ chrp_smp_topo(platform_t plat) > ncpus++; > } > > + mp_ncores = ncores; > + > if (ncpus % ncores != 0) { > printf("WARNING: Irregular SMP topology. Performance may be " > "suboptimal (%d CPUS, %d cores)\n", ncpus, ncores); > @@ -527,6 +529,7 @@ chrp_smp_topo(platform_t plat) > if (ncpus == ncores) > return (smp_topo_none()); > > + smp_threads_per_core = ncpus / ncores; > return (smp_topo_1level(CG_SHARE_L1, ncpus / ncores, CG_FLAG_SMT)); > } > #endif > > Modified: head/sys/sys/smp.h > ============================================================================== > --- head/sys/sys/smp.h Fri Jan 4 18:21:49 2019 (r342770) > +++ head/sys/sys/smp.h Fri Jan 4 18:31:17 2019 (r342771) > @@ -167,8 +167,10 @@ extern cpuset_t logical_cpus_mask; > > extern u_int mp_maxid; > extern int mp_maxcpus; > +extern int mp_ncores; > extern int mp_ncpus; > extern volatile int smp_started; > +extern int smp_threads_per_core; > > extern cpuset_t all_cpus; > extern cpuset_t cpuset_domain[MAXMEMDOM]; /* CPUs in each NUMA domain. */ > > Modified: head/sys/x86/x86/mp_x86.c > ============================================================================== > --- head/sys/x86/x86/mp_x86.c Fri Jan 4 18:21:49 2019 (r342770) > +++ head/sys/x86/x86/mp_x86.c Fri Jan 4 18:31:17 2019 (r342771) > @@ -607,6 +607,7 @@ assign_cpu_ids(void) > { > struct topo_node *node; > u_int smt_mask; > + int nhyper; > > smt_mask = (1u << core_id_shift) - 1; > > @@ -615,6 +616,7 @@ assign_cpu_ids(void) > * beyond MAXCPU. CPU 0 is always assigned to the BSP. > */ > mp_ncpus = 0; > + nhyper = 0; > TOPO_FOREACH(node, &topo_root) { > if (node->type != TOPO_TYPE_PU) > continue; > @@ -642,6 +644,9 @@ assign_cpu_ids(void) > continue; > } > > + if (cpu_info[node->hwid].cpu_hyperthread) > + nhyper++; > + > cpu_apic_ids[mp_ncpus] = node->hwid; > apic_cpuids[node->hwid] = mp_ncpus; > topo_set_pu_id(node, mp_ncpus); > @@ -651,6 +656,9 @@ assign_cpu_ids(void) > KASSERT(mp_maxid >= mp_ncpus - 1, > ("%s: counters out of sync: max %d, count %d", __func__, mp_maxid, > mp_ncpus)); > + > + mp_ncores = mp_ncpus - nhyper; > + smp_threads_per_core = mp_ncpus / mp_ncores; > } > > /* > > -- Rod Grimes rgrimes@freebsd.org