Date: Sat, 21 Mar 2015 11:48:22 -0400 From: John Baldwin <jhb@FreeBSD.org> To: "K. Macy" <kmacy@freebsd.org>, "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: LOCAL_CPUS vs INTR_CPUS Message-ID: <550D92C6.1080002@FreeBSD.org> In-Reply-To: <CAHM0Q_NRg9b4ZG5cLLO94mzaSWgz0k=TvjdyZffKG6QtNuopLA@mail.gmail.com> References: <CAHM0Q_NRg9b4ZG5cLLO94mzaSWgz0k=TvjdyZffKG6QtNuopLA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 3/19/15 8:54 PM, K. Macy wrote: > I haven't been able to get an answer out of the usual suspects so I > thought I'd ask here. > > In reference to: > https://github.com/freebsd/freebsd/compare/master...bsdjhb:numa_bus_get_cpus > > > The LOCAL_CPUS option will return the cpuset corresponding to the NUMA > (_pxm) domain id of a given device. INTR_CPUS can narrow that further. > However, in this particular branch INTR_CPUS will only ever return the > value given by LOCAL_CPUS. When and where would this come in to > effect? Do the new Haswells have cores in the same memory domain but > with non-uniform access to the PCIe bus? On x86 this means that INTR_CPUS only includes one thread per core in a the NUMA domain a device belongs to whereas LOCAL_CPUS will include all threads in the domain. So if you have a dual-socket 8-core SB/IB system for example, a device on the IO hub for the first processor would use CPUs 0-15 for LOCAL_CPUS and 0,2,4,6,8,10,12,14 for INTR_CPUS if HTT is enabled. If it is disabled, both requests would return CPUs 0-7. For the Haswell CPUs with dual rings, my understanding is that those will be represented as two NUMA domains (since there are two memory controllers) and that _PXM for the Host-PCI bridge will reflect the NUMA domain of the ring that contains the IO Hub. This means that LOCAL_CPUS/INTR_CPUS would only return CPUs on the ring that contains the IO Hub, but not other CPUs in the same package on the second ring. However, I would need to see an acpidump from such a machine to confirm that this what ACPI actually tells us. (I think it kind of has to in that the SRAT will have to treat the memory controllers as separate domains and _PXM can only return a single domain, not a list.) -- John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?550D92C6.1080002>