Date: Fri, 06 Jan 2017 10:56:20 -0800 From: John Baldwin <jhb@freebsd.org> To: freebsd-current@freebsd.org Cc: Konstantin Belousov <kostikbel@gmail.com>, Eric Joyner <erj@freebsd.org> Subject: Re: How many CPU cores does FreeBSD support? Message-ID: <2287894.9nzTCdqko4@ralph.baldwin.cx> In-Reply-To: <20170104195909.GB2533@kib.kiev.ua> References: <CA%2Bb0zg-RYZgK67PsUAiSTss6WK2%2BFg_L4=zya33OxaiH%2BOfQHQ@mail.gmail.com> <CA%2Bb0zg8TEt4foPZhg%2BYLroxHyq--%2B%2BU%2BLR0E0ArNp9Hn=be6iA@mail.gmail.com> <20170104195909.GB2533@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wednesday, January 04, 2017 09:59:09 PM Konstantin Belousov wrote:
> On Wed, Jan 04, 2017 at 06:53:23PM +0000, Eric Joyner wrote:
> > Adding freebsd-current, because that's a good idea.
> >
> > I see these lines in the beginning of dmesg:
> >
> > MADT: Ignoring local APIC ID 256 (too high)
> >
> >
> > [907/911]
> > MADT: Ignoring local APIC ID 260 (too high)
> >
> >
> > [906/911]
> > MADT: Ignoring local APIC ID 264 (too high)
> >
> >
> > [905/911]
> > MADT: Ignoring local APIC ID 268 (too high)
> >
> >
> > [904/911]
> > MADT: Ignoring local APIC ID 272 (too high)
> >
> >
> > [903/911]
> > MADT: Ignoring local APIC ID 276 (too high)
> >
> >
> > [902/911]
> > MADT: Ignoring local APIC ID 280 (too high)
> >
> >
> > [901/911]
> > MADT: Ignoring local APIC ID 272 (too high)
> >
> >
> > [903/911]
> > MADT: Ignoring local APIC ID 276 (too high)
> >
> >
> > [902/911]
> > MADT: Ignoring local APIC ID 280 (too high)
> >
> >
> > [901/911]
> > MADT: Ignoring local APIC ID 276 (too high)
> >
> >
> > [902/911]
> > MADT: Ignoring local APIC ID 280 (too high)
> > MADT: Ignoring local APIC ID 284 (too high)
> > MADT: Ignoring local APIC ID 288 (too high)
> > MADT: Ignoring local APIC ID 292 (too high)
> > MADT: Ignoring local APIC ID 296 (too high)
> > MADT: Ignoring local APIC ID 300 (too high)
> > MADT: Ignoring local APIC ID 257 (too high)
> > MADT: Ignoring local APIC ID 261 (too high)
> > MADT: Ignoring local APIC ID 265 (too high)
> > MADT: Ignoring local APIC ID 269 (too high)
> > MADT: Ignoring local APIC ID 273 (too high)
> > MADT: Ignoring local APIC ID 277 (too high)
> > MADT: Ignoring local APIC ID 281 (too high)
> > MADT: Ignoring local APIC ID 285 (too high)
> > MADT: Ignoring local APIC ID 289 (too high)
> > MADT: Ignoring local APIC ID 293 (too high)
> > MADT: Ignoring local APIC ID 297 (too high)
> > MADT: Ignoring local APIC ID 301 (too high)
> > MADT: Ignoring local APIC ID 258 (too high)
> > MADT: Ignoring local APIC ID 262 (too high)
> > MADT: Ignoring local APIC ID 266 (too high)
> > MADT: Ignoring local APIC ID 270 (too high)
> > MADT: Ignoring local APIC ID 274 (too high)
> > MADT: Ignoring local APIC ID 278 (too high)
> > MADT: Ignoring local APIC ID 282 (too high)
> > MADT: Ignoring local APIC ID 286 (too high)
> > MADT: Ignoring local APIC ID 290 (too high)
> > MADT: Ignoring local APIC ID 294 (too high)
> > MADT: Ignoring local APIC ID 298 (too high)
> > MADT: Ignoring local APIC ID 302 (too high)
> > MADT: Ignoring local APIC ID 255 (too high)
> > MADT: Ignoring local APIC ID 259 (too high)
> > MADT: Ignoring local APIC ID 263 (too high)
> > MADT: Ignoring local APIC ID 267 (too high)
> > MADT: Ignoring local APIC ID 271 (too high)
> > MADT: Ignoring local APIC ID 275 (too high)
> > MADT: Ignoring local APIC ID 279 (too high)
> > MADT: Ignoring local APIC ID 283 (too high)
> > MADT: Ignoring local APIC ID 287 (too high)
> > MADT: Ignoring local APIC ID 291 (too high)
> > MADT: Ignoring local APIC ID 295 (too high)
> > MADT: Ignoring local APIC ID 299 (too high)
> > MADT: Ignoring local APIC ID 303 (too high)
> > Copyright (c) 1992-2016 The FreeBSD Project.
> > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> > The Regents of the University of California. All rights reserved.
> > FreeBSD is a registered trademark of The FreeBSD Foundation.
> > FreeBSD 11.0-RELEASE-p1 #0 r306420: Thu Sep 29 01:43:23 UTC 2016
> > root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64
> > FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM
> > 3.8.0)
> > SRAT: No CPU found for memory domain 1
> > VT(efifb): resolution 800x600
> > CPU: Intel(R) Xeon Phi(TM) CPU 7250 @ 1.40GHz (1396.80-MHz K8-class CPU)
> > Origin="GenuineIntel" Id=0x50671 Family=0x6 Model=0x57 Stepping=1
> >
> > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
> >
> > Features2=0x7ff8f39f<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,EST,TM2,SSSE3,FMA,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
> > AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
> > AMD Features2=0x121<LAHF,ABM,Prefetch>
> > Structured Extended
> > Features=0x1c0d23ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,NFPUSG,AVX512F,RDSEED,ADX,AVX512PF,AVX512ER,AVX512CD>
> > Structured Extended Features2=0x1<PREFETCHWT1>
> > XSAVE Features=0x1<XSAVEOPT>
> > TSC: P-state invariant, performance statistics
> > real memory = 223332007936 (212986 MB)
> > avail memory = 216976429056 (206924 MB)
> > Event timer "LAPIC" quality 600
> > ACPI APIC Table: <INTEL S7200AP>
> > FreeBSD/SMP: Multiprocessor System Detected: 223 CPUs
> > FreeBSD/SMP: Non-uniform topology
> >
> > It sounds like those MADT errors point to the problem?
>
> In sys/x86/acpica/madt.c file, function madt_add_cpu(), just remove
> the block
> if (apic_id > MAX_APIC_ID) {
> printf("MADT: Ignoring local APIC ID %u (too high)\n",
> apic_id);
> return;
> }
> Best to do it on HEAD and not on 11.0.
> If it helps, commit the change.
That will not help. mp_machdep.c for x86 has a static table sized by
MAX_APIC_ID to store info about CPUs. That needs to be reworked (either
having a table sized by MAXCPU instead, or by reworking the interface
with APIC enumerators so that we walk the associated firmware table
multiple times rather than generating an internal table).
> Also, for >= 256 CPUs, you might need to enable interrupt remapping
> in the DMAR, use loader tunables
> hw.dmar.enable <= to enable DMAR
> hw.dmar.ir <= to enable remapping
> hw.dmar.dma <= to disable IOMMU if it interacts badly with your hw
We should have the kernel reject routing interrupts to CPUs with APIC
IDs > 256 if IR isn't supported and enabled.
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2287894.9nzTCdqko4>