Date: Sat, 17 Sep 2016 16:55:14 -0700 From: Peter Grehan <grehan@freebsd.org> To: Trent Thompson <trentnthompson@gmail.com> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: High vCPU Counts in bhyve Message-ID: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> In-Reply-To: <CAE7bWBoH0JWR5m5dFpEW=fg8XUaKjPWak7yTra_PWnJ8GCQLbA@mail.gmail.com> References: <CAE7bWBoH0JWR5m5dFpEW=fg8XUaKjPWak7yTra_PWnJ8GCQLbA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Trent, > If anyone has any hints on how to get more information from this or ideas > on the apparent APIC error would be greatly appreciated. Thanks for the boot logs. The bhyve bug can be seen from: ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, should be 0x0A (20160527/tbprint-229) [7/1843] MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9 .. which is pointing to the MADT table being overwritten. There is only 256 bytes allocated for this table when it is being created: acpi.c * MADT -> 0xf2500 (depends on #CPUs) * FADT -> 0xf2600 (268 bytes) ... #define MADT_OFFSET 0x100 #define FADT_OFFSET 0x200 The MADT* for bhyve has a 44-byte fixed header, followed by a array of 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source Override' entries, and a 6-byte 'Local APIC NMI' entry. Looking at the max #CPUs that can fit into 256 bytes: 256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it worked for slightly larger values is probably due to the table entries at the end being ignored, until eventually the I/O APIC table entry was corrupted by the FADT overwrite. A quick fix to get more vCPUs is to bump the addresses of the tables in acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs. +#define FADT_OFFSET 0x700 +#define HPET_OFFSET 0x840 +#define MCFG_OFFSET 0x880 +#define FACS_OFFSET 0x8C0 +#define DSDT_OFFSET 0x900 -#define FADT_OFFSET 0x200 -#define HPET_OFFSET 0x340 -#define MCFG_OFFSET 0x380 -#define FACS_OFFSET 0x3C0 -#define DSDT_OFFSET 0x400 I'll create a bug for this so that the overwrite will be detected at run-time, and also bump up the space to allow for some growth. later, Peter. * see the ACPI spec at http://www.acpi.info/spec.htm for table details.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1>