From owner-freebsd-virtualization@freebsd.org Sat Sep 17 23:55:26 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9DE0BDE28B for ; Sat, 17 Sep 2016 23:55:26 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 52B40622 for ; Sat, 17 Sep 2016 23:55:25 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 35D3720C1BE8 for ; Sun, 18 Sep 2016 09:55:06 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 3096228099E for ; Sun, 18 Sep 2016 09:55:06 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id okb53C8znr_p for ; Sun, 18 Sep 2016 09:55:06 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-50-184-135-69.hsd1.ca.comcast.net [50.184.135.69]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 5980E280901; Sun, 18 Sep 2016 09:55:03 +1000 (AEST) Subject: Re: High vCPU Counts in bhyve To: Trent Thompson References: From: Peter Grehan Cc: FreeBSD virtualization Message-ID: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> Date: Sat, 17 Sep 2016 16:55:14 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=VuVhOK+n c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=ZMguiOAsC5immlfEDy0HCA==:17 a=N659UExz7-8A:10 a=GW1xBdLrtEIA:10 a=IGgMue6PAAAA:8 a=qrcWqzjMrR9eUpCMWjgA:9 a=pILNOxqGKmIA:10 a=fMlXq2aaNVkA:10 a=-FEs8UIgK8oA:10 a=NWVoK91CQyQA:10 a=URp2tKUB9J8u7HlB38tU:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Sep 2016 23:55:26 -0000 Hi Trent, > If anyone has any hints on how to get more information from this or ideas > on the apparent APIC error would be greatly appreciated. Thanks for the boot logs. The bhyve bug can be seen from: ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, should be 0x0A (20160527/tbprint-229) [7/1843] MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ 9 .. which is pointing to the MADT table being overwritten. There is only 256 bytes allocated for this table when it is being created: acpi.c * MADT -> 0xf2500 (depends on #CPUs) * FADT -> 0xf2600 (268 bytes) ... #define MADT_OFFSET 0x100 #define FADT_OFFSET 0x200 The MADT* for bhyve has a 44-byte fixed header, followed by a array of 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of the table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source Override' entries, and a 6-byte 'Local APIC NMI' entry. Looking at the max #CPUs that can fit into 256 bytes: 256 = 44 + N*8 + 12 + 2*10 + 6, which gives N = 21. The fact that it worked for slightly larger values is probably due to the table entries at the end being ignored, until eventually the I/O APIC table entry was corrupted by the FADT overwrite. A quick fix to get more vCPUs is to bump the addresses of the tables in acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs. +#define FADT_OFFSET 0x700 +#define HPET_OFFSET 0x840 +#define MCFG_OFFSET 0x880 +#define FACS_OFFSET 0x8C0 +#define DSDT_OFFSET 0x900 -#define FADT_OFFSET 0x200 -#define HPET_OFFSET 0x340 -#define MCFG_OFFSET 0x380 -#define FACS_OFFSET 0x3C0 -#define DSDT_OFFSET 0x400 I'll create a bug for this so that the overwrite will be detected at run-time, and also bump up the space to allow for some growth. later, Peter. * see the ACPI spec at http://www.acpi.info/spec.htm for table details.