From owner-freebsd-virtualization@freebsd.org Sun Sep 18 00:31:08 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8EC25BD6210 for ; Sun, 18 Sep 2016 00:31:08 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64AF03D3 for ; Sun, 18 Sep 2016 00:31:08 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id E989DDA36 for ; Sun, 18 Sep 2016 00:31:00 +0000 (UTC) Subject: Re: High vCPU Counts in bhyve To: freebsd-virtualization@freebsd.org References: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> From: Allan Jude Message-ID: <4ede10b9-95b2-f5ba-522e-de08a3937068@freebsd.org> Date: Sat, 17 Sep 2016 20:30:57 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UGNGL4icqnMEjcTvGneWvp6qVMOEBMWSe" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Sep 2016 00:31:08 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --UGNGL4icqnMEjcTvGneWvp6qVMOEBMWSe Content-Type: multipart/mixed; boundary="VxLd2RLeGfegAfNgJlMKv4cjrpEFMdLbC"; protected-headers="v1" From: Allan Jude To: freebsd-virtualization@freebsd.org Message-ID: <4ede10b9-95b2-f5ba-522e-de08a3937068@freebsd.org> Subject: Re: High vCPU Counts in bhyve References: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> In-Reply-To: <2ebbc1f9-4b1e-74f2-60fc-c0976b7924a1@freebsd.org> --VxLd2RLeGfegAfNgJlMKv4cjrpEFMdLbC Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2016-09-17 19:55, Peter Grehan wrote: > Hi Trent, >=20 >> If anyone has any hints on how to get more information from this or id= eas >> on the apparent APIC error would be greatly appreciated. >=20 > Thanks for the boot logs. The bhyve bug can be seen from: >=20 > ACPI BIOS Warning (bug): Incorrect checksum in table [APIC] - 0x1C, > should be 0x0A (20160527/tbprint-229) > [7/1843] > MADT: Ignoring bogus I/O APIC ID 0MADT: Could not find APIC for SCI IRQ= 9 >=20 > .. which is pointing to the MADT table being overwritten. There is onl= y > 256 bytes allocated for this table when it is being created: >=20 > acpi.c > * MADT -> 0xf2500 (depends on #CPUs) > * FADT -> 0xf2600 (268 bytes) > ... > #define MADT_OFFSET 0x100 > #define FADT_OFFSET 0x200 >=20 > The MADT* for bhyve has a 44-byte fixed header, followed by a array of= > 8-byte 'Processor Local APIC' entries, one for each vCPU. The end of th= e > table has a 12-byte 'I/O APIC' entry, 2 10-byte 'Interrupt Source > Override' entries, and a 6-byte 'Local APIC NMI' entry. >=20 > Looking at the max #CPUs that can fit into 256 bytes: >=20 > 256 =3D 44 + N*8 + 12 + 2*10 + 6, which gives N =3D 21. The fact that i= t > worked for slightly larger values is probably due to the table entries > at the end being ignored, until eventually the I/O APIC table entry was= > corrupted by the FADT overwrite. >=20 > A quick fix to get more vCPUs is to bump the addresses of the tables in= > acpi.c following the MADT - adding say 0x500 will give you 128 vCPUs. >=20 > +#define FADT_OFFSET 0x700 > +#define HPET_OFFSET 0x840 > +#define MCFG_OFFSET 0x880 > +#define FACS_OFFSET 0x8C0 > +#define DSDT_OFFSET 0x900 > -#define FADT_OFFSET 0x200 > -#define HPET_OFFSET 0x340 > -#define MCFG_OFFSET 0x380 > -#define FACS_OFFSET 0x3C0 > -#define DSDT_OFFSET 0x400 >=20 > I'll create a bug for this so that the overwrite will be detected at > run-time, and also bump up the space to allow for some growth. >=20 > later, >=20 > Peter. >=20 > * see the ACPI spec at http://www.acpi.info/spec.htm for table details.= > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" Ha, so the only reason I didn't run into this, is the biggest VM i've created just happened to be 20 vCPUs on my 24 CPU system. Dodged a bullet, or, delay the discovery of this bug by 2 years. One of the two. --=20 Allan Jude --VxLd2RLeGfegAfNgJlMKv4cjrpEFMdLbC-- --UGNGL4icqnMEjcTvGneWvp6qVMOEBMWSe Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJX3eBEAAoJEBmVNT4SmAt+gqQP/iwcO7EJc8+8CV5RGXwETN4W xfXo1IvOJzKhmJwtUX+3d7UaWH8eLLpK8d0NIhenlcyKCivhJSQcjnTNkvcUVWiB roFCq3cKEZe116h3Y8on/rbjgmpdIjVP5mOOLZ4qulVVY/iogHVZpX0+WdORpt/q 6yGGurYbyoxQ4fZJcIGDI9A1Rjt7ON9bNkfjPvV8iJ66ZNFkOqUFmjifEq6pKW1w thOMBVWcV033BeRpYAQ41NjM56qF6ek2GXi56p7lxeZSif5yUXsCoEs7GhaZnKPs j19gJ0ofaYhsgNzYeiyMDcqDWufCJkUiiMwjpJVwQlG/5x/qWJpnZeHwOthn1mLE FGLHxOgQw3RU3QNop9N0XMEnsdWrmU2Su5ZqeLK2laT/39VTyf7OndSUavtUrPml zrDcnId14WoOSYMFGTZ9noJTUxtmDMTZBMQ0vNRPgNmm0/JTdOylTGnfY/WO0RaI 5oN9G7y9aIIu+DIvXsYI6Ri1IdqY4j2WSj+hbddiTiTmnBGS3/i0c/PkQ15wcRHj eXZYHRUHHdSbjfrXED423r/2pm/FP45OSit+YYBg17wNLqZ6aWWrW5aIKTHqcRV/ ZfksuMIvC3VOVai3McXSPi59pExp13iiaXg6iVnyt6QmxD6nLk0e5yvOPbsJt6UP VMucH+c0VWETOq60oxEy =eMLS -----END PGP SIGNATURE----- --UGNGL4icqnMEjcTvGneWvp6qVMOEBMWSe--