From owner-freebsd-virtualization@FreeBSD.ORG Thu Jul 17 18:15:46 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 872019B4 for ; Thu, 17 Jul 2014 18:15:46 +0000 (UTC) Received: from mail-vc0-x22e.google.com (mail-vc0-x22e.google.com [IPv6:2607:f8b0:400c:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46AF2217B for ; Thu, 17 Jul 2014 18:15:46 +0000 (UTC) Received: by mail-vc0-f174.google.com with SMTP id la4so5369366vcb.19 for ; Thu, 17 Jul 2014 11:15:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=NY/ZZwgB4eox7nUlGS6PIjYn+Y2Q+lbEoE0JTV/OfDI=; b=fTRKod9b0do86QJt/4rVOFLHuM9aizBNnEcceWNIpWUKrPYDYZZmgRYzaKEHyBLMVH ciD/byo0sl3B/SJ1WX+BdcIpvqc3WnzZlclOQybQrZApMlkSDGA382LUt6UU81XX+U+r 39YtmuvwSYe9Osfvkbcnkexu/AEi/H0s1JBrmej93xgO7WcELLCXlztwe5bN8Oo0j2ez 6K/Y/nP7fbhRPLIO6+CPQN4IZgNJRVBuo3udiSlkuWdcd5EAc2+/fv9worw9c54Kyr2B /nfney9U24yx/SKjQTWvP0GyaO6XcO80jwWDmrESu84wJ9s6NKydVrl9icUkYrmiAkAj xUDA== X-Received: by 10.53.13.200 with SMTP id fa8mr9063676vdd.57.1405620943501; Thu, 17 Jul 2014 11:15:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.221.61.131 with HTTP; Thu, 17 Jul 2014 11:15:23 -0700 (PDT) From: Prakhar Deep Date: Thu, 17 Jul 2014 23:45:23 +0530 Message-ID: Subject: Panic: 32-bit PAE enabled OS with 4GB RAM and 2 vCPUs on Bhyve To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Jul 2014 18:15:46 -0000 Hi, I am facing kernel panic when 2nd processor is executing it's reset code while running a 32-bit OS as a VM (with PAE enabled) on =E2=80=98bhyve= =E2=80=99 with following parameters: 1. Number of vCPUs 2 or more, and 2. Memory 4GB or more When number of vCPU =3D1 and memory > 4GB i.e. PAE mode with only one CPU o= r vCPU =3D 2 or more and memory < 4GB i.e non-PAE mode then VM comes up successfully. The OS is able to bring up the boot processor but it panics when AP is executing it's reset code. The OS uses fixed kernel virtual addresses which are mapped by a level 2 page table which is replicated for each CPU to allow some of these mappings to be different on a per-CPU basis. These pages act as storage for many global variables which are used when processor is executing reset code. When a second (non-boot) CPU comes up, we are seeing that per-CPU pages for this CPU are reset to all zeroes when the second CPU sets the PGE flag in CR4. This then leads to incorrect code paths. I am not able to understand why the pages mapped to AP are getting set to 0 when PGE flag is set in CR4 as this does not happen for boot processor. If i do not set PGE flag then pages are not set to 0 but the code panics some where else but not due to zeroed out global values. I have tried FreeBSD i386 with PAE support compiled into the kernel on Bhyve with >4GB RAM and 2 vCPUS and VM came up successfully. Also, the PGE bit is set in the CR4 register for FreeBSD. It will be helpful if anyone could throw some light on following points: 1. Why are the pages mapped to AP are being zeroed when PGE flag is set for AP in it's CR4 ? 2. Are there are any changes required in a 32-bit OS to boot it in PAE mode with 4GB or more RAM and 2 or more vCPUS on Bhyve ? 3. Was FreeBSD (i386) kernel updated to boot on Bhyve with 4GB or more RAM and 2 or more vCPUS ? Thanks. Prakhar