From owner-freebsd-virtualization@freebsd.org Sat Sep 17 05:54:08 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0755BDDE5C for ; Sat, 17 Sep 2016 05:54:08 +0000 (UTC) (envelope-from trentnthompson@gmail.com) Received: from mail-yb0-x236.google.com (mail-yb0-x236.google.com [IPv6:2607:f8b0:4002:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 71138782 for ; Sat, 17 Sep 2016 05:54:08 +0000 (UTC) (envelope-from trentnthompson@gmail.com) Received: by mail-yb0-x236.google.com with SMTP id u125so60177286ybg.3 for ; Fri, 16 Sep 2016 22:54:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=QjGr0sRYyexjfWzv2kBbBuQee5p9g7yKGG6vTP+8hlg=; b=UQkCylQ9HOoH279urSY97M4Xs4erbVNJTYJdqg+qKEGN7n8og2PLtQLchG4Nd4Q2uB t19rSLfa8xbajkJ8NRQQnvyL6IeIWKja8aq8CHb9cA3oLR95j5MGUUUU9vRLee+EzUw6 gc8JnWAc7wToFDiecK9y9pVi9vo5YUDVnIW8nDcmn+rAZ/xVJM03DMZBt6D31evq6gYi vn3uQDfTcN4lKs8bGX0oBfpiYQyyHIzHCDofzppU7kmnxUACXUQKWz638AJ7XKxVB+7v /eEcP46ec+b9KkpR1eo8irMot0ZYJhtwFoEHYIN0pwQy9bNfJg/WfkpHGE13s7Oot7AC qDjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=QjGr0sRYyexjfWzv2kBbBuQee5p9g7yKGG6vTP+8hlg=; b=UM/Dqh1WbyIet7BXLe887qgaPGSnPpckEtMkx+A7igedpQgj6zdOa9jMwGBbKb5Zin BfDGx1bpY0o1UkiOKZ5gnzNjjwb9ImCLiyLdDi+3bb4H3/KJ3HtROTTj093oNhV0hXX6 P8Wl2N/o6XLyspgKMA6Y1jc/WqWulW6ck1ZrD3xAIeCAW1mQjqMerc2T1IUfpxo41qp5 RCssf7S+8M6Crd58Ertl2twMdmOSPXExXzstj8yqhjjvoc0Rfm15tg04exdNmsH/FM67 vlr6w34BmcijwAeVi7gHuVp6X2Z9LTUEhPhcCh/hI8Sl87QxA0XQc/ROBe1nSiFiORvF vhWA== X-Gm-Message-State: AE9vXwMj/q0yT6S9W1CX7qWx0rtIA22n21HR/hCeNSwjZ+7mrXJA5TP0e/N/b6YfCzIwL1d+JlFoQNc2uwRbcQ== X-Received: by 10.37.215.202 with SMTP id o193mr17505674ybg.95.1474091647353; Fri, 16 Sep 2016 22:54:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.196.196 with HTTP; Fri, 16 Sep 2016 22:54:06 -0700 (PDT) From: Trent Thompson Date: Fri, 16 Sep 2016 23:54:06 -0600 Message-ID: Subject: High vCPU Counts in bhyve To: FreeBSD virtualization Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Sep 2016 05:54:08 -0000 Good day to all! I was recently given the hint that I could edit vmm.h to allow for more than 16 vCPUs in bhyve, so I did just that to see what would happen. I understand it's not supported, just wanted to try it out. When trying to boot FreeBSD 12-CURRENT with anything up to 25 vCPUs the guest will boot and run just fine. At 26+ vCPUs the guest OS will panic at boot time. I have posted all of these outputs and more to GitHub [1]. I started by installing FreeBSD with 16 vCPUs and then incrementing until it wouldn't go any further. You can find the dmesg.boot output of the guest at 16 vCPUs on GitHub [2]. Here is the relevant line in /usr/src/sys/amd64/include/vmm.h: #define VM_MAXCPU 32 /* maximum virtual cpus */ Here is what appears in the serial console when attempting to boot with 26 vCPUs (The full output can also be found on GitHub [3]): Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x60 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff80ff3859 stack pointer = 0x28:0xffffffff821c19d0 frame pointer = 0x28:0xffffffff821c1a50 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 0 (swapper) [ thread pid 0 tid 100000 ] Stopped at acpi_install_wakeup_handler+0x1d9: movq %rdi,0x60(%r14) db> If anyone has any hints on how to get more information from this or ideas on the apparent APIC error would be greatly appreciated. Thanks and happy hacking! -Trent @pr1ntf [1] https://gist.github.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31 [2] https://gist.githubusercontent.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31/raw/09dde8362a5e488794e6d7c8b70bb0368b964b49/3-guest-dmesg.boot.txt [3] https://gist.githubusercontent.com/pr1ntf/270efd6ec3fe376e68d1f9dad2deef31/raw/09dde8362a5e488794e6d7c8b70bb0368b964b49/4-guest-26vcpucrash.txt