From owner-freebsd-virtualization@freebsd.org Wed Feb 21 16:59:55 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7CA2F150C3 for ; Wed, 21 Feb 2018 16:59:55 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 4C2458647A for ; Wed, 21 Feb 2018 16:59:55 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 9F26920B4AB9 for ; Thu, 22 Feb 2018 02:59:52 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 9648F2809C2 for ; Thu, 22 Feb 2018 02:59:52 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id pJki0tY9dXlZ for ; Thu, 22 Feb 2018 02:59:52 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id D806F2804E8; Thu, 22 Feb 2018 02:59:49 +1000 (AEST) Subject: Re: VMX exit reason=33 and general userboot.so questions To: Fabian Freyer References: Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <0142d062-ca13-6917-bb06-28cac727680b@freebsd.org> Date: Wed, 21 Feb 2018 08:59:47 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=KPZ08mNo c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=IkcTkHD0fZMA:10 a=Op4juWPpsa0A:10 a=4q-dA8eIyU7An6lk__gA:9 a=QEXdDO2ut3YA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Feb 2018 16:59:56 -0000 Hi Fabian, > exit_reason 33 From the Intel SDM, vol 3B Appendix C, this error is "VM-entry failure due to invalid guest state". These errors can be difficult to debug given the large amount of guest state involved :( However, looking at the state from your dump: > tr desc[0] 0x0000000000000000/0x00000000/0x00000000 I believe you will have to set this. Here's the comment and relevant code fragment from grub2-bhyve grub-core/kern/emu/bhyve_hostif.c:grub_emu_bhyve_boot32() /* * XXX TR is pointing to null selector even though we set the * TSS segment to be usable with a base address and limit of 0. * Has to be 8b or vmenter will fail */ desc_access = 0x0000008b; assert(vm_set_desc(bhyve_ctx, 0, VM_REG_GUEST_TR, 0x1000, 0x67, desc_access) == 0); grub2-bhyve has been able to load/boot multiboot images, so I suspect the register settings in grub_emu_bhyve_boot32() are a good place to start from. later, Peter.