From owner-freebsd-virtualization@freebsd.org Tue Feb 20 23:09:17 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D31EF0ED42 for ; Tue, 20 Feb 2018 23:09:17 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E18A376738 for ; Tue, 20 Feb 2018 23:09:16 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.119.1] (wlan-141-23-176-213.tubit.tu-berlin.de [141.23.176.213]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id 7DE9761F92 for ; Tue, 20 Feb 2018 22:59:10 +0000 (UTC) From: "Fabian Freyer" To: freebsd-virtualization@freebsd.org Subject: VMX exit reason=33 and general userboot.so questions Date: Tue, 20 Feb 2018 23:58:56 +0100 X-Mailer: MailMate (1.10r5443) Message-ID: MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=_MailMate_36086495-52E9-42F0-92E1-29907CBCFF01_="; micalg=pgp-sha512; protocol="application/pgp-signature" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Feb 2018 23:09:17 -0000 --=_MailMate_36086495-52E9-42F0-92E1-29907CBCFF01_= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi! I=E2=80=99m currently writing a userboot.so-compatible boot loader [1] an= d am slowly getting to a point were I want to do some testing with runnin= g test kernels in bhyve. At the moment, I=E2=80=99m getting the following error after loading my k= ernel: ---8< snip vm exit[0] reason VMX rip 0x000000000010000c inst_length 0 status 0 exit_reason 33 qualification 0x0000000000000000 inst_type 0 inst_error 0 [1] [PID] abort bhyve -H -P -s 0,hostbridge -s 31,lpc -c 1 -m 128= M testing --->8 snap This is the register state I=E2=80=99m setting up before running bhyve: ---8< snip efer[0] 0x0000000000000000 cr0[0] 0x0000000000000021 cr3[0] 0x0000000000000000 cr4[0] 0x0000000000000000 dr7[0] 0x0000000000000000 rsp[0] 0x0000000000000000 rip[0] 0x000000000010000c rax[0] 0x000000002badb002 rbx[0] 0x0000000000100952 rcx[0] 0x0000000000000000 rdx[0] 0x0000000000000000 rsi[0] 0x0000000000000000 rdi[0] 0x0000000000000000 rbp[0] 0x0000000000000000 r8[0] 0x0000000000000000 r9[0] 0x0000000000000000 r10[0] 0x0000000000000000 r11[0] 0x0000000000000000 r12[0] 0x0000000000000000 r13[0] 0x0000000000000000 r14[0] 0x0000000000000000 r15[0] 0x0000000000000000 rflags[0] 0x0000000000000002 ds desc[0] 0x0000000000000000/0xffffffff/0x0000c093 es desc[0] 0x0000000000000000/0xffffffff/0x0000c093 fs desc[0] 0x0000000000000000/0xffffffff/0x0000c093 gs desc[0] 0x0000000000000000/0xffffffff/0x0000c093 ss desc[0] 0x0000000000000000/0xffffffff/0x0000c093 cs desc[0] 0x0000000000000000/0xffffffff/0x0000c09b tr desc[0] 0x0000000000000000/0x00000000/0x00000000 ldtr desc[0] 0x0000000000000000/0x00000000/0x00000000 gdtr[0] 0x0000000000000000/0x00000000 idtr[0] 0x0000000000000000/0x00000000 cs[0] 0x0000 ds[0] 0x0000 es[0] 0x0000 fs[0] 0x0000 gs[0] 0x0000 ss[0] 0x0000 tr[0] 0x0000 ldtr[0] 0x0000 [... omitted some, not sure if relevant] --->8 snap And here=E2=80=99s a diff of the register state before and after running = bhyve: ---8< snip --- before 2018-02-20 22:36:16.001919000 +0000 +++ after 2018-02-20 22:36:27.442941000 +0000 --- before 2018-02-20 22:36:16.001919000 +0000 +++ after 2018-02-20 22:36:27.442941000 +0000 @@ -56 +56 @@ -procbased_ctls[0] 0x00000000b5186572 +procbased_ctls[0] 0x00000000f51865f2 @@ -67 +67 @@ -host_cr3[0] 0x0000000000000000 +host_cr3[0] 0x0000000389cac09a @@ -101,4 +101,4 @@ -exit_reason[0] 0 -rtc nvram[000]: 0x05 -rtc time 0x5: Thu Jan 01 00:00:05 1970 -Capability "hlt_exit" is not set on vcpu 0 +exit_reason[0] 0x80000021 +rtc nvram[000]: 0x26 +rtc time 0x5a8ca2ea: Tue Feb 20 22:36:26 2018 +Capability "hlt_exit" is set on vcpu 0 @@ -106 +106 @@ -Capability "pause_exit" is not set on vcpu 0 +Capability "pause_exit" is set on vcpu 0 @@ -109 +109 @@ -active cpus: (none) +active cpus: 0 @@ -125 +125 @@ -number of vm exits for unknown reason 0 +number of vm exits for unknown reason 1 @@ -128 +128 @@ -number of vm exits handled in userspace 0 +number of vm exits handled in userspace 1 @@ -131 +131 @@ -vcpu total runtime 0 +vcpu total runtime 11904 @@ -165,3 +165,3 @@ -Number of vpid invalidations done 0 -vcpu migration across host cpus 0 -total number of vm exits 0 +Number of vpid invalidations done 1 +vcpu migration across host cpus 1 +total number of vm exits 1 --->8 snap The code at that RIP point also looks correct: xxd -s 0x000000000010000c -l 5 /dev/vmm/testing 0010000c: bc00 2000 00 Which decompiles to: 0010000c: bc00200000 mov esp,0x2000 I=E2=80=99ve been looking at what userboot.so, grub2-bhyve and vm_setup_f= reebsd_registers is doing. However, I left all registers that don=E2=80=99= t have a defined state in the multiboot specification [2] in the state th= ey are. How would I best start debugging this? Fabian [1] https://github.com/fabianfreyer/bhyve-multiboot/tree/multiboot/info [2] https://www.gnu.org/software/grub/manual/multiboot/multiboot.html --=_MailMate_36086495-52E9-42F0-92E1-29907CBCFF01_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQJWBAEBCgBAFiEEX6JoxdmEemcFacQZmealkcs85+YFAlqMqDAiHGZhYmlhbi5m cmV5ZXJAcGh5c2lrLnR1LWJlcmxpbi5kZQAKCRCZ5qWRyzzn5jimEADKTpWioyeS b5HMGfrwKvMwQHA73KKTE86c7Wo1poRuTSKxqAT4WeDio211I5uALJueyCHkFzWf xyI4XoPw97Ueq93y1Kx+/hjxU1/u8Fy64Jk/I9Nt0DA80hCTULzb4m+jl0tqPRNM Z/4JIQWu/RHTlyX4wHFg4w7vfobvg60X/qqXWjm7uaec5eQEgJdNpe1/x4wuyZ5D OgDFoks8gR/QUaXu3A9onc8lw0VAr59HyHWb+kty0QCHl1RKX7QDEPoqJ6lbt7f/ hfzvqi4F/pRHdy7a8CAz0iqUMsMfWdT5t+jZbsB2b3MvG3ZEa2UMX3uO8N38I5TP MEVxPkujR4hM4Yq/S0FpZD3R/5UFbg9d9zXJopI9byv8+gs1mH3d4GETn3DsovWw dv44QMKhOw8l0zmArX4qpGXzq+85CFo0sQ2vgCjdVqDCVgJZxpyGRxfYy42Gajpx PS/6scHL1CCRCAKsJrYHP3rBd+E94J1MCfo71/OVa4ppUnE1oCmu1KF0aoVCAKb5 OGDPdsfvvkCYbTEcWQ8KQwEFMBcFFHuK+stIN8k+J3E3v25lle9M5MQWXhX7DCxh 1QANGZwM9/iJne0QbfzwS2z/3gMZf0T1rv70aEZ1lhqGW1/1K6ZkL77xUE2x/7cJ kf+8ILy/Ry09LlkOCjWIkQx5pWXFnsTYzA== =y/K5 -----END PGP SIGNATURE----- --=_MailMate_36086495-52E9-42F0-92E1-29907CBCFF01_=--