From owner-freebsd-virtualization@freebsd.org Thu Feb 22 03:47:47 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 80FFCF23EE3 for ; Thu, 22 Feb 2018 03:47:47 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: from mail-qk0-x231.google.com (mail-qk0-x231.google.com [IPv6:2607:f8b0:400d:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EC7B9876F9; Thu, 22 Feb 2018 03:47:46 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: by mail-qk0-x231.google.com with SMTP id s188so4873425qkb.2; Wed, 21 Feb 2018 19:47:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=GHzNzHvKlHKGfTV3s1QdF2/G/Wq+57PsE/F6jHqkymY=; b=I96bWgSXdyJ0uXmUWDx1Z6FdKI42C4dm+Sl9s8wJUHCXoDCpx4YUNdHlA2s5Xi3Ky1 G5mCQFyORGLncYIquRPeVkVy2iY5oK7eEM3jXi0B8sNj4SMZGorx7Lig1GnNbMscNnGg DyxWG7jqdztOtDZgN0e0U9sCcaM7tjSRgVot/vgYc2+dvhxSBpFmKesi+wj9a63C27gA 2KRU201CaFOt9HsNnSUsKzWHx7mhhm15B0q3pozBYORtutSCeW1ywJ5GXLyt7N9QzduG Rw3OEZu4rcMp1+d9YRv27rIUob4fOXixog61Bkez/agtShmu/j+6mD1zcuXIvsFLFJOg fZyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=GHzNzHvKlHKGfTV3s1QdF2/G/Wq+57PsE/F6jHqkymY=; b=Xo74EiD48dBpqF0p8RyQXhUsHDoZYTKgVJBHF5IMvIjIz3JrO0bi04dn8cwhah1pNS Z4S5K5B9BQ2FcfxOIpfiRiAt9JCogxtY/oQBdCKSfypY0K3U4Q5BsHgM52Xa7tM3SIV0 MSIL7mCGi/fb+nbhXdCL0wErE8FrE/9fYNKZLnXX1WUBqwspTtSbTpR+e7kJ7VDP1k+j GqvpXmWtSBUh8/m3EusaCyhtLO+SguKiJuYZN/Xo28flZWKlxBOOvvZXH2sDqb5mcvf6 SFboHns/Ykfb5SKtzQZZ9H4hFLnuYcpN942c4QlKKRR7TvXZcOnBJhWq/cZAO6zAqorx /oiw== X-Gm-Message-State: APf1xPAdU6ekZif0YYBqT9oSUu/SeLzRIlkhzm7nfDldNgDzALIuGkJP D6LXRuB/qKTD8ZcWB7gPNRIt3+qMBCnw7m+zr1MEdg== X-Google-Smtp-Source: AH8x224IN8R+AcD1M/dFHbWF+uKpb5h+1nXpWJp5oO0rquL6DHbgP7ftq75tb8RP1HA473wervcIFe3KEVPRsPg0Oi8= X-Received: by 10.55.123.5 with SMTP id w5mr8787925qkc.261.1519271266515; Wed, 21 Feb 2018 19:47:46 -0800 (PST) MIME-Version: 1.0 Received: by 10.12.170.140 with HTTP; Wed, 21 Feb 2018 19:47:45 -0800 (PST) In-Reply-To: <5C51D1DD-BD11-40BD-B2EB-185524774D2C@physik.tu-berlin.de> References: <0142d062-ca13-6917-bb06-28cac727680b@freebsd.org> <5C51D1DD-BD11-40BD-B2EB-185524774D2C@physik.tu-berlin.de> From: Anish Date: Wed, 21 Feb 2018 19:47:45 -0800 Message-ID: Subject: Re: VMX exit reason=33 and general userboot.so questions To: Fabian Freyer Cc: Peter Grehan , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Feb 2018 03:47:47 -0000 >idtr[0] 0x0000000000000000/0x00000000 IDTR is not set, you can set the value same as in GDTR, from libvmm /* GDTR, IDTR */ desc_base =3D 0; desc_limit =3D 0xffff; desc_access =3D 0; error =3D vm_set_desc(vmctx, vcpu, VM_REG_GUEST_GDTR, desc_base, desc_limit, desc_access); if (error !=3D 0) goto done; error =3D vm_set_desc(vmctx, vcpu, VM_REG_GUEST_IDTR, desc_base, desc_limit, desc_access); >I=E2=80=99m assuming entry_ctls[0] =3D 0x91fb is the VM-entry control, Yes, it is VM entry control. Its value need to be determined based on MSR_VMX_ENTRY_CTLS and MSR_VMX_TRUE_ENTRY_CTLS MSRs, see /* Check support for VM-entry controls */ error =3D vmx_set_ctlreg(MSR_VMX_ENTRY_CTLS, MSR_VMX_TRUE_ENTRY_CTL= S, VM_ENTRY_CTLS_ONE_SETTING, VM_ENTRY_CTLS_ZERO_SETTING, &entry_ctls); -Anish On Wed, Feb 21, 2018 at 11:08 AM, Fabian Freyer < fabian.freyer@physik.tu-berlin.de> wrote: > Hi Peter, > > thanks for your response! > > On 21 Feb 2018, at 17:59, Peter Grehan wrote: > > > exit_reason 33 > > > > From the Intel SDM, vol 3B Appendix C, this error is "VM-entry failure > due to invalid guest state". > > Yes, I=E2=80=99m currently going through SDM, vol 3B, section 26.3, and > > > These errors can be difficult to debug given the large amount of guest > state involved :( > > definitely accurately describes the experience. That said, I=E2=80=99m go= ing > through each check and trying to see what is going wrong. > > > However, looking at the state from your dump: > > > > > tr desc[0] 0x0000000000000000/0x00000000/0x00000000 > > > > I believe you will have to set this. Here's the comment and relevant > code fragment from grub2-bhyve grub-core/kern/emu/bhyve_ > hostif.c:grub_emu_bhyve_boot32() > > > > /* > > * XXX TR is pointing to null selector even though we set the > > * TSS segment to be usable with a base address and limit of 0. > > * Has to be 8b or vmenter will fail > > */ > > desc_access =3D 0x0000008b; > > assert(vm_set_desc(bhyve_ctx, 0, VM_REG_GUEST_TR, 0x1000, 0x67, > > desc_access) =3D=3D 0); > > > > grub2-bhyve has been able to load/boot multiboot images, so I suspect > the register settings in grub_emu_bhyve_boot32() are a good place to star= t > from. > > Thanks. I=E2=80=99ve tried setting all registers (especially) the segment > registers as grub2-bhyve does, but I still don=E2=80=99t seem to be able = to boot > correctly. Fixing the TR descriptor didn=E2=80=99t do it. > > Here=E2=80=99s a list of the descriptors I set up: > > ds desc[0] 0x0000000000000000/0xffffffff/0x00000093 > es desc[0] 0x0000000000000000/0xffffffff/0x00000093 > fs desc[0] 0x0000000000000000/0xffffffff/0x00000093 > gs desc[0] 0x0000000000000000/0xffffffff/0x00000093 > ss desc[0] 0x0000000000000000/0xffffffff/0x00000093 > cs desc[0] 0x0000000000000000/0xffffffff/0x0000009b > tr desc[0] 0x0000000000001000/0x00000067/0x0000008b > ldtr desc[0] 0x0000000000000000/0x0000ffff/0x00010082 > > > gdtr points to a default gdt as set up in grub2-bhyve: > > static uint16_t bhyve_gdt[] =3D { > 0x0000, 0x0000, 0x0000, 0x0000, /* Null */ > 0x0000, 0x0000, 0x0000, 0x0000, /* Null #2 */ > 0xffff, 0x0000, 0x9a00, 0x00cf, /* code */ > 0xffff, 0x0000, 0x9200, 0x00cf, /* data */ > 0x0000, 0x0000, 0x8900, 0x0080 /* tss */ > }; > > gdtr[0] 0x0000000000100161/0x00000027 > idtr[0] 0x0000000000000000/0x00000000 > > The rest is set up as by grub2-bhyve: > cs[0] 0x0008 > ds[0] 0x0010 > es[0] 0x0010 > fs[0] 0x0010 > gs[0] 0x0010 > ss[0] 0x0010 > tr[0] 0x0000 > ldtr[0] 0x0000 > > I=E2=80=99m assuming entry_ctls[0] =3D 0x91fb is the VM-entry control, bu= t is it the > value described by Table 24-7 of the SDM Volume 3B? Is this constant for > the first entry after running bhyveload? That is, when writing some > compliance checks against section 26.3.1, can I just hardcode that value? > > I=E2=80=99m starting to dig through the bhyverun code, but I=E2=80=99m st= ill pretty new to > VMX stuff, so I am ending up with more questions than answers. > > Fabian >