Date: Tue, 06 Mar 2018 17:22:03 +0100 From: "Fabian Freyer" <fabian.freyer@physik.tu-berlin.de> To: "Peter Grehan" <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org, rumpkernel-users@freelists.org Subject: Re: rumpkernel and bhyve: triple faults Message-ID: <2AAD4069-7D37-4325-8990-21C5DFD80B3B@physik.tu-berlin.de> In-Reply-To: <651856d3-3c34-9930-cda1-62d41091f91f@freebsd.org> References: <C49D0E56-10A4-49D8-A843-E371395831B5@physik.tu-berlin.de> <651856d3-3c34-9930-cda1-62d41091f91f@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 3156 and 4880). --=_MailMate_915AEEFC-27FB-4C2F-88C0-FF431A3BA11F_= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Peter, On 6 Mar 2018, at 16:15, Peter Grehan wrote: > Exception 14 is a page fault (SDM Vol3 ch 6.15). The exception type is= "fault" which means it is delivered at the address it was detected at. > > This cascaded very quickly into a triple-fault, so it looks like it co= uld possibly be an issue with the stack. One debug tool you do have is to= get a register dump on exit, with 'bhyvectl --get-all --vm=3D<your vn na= me>'. > > For a page-fault, the virtual address that resulted in the fault will = be in the CR2 register. I don=E2=80=99t see a CR2 register in the output of bhyvectl --get-all, I= was looking for that too. > From the code at the faulting address: > > > 0000000000102a50 <cons_init>: > > 102a50: push rbx > > 102a51: call 103540 <hypervisor_detect> > > 102a56: cmp WORD PTR [rip-0x102a5c],0x0 # 2 <curr= ent_lwp+0x2> > > It's using RIP-relative addressing here, but objdump seems to think th= is may be an offset in the current_lwp structure - is it possible that ma= y have an uninitialized value ? I=E2=80=99m pretty sure it=E2=80=99s tooling that=E2=80=99s displaying so= mething off, since hopper is showing me this as 0x0000000000102a56 cmp word [0x2], 0x0 Which is very similar to what r2 is giving me: ;-- cons_init: 0x00102a50 53 push rbx ; /arch/x86:43= 0x00102a51 e8ea0a0000 call sym.hypervisor_detect ; /arch/x86:47= 0x00102a56 66833da4d5ef. cmp word [0x00000002], 0 ; /arch/x86:62= > (I don't believe this has anything to do with VGA). Maybe I=E2=80=99m off with my analysis of the actual fault here, but how = I understand the source (assuming compilers work as I would expect, which is not alway= s true) the values here are initialised from values in the bios data area (which = is zeroed out on bhyve): #define BIOS_COM1_BASE 0x400 #define BIOS_CRTC_BASE 0x463 =2E.. movw BIOS_COM1_BASE, %bx movw %bx, bios_com1_base movw BIOS_CRTC_BASE, %bx movw %bx, bios_crtc_base =2E.. /* * If the BIOS says no CRTC is present use the serial console if * available. */ if (bios_crtc_base =3D=3D 0) prefer_serial =3D 1; Here=E2=80=99s my full output from bhyvectl --get-all: ID Length Name 0 128MB sysmem Address Length Segment Offset Prot Flags 0 128MB sysmem 0 RWX efer[0] 0x0000000000000500 cr0[0] 0x0000000080010031 cr3[0] 0x000000000010b000 cr4[0] 0x0000000000002620 dr7[0] 0x0000000000000400 rsp[0] 0x0000000000100ff0 rip[0] 0x0000000000102a56 rax[0] 0x0000000000000000 rbx[0] 0x00000000003eaa2b rcx[0] 0x0000000068622065 rdx[0] 0x0000000020657679 rsi[0] 0x0000000000100fd0 rdi[0] 0x0000000040000000 rbp[0] 0x0000000000000000 r8[0] 0x0000000000100fdc r9[0] 0x0000000000100fd8 r10[0] 0x0000000000100fd4 r11[0] 0x0000000000000000 r12[0] 0x0000000000000000 r13[0] 0x0000000000000000 r14[0] 0x0000000000000000 r15[0] 0x0000000000000000 rflags[0] 0x0000000000010006 ds desc[0] 0x0000000000000000/0xffffffff/0x0000c093 es desc[0] 0x0000000000000000/0xffffffff/0x0000c093 fs desc[0] 0x0000000000000000/0xffffffff/0x0001c001 gs desc[0] 0x0000000000000000/0xffffffff/0x0001c001 ss desc[0] 0x0000000000000000/0xffffffff/0x0000c093 cs desc[0] 0x0000000000000000/0xffffffff/0x0000a09b tr desc[0] 0x0000000000000000/0x00000000/0x0000008b ldtr desc[0] 0x0000000000000000/0x0000ffff/0x00000082 gdtr[0] 0x0000000000378040/0x0000002f idtr[0] 0x0000000000000000/0x0000ffff cs[0] 0x0008 ds[0] 0x0018 es[0] 0x0018 fs[0] 0x0000 gs[0] 0x0000 ss[0] 0x0018 tr[0] 0x0000 ldtr[0] 0x0000 cr0_mask[0] 0xffffffff60000020 cr0_shadow[0] 0x0000000000000021 cr4_mask[0] 0xffffffffffe8f800 cr4_shadow[0] 0x0000000000000000 cr3_target_count[0] 0x0000000000000000 cr3_target0[0] 0x0000000000000000 cr3_target1[0] 0x0000000000000000 cr3_target2[0] 0x0000000000000000 cr3_target3[0] 0x0000000000000000 pinbased_ctls[0] 0x000000000000003f procbased_ctls[0] 0x00000000f51865f2 procbased_ctls2[0] 0x00000000000010a2 gla[0] 0xfffffe0000c41000 gpa[0] 0x0000000000000000 entry_interruption_info[0] 0x0000000000000000 tpr_threshold[0] 0x0000000000000000 instruction_error[0] 0x0000000000000000 exit_ctls[0] 0x000000000033efff entry_ctls[0] 0x00000000000093ff host_pat[0] 0x0001050600070406 host_cr0[0] 0x000000008005003b host_cr3[0] 0x0000000038045054 host_cr4[0] 0x00000000001726e0 host_rip[0] 0xffffffff81435290 host_rsp[0] 0xfffffe003218d700 vmcs_pointer[0] 0xffffffffffffffff vmcs_exit_interruption_info[0] 0x0000000080000b0e vmcs_exit_interruption_error[0] 0x0000000000000000 vmcs_guest_interruptibility[0] 0x0000000000000000 vmcs_exit_inst_length[0] 0x00000003 vmcs_exit_qualification[0] 0x0000000000000080 x2apic_state[0] 0 eptp[0] 0x000000003817905e exception_bitmap[0] 0xffffffff io_bitmap_a[0] 0 io_bitmap_b[0] 0 tsc_offset[0] 0x0000000000000000 msr_bitmap[0] 0x1adbc000 MSR_TSC [0] R- MSR_EFER [0] RW MSR_STAR [0] RW MSR_LSTAR [0] RW MSR_CSTAR [0] RW MSR_SF_MASK [0] RW MSR_FSBASE [0] RW MSR_GSBASE [0] RW MSR_KGSBASE [0] RW MSR_SYSENTER_CS_MSR [0] RW MSR_SYSENTER_ESP_MSR[0] RW MSR_SYSENTER_EIP_MSR[0] RW vpid[0] 0x0011 guest_pat[0] 0x0000000000000000 guest_sysenter_cs[0] 0 guest_sysenter_sp[0] 0 guest_sysenter_ip[0] 0 exit_reason[0] 0 rtc nvram[000]: 0x34 rtc time 0x5a9ebfd2: Tue Mar 06 16:20:34 2018 Capability "hlt_exit" is set on vcpu 0 Capability "mtrap_exit" is not set on vcpu 0 Capability "pause_exit" is set on vcpu 0 Capability "unrestricted_guest" is set on vcpu 0 Capability "enable_invpcid" is set on vcpu 0 active cpus: 0 suspended cpus: 0 pending: n/a current: n/a vcpu0 stats: number of times in/out was intercepted 0 number of times cpuid was intercepted 3 vm exits due to nested page fault 13 vm exits for instruction emulation 0 number of vm exits for unknown reason 0 number of times astpending at exit 0 number of times idle requested at exit 0 number of vm exits handled in userspace 14 number of times rendezvous pending at exit 0 number of vm exits due to exceptions 3 number of NMIs delivered to vcpu 0 number of ExtINTs delivered to vcpu 0 Resident memory 69632 Wired memory 0 vcpu total runtime 3112708 EOI without any in-service interrupt 0 error interrupts generated by vlapic 0 timer interrupts generated by vlapic 0 corrected machine check interrupts generated by vlapic 0 lvts triggered[0] 0 lvts triggered[1] 0 lvts triggered[2] 0 lvts triggered[3] 0 lvts triggered[4] 0 lvts triggered[5] 0 lvts triggered[6] 0 ipis sent to vcpu[0] 0 ipis sent to vcpu[1] 0 ipis sent to vcpu[2] 0 ipis sent to vcpu[3] 0 ipis sent to vcpu[4] 0 ipis sent to vcpu[5] 0 ipis sent to vcpu[6] 0 ipis sent to vcpu[7] 0 ipis sent to vcpu[8] 0 ipis sent to vcpu[9] 0 ipis sent to vcpu[10] 0 ipis sent to vcpu[11] 0 ipis sent to vcpu[12] 0 ipis sent to vcpu[13] 0 ipis sent to vcpu[14] 0 ipis sent to vcpu[15] 0 number of ticks vcpu was idle 0 vcpu migration across host cpus 1 total number of vm exits 19 vm exits due to external interrupt 0 Number of vpid invalidations saved 0 Number of vpid invalidations done 1 number of times hlt was intercepted 0 number of times %cr access was intercepted 0 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 0 number of monitor trap exits 0 number of times pause was intercepted 0 vm exits due to interrupt window opening 0 vm exits due to nmi window opening 0 Fabian --=_MailMate_915AEEFC-27FB-4C2F-88C0-FF431A3BA11F_= Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQJWBAEBCgBAFiEEX6JoxdmEemcFacQZmealkcs85+YFAlqewCsiHGZhYmlhbi5m cmV5ZXJAcGh5c2lrLnR1LWJlcmxpbi5kZQAKCRCZ5qWRyzzn5munD/9HLibTUH3S 0sRR4LvpKXUZIzoYkIEqHNDtrGh4Y0NaNAJKQx1J6FAzf5lZBHKkcDx83vSfbRs7 C0BT76kq+jxxCgMSPm1gxXmjOJ5wuuM64tFKalEIn8AVwdKfeCiqvL4COM3DbTds obw3MWifiBf4U8p7mZMm+AQmUxZgllmV4Lglzkq79rKWOxHjyUz7yP/sJHOUS+wN +0QP9iFjdv2DFuFJTfa9/d84nhzY6bbdajmwYD2jtGfLnCkQW2JDjkMyk6Q5YT0g fBDard4y2EyOsUx+RZpOWvC09MnuyTfZEVPbVCpyfH9tyRxjDw9ARnOOuWo1267u R742vuZrZumKWV6TygloUS6DDpc6P7DhYPCZKHsrpsBUWwq+HCd52SbcYe3WS+rb DqOHW800KLQlvopg2mQ2nix+f6Bb1pmYmJgvRcp14f+QDihjszJGjTXo8Y3npQqm JsHBXLJsa6Txo1SlESWPg/uTB5fiNt63hL7aI9ElWSeDLalvQwRfPlMLWLXxntIX fIFdRAF5d9nKAKYUYf9O9K94z8yFoIDTirFmm44zaWvwOosHk7WC0iCvdHZL71Xv pXnWOhU5AbOANZgS3xPRgPZNhn2DmJAtUCLTtBiugdzU5IVv92Wmo0ifiNbmihjx nK/IlfQ/CDc9+KcaFexiEkuqOdKaYTTqTw== =z1en -----END PGP SIGNATURE----- --=_MailMate_915AEEFC-27FB-4C2F-88C0-FF431A3BA11F_=--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2AAD4069-7D37-4325-8990-21C5DFD80B3B>