Date: Sat, 09 May 2020 07:42:19 +0000 From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 246168] Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve Message-ID: <bug-246168-27103-xHOFcgiglW@https.bugs.freebsd.org/bugzilla/> In-Reply-To: <bug-246168-27103@https.bugs.freebsd.org/bugzilla/> References: <bug-246168-27103@https.bugs.freebsd.org/bugzilla/>
next in thread | previous in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246168 --- Comment #9 from John Hartley <drum@graphica.com.au> --- (In reply to Peter Grehan from comment #8) Hi Peter, thanks for reviewing. I have done further testing: A. Running bhyveload first as per your feedback Result: Brings up FreeBSD Loader and then get to "/dev/entropy not found message and closes machine <<BHYVECTL GET-STATS>> # bhyvectl --vm=3Dbee --get-stats > bee-vm-stats-01.txt # cat bee-vm-stats-01.txt=20 vcpu0 stats: number of times in/out was intercepted 0 number of times cpuid was intercepted 0 vm exits due to nested page fault 0 vm exits for instruction emulation 0 number of vm exits for unknown reason 0 number of times astpending at exit 0 number of times idle requested at exit 0 number of vm exits handled in userspace 0 number of times rendezvous pending at exit 0 number of vm exits due to exceptions 0 number of NMIs delivered to vcpu 0 number of ExtINTs delivered to vcpu 0 Resident memory 0 Wired memory 0 vcpu total runtime 0 EOI without any in-service interrupt 0 error interrupts generated by vlapic 0 timer interrupts generated by vlapic 0 corrected machine check interrupts generated by vlapic 0 lvts triggered[0] 0 lvts triggered[1] 0 lvts triggered[2] 0 lvts triggered[3] 0 lvts triggered[4] 0 lvts triggered[5] 0 lvts triggered[6] 0 ipis sent to vcpu[0] 0 ipis sent to vcpu[1] 0 ipis sent to vcpu[2] 0 ipis sent to vcpu[3] 0 ipis sent to vcpu[4] 0 ipis sent to vcpu[5] 0 ipis sent to vcpu[6] 0 ipis sent to vcpu[7] 0 ipis sent to vcpu[8] 0 ipis sent to vcpu[9] 0 ipis sent to vcpu[10] 0 ipis sent to vcpu[11] 0 ipis sent to vcpu[12] 0 ipis sent to vcpu[13] 0 ipis sent to vcpu[14] 0 ipis sent to vcpu[15] 0 number of ticks vcpu was idle 0 vcpu migration across host cpus 0 total number of vm exits 0 vm exits due to external interrupt 0 Number of vpid invalidations saved 0 Number of vpid invalidations done 0 number of times hlt was intercepted 0 number of times %cr access was intercepted 0 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 0 number of monitor trap exits 0 number of times pause was intercepted 0 vm exits due to interrupt window opening 0 vm exits due to nmi window opening 0 <<END BHYVECTL GET-STATS>> This was then followed with invoking Bhyve, as per original posting. This again results in Layer 1 - FreeBSD VM going to pause with same error v= ia Layer 0 - /var/log/libvirt/qemu/VM.log <<LAYER 0 - LOG RESULT>> char device redirected to /dev/pts/27 (label charserial0) 2020-05-09T06:28:07.870436Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.870499Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.872528Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.872543Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.873742Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.873757Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.874950Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.874964Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=3D00000000 EBX=3D00000000 ECX=3D00000000 EDX=3D00000000 ESI=3D00000000 EDI=3D00000000 EBP=3D00000000 ESP=3D00000000 EIP=3D00000000 EFL=3D00000000 [-------] CPL=3D0 II=3D0 A20=3D1 SMM=3D0 HLT= =3D0 ES =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> CS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> SS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> DS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> FS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> GS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> LDT=3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> TR =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> GDT=3D 0000000000000000 00000000 IDT=3D 0000000000000000 00000000 CR0=3D80050033 CR2=3D0000000000000000 CR3=3D0000000000000000 CR4=3D00372060 DR0=3D0000000000000000 DR1=3D0000000000000000 DR2=3D0000000000000000 DR3=3D0000000000000000=20 DR6=3D00000000ffff0ff0 DR7=3D0000000000000400 EFER=3D0000000000000d01 Code=3D<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? 2020-05-09T06:42:45.518166Z qemu-system-x86_64: terminating on signal 15 fr= om pid 2119 (/usr/sbin/libvirtd) 2020-05-09 06:42:45.919+0000: shutting down, reason=3Ddestroyed <<END LAYER 0 - LOG RESULT>> B. Running bhyve with UEFI rom rather than BIOS: # bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd bee <<LAYER 0 TAIL RESULT>> $ cd /var/log/libvirt/qemu $ sudo tail -f swarm-bhyve-freebsd.log=20 [sudo] password for XXX:=20 -msg timestamp=3Don char device redirected to /dev/pts/27 (label charserial0) 2020-05-09T07:21:12.156910Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.156975Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.158959Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.158973Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.160187Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.160201Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.161459Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.161471Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=3D00000000 EBX=3D00000000 ECX=3D00000000 EDX=3D00000f00 ESI=3D00000000 EDI=3D00000000 EBP=3D00000000 ESP=3D00000000 EIP=3D00000000 EFL=3D00000000 [-------] CPL=3D0 II=3D0 A20=3D1 SMM=3D0 HLT= =3D0 ES =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> CS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> SS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> DS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> FS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> GS =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> LDT=3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> TR =3D0000 00000000 00000000 00008000 DPL=3D0 <hiword> GDT=3D 0000000000000000 00000000 IDT=3D 0000000000000000 00000000 CR0=3D80050033 CR2=3D0000000000000000 CR3=3D0000000000000000 CR4=3D00372060 DR0=3D0000000000000000 DR1=3D0000000000000000 DR2=3D0000000000000000 DR3=3D0000000000000000=20 DR6=3D00000000ffff0ff0 DR7=3D0000000000000400 EFER=3D0000000000000d01 Code=3D<??> ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? 2020-05-09T07:30:48.814754Z qemu-system-x86_64: terminating on signal 15 fr= om pid 2119 (/usr/sbin/libvirtd) 2020-05-09 07:30:49.015+0000: shutting down, reason=3Ddestroyed <<END LAYER 0 TAIL>> So there must be some variation between my and your setup. I am running test on: Lenovo x3650 M5 Server with 2 x CPU=20 QEMU Q35 V4.2 with OVMF Layer 1 FreeBSD 12.1 VM with 2 x vmxnet3 vNICs (1 for machine access and 1 for tap0 use) Due to bug with FreeBSD 12.1 and QEMU with netmap I have recompiled my Laye= r 1 FreeBSD kernel with "dev netmap" removed. Could you possibly suggest way to get better diagnostics Thanks. Cheers, John Hartley. --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-246168-27103-xHOFcgiglW>