From owner-freebsd-virtualization@freebsd.org Sat May 9 07:42:20 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 25CED2DF805 for ; Sat, 9 May 2020 07:42:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 49JzgJ0Gjfz4PPq for ; Sat, 9 May 2020 07:42:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 090062DF6FB; Sat, 9 May 2020 07:42:20 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 08C202DF6FA for ; Sat, 9 May 2020 07:42:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49JzgH6YV4z4PPn for ; Sat, 9 May 2020 07:42:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id DC7F618763 for ; Sat, 9 May 2020 07:42:19 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0497gJ8Z076591 for ; Sat, 9 May 2020 07:42:19 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0497gJLt076575 for virtualization@FreeBSD.org; Sat, 9 May 2020 07:42:19 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 246168] Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve Date: Sat, 09 May 2020 07:42:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 12.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: drum@graphica.com.au X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.32 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 May 2020 07:42:20 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246168 --- Comment #9 from John Hartley --- (In reply to Peter Grehan from comment #8) Hi Peter, thanks for reviewing. I have done further testing: A. Running bhyveload first as per your feedback Result: Brings up FreeBSD Loader and then get to "/dev/entropy not found message and closes machine <> # bhyvectl --vm=3Dbee --get-stats > bee-vm-stats-01.txt # cat bee-vm-stats-01.txt=20 vcpu0 stats: number of times in/out was intercepted 0 number of times cpuid was intercepted 0 vm exits due to nested page fault 0 vm exits for instruction emulation 0 number of vm exits for unknown reason 0 number of times astpending at exit 0 number of times idle requested at exit 0 number of vm exits handled in userspace 0 number of times rendezvous pending at exit 0 number of vm exits due to exceptions 0 number of NMIs delivered to vcpu 0 number of ExtINTs delivered to vcpu 0 Resident memory 0 Wired memory 0 vcpu total runtime 0 EOI without any in-service interrupt 0 error interrupts generated by vlapic 0 timer interrupts generated by vlapic 0 corrected machine check interrupts generated by vlapic 0 lvts triggered[0] 0 lvts triggered[1] 0 lvts triggered[2] 0 lvts triggered[3] 0 lvts triggered[4] 0 lvts triggered[5] 0 lvts triggered[6] 0 ipis sent to vcpu[0] 0 ipis sent to vcpu[1] 0 ipis sent to vcpu[2] 0 ipis sent to vcpu[3] 0 ipis sent to vcpu[4] 0 ipis sent to vcpu[5] 0 ipis sent to vcpu[6] 0 ipis sent to vcpu[7] 0 ipis sent to vcpu[8] 0 ipis sent to vcpu[9] 0 ipis sent to vcpu[10] 0 ipis sent to vcpu[11] 0 ipis sent to vcpu[12] 0 ipis sent to vcpu[13] 0 ipis sent to vcpu[14] 0 ipis sent to vcpu[15] 0 number of ticks vcpu was idle 0 vcpu migration across host cpus 0 total number of vm exits 0 vm exits due to external interrupt 0 Number of vpid invalidations saved 0 Number of vpid invalidations done 0 number of times hlt was intercepted 0 number of times %cr access was intercepted 0 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 0 number of monitor trap exits 0 number of times pause was intercepted 0 vm exits due to interrupt window opening 0 vm exits due to nmi window opening 0 <> This was then followed with invoking Bhyve, as per original posting. This again results in Layer 1 - FreeBSD VM going to pause with same error v= ia Layer 0 - /var/log/libvirt/qemu/VM.log <> char device redirected to /dev/pts/27 (label charserial0) 2020-05-09T06:28:07.870436Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.870499Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.872528Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.872543Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.873742Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.873757Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T06:28:07.874950Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T06:28:07.874964Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=3D00000000 EBX=3D00000000 ECX=3D00000000 EDX=3D00000000 ESI=3D00000000 EDI=3D00000000 EBP=3D00000000 ESP=3D00000000 EIP=3D00000000 EFL=3D00000000 [-------] CPL=3D0 II=3D0 A20=3D1 SMM=3D0 HLT= =3D0 ES =3D0000 00000000 00000000 00008000 DPL=3D0 CS =3D0000 00000000 00000000 00008000 DPL=3D0 SS =3D0000 00000000 00000000 00008000 DPL=3D0 DS =3D0000 00000000 00000000 00008000 DPL=3D0 FS =3D0000 00000000 00000000 00008000 DPL=3D0 GS =3D0000 00000000 00000000 00008000 DPL=3D0 LDT=3D0000 00000000 00000000 00008000 DPL=3D0 TR =3D0000 00000000 00000000 00008000 DPL=3D0 GDT=3D 0000000000000000 00000000 IDT=3D 0000000000000000 00000000 CR0=3D80050033 CR2=3D0000000000000000 CR3=3D0000000000000000 CR4=3D00372060 DR0=3D0000000000000000 DR1=3D0000000000000000 DR2=3D0000000000000000 DR3=3D0000000000000000=20 DR6=3D00000000ffff0ff0 DR7=3D0000000000000400 EFER=3D0000000000000d01 Code=3D ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? 2020-05-09T06:42:45.518166Z qemu-system-x86_64: terminating on signal 15 fr= om pid 2119 (/usr/sbin/libvirtd) 2020-05-09 06:42:45.919+0000: shutting down, reason=3Ddestroyed <> B. Running bhyve with UEFI rom rather than BIOS: # bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd bee <> $ cd /var/log/libvirt/qemu $ sudo tail -f swarm-bhyve-freebsd.log=20 [sudo] password for XXX:=20 -msg timestamp=3Don char device redirected to /dev/pts/27 (label charserial0) 2020-05-09T07:21:12.156910Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.156975Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.158959Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.158973Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.160187Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.160201Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-09T07:21:12.161459Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-09T07:21:12.161471Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=3D00000000 EBX=3D00000000 ECX=3D00000000 EDX=3D00000f00 ESI=3D00000000 EDI=3D00000000 EBP=3D00000000 ESP=3D00000000 EIP=3D00000000 EFL=3D00000000 [-------] CPL=3D0 II=3D0 A20=3D1 SMM=3D0 HLT= =3D0 ES =3D0000 00000000 00000000 00008000 DPL=3D0 CS =3D0000 00000000 00000000 00008000 DPL=3D0 SS =3D0000 00000000 00000000 00008000 DPL=3D0 DS =3D0000 00000000 00000000 00008000 DPL=3D0 FS =3D0000 00000000 00000000 00008000 DPL=3D0 GS =3D0000 00000000 00000000 00008000 DPL=3D0 LDT=3D0000 00000000 00000000 00008000 DPL=3D0 TR =3D0000 00000000 00000000 00008000 DPL=3D0 GDT=3D 0000000000000000 00000000 IDT=3D 0000000000000000 00000000 CR0=3D80050033 CR2=3D0000000000000000 CR3=3D0000000000000000 CR4=3D00372060 DR0=3D0000000000000000 DR1=3D0000000000000000 DR2=3D0000000000000000 DR3=3D0000000000000000=20 DR6=3D00000000ffff0ff0 DR7=3D0000000000000400 EFER=3D0000000000000d01 Code=3D ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? 2020-05-09T07:30:48.814754Z qemu-system-x86_64: terminating on signal 15 fr= om pid 2119 (/usr/sbin/libvirtd) 2020-05-09 07:30:49.015+0000: shutting down, reason=3Ddestroyed <> So there must be some variation between my and your setup. I am running test on: Lenovo x3650 M5 Server with 2 x CPU=20 QEMU Q35 V4.2 with OVMF Layer 1 FreeBSD 12.1 VM with 2 x vmxnet3 vNICs (1 for machine access and 1 for tap0 use) Due to bug with FreeBSD 12.1 and QEMU with netmap I have recompiled my Laye= r 1 FreeBSD kernel with "dev netmap" removed. Could you possibly suggest way to get better diagnostics Thanks. Cheers, John Hartley. --=20 You are receiving this mail because: You are the assignee for the bug.=