From owner-freebsd-virtualization@freebsd.org Mon May 4 08:44:34 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F05232C967D for ; Mon, 4 May 2020 08:44:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 49FxHQ66kDz3Q3G for ; Mon, 4 May 2020 08:44:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id D1F222C967C; Mon, 4 May 2020 08:44:34 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1B372C967B for ; Mon, 4 May 2020 08:44:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49FxHQ5GH1z3Q3F for ; Mon, 4 May 2020 08:44:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id AFF5F217B9 for ; Mon, 4 May 2020 08:44:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 0448iYB2076535 for ; Mon, 4 May 2020 08:44:34 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 0448iY4i076534 for virtualization@FreeBSD.org; Mon, 4 May 2020 08:44:34 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 246168] Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve Date: Mon, 04 May 2020 08:44:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 12.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: drum@graphica.com.au X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 May 2020 08:44:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246168 Bug ID: 246168 Summary: Ubuntu 20.04 QEMU Failure with nested FreeBSD bhyve Product: Base System Version: 12.1-STABLE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: drum@graphica.com.au BUG: Starting FreeBSD Layer 2 bhyve Guest within Layer 1 FreeBSD VM Host on Laye= r 0 Ubuntu 20.04 KVM / QEMU Host result in Layer 1 Guest / Host Pausing with "Emulation Failure" TESTING: My test scenario is nested virtualisation: Layer 0 - Ubuntu 20.04 Host Layer 1 - FreeBSD 12.1 with OVMF + bhyve hypervisor Guest/Host Layer 2 - FreeBSD 12.1 guest Layer 0 Host is: Ubuntu 20.04 LTS KVM / QEMU / libvirt <> $ virsh -c qemu:///system version --daemon Compiled against library: libvirt 6.0.0 Using library: libvirt 6.0.0 Using API: QEMU 6.0.0 Running hypervisor: QEMU 4.2.0 Running against daemon: 6.0.0 < <> $ cat /proc/cpuinfo | grep -c vmx 64 $ cat /sys/module/kvm_intel/parameters/nested Y <> Layer 1 Guest / Host is: FreeBSD Q35 v4.2 with OVMF: Pass Host VMX support to Layer 1 Guest via <> ... ... hvm /usr/share/OVMF/OVMF_CODE.fd /home/USER/swarm.bhyve.freebsd/OVMF_VARS.fd ... ... > Checked that Layer 1 - FreeBSD Quest / Host has VMX feature available: <> # uname -a FreeBSD swarm.DOMAIN.HERE 12.1-RELEASE FreeBSD 12.1-RELEASE GENERIC amd64 # grep Features /var/run/dmesg.boot=20 =20 Features=3D0xf83fbff =20 Features2=3D0xfffa3223 AMD Features=3D0x2c100800 AMD Features2=3D0x121 Structured Extended Features=3D0x1c0fbb Structured Extended Features2=3D0x4 Structured Extended Features3=3D0xac000400 XSAVE Features=3D0x1 < On Layer 1 FreeBSD Guest / Host start up the Layer 2 guest.. <> # ls FreeBSD-11.2-RELEASE-amd64-bootonly.iso FreeBSD-12.1-RELEASE-amd64-dvd1.iso= =20=20=20=20 bee-hd1-01.img # /usr/sbin/bhyve -c 2 -m 2048 -H -A -s 0:0,hostbridge -s 1:0,lpc -s 2:0,e1000,tap0 -s 3:0,ahci-hd,bee-hd1-01.img -l com1,stdio -s 5:0,ahci-cd,./FreeBSD-12.1-RELEASE-amd64-dvd1.iso bee <> Result is that Layer 1 - FreeBSD Host guest "paused". To Layer 1 machines freezes I cannot get any further diagnostics from this machine, so I run tail on libvirt log from Layer 0 - Ubuntu Host <> char device redirected to /dev/pts/29 (label charserial0) 2020-05-04T06:09:15.310474Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.310531Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-04T06:09:15.312533Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.312548Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-04T06:09:15.313828Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.313841Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] 2020-05-04T06:09:15.315185Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12] 2020-05-04T06:09:15.315201Z qemu-system-x86_64: warning: host doesn't suppo= rt requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13] KVM internal error. Suberror: 1 emulation failure EAX=3D00000000 EBX=3D00000000 ECX=3D00000000 EDX=3D00000000 ESI=3D00000000 EDI=3D00000000 EBP=3D00000000 ESP=3D00000000 EIP=3D00000000 EFL=3D00000000 [-------] CPL=3D0 II=3D0 A20=3D1 SMM=3D0 HLT= =3D0 ES =3D0000 00000000 00000000 00008000 DPL=3D0 CS =3D0000 00000000 00000000 00008000 DPL=3D0 SS =3D0000 00000000 00000000 00008000 DPL=3D0 DS =3D0000 00000000 00000000 00008000 DPL=3D0 FS =3D0000 00000000 00000000 00008000 DPL=3D0 GS =3D0000 00000000 00000000 00008000 DPL=3D0 LDT=3D0000 00000000 00000000 00008000 DPL=3D0 TR =3D0000 00000000 00000000 00008000 DPL=3D0 GDT=3D 0000000000000000 00000000 IDT=3D 0000000000000000 00000000 CR0=3D80050033 CR2=3D0000000000000000 CR3=3D0000000000000000 CR4=3D00372060 DR0=3D0000000000000000 DR1=3D0000000000000000 DR2=3D0000000000000000 DR3=3D0000000000000000=20 DR6=3D00000000ffff0ff0 DR7=3D0000000000000400 EFER=3D0000000000000d01 Code=3D ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? = ?? 2020-05-04T06:35:39.186799Z qemu-system-x86_64: terminating on signal 15 fr= om pid 2155 (/usr/sbin/libvirtd) 2020-05-04 06:35:39.386+0000: shutting down, reason=3Ddestroyed <> I am reporting this bug here as result is very similar to that seen with QE= MU seabios failure reported here: https://bugs.launchpad.net/qemu/+bug/1866870 However in this case my VM Layer 1 VM is using OVMF. NOTE 1: I have also tested with Q35 v3.1 and 2.12 and get the same result. NOTE 2: Due to bug in FreeBSD networking code, I had to compile custom kern= el with "netmap driver disabled". This is known bug in FreeBSD that I have reported separately. --=20 You are receiving this mail because: You are the assignee for the bug.=