From owner-freebsd-virtualization@freebsd.org Sat May 9 13:55:03 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 585A02EA684 for ; Sat, 9 May 2020 13:55:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 49K7xM1g7bz3Mrq for ; Sat, 9 May 2020 13:55:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 3720D2EA683; Sat, 9 May 2020 13:55:03 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 36E4E2EA682 for ; Sat, 9 May 2020 13:55:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49K7xM0hHYz3Mrn for ; Sat, 9 May 2020 13:55:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id ED1D41CF1B for ; Sat, 9 May 2020 13:55:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 049Dt23u010620 for ; Sat, 9 May 2020 13:55:02 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 049Dt21Q010619 for virtualization@FreeBSD.org; Sat, 9 May 2020 13:55:02 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 246168] Ubuntu 20.04 KVM / QEMU Failure with nested FreeBSD bhyve Date: Sat, 09 May 2020 13:55:02 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 12.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: drum@graphica.com.au X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.32 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 May 2020 13:55:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246168 --- Comment #11 from John Hartley --- (In reply to Peter Grehan from comment #10) Hi Peter, you have eagle eyes. Your suggestion worked and I now have Layer 3 - FreeBSD UEFI guest doing an install. I also tried to see if disabling X2APIC via QEMU fixed the problem via: This did not work, but it did turn off the feature correct in Layer 1 - Gue= st / Host. Here is output from sysctl on Layer 1 - FreeBSD Guest Host <> # sysctl hw.vmm hw.vmm.amdvi.domain_id: 0 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 0 hw.vmm.amdvi.count: 0 hw.vmm.npt.pmap_flags: 0 hw.vmm.svm.num_asids: 0 hw.vmm.svm.disable_npf_assist: 0 hw.vmm.svm.features: 4294967295 hw.vmm.svm.vmcb_clean: 1023 hw.vmm.vmx.l1d_flush_sw: 0 hw.vmm.vmx.l1d_flush: 0 hw.vmm.vmx.vpid_alloc_failed: 0 hw.vmm.vmx.posted_interrupt_vector: -1 hw.vmm.vmx.cap.posted_interrupts: 0 hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 hw.vmm.vmx.cap.invpcid: 1 hw.vmm.vmx.cap.monitor_trap: 1 hw.vmm.vmx.cap.unrestricted_guest: 1 hw.vmm.vmx.cap.pause_exit: 1 hw.vmm.vmx.cap.halt_exit: 1 hw.vmm.vmx.initialized: 1 hw.vmm.vmx.cr4_zeros_mask: 18446744073705934848 hw.vmm.vmx.cr4_ones_mask: 8192 hw.vmm.vmx.cr0_zeros_mask: 18446744071025197056 hw.vmm.vmx.cr0_ones_mask: 32 hw.vmm.vmx.no_flush_rsb: 0 hw.vmm.ept.pmap_flags: 1531 hw.vmm.vrtc.flag_broken_time: 1 hw.vmm.ppt.devices: 0 hw.vmm.iommu.enable: 1 hw.vmm.iommu.initialized: 0 hw.vmm.bhyve_xcpuids: 0 hw.vmm.topology.cpuid_leaf_b: 1 hw.vmm.create: beavis hw.vmm.destroy: beavis hw.vmm.trace_guest_exceptions: 0 hw.vmm.ipinum: 251 hw.vmm.halt_detection: 1 <> So have work around. Do think believe this needs fix within FreeBSD bhyve or is issue with QEMU / KVM ? NOTE: I also tested with FreeBSD 13 CURRENT and this had same problem. Thanks again for help on this. Cheers, John Hartley --=20 You are receiving this mail because: You are the assignee for the bug.=