From owner-freebsd-virtualization@freebsd.org Thu Sep 10 00:36:18 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 94F2C3E1C37 for ; Thu, 10 Sep 2020 00:36:18 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito-vars.onthenet.com.au (vito-vars.OntheNet.com.au [203.13.68.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bn0LV0Tgbz47Wm for ; Thu, 10 Sep 2020 00:36:17 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito-vars.onthenet.com.au (Postfix) with ESMTPS id A756B2083923 for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 9033420AE983 for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 8963B20F4D29 for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id jqRq-8q7EzIB for ; Thu, 10 Sep 2020 10:36:07 +1000 (AEST) Received: from MacBook-Air-4.local (CPE-120-29-42-110.dsl.OntheNet.net [120.29.42.110]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 30D3720F4D27; Thu, 10 Sep 2020 10:36:06 +1000 (AEST) Subject: Re: Nested Virtualization Under KVM Hypervisor To: Pete Wright References: Cc: FreeBSD virtualization From: Peter Grehan Message-ID: <5eacd0a5-89bd-cc98-6c5a-d5490e33b100@freebsd.org> Date: Thu, 10 Sep 2020 10:36:06 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=bM+N3M2Z c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=jXL/WSj5nX36MkVNSbrI6Q==:17 a=IkcTkHD0fZMA:10 a=reM5J-MqmosA:10 a=6I5d2MoRAAAA:8 a=y9LJmbqIxtuG6_hiBngA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=LtAJNkVc c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=jXL/WSj5nX36MkVNSbrI6Q==:17 a=IkcTkHD0fZMA:10 a=reM5J-MqmosA:10 a=6I5d2MoRAAAA:8 a=y9LJmbqIxtuG6_hiBngA:9 a=QEXdDO2ut3YA:10 a=IjZwj45LgO3ly-622nXo:22 X-Rspamd-Queue-Id: 4Bn0LV0Tgbz47Wm X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:9313, ipnet:203.13.68.0/23, country:AU]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Sep 2020 00:36:18 -0000 Hi Pete, > =C2=A0 VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,Post= Intr That's just a warning: bhyve will attempt to enable VT-x on module=20 load, and will only fail if BIOS has locked the disable. Do you know if the vmm module fails to load ? (there will be an error=20 in syslog "vmx_init: VMX operation disabled by BIOS") > Hypervisor: Origin =3D "KVMKVMKVM" > real memory=C2=A0 =3D 34359738368 (32768 MB) > avail memory =3D 33365536768 (31819 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: >=20 > They did verify that VT-x pass through *is* enabled as mentioned above=20 > (they shared with me the config file snippet on their end to verify),=20 > and they did extensive testing on their end and were not able to get=20 > FreeBSD to detect VT-x as being enabled. >=20 > I pinged them asking for more info on which Linux kernel their=20 > hypervisors are running to see if I can help debug this on my end.=C2=A0= I=20 > have two questions: >=20 > 1) is anyone else doing nested virtualization under KVM hypervisors.=C2= =A0 i=20 > am confident that this is possible, but am asking for confirmation on=20 > this :) Yes: A bug report=20 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D203994 ... had the resolution that a kernel > 4.10 has to be used, and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D246168 ... requires posted interrupts to be disabled. > 2) Are there any other requirements that FreeBSD requires to run Bhyve=20 > nested under KVM? No - nested has been enabled by default with KVM for quite a while now. later, Peter.