From nobody Tue Jun 28 14:43:07 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1229E87E000; Tue, 28 Jun 2022 14:43:39 +0000 (UTC) (envelope-from freebsd@aeichner.de) Received: from mail.a-eichner.de (mail.a-eichner.de [136.243.96.80]) by mx1.freebsd.org (Postfix) with ESMTP id 4LXS5Q1SZXz4ZP0; Tue, 28 Jun 2022 14:43:37 +0000 (UTC) (envelope-from freebsd@aeichner.de) Received: from [192.168.2.234] (ip4d17e279.dynamic.kabel-deutschland.de [77.23.226.121]) by mail.a-eichner.de (Postfix) with ESMTPSA id C1DA64C5343; Tue, 28 Jun 2022 16:43:08 +0200 (CEST) From: Alexander Eichner Message-Id: <8166EF9F-7374-4B05-8269-427A76909002@aeichner.de> Content-Type: multipart/alternative; boundary="Apple-Mail=_2457BFC7-3391-42E9-91F0-2E4D1A306A68" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: how to run bhyve and virtualbox at the same time Date: Tue, 28 Jun 2022 16:43:07 +0200 In-Reply-To: Cc: FreeBSD virtualization To: freebsd-hackers@freebsd.org References: <62B9FBF1.6030906@grosbein.net> <1D2BDC88-6739-45E0-A609-98D6FF271733@aeichner.de> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 4LXS5Q1SZXz4ZP0 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@aeichner.de has no SPF policy when checking 136.243.96.80) smtp.mailfrom=freebsd@aeichner.de X-Spamd-Result: default: False [-0.15 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.66)[-0.664]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[aeichner.de]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.991]; RCPT_COUNT_TWO(0.00)[2]; HTTP_TO_IP(1.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers,freebsd-virtualization]; R_SPF_NA(0.00)[no SPF record]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:24940, ipnet:136.243.0.0/16, country:DE]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[77.23.226.121:received] X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_2457BFC7-3391-42E9-91F0-2E4D1A306A68 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 The content of the VBox.log strongly smells like bhyve wants to use = VT-x/SVM exclusively, unfortunately nothing VirtualBox can do anything = about. vmm.ko/bhyve would need to implement a mode for sharing those = capabilities. And because I forgot to add this to my last mail. I had a look at the = vmm.ko API already and found that the guest memory management API looks = quite strange and is completely different from what kvm/AppleHV/Hyper-V etc. do. With = those you allocate page aligned memory using mmap()/VirtualAlloc() and = pass the GPA and the userspace virtual pointer to the hypervisor for the mapping. = With vmm.ko you allocate the memory through the hypervisor directly = using an ioctl providing the GPA and size and you get a user space mapping by a = sub-sequent mmap() call on the character device for the given offset. = And with libvmm you only have the option to allocate all the memory up-front. Regards, Alexander Eichner > On 28.06.2022 15:32, Mario Marietto wrote: >=20 > I've rebooted and then I have launched virtualbox first and when macos = has been loaded totally,I gave the following command : >=20 > sudo kldload vmm >=20 > and this is what happened : >=20 > > I've attached the log files produced.... >=20 > Il giorno mar 28 giu 2022 alle ore 14:48 Mario Marietto = > ha scritto: > Ok. As root I have disabled the hwvirtexclusive property giving this = command : >=20 > VBoxManage setproperty hwvirtexclusive off >=20 > as explained here : https://www.virtualbox.org/manual/ch09.html = >=20 > this is what happens when I launch bhyve first and then virtualbox.... >=20 >=20 > >=20 > More interesting is to see what happens when I launch virtualbox = first,executing this script : >=20 > setxkbmap it > bhyvectl --vm=3Dvm0 --force-reset > bhyvectl --vm=3Dvm0 --destroy > bhyvectl --vm=3Dvm1 --force-reset > bhyvectl --vm=3Dvm1 --destroy > bhyvectl --vm=3Dvm2 --force-reset > bhyvectl --vm=3Dvm2 --destroy > bhyvectl --vm=3Dvm3 --force-reset > bhyvectl --vm=3Dvm3 --destroy > bhyvectl --vm=3Dvm4 --force-reset > bhyvectl --vm=3Dvm4 --destroy > bhyvectl --vm=3Dvm5 --force-reset > bhyvectl --vm=3Dvm5 --destroy > bhyvectl --vm=3Dvm6 --force-reset > bhyvectl --vm=3Dvm6 --destroy > bhyvectl --vm=3Dvm7 --force-reset > bhyvectl --vm=3Dvm7 --destroy > bhyvectl --vm=3Dvm8 --force-reset > bhyvectl --vm=3Dvm8 --destroy > bhyvectl --vm=3Dvm9 --force-reset > bhyvectl --vm=3Dvm9 --destroy > bhyvectl --vm=3Dvm10 --force-reset > bhyvectl --vm=3Dvm10 --destroy > bhyvectl --vm=3Dvm11 --force-reset > bhyvectl --vm=3Dvm11 --destroy > bhyvectl --vm=3Dvm12 --force-reset > bhyvectl --vm=3Dvm12 --destroy > bhyvectl --vm=3Dvm13 --force-reset > bhyvectl --vm=3Dvm13 --destroy > bhyvectl --vm=3Dvm14 --force-reset > bhyvectl --vm=3Dvm14 --destroy > bhyvectl --vm=3Dvm15 --force-reset > bhyvectl --vm=3Dvm15 --destroy > bhyvectl --vm=3Dvm16 --force-reset > bhyvectl --vm=3Dvm16 --destroy > bhyvectl --vm=3Dvm17 --force-reset > bhyvectl --vm=3Dvm17 --destroy > bhyvectl --vm=3Dvm18 --force-reset > bhyvectl --vm=3Dvm18 --destroy > bhyvectl --vm=3Dvm19 --force-reset > bhyvectl --vm=3Dvm19 --destroy > bhyvectl --vm=3Dvm20 --force-reset > bhyvectl --vm=3Dvm20 --destroy > sudo kldunload vmm > virtualbox >=20 > At this point MacOS is able to boot and run correctly. At this point I = tried to launch a bhyve vm executing this script : >=20 > bhyve -S -c sockets=3D1,cores=3D2,threads=3D2 -m 4G -w -H -A \ > -s 0,hostbridge \ > -s 1,nvme,/dev/nvd0,bootindex=3D1 \ > -s 2,ahci-hd,/dev/$vmdisk2 \ > -s 3,ahci-hd,/dev/$vmdisk3 \ > -s 4,ahci-hd,/dev/$vmdisk4 \ > -s 7,virtio-net,tap4 \ > -s 10,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > -s 29,fbuf,tcp=3D0.0.0.0:5904 ,w=3D1500,h=3D950 = \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ > vm4 < /dev/null & sleep 2 && vncviewer 0:4 >=20 > my FreeBSD 13.1-RELEASE totally crashed. I've seen a lot of errors on = the screen,a lot of them related to efi and my PC rebooted. It seems = that the file bhyve.core has not produced,even if I read "creating = memory dumping"... >=20 > Il giorno mar 28 giu 2022 alle ore 12:03 Alexander Eichner = > ha scritto: > Hi, >=20 > VirtualBox 6.1.x uses its own hypervisor with kernel modules on macOS, = it just uses some internal kernel interfaces to be able to share > the hardware virtualization functionality with other hypervisors. >=20 > On hosts other than Windows and macOS VirtualBox defaults to an = exclusive use of VT-x/SVM for performance reasons. That mode can be = changed > to do the initialization every time VirtualBox is scheduled on a = particular CPU which has higher overhead but allows sharing the hardware = virtualization capabilities > with other hypervisors. The setting can be changed with VBoxManage. = Look at the manual under [1] and search for =E2=80=9Ehwvirtexclusive=E2=80= =9C. > However bhyve would need to support something similar to be able to = play along, I don=E2=80=99t know whether it does. >=20 > Regards, > Alexander Eichner >=20 > [1] https://www.virtualbox.org/manual/UserManual.html = >=20 > > On 28.06.2022 11:29, David Chisnall wrote: > >=20 > > On 27/06/2022 20:41, Mario Marietto wrote: > >> I did it already and it works. Fact is that if I use virtualbox I = can't use bhyve at the same time. So It's not a good idea to use only = virtualbox machines. > >=20 > > On macOS, VirtualBox can coexist with other hypervisors because it = does not provide a kernel module (it is not allowed to on recent macOS) = but is, instead, layered on top of Apple's Hypervisor framework. > >=20 > > bhyve has also been ported to sit on top of Apple's Hypervisor = framework (xhyve, used by Docker) and I believe that the abstractions = provided by the bhyve kernel module are fairly similar. > >=20 > > It should be possible to port VirtualBox to sit on top of vmm.ko. I = have not been able to find any documentation of the vmm(4) ioctls other = than the source code, so I'd imagine that about 2/3 of this work would = be documenting the kernel interfaces. > >=20 > > David > >=20 >=20 >=20 >=20 >=20 > --=20 > Mario. >=20 >=20 > --=20 > Mario. > --Apple-Mail=_2457BFC7-3391-42E9-91F0-2E4D1A306A68 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 The = content of the VBox.log strongly smells like bhyve wants to use VT-x/SVM = exclusively, unfortunately nothing VirtualBox can do anything about.
vmm.ko/bhyve would need to implement a mode for sharing those = capabilities.

And because I forgot to add this to my last mail. I had a = look at the vmm.ko API already and found that the guest memory = management API looks quite strange and
is = completely different from what kvm/AppleHV/Hyper-V etc. do. With those = you allocate page aligned memory using mmap()/VirtualAlloc() and pass = the GPA
and the userspace virtual pointer to the = hypervisor for the mapping. With vmm.ko you allocate the memory through = the hypervisor directly using an ioctl
providing = the GPA and size and you get a user space mapping by a sub-sequent = mmap() call on the character device for the given offset. And with = libvmm
you only have the option to allocate all the = memory up-front.

Regards,
Alexander Eichner

On 28.06.2022 15:32, Mario Marietto <marietto2008@gmail.com> wrote:

I've rebooted and = then I have launched virtualbox first and when macos has been loaded = totally,I gave the following command :

sudo kldload vmm
and this is what happened :

<Screenshot_2022-06-28_15-09-49.jpg><= br class=3D"">
I've attached the log files = produced....

Il giorno = mar 28 giu 2022 alle ore 14:48 Mario Marietto <marietto2008@gmail.com> ha scritto:
Ok. As root I have disabled the = hwvirtexclusive property giving this command :

VBoxManage setproperty =
hwvirtexclusive off

as explained here : https://www.virtualbox.org/manual/ch09.html

this is = what happens when I launch bhyve first and then virtualbox....


<Screenshot_2022-06-28_14-12-36.jpg><= br class=3D"">

More interesting is to see what happens when I launch = virtualbox first,executing this script :

setxkbmap it
bhyvectl = --vm=3Dvm0 --force-reset
bhyvectl --vm=3Dvm0 --destroy
bhyvectl --vm=3Dvm1 --force-reset
bhyvectl = --vm=3Dvm1 --destroy
bhyvectl --vm=3Dvm2 --force-reset
bhyvectl --vm=3Dvm2 --destroy
bhyvectl --vm=3Dvm3= --force-reset
bhyvectl --vm=3Dvm3 --destroy
bhyvectl --vm=3Dvm4 --force-reset
bhyvectl = --vm=3Dvm4 --destroy
bhyvectl --vm=3Dvm5 --force-reset
bhyvectl --vm=3Dvm5 --destroy
bhyvectl --vm=3Dvm6= --force-reset
bhyvectl --vm=3Dvm6 --destroy
bhyvectl --vm=3Dvm7 --force-reset
bhyvectl = --vm=3Dvm7 --destroy
bhyvectl --vm=3Dvm8 --force-reset
bhyvectl --vm=3Dvm8 --destroy
bhyvectl --vm=3Dvm9= --force-reset
bhyvectl --vm=3Dvm9 --destroy
bhyvectl --vm=3Dvm10 --force-reset
bhyvectl = --vm=3Dvm10 --destroy
bhyvectl --vm=3Dvm11 = --force-reset
bhyvectl --vm=3Dvm11 --destroy
bhyvectl --vm=3Dvm12 --force-reset
bhyvectl = --vm=3Dvm12 --destroy
bhyvectl --vm=3Dvm13 = --force-reset
bhyvectl --vm=3Dvm13 --destroy
bhyvectl --vm=3Dvm14 --force-reset
bhyvectl = --vm=3Dvm14 --destroy
bhyvectl --vm=3Dvm15 = --force-reset
bhyvectl --vm=3Dvm15 --destroy
bhyvectl --vm=3Dvm16 --force-reset
bhyvectl = --vm=3Dvm16 --destroy
bhyvectl --vm=3Dvm17 = --force-reset
bhyvectl --vm=3Dvm17 --destroy
bhyvectl --vm=3Dvm18 --force-reset
bhyvectl = --vm=3Dvm18 --destroy
bhyvectl --vm=3Dvm19 = --force-reset
bhyvectl --vm=3Dvm19 --destroy
bhyvectl --vm=3Dvm20 --force-reset
bhyvectl = --vm=3Dvm20 --destroy
sudo kldunload vmm
virtualbox

At this point MacOS is able to boot and run correctly. At = this point I tried to launch a bhyve vm executing this script = :

bhyve -S -c = sockets=3D1,cores=3D2,threads=3D2 -m 4G -w -H -A \
-s = 0,hostbridge \
-s 1,nvme,/dev/nvd0,bootindex=3D1 \
-s 2,ahci-hd,/dev/$vmdisk2 \
-s = 3,ahci-hd,/dev/$vmdisk3 \
-s 4,ahci-hd,/dev/$vmdisk4 \
-s 7,virtio-net,tap4 \
-s = 10,hda,play=3D/dev/dsp,rec=3D/dev/dsp \
-s 29,fbuf,tcp=3D0.0.0.0:5904,w=3D1500,h=3D950 \
-s = 30,xhci,tablet \
-s 31,lpc \
-l = bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm4 < /dev/null & sleep 2 && vncviewer 0:4

my = FreeBSD 13.1-RELEASE totally crashed. I've seen a lot of errors on the = screen,a lot of them related to efi and my PC rebooted. It seems that = the file bhyve.core has not produced,even if I read "creating memory = dumping"...

Il giorno = mar 28 giu 2022 alle ore 12:03 Alexander Eichner <freebsd@aeichner.de> ha scritto:
Hi,

VirtualBox 6.1.x uses its own hypervisor with kernel modules on macOS, = it just uses some internal kernel interfaces to be able to share
the hardware virtualization functionality with other hypervisors.

On hosts other than Windows and macOS VirtualBox defaults to an = exclusive use of VT-x/SVM for performance reasons. That mode can be = changed
to do the initialization every time VirtualBox is scheduled on a = particular CPU which has higher overhead but allows sharing the hardware = virtualization capabilities
with other hypervisors. The setting can be changed with VBoxManage. Look = at the manual under [1] and search for =E2=80=9Ehwvirtexclusive=E2=80=9C.<= br class=3D""> However bhyve would need to support something similar to be able to play = along, I don=E2=80=99t know whether it does.

Regards,
Alexander Eichner

[1] https://www.virtualbox.org/manual/UserManual.html

> On 28.06.2022 11:29, David Chisnall <theraven@FreeBSD.org> wrote:
>
> On 27/06/2022 20:41, Mario Marietto wrote:
>> I did it already and it works. Fact is that if I use virtualbox = I can't use bhyve at the same time. So It's not a good idea to use only = virtualbox machines.
>
> On macOS, VirtualBox can coexist with other hypervisors because it = does not provide a kernel module (it is not allowed to on recent macOS) = but is, instead, layered on top of Apple's Hypervisor framework.
>
> bhyve has also been ported to sit on top of Apple's Hypervisor = framework (xhyve, used by Docker) and I believe that the abstractions = provided by the bhyve kernel module are fairly similar.
>
> It should be possible to port VirtualBox to sit on top of = vmm.ko.  I have not been able to find any documentation of the = vmm(4) ioctls other than the source code, so I'd imagine that about 2/3 = of this work would be documenting the kernel interfaces.
>
> David
>




--
Mario.


--
Mario.
<logs.zip>
= --Apple-Mail=_2457BFC7-3391-42E9-91F0-2E4D1A306A68--