Date: Tue, 28 Jun 2022 16:43:07 +0200 From: Alexander Eichner <freebsd@aeichner.de> To: freebsd-hackers@freebsd.org Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: how to run bhyve and virtualbox at the same time Message-ID: <8166EF9F-7374-4B05-8269-427A76909002@aeichner.de> In-Reply-To: <CA%2B1FSijU7xMXroVo=utBKzuJcNT2eQP-LUB82BesBtmJ0O7wfQ@mail.gmail.com> References: <CA%2B1FSig5qQRcQMm8uJ0A%2BNO1485MXsh_fEjMtU55iF_Qk0jn8w@mail.gmail.com> <62B9FBF1.6030906@grosbein.net> <CA%2B1FSihK9Q4cmthTyGb_TQLN_LOC%2B2GAiFoOOukQHGQ0Scuzsw@mail.gmail.com> <d76307e6-1e14-f58c-2d70-115b2d7cd5b2@FreeBSD.org> <1D2BDC88-6739-45E0-A609-98D6FF271733@aeichner.de> <CA%2B1FSijbjXTRn8UtiqknA%2BVwW=AXofnw-4QTwpVbJyd85zhmWw@mail.gmail.com> <CA%2B1FSijU7xMXroVo=utBKzuJcNT2eQP-LUB82BesBtmJ0O7wfQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_2457BFC7-3391-42E9-91F0-2E4D1A306A68 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 The content of the VBox.log strongly smells like bhyve wants to use = VT-x/SVM exclusively, unfortunately nothing VirtualBox can do anything = about. vmm.ko/bhyve would need to implement a mode for sharing those = capabilities. And because I forgot to add this to my last mail. I had a look at the = vmm.ko API already and found that the guest memory management API looks = quite strange and is completely different from what kvm/AppleHV/Hyper-V etc. do. With = those you allocate page aligned memory using mmap()/VirtualAlloc() and = pass the GPA and the userspace virtual pointer to the hypervisor for the mapping. = With vmm.ko you allocate the memory through the hypervisor directly = using an ioctl providing the GPA and size and you get a user space mapping by a = sub-sequent mmap() call on the character device for the given offset. = And with libvmm you only have the option to allocate all the memory up-front. Regards, Alexander Eichner > On 28.06.2022 15:32, Mario Marietto <marietto2008@gmail.com> wrote: >=20 > I've rebooted and then I have launched virtualbox first and when macos = has been loaded totally,I gave the following command : >=20 > sudo kldload vmm >=20 > and this is what happened : >=20 > <Screenshot_2022-06-28_15-09-49.jpg> > I've attached the log files produced.... >=20 > Il giorno mar 28 giu 2022 alle ore 14:48 Mario Marietto = <marietto2008@gmail.com <mailto:marietto2008@gmail.com>> ha scritto: > Ok. As root I have disabled the hwvirtexclusive property giving this = command : >=20 > VBoxManage setproperty hwvirtexclusive off >=20 > as explained here : https://www.virtualbox.org/manual/ch09.html = <https://www.virtualbox.org/manual/ch09.html>; >=20 > this is what happens when I launch bhyve first and then virtualbox.... >=20 >=20 > <Screenshot_2022-06-28_14-12-36.jpg> >=20 > More interesting is to see what happens when I launch virtualbox = first,executing this script : >=20 > setxkbmap it > bhyvectl --vm=3Dvm0 --force-reset > bhyvectl --vm=3Dvm0 --destroy > bhyvectl --vm=3Dvm1 --force-reset > bhyvectl --vm=3Dvm1 --destroy > bhyvectl --vm=3Dvm2 --force-reset > bhyvectl --vm=3Dvm2 --destroy > bhyvectl --vm=3Dvm3 --force-reset > bhyvectl --vm=3Dvm3 --destroy > bhyvectl --vm=3Dvm4 --force-reset > bhyvectl --vm=3Dvm4 --destroy > bhyvectl --vm=3Dvm5 --force-reset > bhyvectl --vm=3Dvm5 --destroy > bhyvectl --vm=3Dvm6 --force-reset > bhyvectl --vm=3Dvm6 --destroy > bhyvectl --vm=3Dvm7 --force-reset > bhyvectl --vm=3Dvm7 --destroy > bhyvectl --vm=3Dvm8 --force-reset > bhyvectl --vm=3Dvm8 --destroy > bhyvectl --vm=3Dvm9 --force-reset > bhyvectl --vm=3Dvm9 --destroy > bhyvectl --vm=3Dvm10 --force-reset > bhyvectl --vm=3Dvm10 --destroy > bhyvectl --vm=3Dvm11 --force-reset > bhyvectl --vm=3Dvm11 --destroy > bhyvectl --vm=3Dvm12 --force-reset > bhyvectl --vm=3Dvm12 --destroy > bhyvectl --vm=3Dvm13 --force-reset > bhyvectl --vm=3Dvm13 --destroy > bhyvectl --vm=3Dvm14 --force-reset > bhyvectl --vm=3Dvm14 --destroy > bhyvectl --vm=3Dvm15 --force-reset > bhyvectl --vm=3Dvm15 --destroy > bhyvectl --vm=3Dvm16 --force-reset > bhyvectl --vm=3Dvm16 --destroy > bhyvectl --vm=3Dvm17 --force-reset > bhyvectl --vm=3Dvm17 --destroy > bhyvectl --vm=3Dvm18 --force-reset > bhyvectl --vm=3Dvm18 --destroy > bhyvectl --vm=3Dvm19 --force-reset > bhyvectl --vm=3Dvm19 --destroy > bhyvectl --vm=3Dvm20 --force-reset > bhyvectl --vm=3Dvm20 --destroy > sudo kldunload vmm > virtualbox >=20 > At this point MacOS is able to boot and run correctly. At this point I = tried to launch a bhyve vm executing this script : >=20 > bhyve -S -c sockets=3D1,cores=3D2,threads=3D2 -m 4G -w -H -A \ > -s 0,hostbridge \ > -s 1,nvme,/dev/nvd0,bootindex=3D1 \ > -s 2,ahci-hd,/dev/$vmdisk2 \ > -s 3,ahci-hd,/dev/$vmdisk3 \ > -s 4,ahci-hd,/dev/$vmdisk4 \ > -s 7,virtio-net,tap4 \ > -s 10,hda,play=3D/dev/dsp,rec=3D/dev/dsp \ > -s 29,fbuf,tcp=3D0.0.0.0:5904 <http://0.0.0.0:5904/>,w=3D1500,h=3D950 = \ > -s 30,xhci,tablet \ > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \ > vm4 < /dev/null & sleep 2 && vncviewer 0:4 >=20 > my FreeBSD 13.1-RELEASE totally crashed. I've seen a lot of errors on = the screen,a lot of them related to efi and my PC rebooted. It seems = that the file bhyve.core has not produced,even if I read "creating = memory dumping"... >=20 > Il giorno mar 28 giu 2022 alle ore 12:03 Alexander Eichner = <freebsd@aeichner.de <mailto:freebsd@aeichner.de>> ha scritto: > Hi, >=20 > VirtualBox 6.1.x uses its own hypervisor with kernel modules on macOS, = it just uses some internal kernel interfaces to be able to share > the hardware virtualization functionality with other hypervisors. >=20 > On hosts other than Windows and macOS VirtualBox defaults to an = exclusive use of VT-x/SVM for performance reasons. That mode can be = changed > to do the initialization every time VirtualBox is scheduled on a = particular CPU which has higher overhead but allows sharing the hardware = virtualization capabilities > with other hypervisors. The setting can be changed with VBoxManage. = Look at the manual under [1] and search for =E2=80=9Ehwvirtexclusive=E2=80= =9C. > However bhyve would need to support something similar to be able to = play along, I don=E2=80=99t know whether it does. >=20 > Regards, > Alexander Eichner >=20 > [1] https://www.virtualbox.org/manual/UserManual.html = <https://www.virtualbox.org/manual/UserManual.html>; >=20 > > On 28.06.2022 11:29, David Chisnall <theraven@FreeBSD.org> wrote: > >=20 > > On 27/06/2022 20:41, Mario Marietto wrote: > >> I did it already and it works. Fact is that if I use virtualbox I = can't use bhyve at the same time. So It's not a good idea to use only = virtualbox machines. > >=20 > > On macOS, VirtualBox can coexist with other hypervisors because it = does not provide a kernel module (it is not allowed to on recent macOS) = but is, instead, layered on top of Apple's Hypervisor framework. > >=20 > > bhyve has also been ported to sit on top of Apple's Hypervisor = framework (xhyve, used by Docker) and I believe that the abstractions = provided by the bhyve kernel module are fairly similar. > >=20 > > It should be possible to port VirtualBox to sit on top of vmm.ko. I = have not been able to find any documentation of the vmm(4) ioctls other = than the source code, so I'd imagine that about 2/3 of this work would = be documenting the kernel interfaces. > >=20 > > David > >=20 >=20 >=20 >=20 >=20 > --=20 > Mario. >=20 >=20 > --=20 > Mario. > <logs.zip> --Apple-Mail=_2457BFC7-3391-42E9-91F0-2E4D1A306A68 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D"">The = content of the VBox.log strongly smells like bhyve wants to use VT-x/SVM = exclusively, unfortunately nothing VirtualBox can do anything about.<div = class=3D"">vmm.ko/bhyve would need to implement a mode for sharing those = capabilities.</div><div class=3D""><br class=3D""></div><div = class=3D"">And because I forgot to add this to my last mail. I had a = look at the vmm.ko API already and found that the guest memory = management API looks quite strange and</div><div class=3D"">is = completely different from what kvm/AppleHV/Hyper-V etc. do. With those = you allocate page aligned memory using mmap()/VirtualAlloc() and pass = the GPA</div><div class=3D"">and the userspace virtual pointer to the = hypervisor for the mapping. With vmm.ko you allocate the memory through = the hypervisor directly using an ioctl</div><div class=3D"">providing = the GPA and size and you get a user space mapping by a sub-sequent = mmap() call on the character device for the given offset. And with = libvmm</div><div class=3D"">you only have the option to allocate all the = memory up-front.</div><div class=3D""><br class=3D""></div><div = class=3D"">Regards,</div><div class=3D"">Alexander Eichner<br = class=3D""><div><br class=3D""><blockquote type=3D"cite" class=3D""><div = class=3D"">On 28.06.2022 15:32, Mario Marietto <<a = href=3D"mailto:marietto2008@gmail.com" = class=3D"">marietto2008@gmail.com</a>> wrote:</div><br = class=3D"Apple-interchange-newline"><div class=3D""><div dir=3D"ltr" = class=3D""><div dir=3D"ltr" class=3D""><div class=3D"">I've rebooted and = then I have launched virtualbox first and when macos has been loaded = totally,I gave the following command :</div><div class=3D""><br = class=3D""></div><div class=3D"">sudo kldload vmm</div><div class=3D""><br= class=3D""></div><div class=3D"">and this is what happened :</div><div = class=3D""><br class=3D""></div><div class=3D""><span = id=3D"cid:ii_l4y74e8l1"><Screenshot_2022-06-28_15-09-49.jpg></span><= br class=3D""></div><div class=3D"">I've attached the log files = produced....<br class=3D""></div></div><br class=3D""><div = class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">Il giorno = mar 28 giu 2022 alle ore 14:48 Mario Marietto <<a = href=3D"mailto:marietto2008@gmail.com" target=3D"_blank" = class=3D"">marietto2008@gmail.com</a>> ha scritto:<br = class=3D""></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr" class=3D""><div = dir=3D"ltr" class=3D""><div class=3D"">Ok. As root I have disabled the = hwvirtexclusive property giving this command :</div><div class=3D""><br = class=3D""></div><div class=3D""><pre class=3D"">VBoxManage setproperty = hwvirtexclusive off<br class=3D""><br class=3D""></pre></div><div = class=3D"">as explained here : <a = href=3D"https://www.virtualbox.org/manual/ch09.html" target=3D"_blank" = class=3D"">https://www.virtualbox.org/manual/ch09.html</a></div><div = class=3D""><br class=3D""></div><div class=3D""><div class=3D"">this is = what happens when I launch bhyve first and then virtualbox....</div><div = class=3D""><br class=3D""></div></div><div class=3D""><br = class=3D""></div><div class=3D""><span = id=3D"cid:ii_l4y4vb7k0"><Screenshot_2022-06-28_14-12-36.jpg></span><= br class=3D""></div></div><div class=3D""><br class=3D""></div><div = class=3D"">More interesting is to see what happens when I launch = virtualbox first,executing this script :</div><div class=3D""><br = class=3D""></div><div class=3D"">setxkbmap it<br class=3D"">bhyvectl = --vm=3Dvm0 --force-reset<br class=3D"">bhyvectl --vm=3Dvm0 --destroy<br = class=3D"">bhyvectl --vm=3Dvm1 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm1 --destroy<br class=3D"">bhyvectl --vm=3Dvm2 --force-reset<br = class=3D"">bhyvectl --vm=3Dvm2 --destroy<br class=3D"">bhyvectl --vm=3Dvm3= --force-reset<br class=3D"">bhyvectl --vm=3Dvm3 --destroy<br = class=3D"">bhyvectl --vm=3Dvm4 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm4 --destroy<br class=3D"">bhyvectl --vm=3Dvm5 --force-reset<br = class=3D"">bhyvectl --vm=3Dvm5 --destroy<br class=3D"">bhyvectl --vm=3Dvm6= --force-reset<br class=3D"">bhyvectl --vm=3Dvm6 --destroy<br = class=3D"">bhyvectl --vm=3Dvm7 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm7 --destroy<br class=3D"">bhyvectl --vm=3Dvm8 --force-reset<br = class=3D"">bhyvectl --vm=3Dvm8 --destroy<br class=3D"">bhyvectl --vm=3Dvm9= --force-reset<br class=3D"">bhyvectl --vm=3Dvm9 --destroy<br = class=3D"">bhyvectl --vm=3Dvm10 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm10 --destroy<br class=3D"">bhyvectl --vm=3Dvm11 = --force-reset<br class=3D"">bhyvectl --vm=3Dvm11 --destroy<br = class=3D"">bhyvectl --vm=3Dvm12 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm12 --destroy<br class=3D"">bhyvectl --vm=3Dvm13 = --force-reset<br class=3D"">bhyvectl --vm=3Dvm13 --destroy<br = class=3D"">bhyvectl --vm=3Dvm14 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm14 --destroy<br class=3D"">bhyvectl --vm=3Dvm15 = --force-reset<br class=3D"">bhyvectl --vm=3Dvm15 --destroy<br = class=3D"">bhyvectl --vm=3Dvm16 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm16 --destroy<br class=3D"">bhyvectl --vm=3Dvm17 = --force-reset<br class=3D"">bhyvectl --vm=3Dvm17 --destroy<br = class=3D"">bhyvectl --vm=3Dvm18 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm18 --destroy<br class=3D"">bhyvectl --vm=3Dvm19 = --force-reset<br class=3D"">bhyvectl --vm=3Dvm19 --destroy<br = class=3D"">bhyvectl --vm=3Dvm20 --force-reset<br class=3D"">bhyvectl = --vm=3Dvm20 --destroy<br class=3D"">sudo kldunload vmm<br = class=3D"">virtualbox</div><div class=3D""> <br class=3D""></div><div = class=3D"">At this point MacOS is able to boot and run correctly. At = this point I tried to launch a bhyve vm executing this script = :</div><div class=3D""><br class=3D""></div><div class=3D"">bhyve -S -c = sockets=3D1,cores=3D2,threads=3D2 -m 4G -w -H -A \<br class=3D"">-s = 0,hostbridge \<br class=3D"">-s 1,nvme,/dev/nvd0,bootindex=3D1 \<br = class=3D"">-s 2,ahci-hd,/dev/$vmdisk2 \<br class=3D"">-s = 3,ahci-hd,/dev/$vmdisk3 \<br class=3D"">-s 4,ahci-hd,/dev/$vmdisk4 \<br = class=3D"">-s 7,virtio-net,tap4 \<br class=3D"">-s = 10,hda,play=3D/dev/dsp,rec=3D/dev/dsp \<br class=3D"">-s 29,fbuf,tcp=3D<a = href=3D"http://0.0.0.0:5904/" target=3D"_blank" = class=3D"">0.0.0.0:5904</a>,w=3D1500,h=3D950 \<br class=3D"">-s = 30,xhci,tablet \<br class=3D"">-s 31,lpc \<br class=3D"">-l = bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \<br = class=3D"">vm4 < /dev/null & sleep 2 && vncviewer 0:4<br = class=3D""></div><div class=3D""><br class=3D""></div><div class=3D"">my = FreeBSD 13.1-RELEASE totally crashed. I've seen a lot of errors on the = screen,a lot of them related to efi and my PC rebooted. It seems that = the file bhyve.core has not produced,even if I read "creating memory = dumping"...<br class=3D""></div><div class=3D""><br class=3D""></div><div = class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">Il giorno = mar 28 giu 2022 alle ore 12:03 Alexander Eichner <<a = href=3D"mailto:freebsd@aeichner.de" target=3D"_blank" = class=3D"">freebsd@aeichner.de</a>> ha scritto:<br = class=3D""></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex">Hi,<br class=3D""> <br class=3D""> VirtualBox 6.1.x uses its own hypervisor with kernel modules on macOS, = it just uses some internal kernel interfaces to be able to share<br = class=3D""> the hardware virtualization functionality with other hypervisors.<br = class=3D""> <br class=3D""> On hosts other than Windows and macOS VirtualBox defaults to an = exclusive use of VT-x/SVM for performance reasons. That mode can be = changed<br class=3D""> to do the initialization every time VirtualBox is scheduled on a = particular CPU which has higher overhead but allows sharing the hardware = virtualization capabilities<br class=3D""> with other hypervisors. The setting can be changed with VBoxManage. Look = at the manual under [1] and search for =E2=80=9Ehwvirtexclusive=E2=80=9C.<= br class=3D""> However bhyve would need to support something similar to be able to play = along, I don=E2=80=99t know whether it does.<br class=3D""> <br class=3D""> Regards,<br class=3D""> Alexander Eichner<br class=3D""> <br class=3D""> [1] <a href=3D"https://www.virtualbox.org/manual/UserManual.html" = rel=3D"noreferrer" target=3D"_blank" = class=3D"">https://www.virtualbox.org/manual/UserManual.html</a><br = class=3D""> <br class=3D""> > On 28.06.2022 11:29, David Chisnall <<a = href=3D"mailto:theraven@FreeBSD.org" = class=3D"">theraven@FreeBSD.org</a>> wrote:<br class=3D""> > <br class=3D""> > On 27/06/2022 20:41, Mario Marietto wrote:<br class=3D""> >> I did it already and it works. Fact is that if I use virtualbox = I can't use bhyve at the same time. So It's not a good idea to use only = virtualbox machines.<br class=3D""> > <br class=3D""> > On macOS, VirtualBox can coexist with other hypervisors because it = does not provide a kernel module (it is not allowed to on recent macOS) = but is, instead, layered on top of Apple's Hypervisor framework.<br = class=3D""> > <br class=3D""> > bhyve has also been ported to sit on top of Apple's Hypervisor = framework (xhyve, used by Docker) and I believe that the abstractions = provided by the bhyve kernel module are fairly similar.<br class=3D""> > <br class=3D""> > It should be possible to port VirtualBox to sit on top of = vmm.ko. I have not been able to find any documentation of the = vmm(4) ioctls other than the source code, so I'd imagine that about 2/3 = of this work would be documenting the kernel interfaces.<br class=3D""> > <br class=3D""> > David<br class=3D""> > <br class=3D""> <br class=3D""> <br class=3D""> </blockquote></div><br clear=3D"all" class=3D""><br class=3D"">-- <br = class=3D""><div dir=3D"ltr" class=3D"">Mario.<br class=3D""></div> </div> </blockquote></div><br clear=3D"all" class=3D""><br class=3D"">-- <br = class=3D""><div dir=3D"ltr" class=3D"">Mario.<br class=3D""></div></div> <span = id=3D"cid:f_l4y7k6xl2"><logs.zip></span></div></blockquote></div><br= class=3D""></div></body></html>= --Apple-Mail=_2457BFC7-3391-42E9-91F0-2E4D1A306A68--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?8166EF9F-7374-4B05-8269-427A76909002>