Date: Tue, 8 Feb 2022 18:45:32 +0100 From: Mario Marietto <marietto2008@gmail.com> To: Aryeh Friedman <aryeh.friedman@gmail.com> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: Multiple bugs on bhyve present on FreeBSD 13R p7 while passing thru USB controller and graphic card Message-ID: <CA%2B1FSigCU26zAxQKrayjLuUch__X4=TKeTy0Jm=eV0NtG3q=dQ@mail.gmail.com> In-Reply-To: <CAGBxaXnak5sOpiceofttORyfThXSnby7aw9e5BpODPnbgJ_uNw@mail.gmail.com> References: <CA%2B1FSigMNsuGGAqi7hQZTuAFzMuByKtcFs554SVs6oXrfUZs-g@mail.gmail.com> <CAGBxaXnak5sOpiceofttORyfThXSnby7aw9e5BpODPnbgJ_uNw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] In my system I have two graphic cards. This is the first one,configured as default from the bios and used on the freebsd host : vgapci0@pci0:0:2:0: class=0x030000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x3e98 subvendor=0x1458 subdevice=0xd000 vendor = 'Intel Corporation' device = 'CoffeeLake-S GT2 [UHD Graphics 630]' class = display subclass = VGA the USB controller that I'm trying to pass thru is the following : ppt0@pci0:1:0:0: class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014 subvendor=0x1912 subdevice=0x0015 vendor = 'Renesas Technology Corp.' device = 'uPD720201 USB 3.0 Host Controller' class = serial bus subclass = USB the graphic card that I'm trying to pass thru is the following : ppt1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 [GeForce RTX 2080 Ti]' class = display subclass = VGA ppt2@pci0:2:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 High Definition Audio Controller' class = multimedia subclass = HDA ppt3@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 USB 3.1 Host Controller' class = serial bus subclass = USB ppt4@pci0:2:0:3: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 USB Type-C UCSI Controller' class = serial bus On the file /boot/loader.conf I have reserved the following PCI addresses : pptdevs="1/0/0 2/0/0 2/0/1 2/0/2 2/0/3" If I try to boot a virtual machine in raw format passing thru my usb controller and my graphic card I get a different error : this bhyve commands : bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H \ -s 0,hostbridge \ -s 1,virtio-blk,/mnt/da2p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \ -s 3:0,passthru,2/0/0 \ -s 3:1,passthru,2/0/1 \ -s 3:2,passthru,2/0/2 \ -s 3:3,passthru,2/0/3 \ -s 4:0,passthru,1/0/0 \ -s 9,virtio-net,tap0 \ -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -l com1,stdio \ vm0 produces this error : fbuf frame buffer base: 0xa43800000 [sz 16777216] Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501. Abort. Il giorno mar 8 feb 2022 alle ore 18:17 Aryeh Friedman < aryeh.friedman@gmail.com> ha scritto: > > I'm using bhyve on FreeBSD 13R p7 and I'm trying to pass thru my usb >> controller and / or my graphic card while at the same time I'm trying to >> boot 3 operating systems (Ubuntu,Windows and FreeBSD 13) installed on a >> SATA and / or NVME disk using the virtio-blk driver and the ahci-hd driver. >> > > It is my understand for devices to be successfully passed through (based > on my own semi-working attempts and research) the device must meet the > following two conditions: > > 1. Be used exclusively by the guest OS and not the host OS > 2. The host OS needs to not even load drivers for the device(s) > > For example on the GPU this means you would need to boot the host from a > serial port and/or GPU that is not used by the host and not loaded at boot > time and then, and only then, can you pass it through to the guest OS > (wither or not the guest OS recognizes it is an other question though). > > As to good stable bhyve invocation I recommend you look at the source > code/output of either the shell based ports for making bhyve VM's or at > PetiteCloud (if you do the second I would be very interested in the results > so I can incorporate them into a future version of PC). > > > -- > Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org > -- Mario. [-- Attachment #2 --] <div dir="ltr"><div>In my system I have two graphic cards. This is the first one,configured as default from the bios and used on the freebsd host :</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">vgapci0@pci0:0:2:0: class=0x030000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x3e98 subvendor=0x1458 subdevice=0xd000 </span><br> vendor = 'Intel Corporation' <br> device = 'CoffeeLake-S GT2 [UHD Graphics 630]' <br> class = display <br> subclass = VGA<br> <br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">the USB controller that I'm trying to pass thru is the following :</span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">ppt0@pci0:1:0:0: class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014 subvendor=0x1912 subdevice=0x0015 </span><br> vendor = 'Renesas Technology Corp.' <br> device = 'uPD720201 USB 3.0 Host Controller' <br> class = serial bus <br> subclass = USB</span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"> <br></span></span><div><span style="font-family:monospace">the graphic card that I'm trying to pass thru is the following :</span></div><div><span style="font-family:monospace"><br></span></div><span style="font-family:monospace"><span style="font-family:monospace">ppt1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503 <br> vendor = 'NVIDIA Corporation' <br> device = 'TU102 [GeForce RTX 2080 Ti]' <br> class = display <br> subclass = VGA <br>ppt2@pci0:2:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503 <br> vendor = 'NVIDIA Corporation' <br> device = 'TU102 High Definition Audio Controller' <br> class = multimedia <br> subclass = HDA <br>ppt3@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503 <br> vendor = 'NVIDIA Corporation' <br> device = 'TU102 USB 3.1 Host Controller' <br> class = serial bus <br> subclass = USB <br>ppt4@pci0:2:0:3: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503 <br> vendor = 'NVIDIA Corporation' <br> device = 'TU102 USB Type-C UCSI Controller' <br> class = serial bus<br> <br></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace">On the file /boot/loader.conf I have reserved the following PCI addresses :<br></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">pptdevs="1/0/0 2/0/0 2/0/1 2/0/2 2/0/3"</span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace">If I try to boot a virtual machine in raw format passing thru my usb controller and my graphic card I get a different error :<br></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace">this bhyve commands :</span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H \ </span><br>-s 0,hostbridge \ <br>-s 1,virtio-blk,/mnt/da2p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \ <br>-s 3:0,passthru,2/0/0 \ <br>-s 3:1,passthru,2/0/1 \ <br>-s 3:2,passthru,2/0/2 \ <br>-s 3:3,passthru,2/0/3 \ <br>-s 4:0,passthru,1/0/0 \ <br>-s 9,virtio-net,tap0 \ <br>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5900">0.0.0.0:5900</a>,w=1440,h=900 \ <br>-s 30,xhci,tablet \ <br>-s 31,lpc \ <br>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ <br>-l com1,stdio \ <br>vm0<br></span></span></span></span></div><div><br><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"> </span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace">produces this error :</span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"></span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">fbuf frame buffer base: 0xa43800000 [sz 16777216] </span><br>Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501. Abort.<br></span></span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mar 8 feb 2022 alle ore 18:17 Aryeh Friedman <<a href="mailto:aryeh.friedman@gmail.com">aryeh.friedman@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>I'm using bhyve on FreeBSD 13R p7 and I'm trying to pass thru my usb controller and / or my graphic card while at the same time I'm trying to boot 3 operating systems (Ubuntu,Windows and FreeBSD 13) installed on a SATA and / or NVME disk using the virtio-blk driver and the ahci-hd driver. <br></div></div></div></blockquote><div><br></div><div>It is my understand for devices to be successfully passed through (based on my own semi-working attempts and research) the device must meet the following two conditions:</div><div><br></div><div>1. Be used exclusively by the guest OS and not the host OS</div><div>2. The host OS needs to not even load drivers for the device(s)</div><div><br></div><div>For example on the GPU this means you would need to boot the host from a serial port and/or GPU that is not used by the host and not loaded at boot time and then, and only then, can you pass it through to the guest OS (wither or not the guest OS recognizes it is an other question though).</div><div><br></div><div>As to good stable bhyve invocation I recommend you look at the source code/output of either the shell based ports for making bhyve VM's or at PetiteCloud (if you do the second I would be very interested in the results so I can incorporate them into a future version of PC).<br></div></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr">Aryeh M. Friedman, Lead Developer, <a href="http://www.PetiteCloud.org" target="_blank">http://www.PetiteCloud.org</a><br></div></div></div>; </blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Mario.<br></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSigCU26zAxQKrayjLuUch__X4=TKeTy0Jm=eV0NtG3q=dQ>