Date: Tue, 8 Feb 2022 18:45:32 +0100 From: Mario Marietto <marietto2008@gmail.com> To: Aryeh Friedman <aryeh.friedman@gmail.com> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: Multiple bugs on bhyve present on FreeBSD 13R p7 while passing thru USB controller and graphic card Message-ID: <CA%2B1FSigCU26zAxQKrayjLuUch__X4=TKeTy0Jm=eV0NtG3q=dQ@mail.gmail.com> In-Reply-To: <CAGBxaXnak5sOpiceofttORyfThXSnby7aw9e5BpODPnbgJ_uNw@mail.gmail.com> References: <CA%2B1FSigMNsuGGAqi7hQZTuAFzMuByKtcFs554SVs6oXrfUZs-g@mail.gmail.com> <CAGBxaXnak5sOpiceofttORyfThXSnby7aw9e5BpODPnbgJ_uNw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000c212b005d7854a47 Content-Type: text/plain; charset="UTF-8" In my system I have two graphic cards. This is the first one,configured as default from the bios and used on the freebsd host : vgapci0@pci0:0:2:0: class=0x030000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x3e98 subvendor=0x1458 subdevice=0xd000 vendor = 'Intel Corporation' device = 'CoffeeLake-S GT2 [UHD Graphics 630]' class = display subclass = VGA the USB controller that I'm trying to pass thru is the following : ppt0@pci0:1:0:0: class=0x0c0330 rev=0x03 hdr=0x00 vendor=0x1912 device=0x0014 subvendor=0x1912 subdevice=0x0015 vendor = 'Renesas Technology Corp.' device = 'uPD720201 USB 3.0 Host Controller' class = serial bus subclass = USB the graphic card that I'm trying to pass thru is the following : ppt1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 [GeForce RTX 2080 Ti]' class = display subclass = VGA ppt2@pci0:2:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 High Definition Audio Controller' class = multimedia subclass = HDA ppt3@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 USB 3.1 Host Controller' class = serial bus subclass = USB ppt4@pci0:2:0:3: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503 vendor = 'NVIDIA Corporation' device = 'TU102 USB Type-C UCSI Controller' class = serial bus On the file /boot/loader.conf I have reserved the following PCI addresses : pptdevs="1/0/0 2/0/0 2/0/1 2/0/2 2/0/3" If I try to boot a virtual machine in raw format passing thru my usb controller and my graphic card I get a different error : this bhyve commands : bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H \ -s 0,hostbridge \ -s 1,virtio-blk,/mnt/da2p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \ -s 3:0,passthru,2/0/0 \ -s 3:1,passthru,2/0/1 \ -s 3:2,passthru,2/0/2 \ -s 3:3,passthru,2/0/3 \ -s 4:0,passthru,1/0/0 \ -s 9,virtio-net,tap0 \ -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -l com1,stdio \ vm0 produces this error : fbuf frame buffer base: 0xa43800000 [sz 16777216] Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501. Abort. Il giorno mar 8 feb 2022 alle ore 18:17 Aryeh Friedman < aryeh.friedman@gmail.com> ha scritto: > > I'm using bhyve on FreeBSD 13R p7 and I'm trying to pass thru my usb >> controller and / or my graphic card while at the same time I'm trying to >> boot 3 operating systems (Ubuntu,Windows and FreeBSD 13) installed on a >> SATA and / or NVME disk using the virtio-blk driver and the ahci-hd driver. >> > > It is my understand for devices to be successfully passed through (based > on my own semi-working attempts and research) the device must meet the > following two conditions: > > 1. Be used exclusively by the guest OS and not the host OS > 2. The host OS needs to not even load drivers for the device(s) > > For example on the GPU this means you would need to boot the host from a > serial port and/or GPU that is not used by the host and not loaded at boot > time and then, and only then, can you pass it through to the guest OS > (wither or not the guest OS recognizes it is an other question though). > > As to good stable bhyve invocation I recommend you look at the source > code/output of either the shell based ports for making bhyve VM's or at > PetiteCloud (if you do the second I would be very interested in the results > so I can incorporate them into a future version of PC). > > > -- > Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org > -- Mario. --000000000000c212b005d7854a47 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>In my system I have two graphic cards. This is the fi= rst one,configured as default from the bios and used on the freebsd host :<= /div><div><br></div><div><span style=3D"font-family:monospace"><span style= =3D"color:rgb(0,0,0);background-color:rgb(255,255,255)">vgapci0@pci0:0:2:0:= =C2=A0=C2=A0=C2=A0=C2=A0class=3D0x030000 rev=3D0x02 hdr=3D0x00 vendor=3D0x= 8086 device=3D0x3e98 subvendor=3D0x1458 subdevice=3D0xd000 </span><br> =C2=A0=C2=A0=C2=A0vendor =C2=A0=C2=A0=C2=A0=C2=A0=3D 'Intel= Corporation' <br> =C2=A0=C2=A0=C2=A0device =C2=A0=C2=A0=C2=A0=C2=A0=3D 'CoffeeLake-S= GT2 [UHD Graphics 630]' <br> =C2=A0=C2=A0=C2=A0class =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=3D display <br> =C2=A0=C2=A0=C2=A0subclass =C2=A0=C2=A0=3D VGA<br> <br></span></div><div><span style=3D"font-family:monospace"><br></span></di= v><div><span style=3D"font-family:monospace"><span style=3D"font-family:mon= ospace"><span style=3D"color:rgb(0,0,0);background-color:rgb(255,255,255)">= the USB controller that I'm trying to pass thru is the following :</spa= n></span></span></div><div><span style=3D"font-family:monospace"><span styl= e=3D"font-family:monospace"><span style=3D"color:rgb(0,0,0);background-colo= r:rgb(255,255,255)"><br></span></span></span></div><div><span style=3D"font= -family:monospace"><span style=3D"font-family:monospace"><span style=3D"col= or:rgb(0,0,0);background-color:rgb(255,255,255)">ppt0@pci0:1:0:0: =C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c0330 rev=3D0x03 hdr=3D0x00 ven= dor=3D0x1912 device=3D0x0014 subvendor=3D0x1912 subdevice=3D0x0015 </span><br> =C2=A0=C2=A0=C2=A0vendor =C2=A0=C2=A0=C2=A0=C2=A0=3D 'Renes= as Technology Corp.' <br> =C2=A0=C2=A0=C2=A0device =C2=A0=C2=A0=C2=A0=C2=A0=3D 'uPD720201 US= B 3.0 Host Controller' <br> =C2=A0=C2=A0=C2=A0class =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=3D serial bus <br> =C2=A0=C2=A0=C2=A0subclass =C2=A0=C2=A0=3D USB</span></span></div><div= ><span style=3D"font-family:monospace"><span style=3D"font-family:monospace= ">=C2=A0<br></span></span><div><span style=3D"font-family:monospace">the gr= aphic card that I'm trying to pass thru is the following :</span></div>= <div><span style=3D"font-family:monospace"><br></span></div><span style=3D"= font-family:monospace"><span style=3D"font-family:monospace">ppt1@pci0:2:0:= 0: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x030000 rev=3D0xa1 hd= r=3D0x00 vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x2= 503 <br> =C2=A0=C2=A0=C2=A0vendor =C2=A0=C2=A0=C2=A0=C2=A0=3D 'NVIDIA Corpo= ration' <br> =C2=A0=C2=A0=C2=A0device =C2=A0=C2=A0=C2=A0=C2=A0=3D 'TU102 [GeFor= ce RTX 2080 Ti]' <br> =C2=A0=C2=A0=C2=A0class =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=3D display <br> =C2=A0=C2=A0=C2=A0subclass =C2=A0=C2=A0=3D VGA <br>ppt2@pci0:2:0:1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x04= 0300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x10f7 subvendor=3D0x19= da subdevice=3D0x2503 <br> =C2=A0=C2=A0=C2=A0vendor =C2=A0=C2=A0=C2=A0=C2=A0=3D 'NVIDIA Corpo= ration' <br> =C2=A0=C2=A0=C2=A0device =C2=A0=C2=A0=C2=A0=C2=A0=3D 'TU102 High D= efinition Audio Controller' <br> =C2=A0=C2=A0=C2=A0class =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=3D multimedia <br> =C2=A0=C2=A0=C2=A0subclass =C2=A0=C2=A0=3D HDA <br>ppt3@pci0:2:0:2: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 0330 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad6 subvendor=3D0x19= da subdevice=3D0x2503 <br> =C2=A0=C2=A0=C2=A0vendor =C2=A0=C2=A0=C2=A0=C2=A0=3D 'NVIDIA Corpo= ration' <br> =C2=A0=C2=A0=C2=A0device =C2=A0=C2=A0=C2=A0=C2=A0=3D 'TU102 USB 3.= 1 Host Controller' <br> =C2=A0=C2=A0=C2=A0class =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=3D serial bus <br> =C2=A0=C2=A0=C2=A0subclass =C2=A0=C2=A0=3D USB <br>ppt4@pci0:2:0:3: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 8000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad7 subvendor=3D0x19= da subdevice=3D0x2503 <br> =C2=A0=C2=A0=C2=A0vendor =C2=A0=C2=A0=C2=A0=C2=A0=3D 'NVIDIA Corpo= ration' <br> =C2=A0=C2=A0=C2=A0device =C2=A0=C2=A0=C2=A0=C2=A0=3D 'TU102 USB Ty= pe-C UCSI Controller' <br> =C2=A0=C2=A0=C2=A0class =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=3D serial bus<b= r> <br></span></span></div><div><span style=3D"font-family:monospace"><span st= yle=3D"font-family:monospace">On the file /boot/loader.conf I have reserved= the following PCI addresses :<br></span></span></div><div><span style=3D"f= ont-family:monospace"><span style=3D"font-family:monospace"><br></span></sp= an></div><div><span style=3D"font-family:monospace"><span style=3D"font-fam= ily:monospace"><span style=3D"font-family:monospace"><span style=3D"color:r= gb(0,0,0);background-color:rgb(255,255,255)">pptdevs=3D"1/0/0 2/0/0 2/= 0/1 2/0/2 2/0/3"</span></span></span></span></div><div><span style=3D"= font-family:monospace"><span style=3D"font-family:monospace"><span style=3D= "font-family:monospace"><br></span></span></span></div><div><span style=3D"= font-family:monospace"><span style=3D"font-family:monospace"><span style=3D= "font-family:monospace">If I try to boot a virtual machine in raw format pa= ssing thru my usb controller and my graphic card I get a different error :<= br></span></span></span></div><div><span style=3D"font-family:monospace"><s= pan style=3D"font-family:monospace"><span style=3D"font-family:monospace"><= br></span></span></span></div><div><span style=3D"font-family:monospace"><s= pan style=3D"font-family:monospace"><span style=3D"font-family:monospace">t= his bhyve commands :</span></span></span></div><div><span style=3D"font-fam= ily:monospace"><span style=3D"font-family:monospace"><span style=3D"font-fa= mily:monospace"><br></span></span></span></div><div><span style=3D"font-fam= ily:monospace"><span style=3D"font-family:monospace"><span style=3D"font-fa= mily:monospace"><span style=3D"font-family:monospace"><span style=3D"color:= rgb(0,0,0);background-color:rgb(255,255,255)">bhyve -S -c sockets=3D2,cores= =3D2,threads=3D2 -m 8G -w -H \ </span><br>-s 0,hostbridge \ <br>-s 1,virtio-blk,/mnt/da2p2/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.i= mg \ <br>-s 3:0,passthru,2/0/0 \ <br>-s 3:1,passthru,2/0/1 \ <br>-s 3:2,passthru,2/0/2 \ <br>-s 3:3,passthru,2/0/3 \ <br>-s 4:0,passthru,1/0/0 \ <br>-s 9,virtio-net,tap0 \ <br>-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5900">0.0.0.0:5900</a>,w=3D1= 440,h=3D900 \ <br>-s 30,xhci,tablet \ <br>-s 31,lpc \ =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0<br>-l bootrom,/usr/local/share/uefi-firmware= /BHYVE_UEFI.fd \ <br>-l com1,stdio \ <br>vm0<br></span></span></span></span></div><div><br><span style=3D"font-f= amily:monospace"><span style=3D"font-family:monospace"><span style=3D"font-= family:monospace"><span style=3D"font-family:monospace"> </span></span></span></span></div><div><span style=3D"font-family:monospace= "><span style=3D"font-family:monospace"><span style=3D"font-family:monospac= e"><span style=3D"font-family:monospace">produces this error :</span></span= ></span></span></div><div><span style=3D"font-family:monospace"><span style= =3D"font-family:monospace"><span style=3D"font-family:monospace"><span styl= e=3D"font-family:monospace"></span></span></span></span></div><div><span st= yle=3D"font-family:monospace"><span style=3D"font-family:monospace"><span s= tyle=3D"font-family:monospace"><span style=3D"font-family:monospace"><br></= span></span></span></span></div><div><span style=3D"font-family:monospace">= <span style=3D"font-family:monospace"><span style=3D"font-family:monospace"= ><span style=3D"font-family:monospace"><span style=3D"color:rgb(0,0,0);back= ground-color:rgb(255,255,255)">fbuf frame buffer base: 0xa43800000 [sz 1677= 7216] </span><br>Assertion failed: (error =3D=3D 0), function modify_bar_registra= tion, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501. Abort.<br></span><= /span></span></span></div><div><span style=3D"font-family:monospace"><span = style=3D"font-family:monospace"><span style=3D"font-family:monospace"><br><= /span></span></span></div></div><br><div class=3D"gmail_quote"><div dir=3D"= ltr" class=3D"gmail_attr">Il giorno mar 8 feb 2022 alle ore 18:17 Aryeh Fri= edman <<a href=3D"mailto:aryeh.friedman@gmail.com">aryeh.friedman@gmail.= com</a>> ha scritto:<br></div><blockquote class=3D"gmail_quote" style=3D= "margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le= ft:1ex"><div dir=3D"ltr"><div dir=3D"ltr"><br></div><div class=3D"gmail_quo= te"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor= der-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=3D"ltr"><div= ><div>I'm using bhyve on FreeBSD 13R p7 and I'm trying to pass thru= my usb controller and / or my graphic card while at the same time I'm = trying to boot 3 operating systems (Ubuntu,Windows and FreeBSD 13) installe= d on a SATA and / or NVME disk using the virtio-blk driver and the ahci-hd = driver. <br></div></div></div></blockquote><div><br></div><div>It is my und= erstand for devices to be successfully passed through (based on my own semi= -working attempts and research) the device must meet the following two cond= itions:</div><div><br></div><div>1. Be used exclusively by the guest OS and= not the host OS</div><div>2. The host OS needs to not even load drivers fo= r the device(s)</div><div><br></div><div>For example on the GPU this means = you would need to boot the host from a serial port and/or GPU that is not u= sed by the host and not loaded at boot time and then, and only then, can yo= u pass it through to the guest OS (wither or not the guest OS recognizes it= is an other question though).</div><div><br></div><div>As to good stable b= hyve invocation I recommend you look at the source code/output of either th= e shell based ports for making bhyve VM's or at PetiteCloud (if you do = the second I would be very interested in the results so I can incorporate t= hem into a future version of PC).<br></div></div><br clear=3D"all"><br>-- <= br><div dir=3D"ltr"><div dir=3D"ltr">Aryeh M. Friedman, Lead Developer, <a = href=3D"http://www.PetiteCloud.org" target=3D"_blank">http://www.PetiteClou= d.org</a><br></div></div></div> </blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g= mail_signature">Mario.<br></div> --000000000000c212b005d7854a47--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSigCU26zAxQKrayjLuUch__X4=TKeTy0Jm=eV0NtG3q=dQ>