Date: Tue, 27 Sep 2022 14:21:18 +0200 From: Mario Marietto <marietto2008@gmail.com> To: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: My nvidia gpu can't be passed through from Linux (qemu+kvm) to the 2. guest OS (Puppy Linux) that I tried to virtualize on the 1. (FreeBSD 13.1) Message-ID: <CA%2B1FSijOAPCJ6ry2cb_6Vf2UuOmH5r8aoWNSdHZ=ewfSwGWcVg@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
--0000000000008f3feb05e9a7b03d
Content-Type: text/plain; charset="UTF-8"
Hello.
I've configured Xubuntu 22.04 so that it can allow me to pass thru my
nvidia gpu on the guest os vms. This time I've chosen freebsd 13.1 as a
guest,because I was curious to see if bhyve supports the passthrough of my
nvidia gpu within another guest os (Puppy Linux) that I have virtualized
with bhyve. This is how I have configured the FreeBSD 13.1 guest vm :
/boot/loader.conf
vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.racct.enable=1
aio_load="YES"
cryptodev_load="YES"
zfs_load="YES"
verbose_loading="YES"
pptdevs="8/0/0 9/0/0 10/0/0 11/0/0"
pciconf -vl says that pptdevs reserved correctly the addresses 8/0/0 9/0/0
10/0/0 11/0/0 which belong to the nvidia GPU :
ppt0@pci0:8:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de
device=0x1e04 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 [GeForce RTX 2080 Ti]'
class = display
subclass = VGA
ppt1@pci0:9:0:0: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de
device=0x10f7 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 High Definition Audio Controller'
class = multimedia
subclass = HDA
ppt2@pci0:10:0:0: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de
device=0x1ad6 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 USB 3.1 Host Controller'
class = serial bus
subclass = USB
ppt3@pci0:11:0:0: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de
device=0x1ad7 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 USB Type-C UCSI Controller'
At this point,inside FreeBSD 13.1 guest OS,I tried to virtualize another
OS,the puppy Linux :
bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \
-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \
-s 8:0,passthru,8/0/0 \
-s 8:1,passthru,9/0/0 \
-s 8:2,passthru,10/0/0 \
-s 8:3,passthru,11/0/0 \
-s 10,virtio-net,tap18 \
-s 11,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5918,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18
unfortunately it gives this error :
*bhyve: PCI device at 8/0/0 is not using the ppt(4) driverdevice emulation
initialization error: Device busy*
I tried to remove the slots related to the passthru of the gpu and puppy
booted like a charm,like this one :
bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \
-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \
-s 10,virtio-net,tap18 \
-s 11,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5918,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18
So,where could be the error in this specific scenario ? Nested vm works,but
I can't pass through the gpu,even if the host OS (xubuntu) makes it
available to the guest.
--
Mario.
--0000000000008f3feb05e9a7b03d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div class=3D"gmail-_3xX726aBn29LDbsDtzr_6E gmail-_1Ap4F5m=
aDtT1E1YuCiaO0r gmail-D3IL3FD0RFy_mkKLPwL4" style=3D"max-width:800px"><div =
class=3D"gmail-_292iotee39Lmt0MkQZ2hPV gmail-RichTextJSON-root"><p class=3D=
"gmail-_1qeIAgB0cPwnLhDF9XSiJM">Hello.</p><p class=3D"gmail-_1qeIAgB0cPwnLh=
DF9XSiJM">I've
configured Xubuntu 22.04 so that it can allow me to pass thru my =20
nvidia gpu on the guest os vms. This time I've chosen freebsd 13.1 as a=
=20
guest,because I was curious to see if bhyve supports the passthrough of
my nvidia gpu within another guest os (Puppy Linux) that I have =20
virtualized with bhyve. This is how I have configured the FreeBSD 13.1=20
guest vm :</p><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><br></p><p class=
=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM">/boot/loader.conf</p><p class=3D"gmail-_=
1qeIAgB0cPwnLhDF9XSiJM"><br></p><pre class=3D"gmail-_3GnarIQX9tD_qsgXkfSDz1=
"><code class=3D"gmail-_34q3PgLsx9zIU5BiSOjFoM">vmm_load=3D"YES"
nmdm_load=3D"YES"
if_tap_load=3D"YES"
if_bridge_load=3D"YES"
bridgestp_load=3D"YES"
kern.geom.label.disk_ident.enable=3D"0"
kern.geom.label.gptid.enable=3D"0"
kern.racct.enable=3D1
aio_load=3D"YES"
cryptodev_load=3D"YES"
zfs_load=3D"YES"
verbose_loading=3D"YES"
pptdevs=3D"8/0/0 9/0/0 10/0/0 11/0/0"
</code></pre><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><br></p><p class=3D=
"gmail-_1qeIAgB0cPwnLhDF9XSiJM">pciconf -vl says that pptdevs reserved corr=
ectly the addresses 8/0/0 9/0/0 10/0/0 11/0/0 which belong to the nvidia GP=
U :</p><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><br></p><pre class=3D"gma=
il-_3GnarIQX9tD_qsgXkfSDz1"><code class=3D"gmail-_34q3PgLsx9zIU5BiSOjFoM">p=
pt0@pci0:8:0:0: class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de d=
evice=3D0x1e04 subvendor=3D0x19da subdevice=3D0x2503
vendor =3D 'NVIDIA Corporation'
device =3D 'TU102 [GeForce RTX 2080 Ti]'
class =3D display
subclass =3D VGA
ppt1@pci0:9:0:0: class=3D0x040300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de =
device=3D0x10f7 subvendor=3D0x19da subdevice=3D0x2503
vendor =3D 'NVIDIA Corporation'
device =3D 'TU102 High Definition Audio Controller'
class =3D multimedia
subclass =3D HDA
ppt2@pci0:10:0:0: class=3D0x0c0330 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de=
device=3D0x1ad6 subvendor=3D0x19da subdevice=3D0x2503
vendor =3D 'NVIDIA Corporation'
device =3D 'TU102 USB 3.1 Host Controller'
class =3D serial bus
subclass =3D USB
ppt3@pci0:11:0:0: class=3D0x0c8000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de=
device=3D0x1ad7 subvendor=3D0x19da subdevice=3D0x2503
vendor =3D 'NVIDIA Corporation'
device =3D 'TU102 USB Type-C UCSI Controller'
</code></pre><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><br></p><p class=3D=
"gmail-_1qeIAgB0cPwnLhDF9XSiJM">At this point,inside FreeBSD 13.1 guest OS,=
I tried to virtualize another OS,the puppy Linux :</p><p class=3D"gmail-_1q=
eIAgB0cPwnLhDF9XSiJM"><br></p><pre class=3D"gmail-_3GnarIQX9tD_qsgXkfSDz1">=
<code class=3D"gmail-_34q3PgLsx9zIU5BiSOjFoM">bhyve -S -c sockets=3D1,cores=
=3D1,threads=3D1 -m 2G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootinde=
x=3D1 \
-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=3D2 =
\
-s 8:0,passthru,8/0/0 \
-s 8:1,passthru,9/0/0 \
-s 8:2,passthru,10/0/0 \
-s 8:3,passthru,11/0/0 \
-s 10,virtio-net,tap18 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5918">0.0.0.0:5918</a>,w=3D800,h=
=3D600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18
</code></pre><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><br></p><p class=3D=
"gmail-_1qeIAgB0cPwnLhDF9XSiJM">unfortunately it gives this error :</p><p c=
lass=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><strong class=3D"gmail-_12FoOEddL7j_=
RgMQN0SNeU">bhyve: PCI device at 8/0/0 is not using the ppt(4) driverdevice=
emulation initialization error: Device busy</strong></p><p class=3D"gmail-=
_1qeIAgB0cPwnLhDF9XSiJM"><strong class=3D"gmail-_12FoOEddL7j_RgMQN0SNeU"><b=
r></strong></p><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM">I tried to remove=
the slots related to the passthru of the gpu and puppy booted like a charm=
,like this one :</p><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><br></p><pre=
class=3D"gmail-_3GnarIQX9tD_qsgXkfSDz1"><code class=3D"gmail-_34q3PgLsx9zI=
U5BiSOjFoM">bhyve -S -c sockets=3D1,cores=3D1,threads=3D1 -m 2G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootinde=
x=3D1 \
-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=3D2 =
\
-s 10,virtio-net,tap18 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5918">0.0.0.0:5918</a>,w=3D800,h=
=3D600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18
</code></pre><p class=3D"gmail-_1qeIAgB0cPwnLhDF9XSiJM"><br></p><p class=3D=
"gmail-_1qeIAgB0cPwnLhDF9XSiJM">So,where
could be the error in this specific scenario ? Nested vm works,but I=20
can't pass through the gpu,even if the host OS (xubuntu) makes it=20
available to the guest.</p></div></div>-- <br><div dir=3D"ltr" class=3D"gma=
il_signature" data-smartmail=3D"gmail_signature">Mario.<br></div></div>
--0000000000008f3feb05e9a7b03d--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSijOAPCJ6ry2cb_6Vf2UuOmH5r8aoWNSdHZ=ewfSwGWcVg>