Date: Mon, 10 Oct 2022 06:06:18 +0000 From: =?utf-8?B?Q29ydmluIEvDtmhuZQ==?= <C.Koehne@beckhoff.com> To: Mario Marietto <marietto2008@gmail.com>, FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: RE: My nvidia gpu can't be passed through from Linux (qemu+kvm) to the 2. guest OS (Puppy Linux) that I tried to virtualize on the 1. (FreeBSD 13.1) Message-ID: <42b3e0459e8246d18704e6f69ad35e7a@beckhoff.com> In-Reply-To: <CA%2B1FSijOAPCJ6ry2cb_6Vf2UuOmH5r8aoWNSdHZ=ewfSwGWcVg@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --]
Hi,
Your nested virtualization may not support all features required for nested passthrough.
Best regards
Corvin
This email contains confidential information. If you have received it in error, you must not read, use, copy or pass on this e-mail or its attachments.
If you have received the e-mail in error, please inform me immediately by reply e-mail and then delete this e-mail from your system. Thank you
Diese E-Mail enthaelt vertrauliche Informationen. Sollten Sie sie irrtuemlich erhalten haben, duerfen Sie diese E-Mail oder ihre Anhaenge nicht lesen, verwenden, kopieren oder weitergeben.
Sollten Sie die Mail versehentlich erhalten haben, teilen Sie mir dies bitte umgehend per Antwort-E-Mail mit und loeschen Sie diese E-Mail dann aus Ihrem System. Vielen Dank
Beckhoff Automation GmbH & Co. KG | Managing Director: Dipl. Phys. Hans Beckhoff
Registered office: Verl, Germany | Register court: Guetersloh HRA 7075
From: owner-freebsd-virtualization@freebsd.org <owner-freebsd-virtualization@freebsd.org> On Behalf Of Mario Marietto
Sent: Tuesday, September 27, 2022 2:21 PM
To: FreeBSD virtualization <freebsd-virtualization@freebsd.org>
Subject: My nvidia gpu can't be passed through from Linux (qemu+kvm) to the 2. guest OS (Puppy Linux) that I tried to virtualize on the 1. (FreeBSD 13.1)
CAUTION: External Email!!
Hello.
I've configured Xubuntu 22.04 so that it can allow me to pass thru my nvidia gpu on the guest os vms. This time I've chosen freebsd 13.1 as a guest,because I was curious to see if bhyve supports the passthrough of my nvidia gpu within another guest os (Puppy Linux) that I have virtualized with bhyve. This is how I have configured the FreeBSD 13.1 guest vm :
/boot/loader.conf
vmm_load="YES"
nmdm_load="YES"
if_tap_load="YES"
if_bridge_load="YES"
bridgestp_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.racct.enable=1
aio_load="YES"
cryptodev_load="YES"
zfs_load="YES"
verbose_loading="YES"
pptdevs="8/0/0 9/0/0 10/0/0 11/0/0"
pciconf -vl says that pptdevs reserved correctly the addresses 8/0/0 9/0/0 10/0/0 11/0/0 which belong to the nvidia GPU :
ppt0@pci0:8:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 [GeForce RTX 2080 Ti]'
class = display
subclass = VGA
ppt1@pci0:9:0:0: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 High Definition Audio Controller'
class = multimedia
subclass = HDA
ppt2@pci0:10:0:0: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 USB 3.1 Host Controller'
class = serial bus
subclass = USB
ppt3@pci0:11:0:0: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503
vendor = 'NVIDIA Corporation'
device = 'TU102 USB Type-C UCSI Controller'
At this point,inside FreeBSD 13.1 guest OS,I tried to virtualize another OS,the puppy Linux :
bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \
-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \
-s 8:0,passthru,8/0/0 \
-s 8:1,passthru,9/0/0 \
-s 8:2,passthru,10/0/0 \
-s 8:3,passthru,11/0/0 \
-s 10,virtio-net,tap18 \
-s 11,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5918<http://0.0.0.0:5918>,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18
unfortunately it gives this error :
bhyve: PCI device at 8/0/0 is not using the ppt(4) driverdevice emulation initialization error: Device busy
I tried to remove the slots related to the passthru of the gpu and puppy booted like a charm,like this one :
bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \
-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \
-s 10,virtio-net,tap18 \
-s 11,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5918<http://0.0.0.0:5918>,w=800,h=600,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0:18 < /dev/null & sleep 2 && vncviewer 0:18
So,where could be the error in this specific scenario ? Nested vm works,but I can't pass through the gpu,even if the host OS (xubuntu) makes it available to the guest.
--
Mario.
[-- Attachment #2 --]
<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">;
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.gmail-1qeiagb0cpwnlhdf9xsijm, li.gmail-1qeiagb0cpwnlhdf9xsijm, div.gmail-1qeiagb0cpwnlhdf9xsijm
{mso-style-name:gmail-_1qeiagb0cpwnlhdf9xsijm;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div id="OLX_DISCLAIMER_MARKER" style="visibility:hidden;mso-line-height-rule:exactly;line-height:0px">
</div>
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Your nested virtualization may not support all features required for nested passthrough.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best regards<o:p></o:p></p>
<p class="MsoNormal">Corvin<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<br>
<span style="FONT-FAMILY: Arial, Helvetica, sans-serif; FONT-SIZE: 8pt">This email contains confidential information. If you have received it in error, you must not read, use, copy or pass on this e-mail or its attachments.<br>
If you have received the e-mail in error, please inform me immediately by reply e-mail and then delete this e-mail from your system. Thank you<br>
<br>
Diese E-Mail enthaelt vertrauliche Informationen. Sollten Sie sie irrtuemlich erhalten haben, duerfen Sie diese E-Mail oder ihre Anhaenge nicht lesen, verwenden, kopieren oder weitergeben.<br>
Sollten Sie die Mail versehentlich erhalten haben, teilen Sie mir dies bitte umgehend per Antwort-E-Mail mit und loeschen Sie diese E-Mail dann aus Ihrem System. Vielen Dank<br>
<br>
Beckhoff Automation GmbH & Co. KG | Managing Director: Dipl. Phys. Hans Beckhoff<br>
Registered office: Verl, Germany | Register court: Guetersloh HRA 7075</span> <br>
<br>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> owner-freebsd-virtualization@freebsd.org <owner-freebsd-virtualization@freebsd.org>
<b>On Behalf Of </b>Mario Marietto<br>
<b>Sent:</b> Tuesday, September 27, 2022 2:21 PM<br>
<b>To:</b> FreeBSD virtualization <freebsd-virtualization@freebsd.org><br>
<b>Subject:</b> My nvidia gpu can't be passed through from Linux (qemu+kvm) to the 2. guest OS (Puppy Linux) that I tried to virtualize on the 1. (FreeBSD 13.1)<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:solid #9C6500 1.0pt;padding:2.0pt 2.0pt 2.0pt 2.0pt">
<p class="MsoNormal" style="line-height:12.0pt;background:#FFEB9C"><b><span style="font-size:10.0pt;color:#9C6500">CAUTION:</span></b><span style="font-size:10.0pt;color:black"> External Email!!
<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">Hello.<o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">I've configured Xubuntu 22.04 so that it can allow me to pass thru my nvidia gpu on the guest os vms. This time I've chosen freebsd 13.1 as a guest,because I was curious to see if bhyve supports the passthrough of my
nvidia gpu within another guest os (Puppy Linux) that I have virtualized with bhyve. This is how I have configured the FreeBSD 13.1 guest vm :<o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">/boot/loader.conf<o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<pre><code>vmm_load="YES"<o:p></o:p></code></pre>
<pre><code>nmdm_load="YES"<o:p></o:p></code></pre>
<pre><code>if_tap_load="YES"<o:p></o:p></code></pre>
<pre><code>if_bridge_load="YES"<o:p></o:p></code></pre>
<pre><code>bridgestp_load="YES"<o:p></o:p></code></pre>
<pre><code>kern.geom.label.disk_ident.enable="0"<o:p></o:p></code></pre>
<pre><code>kern.geom.label.gptid.enable="0"<o:p></o:p></code></pre>
<pre><code>kern.racct.enable=1<o:p></o:p></code></pre>
<pre><code>aio_load="YES"<o:p></o:p></code></pre>
<pre><code>cryptodev_load="YES"<o:p></o:p></code></pre>
<pre><code>zfs_load="YES"<o:p></o:p></code></pre>
<pre><code>verbose_loading="YES"<o:p></o:p></code></pre>
<pre><code>pptdevs="8/0/0 9/0/0 10/0/0 11/0/0"<o:p></o:p></code></pre>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">pciconf -vl says that pptdevs reserved correctly the addresses 8/0/0 9/0/0 10/0/0 11/0/0 which belong to the nvidia GPU :<o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<pre><code>ppt0@pci0:8:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503<o:p></o:p></code></pre>
<pre><code> vendor = 'NVIDIA Corporation'<o:p></o:p></code></pre>
<pre><code> device = 'TU102 [GeForce RTX 2080 Ti]'<o:p></o:p></code></pre>
<pre><code> class = display<o:p></o:p></code></pre>
<pre><code> subclass = VGA<o:p></o:p></code></pre>
<pre><code><o:p> </o:p></code></pre>
<pre><code>ppt1@pci0:9:0:0: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503<o:p></o:p></code></pre>
<pre><code> vendor = 'NVIDIA Corporation'<o:p></o:p></code></pre>
<pre><code> device = 'TU102 High Definition Audio Controller'<o:p></o:p></code></pre>
<pre><code> class = multimedia<o:p></o:p></code></pre>
<pre><code> subclass = HDA<o:p></o:p></code></pre>
<pre><code><o:p> </o:p></code></pre>
<pre><code>ppt2@pci0:10:0:0: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503<o:p></o:p></code></pre>
<pre><code> vendor = 'NVIDIA Corporation'<o:p></o:p></code></pre>
<pre><code> device = 'TU102 USB 3.1 Host Controller'<o:p></o:p></code></pre>
<pre><code> class = serial bus<o:p></o:p></code></pre>
<pre><code> subclass = USB<o:p></o:p></code></pre>
<pre><code><o:p> </o:p></code></pre>
<pre><code>ppt3@pci0:11:0:0: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503<o:p></o:p></code></pre>
<pre><code> vendor = 'NVIDIA Corporation'<o:p></o:p></code></pre>
<pre><code> device = 'TU102 USB Type-C UCSI Controller'<o:p></o:p></code></pre>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">At this point,inside FreeBSD 13.1 guest OS,I tried to virtualize another OS,the puppy Linux :<o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<pre><code>bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \<o:p></o:p></code></pre>
<pre><code>-s 0,hostbridge \<o:p></o:p></code></pre>
<pre><code>-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \<o:p></o:p></code></pre>
<pre><code>-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \<o:p></o:p></code></pre>
<pre><code>-s 8:0,passthru,8/0/0 \<o:p></o:p></code></pre>
<pre><code>-s 8:1,passthru,9/0/0 \<o:p></o:p></code></pre>
<pre><code>-s 8:2,passthru,10/0/0 \<o:p></o:p></code></pre>
<pre><code>-s 8:3,passthru,11/0/0 \<o:p></o:p></code></pre>
<pre><code>-s 10,virtio-net,tap18 \<o:p></o:p></code></pre>
<pre><code>-s 11,virtio-9p,sharename=/ \<o:p></o:p></code></pre>
<pre><code>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5918">0.0.0.0:5918</a>,w=800,h=600,wait \<o:p></o:p></code></pre>
<pre><code>-s 30,xhci,tablet \<o:p></o:p></code></pre>
<pre><code>-s 31,lpc \<o:p></o:p></code></pre>
<pre><code>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \<o:p></o:p></code></pre>
<pre><code>vm0:18 < /dev/null & sleep 2 && vncviewer 0:18<o:p></o:p></code></pre>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">unfortunately it gives this error :<o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><strong><span style="font-family:"Calibri",sans-serif">bhyve: PCI device at 8/0/0 is not using the ppt(4) driverdevice emulation initialization error: Device busy</span></strong><o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">I tried to remove the slots related to the passthru of the gpu and puppy booted like a charm,like this one :<o:p></o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<pre><code>bhyve -S -c sockets=1,cores=1,threads=1 -m 2G -w -H -A \<o:p></o:p></code></pre>
<pre><code>-s 0,hostbridge \<o:p></o:p></code></pre>
<pre><code>-s 1,ahci-cd,/home/marietto/Desktop/bhyve/Files/fossapup64-9.5.iso,bootindex=1 \<o:p></o:p></code></pre>
<pre><code>-s 2,virtio-blk,/home/marietto/Desktop/bhyve/Files/puppy.img,bootindex=2 \<o:p></o:p></code></pre>
<pre><code>-s 10,virtio-net,tap18 \<o:p></o:p></code></pre>
<pre><code>-s 11,virtio-9p,sharename=/ \<o:p></o:p></code></pre>
<pre><code>-s 29,fbuf,tcp=<a href="http://0.0.0.0:5918">0.0.0.0:5918</a>,w=800,h=600,wait \<o:p></o:p></code></pre>
<pre><code>-s 30,xhci,tablet \<o:p></o:p></code></pre>
<pre><code>-s 31,lpc \<o:p></o:p></code></pre>
<pre><code>-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \<o:p></o:p></code></pre>
<pre><code>vm0:18 < /dev/null & sleep 2 && vncviewer 0:18<o:p></o:p></code></pre>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm"><o:p> </o:p></p>
<p class="gmail-1qeiagb0cpwnlhdf9xsijm">So,where could be the error in this specific scenario ? Nested vm works,but I can't pass through the gpu,even if the host OS (xubuntu) makes it available to the guest.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">Mario.<o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?42b3e0459e8246d18704e6f69ad35e7a>