Date: Thu, 27 Mar 2025 10:22:18 +0000 From: Shamim Shahriar <shamim.shahriar@gmail.com> To: Alex <9vlc@proton.me> Cc: "freebsd-virtualization@FreeBSD.org" <freebsd-virtualization@freebsd.org> Subject: Re: In response to "GPU Passthrough with FreeBSD 14.2 bhyve" Message-ID: <CAOyJeZRbR0rK88OrH0-vfxm_5J-o1Cn7dQ0jbFB%2BmEy79MWkcw@mail.gmail.com> In-Reply-To: <lCxhecMbYUA09VPSS9N0UCDIrsFfxCXwCh5de8fcp_VIXlXpCyZwKP3crlelEx8YHP2wmdzp4GS74XOUwzlVy4eJhl2YToMhOV5Lfz95U0k=@proton.me> References: <lCxhecMbYUA09VPSS9N0UCDIrsFfxCXwCh5de8fcp_VIXlXpCyZwKP3crlelEx8YHP2wmdzp4GS74XOUwzlVy4eJhl2YToMhOV5Lfz95U0k=@proton.me>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Good morning Alex, and thank you for the detailed help, it is very much appreciated. For some reason your email went into google's spam box, I just noticed it this morning. I'll try this as well and keep the list posted as to how it goes. Best regards SS On Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote: > This is a hefty help email for Shamim Shahriar in regards to passing > through Nvidia GPU to a bhyve VM. > > Hi. > I have recently gotten this mailing list message forwarded to me by > Michael Dexter > https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html > and it alone made me sign up here. > > So, here's a big mail containing a bunch of useful GPU passthrough info! > > To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, > let me help you out make it work correctly! > > Parts and short description: > > 1. Pass through the GPU audio and other devices (if exist) > Modern GPUs usually create two PCI devices that you can see in > pciconf - VGA and audio. They come on the same PCI function (i think it is > called that), for example GPU 19/0/0 will also have 19/0/1 for audio and > may have 19/0/2 for usb. > > 2. Give the GPU a VGA BIOS as PCI Option ROM > GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a > trimmed VGA BIOS since the PCI option rom that is used by the VM is nested > inside the ROM file. > (from now on i will refer to the vga bios as vbios) > > 3. Possibly, re-BAR in BIOS? > Some UEFI firmware have a feature called re-BAR and "above 4G > decoding / mining", it makes some GPUs in some configurations not work in > passthru. > > 5. Possibly, Try newer EDK2 > The edk2-bhyve package (provides UEFI firmware for bhyve) on the > repos is still using the 2023 release. You can compile a 2025 version with > my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 > > Full parts and HOWTOs: > > !!!! MISC GPU DEVICES > > As decribed before, pass through all the devices that your GPU creates in > pciconf right after the GPU device on the same PCI function. > All audio, USB, etc. devices belonging to the GPU should be passed through > with it since they are a part of it, else the passthru will NOT work > correctly. > If that is not the case and there is only a single GPU VGA device with > nothing else Nvidia to be found, skip this step. > > !!!! VBIOS > > You first need to get the pure vBIOS file. To do that you can either: > 1. Use a Linux system and dump it with > https://github.com/ilya-zlobintsev/LACT > 2. Use a Windows system and dump it with https://www.techpowerup.com/gpuz > 3. Download one that someone has dumped from their GPU using GPU-z from > https://www.techpowerup.com/vgabios > > If you don't want to install a different OS on the PC you're currently > running, we can just find and download a correct vBIOS from Techpowerup. > But we stumble across an issue! You specified "NVidia Quadro RTX > 6000/8000", so which GPU is it really? Those are two different GPUs with > different specifications! > Thankfully, we have your pciconf output which says "subdevice=0x13d8" and > if we refer to Nvidias documentation, on page 7 over here: > > https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf > we can see that it is a RTX 8000! > Here is an unverified but most likely working vBIOS: > https://www.techpowerup.com/vgabios/256271/256271 > > Now after we've gotten the vBIOS, we need to trim the start of the file to > get to the embedded PCI Option ROM. > For that we will use my tool "trim_nvidia_vbios" from > https://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios > Compile and run the resulting executable with the first argument being the > downloaded vBIOS and the second argument being an output PCI OpROM. Example: > $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c > $ ./trim.elf techpowerup.rom trimmed-vbios.rom > > And we come upon a road block: vm-bhyve does not support passing a PCI > Option ROMs in "passthru=", so we're going to use "bhyve_options=" instead > to define the GPU VGA device: > bhyve_options="-s 6:0,passthru,19/0/0,rom=</path/to/trimmed-vbios.rom>" > Other devices that do not require a PCI Option ROM can be added with > "passthru=". These devices include the rest of the GPU pci functions like > audio and usb. > > !!!! RE-BAR > > If after everything you do get the GPU to work in the VM at least in the > edk2 firmware, > Go to your regular UEFI BIOS setup utility and search for options titled > "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for > Mining". > If you do see re-BAR but cannot disable it, you could try using > https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI > settings. Be warned though, smokeless is a heck of an unstable MESS. > > !!!! UPDATED EDK2 > > As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some > rare cases, that might cause issues. > I have submitted a package update bug report for sysutils/edk2 to FreeBSD > bugzilla over here: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 > You can download the patch, apply it to your ports tree, then compile and > install the edk2 package with "FLAVOR=bhyve" to get the fresh UEFI firmware. > > ~~~~ > > Hope I didn't forget anything! > I wish you success in the GPU passthrough journey!! > - Alexey > > > [-- Attachment #2 --] <div dir="ltr">Good morning Alex, and thank you for the detailed help, it is very much appreciated.<div><br></div><div>For some reason your email went into google's spam box, I just noticed it this morning. I'll try this as well and keep the list posted as to how it goes.</div><div><br></div><div>Best regards</div><div>SS</div></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Tue, 25 Mar 2025 at 13:27, Alex <<a href="mailto:9vlc@proton.me">9vlc@proton.me</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">This is a hefty help email for Shamim Shahriar in regards to passing through Nvidia GPU to a bhyve VM.<br> <br> Hi.<br> I have recently gotten this mailing list message forwarded to me by Michael Dexter <a href="https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html" rel="noreferrer" target="_blank">https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html</a>; and it alone made me sign up here.<br> <br> So, here's a big mail containing a bunch of useful GPU passthrough info!<br> <br> To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, let me help you out make it work correctly!<br> <br> Parts and short description:<br> <br> 1. Pass through the GPU audio and other devices (if exist)<br> Modern GPUs usually create two PCI devices that you can see in pciconf - VGA and audio. They come on the same PCI function (i think it is called that), for example GPU 19/0/0 will also have 19/0/1 for audio and may have 19/0/2 for usb.<br> <br> 2. Give the GPU a VGA BIOS as PCI Option ROM<br> GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a trimmed VGA BIOS since the PCI option rom that is used by the VM is nested inside the ROM file.<br> (from now on i will refer to the vga bios as vbios)<br> <br> 3. Possibly, re-BAR in BIOS?<br> Some UEFI firmware have a feature called re-BAR and "above 4G decoding / mining", it makes some GPUs in some configurations not work in passthru.<br> <br> 5. Possibly, Try newer EDK2<br> The edk2-bhyve package (provides UEFI firmware for bhyve) on the repos is still using the 2023 release. You can compile a 2025 version with my patch: <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554" rel="noreferrer" target="_blank">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554</a><br>; <br> Full parts and HOWTOs:<br> <br> !!!! MISC GPU DEVICES<br> <br> As decribed before, pass through all the devices that your GPU creates in pciconf right after the GPU device on the same PCI function.<br> All audio, USB, etc. devices belonging to the GPU should be passed through with it since they are a part of it, else the passthru will NOT work correctly.<br> If that is not the case and there is only a single GPU VGA device with nothing else Nvidia to be found, skip this step.<br> <br> !!!! VBIOS<br> <br> You first need to get the pure vBIOS file. To do that you can either:<br> 1. Use a Linux system and dump it with <a href="https://github.com/ilya-zlobintsev/LACT" rel="noreferrer" target="_blank">https://github.com/ilya-zlobintsev/LACT</a><br>; 2. Use a Windows system and dump it with <a href="https://www.techpowerup.com/gpuz" rel="noreferrer" target="_blank">https://www.techpowerup.com/gpuz</a><br>; 3. Download one that someone has dumped from their GPU using GPU-z from <a href="https://www.techpowerup.com/vgabios" rel="noreferrer" target="_blank">https://www.techpowerup.com/vgabios</a><br>; <br> If you don't want to install a different OS on the PC you're currently running, we can just find and download a correct vBIOS from Techpowerup.<br> But we stumble across an issue! You specified "NVidia Quadro RTX 6000/8000", so which GPU is it really? Those are two different GPUs with different specifications!<br> Thankfully, we have your pciconf output which says "subdevice=0x13d8" and if we refer to Nvidias documentation, on page 7 over here:<br> <a href="https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf" rel="noreferrer" target="_blank">https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf</a><br>; we can see that it is a RTX 8000!<br> Here is an unverified but most likely working vBIOS: <a href="https://www.techpowerup.com/vgabios/256271/256271" rel="noreferrer" target="_blank">https://www.techpowerup.com/vgabios/256271/256271</a><br>; <br> Now after we've gotten the vBIOS, we need to trim the start of the file to get to the embedded PCI Option ROM.<br> For that we will use my tool "trim_nvidia_vbios" from <a href="https://github.com/9vlc/ptnotes" rel="noreferrer" target="_blank">https://github.com/9vlc/ptnotes</a>; located in progs/trim_nvidia_vbios<br> Compile and run the resulting executable with the first argument being the downloaded vBIOS and the second argument being an output PCI OpROM. Example:<br> $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c<br> $ ./trim.elf techpowerup.rom trimmed-vbios.rom<br> <br> And we come upon a road block: vm-bhyve does not support passing a PCI Option ROMs in "passthru=", so we're going to use "bhyve_options=" instead to define the GPU VGA device:<br> bhyve_options="-s 6:0,passthru,19/0/0,rom=</path/to/trimmed-vbios.rom>"<br> Other devices that do not require a PCI Option ROM can be added with "passthru=". These devices include the rest of the GPU pci functions like audio and usb.<br> <br> !!!! RE-BAR<br> <br> If after everything you do get the GPU to work in the VM at least in the edk2 firmware,<br> Go to your regular UEFI BIOS setup utility and search for options titled "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for Mining".<br> If you do see re-BAR but cannot disable it, you could try using <a href="https://github.com/DavidS95/Smokeless_UMAF" rel="noreferrer" target="_blank">https://github.com/DavidS95/Smokeless_UMAF</a>; to modify locked UEFI settings. Be warned though, smokeless is a heck of an unstable MESS.<br> <br> !!!! UPDATED EDK2<br> <br> As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some rare cases, that might cause issues.<br> I have submitted a package update bug report for sysutils/edk2 to FreeBSD bugzilla over here: <a href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554" rel="noreferrer" target="_blank">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554</a><br>; You can download the patch, apply it to your ports tree, then compile and install the edk2 package with "FLAVOR=bhyve" to get the fresh UEFI firmware.<br> <br> ~~~~<br> <br> Hope I didn't forget anything!<br> I wish you success in the GPU passthrough journey!!<br> - Alexey<br> <br> <br> </blockquote></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOyJeZRbR0rK88OrH0-vfxm_5J-o1Cn7dQ0jbFB%2BmEy79MWkcw>