From nobody Thu Mar 27 10:22:18 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNfqF0SV4z5r7xT for ; Thu, 27 Mar 2025 10:22:33 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Received: from mail-yw1-x1136.google.com (mail-yw1-x1136.google.com [IPv6:2607:f8b0:4864:20::1136]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNfqD1d6Lz3Tyg for ; Thu, 27 Mar 2025 10:22:32 +0000 (UTC) (envelope-from shamim.shahriar@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=gCxunomh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of shamim.shahriar@gmail.com designates 2607:f8b0:4864:20::1136 as permitted sender) smtp.mailfrom=shamim.shahriar@gmail.com Received: by mail-yw1-x1136.google.com with SMTP id 00721157ae682-6fee50bfea5so7434327b3.1 for ; Thu, 27 Mar 2025 03:22:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743070951; x=1743675751; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=cUiJnMPRZZy9OI38gszjHiFxnVMGIihzoE0nSoowfDc=; b=gCxunomhBoNFGJS+ebwYtL3wHVxT3lWGTgm3jq23d4pA/+V1NlWeRC5ARGo+SBUNE4 9fwduOG0AERKmNec3/5A58M1mYragxgFMtFJIqBDBJapF7CcbuXhYOx4v1bEiIGnYGH3 zRi+p6okhRqJHWQ7iseK1Sjpp61lZVMLjbd6M14Ogl4Hrs1ro1wDffoEKb8wOkbc9uWw ucRUxSNef3uPoMq5moMiR8QUsitMz02a9S138p6IucQOhV1XHqv5V4a/8G0O+xrDUfBX O7sD/NTU4X64cAf1r/SOXfNbBoisPav9zj+4AQR4pXimDw+7cO0+uh6ff8wTKkBSnk2b 2CjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743070951; x=1743675751; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cUiJnMPRZZy9OI38gszjHiFxnVMGIihzoE0nSoowfDc=; b=UyeGzr84cZ38W3qqVgPof5Yu3nWZkm6LEUSyx7EjAemRdjarxQZzEMzBtX+vBHvyoA BlJTN1B5SIm7qcOrHiE8tkxSsuNH3rWJtx8UJw6hHzOpOIAgtuz+ednQX5EJR2KrDzFd XbywUR7P5XxzizHgYqArobNR+64TpA+uqKezOv6YR0EsyIAk8NZbic2vrxYkEHKcvTYQ fghJAtNZTUl3XLQKfoDPFv4P5FUXGAyeSinZkNJ0WybUYSrg48Q1cYPnNaO/EJhw8qlh 3vHZ9XR3Qcn0LZmp/BbgsBqEcXtJYev0UC99PQ0kjagRb0RUxE/B3K3lNjvIZvx4qKqK QS4g== X-Gm-Message-State: AOJu0YyHYsmJvAHsA3hnixf6J3SVG60Gf6EmwwFmNZUnOPdcysRymQsw oC0t+A158F9aKDWU1GcBGCKxtiAn2JSWByrEKAj27IEKS7uZHbc3FmWUZF/wwCM9wHAw5iGwjxc 8Wfc7f6LRhn7vvFuVQouytn4CeQ== X-Gm-Gg: ASbGncsolwq1oKvvL+IZsrdsARmlxeWrE9i3oD0KbMSS4WfgR9gQFUMuPcSFgcJhmX6 p7m4asF5RMm1MxO194y645if6X1Sm//vHyww8GLIszo3OU4e6ylBa0pqDSfIIflr4Ktg23hATtf 7FMCNyFSw+ysP5dZjKJVpJ9w== X-Google-Smtp-Source: AGHT+IFPxjmEHy3rOfyXHGBBXzSBD67yBcKbbORxtBy6Or5eI7qGsBYsUDjLvI0ygMcpi0tPfsWnStDlZWXSc2dIjog= X-Received: by 2002:a05:690c:6906:b0:6f9:7ce7:b107 with SMTP id 00721157ae682-702250c41a1mr37409167b3.29.1743070950787; Thu, 27 Mar 2025 03:22:30 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Shamim Shahriar Date: Thu, 27 Mar 2025 10:22:18 +0000 X-Gm-Features: AQ5f1JoVCKeqT0KlPuv7ysDne3RX1qBtnvNXi0VKnVbcBP2157n2HsLKyDMIqfI Message-ID: Subject: Re: In response to "GPU Passthrough with FreeBSD 14.2 bhyve" To: Alex <9vlc@proton.me> Cc: "freebsd-virtualization@FreeBSD.org" Content-Type: multipart/alternative; boundary="000000000000d71a8d0631505401" X-Spamd-Result: default: False [-3.41 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.97)[-0.969]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_LONG(-0.44)[-0.443]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; TAGGED_FROM(0.00)[]; MISSING_XM_UA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1136:from] X-Rspamd-Queue-Id: 4ZNfqD1d6Lz3Tyg X-Spamd-Bar: --- --000000000000d71a8d0631505401 Content-Type: text/plain; charset="UTF-8" Good morning Alex, and thank you for the detailed help, it is very much appreciated. For some reason your email went into google's spam box, I just noticed it this morning. I'll try this as well and keep the list posted as to how it goes. Best regards SS On Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote: > This is a hefty help email for Shamim Shahriar in regards to passing > through Nvidia GPU to a bhyve VM. > > Hi. > I have recently gotten this mailing list message forwarded to me by > Michael Dexter > https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html > and it alone made me sign up here. > > So, here's a big mail containing a bunch of useful GPU passthrough info! > > To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, > let me help you out make it work correctly! > > Parts and short description: > > 1. Pass through the GPU audio and other devices (if exist) > Modern GPUs usually create two PCI devices that you can see in > pciconf - VGA and audio. They come on the same PCI function (i think it is > called that), for example GPU 19/0/0 will also have 19/0/1 for audio and > may have 19/0/2 for usb. > > 2. Give the GPU a VGA BIOS as PCI Option ROM > GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a > trimmed VGA BIOS since the PCI option rom that is used by the VM is nested > inside the ROM file. > (from now on i will refer to the vga bios as vbios) > > 3. Possibly, re-BAR in BIOS? > Some UEFI firmware have a feature called re-BAR and "above 4G > decoding / mining", it makes some GPUs in some configurations not work in > passthru. > > 5. Possibly, Try newer EDK2 > The edk2-bhyve package (provides UEFI firmware for bhyve) on the > repos is still using the 2023 release. You can compile a 2025 version with > my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 > > Full parts and HOWTOs: > > !!!! MISC GPU DEVICES > > As decribed before, pass through all the devices that your GPU creates in > pciconf right after the GPU device on the same PCI function. > All audio, USB, etc. devices belonging to the GPU should be passed through > with it since they are a part of it, else the passthru will NOT work > correctly. > If that is not the case and there is only a single GPU VGA device with > nothing else Nvidia to be found, skip this step. > > !!!! VBIOS > > You first need to get the pure vBIOS file. To do that you can either: > 1. Use a Linux system and dump it with > https://github.com/ilya-zlobintsev/LACT > 2. Use a Windows system and dump it with https://www.techpowerup.com/gpuz > 3. Download one that someone has dumped from their GPU using GPU-z from > https://www.techpowerup.com/vgabios > > If you don't want to install a different OS on the PC you're currently > running, we can just find and download a correct vBIOS from Techpowerup. > But we stumble across an issue! You specified "NVidia Quadro RTX > 6000/8000", so which GPU is it really? Those are two different GPUs with > different specifications! > Thankfully, we have your pciconf output which says "subdevice=0x13d8" and > if we refer to Nvidias documentation, on page 7 over here: > > https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf > we can see that it is a RTX 8000! > Here is an unverified but most likely working vBIOS: > https://www.techpowerup.com/vgabios/256271/256271 > > Now after we've gotten the vBIOS, we need to trim the start of the file to > get to the embedded PCI Option ROM. > For that we will use my tool "trim_nvidia_vbios" from > https://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios > Compile and run the resulting executable with the first argument being the > downloaded vBIOS and the second argument being an output PCI OpROM. Example: > $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c > $ ./trim.elf techpowerup.rom trimmed-vbios.rom > > And we come upon a road block: vm-bhyve does not support passing a PCI > Option ROMs in "passthru=", so we're going to use "bhyve_options=" instead > to define the GPU VGA device: > bhyve_options="-s 6:0,passthru,19/0/0,rom=" > Other devices that do not require a PCI Option ROM can be added with > "passthru=". These devices include the rest of the GPU pci functions like > audio and usb. > > !!!! RE-BAR > > If after everything you do get the GPU to work in the VM at least in the > edk2 firmware, > Go to your regular UEFI BIOS setup utility and search for options titled > "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for > Mining". > If you do see re-BAR but cannot disable it, you could try using > https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI > settings. Be warned though, smokeless is a heck of an unstable MESS. > > !!!! UPDATED EDK2 > > As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some > rare cases, that might cause issues. > I have submitted a package update bug report for sysutils/edk2 to FreeBSD > bugzilla over here: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=285554 > You can download the patch, apply it to your ports tree, then compile and > install the edk2 package with "FLAVOR=bhyve" to get the fresh UEFI firmware. > > ~~~~ > > Hope I didn't forget anything! > I wish you success in the GPU passthrough journey!! > - Alexey > > > --000000000000d71a8d0631505401 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Good morning Alex, and thank you for the detailed help, it= is very much appreciated.

For some reason your email we= nt into google's spam box, I just noticed it this morning. I'll try= this as well and keep the list posted as to how it goes.

Best regards
SS

On Tue, 25 Mar= 2025 at 13:27, Alex <9vlc@proton.me> wrote:
Th= is is a hefty help email for Shamim Shahriar in regards to passing through = Nvidia GPU to a bhyve VM.

Hi.
I have recently gotten this mailing list message forwarded to me by Michael= Dexter https://list= s.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html an= d it alone made me sign up here.

So, here's a big mail containing a bunch of useful GPU passthrough info= !

To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, l= et me help you out make it work correctly!

Parts and short description:

1. Pass through the GPU audio and other devices (if exist)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Modern GPUs usually create two PCI devices that= you can see in pciconf - VGA and audio. They come on the same PCI function= (i think it is called that), for example GPU 19/0/0 will also have 19/0/1 = for audio and may have 19/0/2 for usb.

2. Give the GPU a VGA BIOS as PCI Option ROM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GPUs need a VGA BIOS to work. In addition, Nvid= ia GPUs need a trimmed VGA BIOS since the PCI option rom that is used by th= e VM is nested inside the ROM file.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (from now on i will refer to the vga bios as vb= ios)

3. Possibly, re-BAR in BIOS?
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Some UEFI firmware have a feature called re-BAR= and "above 4G decoding / mining", it makes some GPUs in some con= figurations not work in passthru.

5. Possibly, Try newer EDK2
=C2=A0 =C2=A0 =C2=A0 =C2=A0 The edk2-bhyve package (provides UEFI firmware = for bhyve) on the repos is still using the 2023 release. You can compile a = 2025 version with my patch: https://bugs.f= reebsd.org/bugzilla/show_bug.cgi?id=3D285554

Full parts and HOWTOs:

!!!! MISC GPU DEVICES

As decribed before, pass through all the devices that your GPU creates in p= ciconf right after the GPU device on the same PCI function.
All audio, USB, etc. devices belonging to the GPU should be passed through = with it since they are a part of it, else the passthru will NOT work correc= tly.
If that is not the case and there is only a single GPU VGA device with noth= ing else Nvidia to be found, skip this step.

!!!! VBIOS

You first need to get the pure vBIOS file. To do that you can either:
1. Use a Linux system and dump it with https://github.com/ily= a-zlobintsev/LACT
2. Use a Windows system and dump it with https://www.techpowerup.com= /gpuz
3. Download one that someone has dumped from their GPU using GPU-z from https://www.techpowerup.com/vgabios

If you don't want to install a different OS on the PC you're curren= tly running, we can just find and download a correct vBIOS from Techpowerup= .
But we stumble across an issue! You specified "NVidia Quadro RTX 6000/= 8000", so which GPU is it really? Those are two different GPUs with di= fferent specifications!
Thankfully, we have your pciconf output which says "subdevice=3D0x13d8= " and if we refer to Nvidias documentation, on page 7 over here:
https://www.nvidia.c= om/content/dam/en-zz/Solutions/design-visualization/quadro-product-literatu= re/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf
we can see that it is a RTX 8000!
Here is an unverified but most likely working vBIOS: https://www.techpowerup.com/vgabios/256271/256271

Now after we've gotten the vBIOS, we need to trim the start of the file= to get to the embedded PCI Option ROM.
For that we will use my tool "trim_nvidia_vbios" from https= ://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios
Compile and run the resulting executable with the first argument being the = downloaded vBIOS and the second argument being an output PCI OpROM. Example= :
$ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c
$ ./trim.elf techpowerup.rom trimmed-vbios.rom

And we come upon a road block: vm-bhyve does not support passing a PCI Opti= on ROMs in "passthru=3D", so we're going to use "bhyve_o= ptions=3D" instead to define the GPU VGA device:
bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D</path/to/trimmed-vbi= os.rom>"
Other devices that do not require a PCI Option ROM can be added with "= passthru=3D". These devices include the rest of the GPU pci functions = like audio and usb.

!!!! RE-BAR

If after everything you do get the GPU to work in the VM at least in the ed= k2 firmware,
Go to your regular UEFI BIOS setup utility and search for options titled &q= uot;BAR", "PCI Re-size BAR", "re-BAR", "Above= 4G Encoding" or "Above 4G for Mining".
If you do see re-BAR but cannot disable it, you could try using https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI = settings. Be warned though, smokeless is a heck of an unstable MESS.

!!!! UPDATED EDK2

As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some ra= re cases, that might cause issues.
I have submitted a package update bug report for sysutils/edk2 to FreeBSD b= ugzilla over here: https://bugs.freebsd.or= g/bugzilla/show_bug.cgi?id=3D285554
You can download the patch, apply it to your ports tree, then compile and i= nstall the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UE= FI firmware.

~~~~

Hope I didn't forget anything!
I wish you success in the GPU passthrough journey!!
- Alexey


--000000000000d71a8d0631505401--