From nobody Thu Mar 27 10:37:26 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZNg986nL3z5r904 for ; Thu, 27 Mar 2025 10:38:04 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZNg981Sd1z3ZW2 for ; Thu, 27 Mar 2025 10:38:04 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=C+PQm+5W; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::62a as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pl1-x62a.google.com with SMTP id d9443c01a7336-227d6b530d8so15565375ad.3 for ; Thu, 27 Mar 2025 03:38:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1743071883; x=1743676683; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=jBoOVGqT6Ip57UzpxL5s7JLChKH43ndtcOfzqI57IIo=; b=C+PQm+5WsUvia+pzkthuKfILgeMHtbSy/QoDEq/A/aT4IpWnrfYb5SBQG8a05pJtA+ KqY9mRKep3BVaYSPcoQEJg7xngAvIPSCmg9uVuRZh6VYfiuYWkiAc3OFS1Xx2wJU64CW zG2CeuLeRUveLKTvJX+OGAnRInHmEWu1Ndo24rtgGZYipnG21QlOl39eDuCR6W4Yc3ik mjlSfuB1DQPLfSd5M/SIpFuLPvGg1LhPVCOqGeO1nghi0baPy+LPi6JQ9RRXqA3I3DdK Fg6P6AFt17Wxjt+OyPofgmCMo+jlDOSYUqtLW2U03wtgvsHJHwpfish5zUirQP/nD/2N 6mIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1743071883; x=1743676683; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jBoOVGqT6Ip57UzpxL5s7JLChKH43ndtcOfzqI57IIo=; b=hNAHbkuuV3/WIzBFk3JGk2FMERPpm3WF2ZX1TAiTYWJ2mheX5ts5hCrPlcvZDZsq8P asizWOSk1cOHiDqQAW33TahM/Ojdw8wwYLijOWIwNCYO7kAC0MObiY8lSLEjPeKmTlN4 E407zfSsTP92BemE+lrq8oos3DNLBwbpN1T2i91Lbv15aA1FhPROLWU1AY23M0C0a/Ah 3DfQKziwe8wT07F7gDOLiadp/2o34dxAMX3kMh03MjDd3GtrQe9ne6DY4vHU89pKLlRu LeLuicc5hdHkpysNT7isSXV1GM3Qo2QWd1l1iu6yvSSMr4eCOZ2nPgm5RvZlIyUNMj6i FNRw== X-Forwarded-Encrypted: i=1; AJvYcCXErkKKAdqzWMFxsfRGtgXnWnoJLwA+nDokkm1ucLEyrS2WQvZPQ9s+WEEceoGmLCelxkYiqSRf2g1wiGbv8SpDZpPbslxf@freebsd.org X-Gm-Message-State: AOJu0Ywi+/PNdkCQKQ19tlxpIAKpiqczIS0LeWBrYVjQZ1mzI1604BqM JwmXu/VSs/3dySAkzzUTbNTRBLdgV4UHQG/IGjQzy1C021uKcTGkdVXeoio2rt1Ddm+YJjy04P/ 2o2dXdrcR14kw50eMypxal0aX+qL339mb X-Gm-Gg: ASbGncsd76vqjXUDfM8tpXCaQRH8jBlEugm527mBOn+Uuf7j4FkyMJeqpqd3n+am8f5 LcJ7nlVIv218YSDzDxMwiVI7tAB49eT7cZ1VMAV6W8q9VWt16TPBq0bGRzxt+LvjAVFRvflBTgg nI1Oy61DYbaNBkfho5uk164TpUI4k= X-Google-Smtp-Source: AGHT+IEmC4c5ERv0jTpnKSIwFwsdyl7uGd7x/wL9txPipcH4tsdz8lGFXT5Kx9UuljlBLKsR8Mk1lcYgVNHLLmDNkTk= X-Received: by 2002:a17:90b:2f4d:b0:2fe:8a84:e033 with SMTP id 98e67ed59e1d1-303a7b5a137mr4820698a91.2.1743071882581; Thu, 27 Mar 2025 03:38:02 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Thu, 27 Mar 2025 11:37:26 +0100 X-Gm-Features: AQ5f1JpRm8Jn8y2TQs_gqKT4U7F-XkuiIOrU0wmUqZKHQf127WShs3Nqq8GXGco Message-ID: Subject: Re: In response to "GPU Passthrough with FreeBSD 14.2 bhyve" To: Shamim Shahriar Cc: Alex <9vlc@proton.me>, "freebsd-virtualization@FreeBSD.org" Content-Type: multipart/alternative; boundary="0000000000006123850631508c90" X-Spamd-Result: default: False [-3.80 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997]; NEURAL_HAM_SHORT(-0.99)[-0.988]; NEURAL_HAM_LONG(-0.81)[-0.812]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; TO_DN_EQ_ADDR_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::62a:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MID_RHS_MATCH_FROMTLD(0.00)[]; TAGGED_RCPT(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4ZNg981Sd1z3ZW2 X-Spamd-Bar: --- --0000000000006123850631508c90 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Shamim. Without applying the patches that I sent to you will not be able to pass through your nVidia Gpu. Probably your problem is due to this. You can try what you want,but the risk is that you go nowhere. If I were you,I would try first to apply the patches of Corvin. On Thu, Mar 27, 2025 at 11:22=E2=80=AFAM Shamim Shahriar wrote: > Good morning Alex, and thank you for the detailed help, it is very much > appreciated. > > For some reason your email went into google's spam box, I just noticed it > this morning. I'll try this as well and keep the list posted as to how it > goes. > > Best regards > SS > > On Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote: > >> This is a hefty help email for Shamim Shahriar in regards to passing >> through Nvidia GPU to a bhyve VM. >> >> Hi. >> I have recently gotten this mailing list message forwarded to me by >> Michael Dexter >> https://lists.freebsd.org/archives/freebsd-virtualization/2025-March/002= 818.html >> and it alone made me sign up here. >> >> So, here's a big mail containing a bunch of useful GPU passthrough info! >> >> To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config= , >> let me help you out make it work correctly! >> >> Parts and short description: >> >> 1. Pass through the GPU audio and other devices (if exist) >> Modern GPUs usually create two PCI devices that you can see in >> pciconf - VGA and audio. They come on the same PCI function (i think it = is >> called that), for example GPU 19/0/0 will also have 19/0/1 for audio and >> may have 19/0/2 for usb. >> >> 2. Give the GPU a VGA BIOS as PCI Option ROM >> GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a >> trimmed VGA BIOS since the PCI option rom that is used by the VM is nest= ed >> inside the ROM file. >> (from now on i will refer to the vga bios as vbios) >> >> 3. Possibly, re-BAR in BIOS? >> Some UEFI firmware have a feature called re-BAR and "above 4G >> decoding / mining", it makes some GPUs in some configurations not work i= n >> passthru. >> >> 5. Possibly, Try newer EDK2 >> The edk2-bhyve package (provides UEFI firmware for bhyve) on the >> repos is still using the 2023 release. You can compile a 2025 version wi= th >> my patch: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285554 >> >> Full parts and HOWTOs: >> >> !!!! MISC GPU DEVICES >> >> As decribed before, pass through all the devices that your GPU creates i= n >> pciconf right after the GPU device on the same PCI function. >> All audio, USB, etc. devices belonging to the GPU should be passed >> through with it since they are a part of it, else the passthru will NOT >> work correctly. >> If that is not the case and there is only a single GPU VGA device with >> nothing else Nvidia to be found, skip this step. >> >> !!!! VBIOS >> >> You first need to get the pure vBIOS file. To do that you can either: >> 1. Use a Linux system and dump it with >> https://github.com/ilya-zlobintsev/LACT >> 2. Use a Windows system and dump it with https://www.techpowerup.com/gpu= z >> 3. Download one that someone has dumped from their GPU using GPU-z from >> https://www.techpowerup.com/vgabios >> >> If you don't want to install a different OS on the PC you're currently >> running, we can just find and download a correct vBIOS from Techpowerup. >> But we stumble across an issue! You specified "NVidia Quadro RTX >> 6000/8000", so which GPU is it really? Those are two different GPUs with >> different specifications! >> Thankfully, we have your pciconf output which says "subdevice=3D0x13d8" = and >> if we refer to Nvidias documentation, on page 7 over here: >> >> https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/= quadro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-= 1219.pdf >> we can see that it is a RTX 8000! >> Here is an unverified but most likely working vBIOS: >> https://www.techpowerup.com/vgabios/256271/256271 >> >> Now after we've gotten the vBIOS, we need to trim the start of the file >> to get to the embedded PCI Option ROM. >> For that we will use my tool "trim_nvidia_vbios" from >> https://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios >> Compile and run the resulting executable with the first argument being >> the downloaded vBIOS and the second argument being an output PCI OpROM. >> Example: >> $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c >> $ ./trim.elf techpowerup.rom trimmed-vbios.rom >> >> And we come upon a road block: vm-bhyve does not support passing a PCI >> Option ROMs in "passthru=3D", so we're going to use "bhyve_options=3D" i= nstead >> to define the GPU VGA device: >> bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D" >> Other devices that do not require a PCI Option ROM can be added with >> "passthru=3D". These devices include the rest of the GPU pci functions l= ike >> audio and usb. >> >> !!!! RE-BAR >> >> If after everything you do get the GPU to work in the VM at least in the >> edk2 firmware, >> Go to your regular UEFI BIOS setup utility and search for options titled >> "BAR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for >> Mining". >> If you do see re-BAR but cannot disable it, you could try using >> https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI >> settings. Be warned though, smokeless is a heck of an unstable MESS. >> >> !!!! UPDATED EDK2 >> >> As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some >> rare cases, that might cause issues. >> I have submitted a package update bug report for sysutils/edk2 to FreeBS= D >> bugzilla over here: >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285554 >> You can download the patch, apply it to your ports tree, then compile an= d >> install the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UEFI fir= mware. >> >> ~~~~ >> >> Hope I didn't forget anything! >> I wish you success in the GPU passthrough journey!! >> - Alexey >> >> >> --=20 Mario. --0000000000006123850631508c90 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Shamim.

Without applyi= ng the patches that I sent to you will not be able to pass through your nVi= dia Gpu. Probably your problem is due to this. You can try what you want,bu= t the risk is that you go nowhere. If I were you,I would try first to apply= the patches of Corvin.

On Thu, Mar 27, 2025 at= 11:22=E2=80=AFAM Shamim Shahriar <shamim.shahriar@gmail.com> wrote:
Good morning Alex, and th= ank you for the detailed help, it is very much appreciated.

<= div>For some reason your email went into google's spam box, I just noti= ced it this morning. I'll try this as well and keep the list posted as = to how it goes.

Best regards
SS

On = Tue, 25 Mar 2025 at 13:27, Alex <9vlc@proton.me> wrote:
This is a hefty help email for Shamim Shahriar i= n regards to passing through Nvidia GPU to a bhyve VM.

Hi.
I have recently gotten this mailing list message forwarded to me by Michael= Dexter https://list= s.freebsd.org/archives/freebsd-virtualization/2025-March/002818.html an= d it alone made me sign up here.

So, here's a big mail containing a bunch of useful GPU passthrough info= !

To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, l= et me help you out make it work correctly!

Parts and short description:

1. Pass through the GPU audio and other devices (if exist)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Modern GPUs usually create two PCI devices that= you can see in pciconf - VGA and audio. They come on the same PCI function= (i think it is called that), for example GPU 19/0/0 will also have 19/0/1 = for audio and may have 19/0/2 for usb.

2. Give the GPU a VGA BIOS as PCI Option ROM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 GPUs need a VGA BIOS to work. In addition, Nvid= ia GPUs need a trimmed VGA BIOS since the PCI option rom that is used by th= e VM is nested inside the ROM file.
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (from now on i will refer to the vga bios as vb= ios)

3. Possibly, re-BAR in BIOS?
=C2=A0 =C2=A0 =C2=A0 =C2=A0 Some UEFI firmware have a feature called re-BAR= and "above 4G decoding / mining", it makes some GPUs in some con= figurations not work in passthru.

5. Possibly, Try newer EDK2
=C2=A0 =C2=A0 =C2=A0 =C2=A0 The edk2-bhyve package (provides UEFI firmware = for bhyve) on the repos is still using the 2023 release. You can compile a = 2025 version with my patch: https://bugs.f= reebsd.org/bugzilla/show_bug.cgi?id=3D285554

Full parts and HOWTOs:

!!!! MISC GPU DEVICES

As decribed before, pass through all the devices that your GPU creates in p= ciconf right after the GPU device on the same PCI function.
All audio, USB, etc. devices belonging to the GPU should be passed through = with it since they are a part of it, else the passthru will NOT work correc= tly.
If that is not the case and there is only a single GPU VGA device with noth= ing else Nvidia to be found, skip this step.

!!!! VBIOS

You first need to get the pure vBIOS file. To do that you can either:
1. Use a Linux system and dump it with https://github.com/ily= a-zlobintsev/LACT
2. Use a Windows system and dump it with https://www.techpowerup.com= /gpuz
3. Download one that someone has dumped from their GPU using GPU-z from https://www.techpowerup.com/vgabios

If you don't want to install a different OS on the PC you're curren= tly running, we can just find and download a correct vBIOS from Techpowerup= .
But we stumble across an issue! You specified "NVidia Quadro RTX 6000/= 8000", so which GPU is it really? Those are two different GPUs with di= fferent specifications!
Thankfully, we have your pciconf output which says "subdevice=3D0x13d8= " and if we refer to Nvidias documentation, on page 7 over here:
https://www.nvidia.c= om/content/dam/en-zz/Solutions/design-visualization/quadro-product-literatu= re/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-1219.pdf
we can see that it is a RTX 8000!
Here is an unverified but most likely working vBIOS: https://www.techpowerup.com/vgabios/256271/256271

Now after we've gotten the vBIOS, we need to trim the start of the file= to get to the embedded PCI Option ROM.
For that we will use my tool "trim_nvidia_vbios" from https= ://github.com/9vlc/ptnotes located in progs/trim_nvidia_vbios
Compile and run the resulting executable with the first argument being the = downloaded vBIOS and the second argument being an output PCI OpROM. Example= :
$ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c
$ ./trim.elf techpowerup.rom trimmed-vbios.rom

And we come upon a road block: vm-bhyve does not support passing a PCI Opti= on ROMs in "passthru=3D", so we're going to use "bhyve_o= ptions=3D" instead to define the GPU VGA device:
bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D</path/to/trimmed-vbi= os.rom>"
Other devices that do not require a PCI Option ROM can be added with "= passthru=3D". These devices include the rest of the GPU pci functions = like audio and usb.

!!!! RE-BAR

If after everything you do get the GPU to work in the VM at least in the ed= k2 firmware,
Go to your regular UEFI BIOS setup utility and search for options titled &q= uot;BAR", "PCI Re-size BAR", "re-BAR", "Above= 4G Encoding" or "Above 4G for Mining".
If you do see re-BAR but cannot disable it, you could try using https://github.com/DavidS95/Smokeless_UMAF to modify locked UEFI = settings. Be warned though, smokeless is a heck of an unstable MESS.

!!!! UPDATED EDK2

As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some ra= re cases, that might cause issues.
I have submitted a package update bug report for sysutils/edk2 to FreeBSD b= ugzilla over here: https://bugs.freebsd.or= g/bugzilla/show_bug.cgi?id=3D285554
You can download the patch, apply it to your ports tree, then compile and i= nstall the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UE= FI firmware.

~~~~

Hope I didn't forget anything!
I wish you success in the GPU passthrough journey!!
- Alexey




--
Ma= rio.
--0000000000006123850631508c90--