From nobody Tue Mar 25 13:26:17 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZMW0L0x6rz5rHgK for ; Tue, 25 Mar 2025 13:26:26 +0000 (UTC) (envelope-from 9vlc@proton.me) Received: from mail-106101.protonmail.ch (mail-106101.protonmail.ch [79.135.106.101]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZMW0H682fz3KF5 for ; Tue, 25 Mar 2025 13:26:23 +0000 (UTC) (envelope-from 9vlc@proton.me) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=proton.me header.s=protonmail header.b=eL18J1Rt; dmarc=pass (policy=quarantine) header.from=proton.me; spf=pass (mx1.freebsd.org: domain of 9vlc@proton.me designates 79.135.106.101 as permitted sender) smtp.mailfrom=9vlc@proton.me DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=proton.me; s=protonmail; t=1742909180; x=1743168380; bh=xchB8yKj79mO2OyRXI+FuBQoTQF86Mba5CrEYj6BJXc=; h=Date:To:From:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector: List-Unsubscribe:List-Unsubscribe-Post; b=eL18J1RtDmFcBiyRf3UTveCrmOimO7S2Oar0LhCM0kH0emKq8sxOdEvs0dXq1QrBK bYhxa2Hts2i0Q1WEgEq/mBww0Qo+WfuKGL4QXydPpIzBZUvwnxm1lHUu5yqpuqDSkH 07dLSumY1Dbxbxrk6WijAzqplo5SolvilH/kIamIBZmmJSj2JFz+jkruSDOBR77myM WtnYq12ecR6fcSf6f7fPAmejtpInOrGvgTeA/zgJGHzq54lqaakPDkAlIw/BrIDs/E qYViv7RjiAQzR6rn0Jc0Xs5YlQtK13FC0F1w8oOtiSE67/pWUEpxKbpkzCy9U41Lyw XdcplrGJMA8OQ== Date: Tue, 25 Mar 2025 13:26:17 +0000 To: "freebsd-virtualization@FreeBSD.org" From: Alex <9vlc@proton.me> Subject: In response to "GPU Passthrough with FreeBSD 14.2 bhyve" Message-ID: Feedback-ID: 117452339:user:proton X-Pm-Message-ID: 57f29f9129675e010c012cc4685eeacefb83536b List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-3.32 / 15.00]; RBL_SENDERSCORE_REPUT_9(-1.00)[79.135.106.101:from]; NEURAL_HAM_SHORT(-1.00)[-0.998]; NEURAL_HAM_MEDIUM(-0.99)[-0.988]; NEURAL_SPAM_LONG(0.77)[0.766]; DMARC_POLICY_ALLOW(-0.50)[proton.me,quarantine]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; R_DKIM_ALLOW(-0.20)[proton.me:s=protonmail]; RWL_MAILSPIKE_GOOD(-0.10)[79.135.106.101:from]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@FreeBSD.org]; FROM_HAS_DN(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[proton.me:+] X-Rspamd-Queue-Id: 4ZMW0H682fz3KF5 X-Spamd-Bar: --- This is a hefty help email for Shamim Shahriar in regards to passing throug= h Nvidia GPU to a bhyve VM. Hi. I have recently gotten this mailing list message forwarded to me by Michael= Dexter https://lists.freebsd.org/archives/freebsd-virtualization/2025-Marc= h/002818.html and it alone made me sign up here. So, here's a big mail containing a bunch of useful GPU passthrough info! To Shamim, I noticed quite a bit of broken stuff in your vm-bhyve config, l= et me help you out make it work correctly! Parts and short description: 1. Pass through the GPU audio and other devices (if exist) =09Modern GPUs usually create two PCI devices that you can see in pciconf -= VGA and audio. They come on the same PCI function (i think it is called th= at), for example GPU 19/0/0 will also have 19/0/1 for audio and may have 19= /0/2 for usb. 2. Give the GPU a VGA BIOS as PCI Option ROM =09GPUs need a VGA BIOS to work. In addition, Nvidia GPUs need a trimmed VG= A BIOS since the PCI option rom that is used by the VM is nested inside the= ROM file. =09(from now on i will refer to the vga bios as vbios) 3. Possibly, re-BAR in BIOS? =09Some UEFI firmware have a feature called re-BAR and "above 4G decoding /= mining", it makes some GPUs in some configurations not work in passthru. 5. Possibly, Try newer EDK2 =09The edk2-bhyve package (provides UEFI firmware for bhyve) on the repos i= s still using the 2023 release. You can compile a 2025 version with my patc= h: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D285554 Full parts and HOWTOs: !!!! MISC GPU DEVICES As decribed before, pass through all the devices that your GPU creates in p= ciconf right after the GPU device on the same PCI function. All audio, USB, etc. devices belonging to the GPU should be passed through = with it since they are a part of it, else the passthru will NOT work correc= tly. If that is not the case and there is only a single GPU VGA device with noth= ing else Nvidia to be found, skip this step. !!!! VBIOS You first need to get the pure vBIOS file. To do that you can either: 1. Use a Linux system and dump it with https://github.com/ilya-zlobintsev/L= ACT 2. Use a Windows system and dump it with https://www.techpowerup.com/gpuz 3. Download one that someone has dumped from their GPU using GPU-z from htt= ps://www.techpowerup.com/vgabios If you don't want to install a different OS on the PC you're currently runn= ing, we can just find and download a correct vBIOS from Techpowerup. But we stumble across an issue! You specified "NVidia Quadro RTX 6000/8000"= , so which GPU is it really? Those are two different GPUs with different sp= ecifications! Thankfully, we have your pciconf output which says "subdevice=3D0x13d8" and= if we refer to Nvidias documentation, on page 7 over here: https://www.nvidia.com/content/dam/en-zz/Solutions/design-visualization/qua= dro-product-literature/NVIDIA-Quadro-RTX-8000-PCIe-Server-Card-PB-FINAL-121= 9.pdf we can see that it is a RTX 8000! Here is an unverified but most likely working vBIOS: https://www.techpoweru= p.com/vgabios/256271/256271 Now after we've gotten the vBIOS, we need to trim the start of the file to = get to the embedded PCI Option ROM. For that we will use my tool "trim_nvidia_vbios" from https://github.com/9v= lc/ptnotes located in progs/trim_nvidia_vbios Compile and run the resulting executable with the first argument being the = downloaded vBIOS and the second argument being an output PCI OpROM. Example= : $ cc -o trim.elf ptnotes/progs/trim_nvidia_vbios/trim.c $ ./trim.elf techpowerup.rom trimmed-vbios.rom And we come upon a road block: vm-bhyve does not support passing a PCI Opti= on ROMs in "passthru=3D", so we're going to use "bhyve_options=3D" instead = to define the GPU VGA device: bhyve_options=3D"-s 6:0,passthru,19/0/0,rom=3D" Other devices that do not require a PCI Option ROM can be added with "passt= hru=3D". These devices include the rest of the GPU pci functions like audio= and usb. !!!! RE-BAR If after everything you do get the GPU to work in the VM at least in the ed= k2 firmware, Go to your regular UEFI BIOS setup utility and search for options titled "B= AR", "PCI Re-size BAR", "re-BAR", "Above 4G Encoding" or "Above 4G for Mini= ng". If you do see re-BAR but cannot disable it, you could try using https://git= hub.com/DavidS95/Smokeless_UMAF to modify locked UEFI settings. Be warned t= hough, smokeless is a heck of an unstable MESS. !!!! UPDATED EDK2 As noted previously, EDK2 on FreeBSD ports is pretty outdated so in some ra= re cases, that might cause issues. I have submitted a package update bug report for sysutils/edk2 to FreeBSD b= ugzilla over here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D2855= 54 You can download the patch, apply it to your ports tree, then compile and i= nstall the edk2 package with "FLAVOR=3Dbhyve" to get the fresh UEFI firmwar= e. ~~~~ Hope I didn't forget anything! I wish you success in the GPU passthrough journey!! - Alexey