From nobody Sun Aug 31 07:43:32 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cF3sR125lz66WFb for ; Sun, 31 Aug 2025 07:43:39 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Received: from mo4-p05-ob.smtp.rzone.de (mo4-p05-ob.smtp.rzone.de [85.215.255.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "*.smtp.rzone.de", Issuer "Telekom Security ServerID OV Class 2 CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cF3sN66Tsz3SpD for ; Sun, 31 Aug 2025 07:43:36 +0000 (UTC) (envelope-from Stephan.Althaus@Duedinghausen.eu) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0002 header.b=kQB2Fnq5; dkim=pass header.d=hoewweken.de header.s=strato-dkim-0003 header.b=gRwPTjtI; dkim=pass header.d=duedinghausen.eu header.s=default header.b=EFgtmwuk; dmarc=pass (policy=reject) header.from=duedinghausen.eu; spf=none (mx1.freebsd.org: domain of Stephan.Althaus@Duedinghausen.eu has no SPF policy when checking 85.215.255.132) smtp.mailfrom=Stephan.Althaus@Duedinghausen.eu; arc=pass ("strato.com:s=strato-dkim-0002:i=1") ARC-Seal: i=1; a=rsa-sha256; t=1756626213; cv=none; d=strato.com; s=strato-dkim-0002; b=oC4KAZA3m2VYk86gFQEb8v+ITrghoPCBiAOeCb+2KIa8yh81H5OFOSigcKkWKE4eMP fzJwMwCxWbzeoVBrmSYy5EFVZeW8JdVrf4/WTwyLcnitsRjNI54h4V5IqPVvJs8Sdhfd ARBFKzmt+iCFumxqW14S6bpQFO1hA1S/nWtWU9JbcEwstW81qnVS+JEBIHyYI4AHN5KS yKz5E/N2KZ3ydkV8ld295ML2VO7Ay7540MugOWJHcuxJGxf0e/kHJXL7Yh8eFCW+9ohJ sdOhQoWDAPue6qub2moZ3aVTveWaiDwLTm1+pwcbIGtpiGJthRmmVX4+dh6nVisgB47P zzBA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; t=1756626213; s=strato-dkim-0002; d=strato.com; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=qaGzaZmY6uYUIXtJVNwoirgSB9MGzrWz5wTZNCsRXgw=; b=X5vAKDWNAhKTThvH6Xrlk+fHs390xf/PYbeAz0RnpJjVZZr3Ew2wfkgRe/FnHmYKDk ldhEMUYveDsBPRx/lNDv0jKnM3AqqXk9V3WuRkKTbjT7It9ZFNiX+rd2BbHBQudDJiNy eYAYLcQKx2a6lZ+E4L981Uz8Xy319sSUAmo3+ji5Hq193JFhvU8ZK9+PIBhoNmVE0SLY VZ8YwfdCD66bvMODaC+/BaQ685eQczKZPFeqigF8lsb2C6v9zSv4YxXUbXERKWMgJI87 8H81dNPmVOJgDzRcPLPbGVsQOP7hRc6w9fOVlwsEfM9m0pVbrXaIAtJHW1x5uWF8Mt4j FQSw== ARC-Authentication-Results: i=1; strato.com; arc=none; dkim=pass header.d="duedinghausen.eu" header.s="default" header.a="rsa-sha256" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; t=1756626213; s=strato-dkim-0002; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=qaGzaZmY6uYUIXtJVNwoirgSB9MGzrWz5wTZNCsRXgw=; b=kQB2Fnq5emQAni9lYvj6O1dkwgdWVlTs8Rjl5PJeuz5F+Cxf+DYG5S3HPw/mAwqwkV dktwJCJsAibgrX/kdw4+TmQySW3y/zFpreILPbmU9L5/6WnhwWr71WXAlSG7hWWgAFm7 OFWcB9v9vcOc9Ll3g0jzedBIhxO273Ur1G+puAmcbOIYlRyvKCubJ31FAT8grS5AdGBF hpiGjAV3AEuiav+HvYw05bG5fb0BCLvVa85IGJN/LzNsMpwWTBHHYNn8m77Ae7LMTW29 osTr28TchJ+lrKub9agq7I6r3rUE4pJDPmPf7JwS/5taNtjj3b7nfKLLToQpocEzP4r+ XeIA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; t=1756626213; s=strato-dkim-0003; d=hoewweken.de; h=In-Reply-To:From:References:To:Subject:Date:Message-ID:Cc:Date:From: Subject:Sender; bh=qaGzaZmY6uYUIXtJVNwoirgSB9MGzrWz5wTZNCsRXgw=; b=gRwPTjtIVo8EAe082HvTqcP0HnYI1nVVR3nhgQwpWt7cvfuwH8GymUiAHkI2yk+IDE ZR6oG9cnNYFyQB+frZBg== X-RZG-AUTH: ":O2kGeEG7b/pS1EW2TmikjLDsYYueHLp2aWg0q38nsxvThUrH35SBpgREu1OVr06TMgNQOGA=" X-RZG-CLASS-ID: mo05 Received: from www.duedinghausen.eu by smtp.strato.de (RZmta 52.1.2 DYNA|AUTH) with ESMTPSA id q9bf2e17V7hXUap (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256 bits)) (Client did not present a certificate) for ; Sun, 31 Aug 2025 09:43:33 +0200 (CEST) Received: from [192.168.2.63] (p5dde9411.dip0.t-ipconnect.de [93.222.148.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange x25519 server-signature ECDSA (secp384r1) server-digest SHA384) (No client certificate requested) (Authenticated sender: steven) by www.duedinghausen.eu (Postfix) with ESMTPSA id 4A5A814EA46 for ; Sun, 31 Aug 2025 09:43:33 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=duedinghausen.eu; s=default; t=1756626213; bh=yaiTDYhJlmC6cw7H34CWA3w+lf9P/XFsBLI4+kkX7XQ=; h=Date:Subject:To:References:From:In-Reply-To; b=EFgtmwuk6aHfci3fd9Nx+apBu9k80p+x6d9csolDmofd2DkLijBaMhUC2OUb4vId0 mfWeGx/nRpuXY2zzo4jU6CvEx5Zm3JisfQ/p9L23sBIFafvVuJhgQS+fc33bMf+G8O fZJeHN8TVrJ7bG5eB6TSd1lAPgpP1hL6UlRu5wSIpbZU09BJp10AKL5IFTl21qKcxR zsa78Vw846RT1NzXYHRh2b7O1PTEnRJRmt05nlCjC623Rf1ZNjCek96HgNRvBuF7yd J3r6lcAfQuxxtq6dPHC4p1LRKO649A/LOgkAmTnsMi7WgSQ3dM17XJgKgx3MU0yIsl Gp0ZEAxmUpCCPIRJ3+Mw9ElVx8tJ6f7996KwkLOufaIbUBQSTptShxo6z0lcAHgp8H eMUwRMFPhca6gr9Fr13qqThGfRFkBN6kWZhKUVg00VpSdchgCdNshL+jTf2JfxFGab o89B2kfL+fLZFsg5JFDno9RiUhqQAfD7/l3dr62imVHTVgGKt18ihfFCJb7l81Q3BI PjL3SSGmq9pWmEK2BDmwmRNt7T1rrh0/pEikFMIbbb95uZ/dY+g85DBl2KgEYi6pZo U8XU/dKWlUkIaRRt2hx0UMfMXHCsmKvuiStx+uoKZFMdFp6zIzOo0hEeA8OR/En4SC VQVUw9JFScB+9Zzf00NwVdFc= Content-Type: multipart/alternative; boundary="------------jjSBjicZVxOhQccmksACXFR6" Message-ID: <346a503c-e83a-4833-ac62-d7969f8728ef@Duedinghausen.eu> Date: Sun, 31 Aug 2025 09:43:32 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) To: "virtualization@freebsd.org" References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <98007d7f-427e-461a-adf1-7cb55f777662@Duedinghausen.eu> <9573a65a-ef25-416b-a5b6-e49b28c6086c@Duedinghausen.eu> <8e7f6904-8188-4941-827b-c874d5da586d@Duedinghausen.eu> Content-Language: en-US From: Stephan Althaus In-Reply-To: Content-Transfer-Encoding: 8bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.42 / 15.00]; URI_COUNT_ODD(1.00)[17]; NEURAL_HAM_LONG(-1.00)[-1.000]; ARC_ALLOW(-1.00)[strato.com:s=strato-dkim-0002:i=1]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.62)[-0.616]; DMARC_POLICY_ALLOW(-0.50)[duedinghausen.eu,reject]; R_DKIM_ALLOW(-0.20)[hoewweken.de:s=strato-dkim-0002,hoewweken.de:s=strato-dkim-0003,duedinghausen.eu:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:6724, ipnet:85.215.255.0/24, country:DE]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[85.215.255.132:from]; RCVD_COUNT_TWO(0.00)[2]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[hoewweken.de:+,duedinghausen.eu:+] X-Rspamd-Queue-Id: 4cF3sN66Tsz3SpD This is a multi-part message in MIME format. --------------jjSBjicZVxOhQccmksACXFR6 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 8/31/25 02:10, Jonathan Vasquez wrote: > Hey all, > > I've updated my blog post with more information and some useful > screenshots: > https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd > > The important stuff to mention is that I've also gotten a chance to > play with vm-bhyve, but for the purposes of making sure we get as > clear information as possible without having to unwrap the vm-bhyve > scripts, I'll continue to use bhyve directly. With that said, some > useful info: > > * The vBIOS is not required when running on Windows as mentioned by > Corvin here: > https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html, > thus I've stopped using it. I was however also able to > successfully use his "echo 1 ..." recommendation to dump the bios > from within Linux (Fedora 42). I actually failed to extract the > bios before (without using amdvbflash) because most likely my > Linux Mint 22.1's Kernel was too old to extract the ROM. When I > tried to do the same exact thing today with Fedora 42, it worked, > and I was able to see the "rom" file from Fedora 42, but not from > Linux Mint 22.1. > * I was able to get the GPU to load without a Code 43 error. But > it's still unusable. I was actually able to get to this state > before but it was inconsistent. Through more research I was able > to get it to be reproducible. There are a few things to mention here: > o The card will always start in a Code 43 failed state > regardless of what I mention below (including if I reboot the > host). > o If you use RDP, the card will always be in a Code 43 state. > o If I use VNC, the card will start in a failed state (as > mentioned above), HOWEVER, you will be able to go into the > Device Manager, Disable the card, and then Enable the card, > and it will be enabled successfully as you can see in my > screenshots. The OS will also give it resources from what we > can see in the Device's Properties' Resources Tab. > > > Given what I mentioned above, I would recommend (as some other people > recommended in other links I've found), to disable the card in the VM > before you shut it down, and then enable it from inside of Windows > when you turn it back on. Although I don't know how necessary this is > given that if I use VNC, even if it starts off in a failed state > (let's say from a fresh host reboot), doing the "device re-nabling > trick" will let it load successfully again. At this point since I've > gotten the card to say it worked, I'm not sure how to tell Windows to > use that as the main GPU when I run my apps. I've looked at some > examples online of using display settings to change the preferred > card, but I don't see it listed there. Am I suppose to see the card in > dxdiag if I'm using VNC? I'm just running the bhyve VM on my server, > and I VNC over from my laptop on the same LAN network. I'm guessing if > things are working correctly I should be able to do that right? I know > there are also some benchmarking apps that show you what GPUs its > detected and maybe allow you to pick one to run a benchmark on. I > tried opening FurMark but it fails saying that it only detects Open GL > 1.1 and all of that... I actually tried installing Mesa for windows to > see what happens and that does help but it's slow and it's clearly not > using the card and most likely doing some sort of software rendering > (maybe like a Windows llvmpipe). > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez > wrote: >> No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and >> passing in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to >> extract some of the "dmesg" and "eglinfo" output. Running "glxinfo" >> also showed that the OpenGL renderer was set to llvmpipe, so software >> rendering :(. "lspci -k" detected it and showed that it was using the >> "amdgpu" kernel driver. Since I'm using vm-bhyve atm I didn't test >> this with the dumped rom, just directly passing them through. I've >> uploaded some of the logs here for now in case anyone wants to take a >> glance at them: >> >> https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/ >> >> Jonathan Vasquez >> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >> Sent with ProtonMail Secure Email >> >> >> On Monday, August 25th, 2025 at 14:37, Robert Cocker >> wrote: >>> Hey group has bhvye core dev baked out an update for this >>> >>> Robert >>> ------------------------------------------------------------------------ >>> *From:* owner-freebsd-virtualization@FreeBSD.org >>> on behalf of Jonathan >>> Vasquez >>> *Sent:* Monday, August 25, 2025 11:35:31 AM >>> *To:* Jonathan Vasquez >>> *Cc:* Stephan Althaus ; >>> virtualization@freebsd.org >>> *Subject:* Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 >>> XT and Windows 10 Pro) >>> I was able to get a fresh Win 10 Pro VM running through vm-bhyve but >>> no progress. Still same Code 43 error as before. I wasn't able to >>> figure out how to pass the |rom=bios.rom|​ through vm-bhyve, I >>> didn't see anything about it in the config.sample file and I also >>> tried to bypass that by adding it in bhyve_options but no go, the VM >>> wouldn't even start if I did that. I see there is a "bhyve_devices" >>> that contains all of the slot devices but I don't see that as an >>> adjustable variable based on the config.sample info. I've pasted my >>> current vm-bhyve template below. My next test is to try the same >>> thing inside a Linux guest to see what happens. >>> >>> ---------------- >>> >>> loader="uefi" >>> cpu="4" >>> cpu_sockets="1" >>> cpu_cores="4" >>> cpu_threads="1" >>> memory="16G" >>> wired_memory="yes" >>> utctime="no" >>> ahci_device_limit="8" >>> disk0_type="nvme" >>> disk0_name="disk0.img" >>> network0_type="e1000" >>> network0_switch="public" >>> passthru0="3/0/0=2:0" >>> passthru1="3/0/1=2:1" >>> #bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom" >>> >>> # Using RDP post-install. >>> graphics="no" >>> graphics_port="5900" >>> graphics_listen="0.0.0.0" >>> graphics_res="1920x1080" >>> graphics_wait="yes" >>> xhci_mouse="yes" >>> uuid="535d54b5-81cb-11f0-93a2-98b78501bda8" >>> network0_mac="58:9c:fc:04:5a:eb" >>> >>> ---------------- >>> >>> Jonathan Vasquez >>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>> Sent with ProtonMail Secure Email >>> >>> >>> On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez >>> wrote: >>>> Unfortunately I don't have an intel machine with an available PCIe >>>> slot. However I'm trying another thing which is to passthrough the >>>> card to a Linux guest. I need to run more experiments but a >>>> preliminary boot of a Linux Mint ISO did show the AMD Radeon RX >>>> 6900 XT card in |lspci|​ and showed that it was using the |amdgpu|​ >>>> driver. I then did a basic |glxgears|​ and it opened up in my VNC >>>> window and showed the FPS counter at over 2000 fps. The only issue >>>> I noticed was when shutting down the live CD environment there were >>>> |amdgpu|​ failure messages where it was failing to write something >>>> to it. I'm currently redoing my "basic bhyve script" and moving it >>>> over to "vm-bhyve" since it seems a lot of people are using that >>>> instead of the more manual approach. This should hopefully help >>>> simplify things for debugging. If the card works on Linux via >>>> passthrough this would also be useful info and can allow me to see >>>> the performance of Cyperpunk 2077, and some other games (even if >>>> I'm not getting 100% of the card's performance). >>>> >>>> Jonathan Vasquez >>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>> Sent with ProtonMail Secure Email >>>> >>>> >>>> On Monday, August 25th, 2025 at 11:00, Stephan Althaus >>>> wrote: >>>>> On 8/25/25 16:56, Jonathan Vasquez wrote: >>>>>> Hey Stephan, >>>>>> >>>>>> Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I >>>>>> also make sure to reboot the host in between runs due to the amd >>>>>> hardware issue. I don't believe 3/0/1 is necessary though based >>>>>> on the examples by Corvin in his talk. He only passed in 3/0/0 >>>>>> (I'm guessing his card also has a 3/0/1 function that he isn't >>>>>> using since that's the audio channel). >>>>>> >>>>>> Jonathan Vasquez >>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>> Sent with ProtonMail Secure Email >>>>>> >>>>>> >>>>>> On Monday, August 25th, 2025 at 10:52, Stephan Althaus >>>>>> >>>>>> wrote: >>>>>>> On 8/25/25 16:47, Stephan Althaus wrote: >>>>>>>> On 8/24/25 23:58, Jonathan Vasquez wrote: >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> I've been experimenting with trying to get my server's >>>>>>>>> dedicated AMD Radeon RX 6900 XT working on a bhyve Windows 10 >>>>>>>>> Pro client for gaming purposes. I've been reading a bunch of >>>>>>>>> blog posts, sites, files, and videos (thanks Corvin) to try >>>>>>>>> and piece things together. I've been able to get the VM to >>>>>>>>> recognize the card, but still stuck on the Code 43 issue (with >>>>>>>>> or without using the vBIOS I've extracted from my dedicated >>>>>>>>> card and also ensuring any options needed to be on in the UEFI >>>>>>>>> settings are enabled). This machine is running on an AMD Ryzen >>>>>>>>> 9 7950X 16-Core Processor, so it's a full AMD setup, no Intel >>>>>>>>> stuff. It technically should be relatively "easy" to get AMD >>>>>>>>> working from what I've read (excluding the known AMD Hardware >>>>>>>>> reset issue). I've documented everything I've done so far, and >>>>>>>>> will be documenting more and simplifying the instructions as I >>>>>>>>> make progress. Please let me know if you see any thing weird >>>>>>>>> or anything I could try based on what I've rewritten. I'll try >>>>>>>>> to give it a go and provide feedback. Thank you! >>>>>>>>> >>>>>>>>> https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd >>>>>>>>> >>>>>>>>> >>>>>>>>> Jonathan Vasquez >>>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>>>>> Sent with ProtonMail Secure Email >>>>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, >>>>>>>> host is a intel system. >>>>>>>> >>>>>>>> i think you have to passthrough the 3/0/1 device as well! >>>>>>>> >>>>>>>> The bios.rom is extracted from this card in place ? >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Stephan >>>>>>>> >>>>>>>> >>>>>>> Ah, and be shure to reboot the whole setup, inclusive host, for >>>>>>> every new test-case, >>>>>>> at least as long at is not guaranteed that we can live without.. >>>>>>> >>>>>>> Regars, >>>>>>> Stephan >>>>>>> >>>>>>> >>>>>> >>>>> Hi! >>>>> >>>>> My only next assumption would be that this only does not work on >>>>> AMD host. >>>>> >>>>> You don't have an intel system to test, you would havedone it, if >>>>> so.. (?) >>>>> >>>>> Stephan >>>>> >>>>> >>>> >>> >> > Hello! I have a successful GPU passthrough of an AMD RX6400 to Win10 on an Intel host. Just want to share some Infos + no problem with RDP here, that's my use case + i used GPU-Z within the vm to check if the gpu is actually useable, it shows GPU frequency > 0 , and OpenGL >1.1 when it is. + i have thr ROM in my bhyve start script, the vm has no problem with it. i don't use vm-bhyve so far. - your win 10 vm is EFI install ? + yes, mesa is software rendering, sort of 'software GPU' It seems to me that the only noticeable difference between our systems is that mine is an intel host.. Regards, Stephan --------------jjSBjicZVxOhQccmksACXFR6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 8/31/25 02:10, Jonathan Vasquez wrote:
Hey all,

I've updated my blog post with more information and some useful screenshots: https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd

The important stuff to mention is that I've also gotten a chance to play with vm-bhyve, but for the purposes of making sure we get as clear information as possible without having to unwrap the vm-bhyve scripts, I'll continue to use bhyve directly. With that said, some useful info:

  • The vBIOS is not required when running on Windows as mentioned by Corvin here: https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html, thus I've stopped using it. I was however also able to successfully use his "echo 1 ..." recommendation to dump the bios from within Linux (Fedora 42). I actually failed to extract the bios before (without using amdvbflash) because most likely my Linux Mint 22.1's Kernel was too old to extract the ROM. When I tried to do the same exact thing today with Fedora 42, it worked, and I was able to see the "rom" file from Fedora 42, but not from Linux Mint 22.1.
  • I was able to get the GPU to load without a Code 43 error. But it's still unusable. I was actually able to get to this state before but it was inconsistent. Through more research I was able to get it to be reproducible. There are a few things to mention here:
    • The card will always start in a Code 43 failed state regardless of what I mention below (including if I reboot the host).
    • If you use RDP, the card will always be in a Code 43 state.
    • If I use VNC, the card will start in a failed state (as mentioned above), HOWEVER, you will be able to go into the Device Manager, Disable the card, and then Enable the card, and it will be enabled successfully as you can see in my screenshots. The OS will also give it resources from what we can see in the Device's Properties' Resources Tab.

Given what I mentioned above, I would recommend (as some other people recommended in other links I've found), to disable the card in the VM before you shut it down, and then enable it from inside of Windows when you turn it back on. Although I don't know how necessary this is given that if I use VNC, even if it starts off in a failed state (let's say from a fresh host reboot), doing the "device re-nabling trick" will let it load successfully again. At this point since I've gotten the card to say it worked, I'm not sure how to tell Windows to use that as the main GPU when I run my apps. I've looked at some examples online of using display settings to change the preferred card, but I don't see it listed there. Am I suppose to see the card in dxdiag if I'm using VNC? I'm just running the bhyve VM on my server, and I VNC over from my laptop on the same LAN network. I'm guessing if things are working correctly I should be able to do that right? I know there are also some benchmarking apps that show you what GPUs its detected and maybe allow you to pick one to run a benchmark on. I tried opening FurMark but it fails saying that it only detects Open GL 1.1 and all of that... I actually tried installing Mesa for windows to see what happens and that does help but it's slow and it's clearly not using the card and most likely doing some sort of software rendering (maybe like a Windows llvmpipe).

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez <jon@xyinn.org> wrote:
No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and passing in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to extract some of the "dmesg" and "eglinfo" output. Running "glxinfo" also showed that the OpenGL renderer was set to llvmpipe, so software rendering :(. "lspci -k" detected it and showed that it was using the "amdgpu" kernel driver. Since I'm using vm-bhyve atm I didn't test this with the dumped rom, just directly passing them through. I've uploaded some of the logs here for now in case anyone wants to take a glance at them:


Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Monday, August 25th, 2025 at 14:37, Robert Cocker <rcocker87@gmail.com> wrote:
Hey group has bhvye core dev baked out an update for this 

Robert
From: owner-freebsd-virtualization@FreeBSD.org <owner-freebsd-virtualization@FreeBSD.org> on behalf of Jonathan Vasquez <jon@xyinn.org>
Sent: Monday, August 25, 2025 11:35:31 AM
To: Jonathan Vasquez <jon@xyinn.org>
Cc: Stephan Althaus <Stephan.Althaus@Duedinghausen.eu>; virtualization@freebsd.org <virtualization@freebsd.org>
Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)
 
I was able to get a fresh Win 10 Pro VM running through vm-bhyve but no progress. Still same Code 43 error as before. I wasn't able to figure out how to pass the rom=bios.rom​ through vm-bhyve, I didn't see anything about it in the config.sample file and I also tried to bypass that by adding it in bhyve_options but no go, the VM wouldn't even start if I did that. I see there is a "bhyve_devices" that contains all of the slot devices but I don't see that as an adjustable variable based on the config.sample info. I've pasted my current vm-bhyve template below. My next test is to try the same thing inside a Linux guest to see what happens.

----------------

loader="uefi"    
cpu="4"    
cpu_sockets="1"    
cpu_cores="4"    
cpu_threads="1"    
memory="16G"    
wired_memory="yes"    
   
utctime="no"    
ahci_device_limit="8"    
   
disk0_type="nvme"    
disk0_name="disk0.img"    
   
network0_type="e1000"    
network0_switch="public"    
   
passthru0="3/0/0=2:0"    
passthru1="3/0/1=2:1"    
   
#bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom"    

# Using RDP post-install.   
graphics="no"    
graphics_port="5900"    
graphics_listen="0.0.0.0"    
graphics_res="1920x1080"    
graphics_wait="yes"    
   
xhci_mouse="yes"    
uuid="535d54b5-81cb-11f0-93a2-98b78501bda8"    
network0_mac="58:9c:fc:04:5a:eb"   

----------------

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez <jon@xyinn.org> wrote:
Unfortunately I don't have an intel machine with an available PCIe slot. However I'm trying another thing which is to passthrough the card to a Linux guest. I need to run more experiments but a preliminary boot of a Linux Mint ISO did show the AMD Radeon RX 6900 XT card in lspci​ and showed that it was using the amdgpu​ driver. I then did a basic glxgears​ and it opened up in my VNC window and showed the FPS counter at over 2000 fps. The only issue I noticed was when shutting down the live CD environment there were amdgpu​ failure messages where it was failing to write something to it. I'm currently redoing my "basic bhyve script" and moving it over to "vm-bhyve" since it seems a lot of people are using that instead of the more manual approach. This should hopefully help simplify things for debugging. If the card works on Linux via passthrough this would also be useful info and can allow me to see the performance of Cyperpunk 2077, and some other games (even if I'm not getting 100% of the card's performance).

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Monday, August 25th, 2025 at 11:00, Stephan Althaus <Stephan.Althaus@Duedinghausen.eu> wrote:
On 8/25/25 16:56, Jonathan Vasquez wrote:
Hey Stephan,

Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I also make sure to reboot the host in between runs due to the amd hardware issue. I don't believe 3/0/1 is necessary though based on the examples by Corvin in his talk. He only passed in 3/0/0 (I'm guessing his card also has a 3/0/1 function that he isn't using since that's the audio channel).

Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email


On Monday, August 25th, 2025 at 10:52, Stephan Althaus <Stephan.Althaus@Duedinghausen.eu> wrote:
On 8/25/25 16:47, Stephan Althaus wrote:
On 8/24/25 23:58, Jonathan Vasquez wrote:
Hey all,

I've been experimenting with trying to get my server's dedicated AMD Radeon RX 6900 XT working on a bhyve Windows 10 Pro client for gaming purposes. I've been reading a bunch of blog posts, sites, files, and videos (thanks Corvin) to try and piece things together. I've been able to get the VM to recognize the card, but still stuck on the Code 43 issue (with or without using the vBIOS I've extracted from my dedicated card and also ensuring any options needed to be on in the UEFI settings are enabled). This machine is running on an AMD Ryzen 9 7950X 16-Core Processor, so it's a full AMD setup, no Intel stuff. It technically should be relatively "easy" to get AMD working from what I've read (excluding the known AMD Hardware reset issue). I've documented everything I've done so far, and will be documenting more and simplifying the instructions as I make progress. Please let me know if you see any thing weird or anything I could try based on what I've rewritten. I'll try to give it a go and provide feedback. Thank you!


Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email

Hi!

i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, host is a intel system.

i think you have to passthrough the 3/0/1 device as well!

The bios.rom is extracted from this card in place ?


Regards,
Stephan


Ah, and be shure to reboot the whole setup, inclusive host, for every new test-case,
at least as long at is not guaranteed that we can live without..

Regars,
Stephan



Hi!

My only next assumption would be that this only does not work on AMD host.

You don't have an intel system to test, you would havedone it, if so.. (?)

Stephan






Hello!

I have a successful GPU passthrough of an AMD RX6400 to Win10 on an Intel host.

Just want to share some Infos

+ no problem with RDP here, that's my use case

+ i used GPU-Z within the vm to check if the gpu is actually useable, it shows GPU frequency > 0 , and OpenGL >1.1 when it is.

+ i have thr ROM in my bhyve start script, the vm has no problem with it. i don't use vm-bhyve so far.

- your win 10 vm is EFI install ?

+ yes, mesa is software rendering, sort of 'software GPU' 

It seems to me that the only noticeable difference between our systems is that mine is an intel host..

Regards,
Stephan

--------------jjSBjicZVxOhQccmksACXFR6--