Date: Sun, 31 Aug 2025 09:43:32 +0200 From: Stephan Althaus <Stephan.Althaus@Duedinghausen.eu> To: "virtualization@freebsd.org" <virtualization@freebsd.org> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <346a503c-e83a-4833-ac62-d7969f8728ef@Duedinghausen.eu> In-Reply-To: <FIR4iioBciiFZPGzeCeKoY_RKIdxeWWraK2Dr98ag7G7X6n2u5CWtBc2GT7CAzc9sq9JUREXfr7tbj9Bpd341cp7YxJFyM04FczcztnSb_I=@xyinn.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <98007d7f-427e-461a-adf1-7cb55f777662@Duedinghausen.eu> <9573a65a-ef25-416b-a5b6-e49b28c6086c@Duedinghausen.eu> <obPmbtHAn7PsdPlHCvUijyGOcJYzN8G0-VWTOzZnDzOnPAZcy5PNe6hU3JUSjuKcWWfpt8HzTQI47DdMymQP89JKuTIEd7uMHZf2xx_mtcM=@xyinn.org> <8e7f6904-8188-4941-827b-c874d5da586d@Duedinghausen.eu> <qL4a3FqFmcZXpCOOozFH1j4h344deHkH5YBeVAcnZFvGDJI-i6ogV50uZ-xQioPf1C9MBu-ShQLHJnT-BWFgEOr27F8hNq3MmkuiWyDu0xY=@xyinn.org> <GXXvtGkj6StE4BIkkGS7gd01j9K7K3IIGkBOHHqgomnZuNxo-5Sz4WjNKV6HYi5HhPXeBIwsdkAuR71i7hJ8QEPaH4bqkhrTIaA2ti0IXoE=@xyinn.org> <MN2PR20MB3168842FD4AAADEB8A29DC5DFE3EA@MN2PR20MB3168.namprd20.prod.outlook.com> <fb59ninGeo2cCSN7tULNr1ZaXzLFaZn1Lh_SuGWMBCv63RAbqgZetMhn63dppXM7quUBuGF8U9N8-furgb-088Gn-LHr9-e9JYUPvHIvuqU=@xyinn.org> <FIR4iioBciiFZPGzeCeKoY_RKIdxeWWraK2Dr98ag7G7X6n2u5CWtBc2GT7CAzc9sq9JUREXfr7tbj9Bpd341cp7YxJFyM04FczcztnSb_I=@xyinn.org>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On 8/31/25 02:10, Jonathan Vasquez wrote: > Hey all, > > I've updated my blog post with more information and some useful > screenshots: > https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd > > The important stuff to mention is that I've also gotten a chance to > play with vm-bhyve, but for the purposes of making sure we get as > clear information as possible without having to unwrap the vm-bhyve > scripts, I'll continue to use bhyve directly. With that said, some > useful info: > > * The vBIOS is not required when running on Windows as mentioned by > Corvin here: > https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html, > thus I've stopped using it. I was however also able to > successfully use his "echo 1 ..." recommendation to dump the bios > from within Linux (Fedora 42). I actually failed to extract the > bios before (without using amdvbflash) because most likely my > Linux Mint 22.1's Kernel was too old to extract the ROM. When I > tried to do the same exact thing today with Fedora 42, it worked, > and I was able to see the "rom" file from Fedora 42, but not from > Linux Mint 22.1. > * I was able to get the GPU to load without a Code 43 error. But > it's still unusable. I was actually able to get to this state > before but it was inconsistent. Through more research I was able > to get it to be reproducible. There are a few things to mention here: > o The card will always start in a Code 43 failed state > regardless of what I mention below (including if I reboot the > host). > o If you use RDP, the card will always be in a Code 43 state. > o If I use VNC, the card will start in a failed state (as > mentioned above), HOWEVER, you will be able to go into the > Device Manager, Disable the card, and then Enable the card, > and it will be enabled successfully as you can see in my > screenshots. The OS will also give it resources from what we > can see in the Device's Properties' Resources Tab. > > > Given what I mentioned above, I would recommend (as some other people > recommended in other links I've found), to disable the card in the VM > before you shut it down, and then enable it from inside of Windows > when you turn it back on. Although I don't know how necessary this is > given that if I use VNC, even if it starts off in a failed state > (let's say from a fresh host reboot), doing the "device re-nabling > trick" will let it load successfully again. At this point since I've > gotten the card to say it worked, I'm not sure how to tell Windows to > use that as the main GPU when I run my apps. I've looked at some > examples online of using display settings to change the preferred > card, but I don't see it listed there. Am I suppose to see the card in > dxdiag if I'm using VNC? I'm just running the bhyve VM on my server, > and I VNC over from my laptop on the same LAN network. I'm guessing if > things are working correctly I should be able to do that right? I know > there are also some benchmarking apps that show you what GPUs its > detected and maybe allow you to pick one to run a benchmark on. I > tried opening FurMark but it fails saying that it only detects Open GL > 1.1 and all of that... I actually tried installing Mesa for windows to > see what happens and that does help but it's slow and it's clearly not > using the card and most likely doing some sort of software rendering > (maybe like a Windows llvmpipe). > > Jonathan Vasquez > PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 > Sent with ProtonMail Secure Email > > > On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez > <jon@xyinn.org> wrote: >> No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and >> passing in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to >> extract some of the "dmesg" and "eglinfo" output. Running "glxinfo" >> also showed that the OpenGL renderer was set to llvmpipe, so software >> rendering :(. "lspci -k" detected it and showed that it was using the >> "amdgpu" kernel driver. Since I'm using vm-bhyve atm I didn't test >> this with the dumped rom, just directly passing them through. I've >> uploaded some of the logs here for now in case anyone wants to take a >> glance at them: >> >> https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/ >> >> Jonathan Vasquez >> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >> Sent with ProtonMail Secure Email >> >> >> On Monday, August 25th, 2025 at 14:37, Robert Cocker >> <rcocker87@gmail.com> wrote: >>> Hey group has bhvye core dev baked out an update for this >>> >>> Robert >>> ------------------------------------------------------------------------ >>> *From:* owner-freebsd-virtualization@FreeBSD.org >>> <owner-freebsd-virtualization@FreeBSD.org> on behalf of Jonathan >>> Vasquez <jon@xyinn.org> >>> *Sent:* Monday, August 25, 2025 11:35:31 AM >>> *To:* Jonathan Vasquez <jon@xyinn.org> >>> *Cc:* Stephan Althaus <Stephan.Althaus@Duedinghausen.eu>; >>> virtualization@freebsd.org <virtualization@freebsd.org> >>> *Subject:* Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 >>> XT and Windows 10 Pro) >>> I was able to get a fresh Win 10 Pro VM running through vm-bhyve but >>> no progress. Still same Code 43 error as before. I wasn't able to >>> figure out how to pass the |rom=bios.rom| through vm-bhyve, I >>> didn't see anything about it in the config.sample file and I also >>> tried to bypass that by adding it in bhyve_options but no go, the VM >>> wouldn't even start if I did that. I see there is a "bhyve_devices" >>> that contains all of the slot devices but I don't see that as an >>> adjustable variable based on the config.sample info. I've pasted my >>> current vm-bhyve template below. My next test is to try the same >>> thing inside a Linux guest to see what happens. >>> >>> ---------------- >>> >>> loader="uefi" >>> cpu="4" >>> cpu_sockets="1" >>> cpu_cores="4" >>> cpu_threads="1" >>> memory="16G" >>> wired_memory="yes" >>> utctime="no" >>> ahci_device_limit="8" >>> disk0_type="nvme" >>> disk0_name="disk0.img" >>> network0_type="e1000" >>> network0_switch="public" >>> passthru0="3/0/0=2:0" >>> passthru1="3/0/1=2:1" >>> #bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom" >>> >>> # Using RDP post-install. >>> graphics="no" >>> graphics_port="5900" >>> graphics_listen="0.0.0.0" >>> graphics_res="1920x1080" >>> graphics_wait="yes" >>> xhci_mouse="yes" >>> uuid="535d54b5-81cb-11f0-93a2-98b78501bda8" >>> network0_mac="58:9c:fc:04:5a:eb" >>> >>> ---------------- >>> >>> Jonathan Vasquez >>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>> Sent with ProtonMail Secure Email >>> >>> >>> On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez >>> <jon@xyinn.org> wrote: >>>> Unfortunately I don't have an intel machine with an available PCIe >>>> slot. However I'm trying another thing which is to passthrough the >>>> card to a Linux guest. I need to run more experiments but a >>>> preliminary boot of a Linux Mint ISO did show the AMD Radeon RX >>>> 6900 XT card in |lspci| and showed that it was using the |amdgpu| >>>> driver. I then did a basic |glxgears| and it opened up in my VNC >>>> window and showed the FPS counter at over 2000 fps. The only issue >>>> I noticed was when shutting down the live CD environment there were >>>> |amdgpu| failure messages where it was failing to write something >>>> to it. I'm currently redoing my "basic bhyve script" and moving it >>>> over to "vm-bhyve" since it seems a lot of people are using that >>>> instead of the more manual approach. This should hopefully help >>>> simplify things for debugging. If the card works on Linux via >>>> passthrough this would also be useful info and can allow me to see >>>> the performance of Cyperpunk 2077, and some other games (even if >>>> I'm not getting 100% of the card's performance). >>>> >>>> Jonathan Vasquez >>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>> Sent with ProtonMail Secure Email >>>> >>>> >>>> On Monday, August 25th, 2025 at 11:00, Stephan Althaus >>>> <Stephan.Althaus@Duedinghausen.eu> wrote: >>>>> On 8/25/25 16:56, Jonathan Vasquez wrote: >>>>>> Hey Stephan, >>>>>> >>>>>> Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I >>>>>> also make sure to reboot the host in between runs due to the amd >>>>>> hardware issue. I don't believe 3/0/1 is necessary though based >>>>>> on the examples by Corvin in his talk. He only passed in 3/0/0 >>>>>> (I'm guessing his card also has a 3/0/1 function that he isn't >>>>>> using since that's the audio channel). >>>>>> >>>>>> Jonathan Vasquez >>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>> Sent with ProtonMail Secure Email >>>>>> >>>>>> >>>>>> On Monday, August 25th, 2025 at 10:52, Stephan Althaus >>>>>> <Stephan.Althaus@Duedinghausen.eu> >>>>>> <mailto:Stephan.Althaus@Duedinghausen.eu> wrote: >>>>>>> On 8/25/25 16:47, Stephan Althaus wrote: >>>>>>>> On 8/24/25 23:58, Jonathan Vasquez wrote: >>>>>>>>> Hey all, >>>>>>>>> >>>>>>>>> I've been experimenting with trying to get my server's >>>>>>>>> dedicated AMD Radeon RX 6900 XT working on a bhyve Windows 10 >>>>>>>>> Pro client for gaming purposes. I've been reading a bunch of >>>>>>>>> blog posts, sites, files, and videos (thanks Corvin) to try >>>>>>>>> and piece things together. I've been able to get the VM to >>>>>>>>> recognize the card, but still stuck on the Code 43 issue (with >>>>>>>>> or without using the vBIOS I've extracted from my dedicated >>>>>>>>> card and also ensuring any options needed to be on in the UEFI >>>>>>>>> settings are enabled). This machine is running on an AMD Ryzen >>>>>>>>> 9 7950X 16-Core Processor, so it's a full AMD setup, no Intel >>>>>>>>> stuff. It technically should be relatively "easy" to get AMD >>>>>>>>> working from what I've read (excluding the known AMD Hardware >>>>>>>>> reset issue). I've documented everything I've done so far, and >>>>>>>>> will be documenting more and simplifying the instructions as I >>>>>>>>> make progress. Please let me know if you see any thing weird >>>>>>>>> or anything I could try based on what I've rewritten. I'll try >>>>>>>>> to give it a go and provide feedback. Thank you! >>>>>>>>> >>>>>>>>> https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd >>>>>>>>> <https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd>; >>>>>>>>> >>>>>>>>> Jonathan Vasquez >>>>>>>>> PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 >>>>>>>>> Sent with ProtonMail Secure Email >>>>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, >>>>>>>> host is a intel system. >>>>>>>> >>>>>>>> i think you have to passthrough the 3/0/1 device as well! >>>>>>>> >>>>>>>> The bios.rom is extracted from this card in place ? >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Stephan >>>>>>>> >>>>>>>> >>>>>>> Ah, and be shure to reboot the whole setup, inclusive host, for >>>>>>> every new test-case, >>>>>>> at least as long at is not guaranteed that we can live without.. >>>>>>> >>>>>>> Regars, >>>>>>> Stephan >>>>>>> >>>>>>> >>>>>> >>>>> Hi! >>>>> >>>>> My only next assumption would be that this only does not work on >>>>> AMD host. >>>>> >>>>> You don't have an intel system to test, you would havedone it, if >>>>> so.. (?) >>>>> >>>>> Stephan >>>>> >>>>> >>>> >>> >> > Hello! I have a successful GPU passthrough of an AMD RX6400 to Win10 on an Intel host. Just want to share some Infos + no problem with RDP here, that's my use case + i used GPU-Z within the vm to check if the gpu is actually useable, it shows GPU frequency > 0 , and OpenGL >1.1 when it is. + i have thr ROM in my bhyve start script, the vm has no problem with it. i don't use vm-bhyve so far. - your win 10 vm is EFI install ? + yes, mesa is software rendering, sort of 'software GPU' It seems to me that the only noticeable difference between our systems is that mine is an intel host.. Regards, Stephan [-- Attachment #2 --] <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <div class="moz-cite-prefix">On 8/31/25 02:10, Jonathan Vasquez wrote:<br> </div> <blockquote type="cite" cite="mid:FIR4iioBciiFZPGzeCeKoY_RKIdxeWWraK2Dr98ag7G7X6n2u5CWtBc2GT7CAzc9sq9JUREXfr7tbj9Bpd341cp7YxJFyM04FczcztnSb_I=@xyinn.org"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> <div style="font-family: Arial, sans-serif; font-size: 14px;">Hey all,</div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;">I've updated my blog post with more information and some useful screenshots: <span><a target="_blank" rel="noreferrer nofollow noopener" href="https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd" moz-do-not-send="true" class="moz-txt-link-freetext">https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd</a></span></div>; <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;">The important stuff to mention is that I've also gotten a chance to play with vm-bhyve, but for the purposes of making sure we get as clear information as possible without having to unwrap the vm-bhyve scripts, I'll continue to use bhyve directly. With that said, some useful info:</div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"> <ul style="margin-top: 0px; margin-bottom: 0px;" data-editing-info="{"orderedStyleType":1,"unorderedStyleType":2}"> <li style="list-style-type: "- ";"><span>The vBIOS is not required when running on Windows as mentioned by Corvin here: <span><a target="_blank" rel="noreferrer nofollow noopener" href="https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html" moz-do-not-send="true" class="moz-txt-link-freetext">https://lists.freebsd.org/archives/freebsd-virtualization/2022-June/000797.html</a></span>, thus I've stopped using it. I was however also able to successfully use his "echo 1 ..." recommendation to dump the bios from within Linux (Fedora 42). I actually failed to extract the bios before (without using amdvbflash) because most likely my Linux Mint 22.1's Kernel was too old to extract the ROM. When I tried to do the same exact thing today with Fedora 42, it worked, and I was able to see the "rom" file from Fedora 42, but not from Linux Mint 22.1.</span></li> <li style="list-style-type: "- ";">I was able to get the GPU to load without a Code 43 error. But it's still unusable. I was actually able to get to this state before but it was inconsistent. Through more research I was able to get it to be reproducible. There are a few things to mention here:</li> <ul style="margin-top: 0px; margin-bottom: 0px; list-style-type: circle;"> <li style="">The card will always start in a Code 43 failed state regardless of what I mention below (including if I reboot the host).</li> <li style="">If you use RDP, the card will always be in a Code 43 state.</li> <li style="">If I use VNC, the card will start in a failed state (as mentioned above), HOWEVER, you will be able to go into the Device Manager, Disable the card, and then Enable the card, and it will be enabled successfully as you can see in my screenshots. The OS will also give it resources from what we can see in the Device's Properties' Resources Tab.</li> </ul> </ul> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;">Given what I mentioned above, I would recommend (as some other people recommended in other links I've found), to disable the card in the VM before you shut it down, and then enable it from inside of Windows when you turn it back on. Although I don't know how necessary this is given that if I use VNC, even if it starts off in a failed state (let's say from a fresh host reboot), doing the "device re-nabling trick" will let it load successfully again. At this point since I've gotten the card to say it worked, I'm not sure how to tell Windows to use that as the main GPU when I run my apps. I've looked at some examples online of using display settings to change the preferred card, but I don't see it listed there. Am I suppose to see the card in dxdiag if I'm using VNC? I'm just running the bhyve VM on my server, and I VNC over from my laptop on the same LAN network. I'm guessing if things are working correctly I should be able to do that right? I know there are also some benchmarking apps that show you what GPUs its detected and maybe allow you to pick one to run a benchmark on. I tried opening FurMark but it fails saying that it only detects Open GL 1.1 and all of that... I actually tried installing Mesa for windows to see what happens and that does help but it's slow and it's clearly not using the card and most likely doing some sort of software rendering (maybe like a Windows llvmpipe).</div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;" class="protonmail_signature_block"> <div class="protonmail_signature_block-user"> <div>Jonathan Vasquez<br> </div> <div>PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279<br> </div> <div>Sent with ProtonMail Secure Email<br> </div> <div><br> </div> </div> <div class="protonmail_signature_block-proton protonmail_signature_block-empty"> </div> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div class="protonmail_quote"> On Monday, August 25th, 2025 at 14:51, Jonathan Vasquez <a class="moz-txt-link-rfc2396E" href="mailto:jon@xyinn.org"><jon@xyinn.org></a> wrote:<br> <blockquote class="protonmail_quote" type="cite"> <div style="font-family: Arial, sans-serif; font-size: 14px;">No go for my testing inside a Linux Mint 22.1 VM. Fully updated, and passing in both 3/0/0 and 3/0/1 for my AMD Radeon card. I was able to extract some of the "dmesg" and "eglinfo" output. Running "glxinfo" also showed that the OpenGL renderer was set to llvmpipe, so software rendering :(. "lspci -k" detected it and showed that it was using the "amdgpu" kernel driver. Since I'm using vm-bhyve atm I didn't test this with the dumped rom, just directly passing them through. I've uploaded some of the logs here for now in case anyone wants to take a glance at them:</div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><span><a target="_blank" rel="noreferrer nofollow noopener" href="https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/" moz-do-not-send="true" class="moz-txt-link-freetext">https://xyinn.org/freebsd/files/gpu_pass/2025-08-25/</a></span><br>; </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;" class="protonmail_signature_block"> <div class="protonmail_signature_block-user"> <div>Jonathan Vasquez<br> </div> <div>PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279<br> </div> <div>Sent with ProtonMail Secure Email<br> </div> <div><br> </div> </div> <div class="protonmail_signature_block-proton protonmail_signature_block-empty"> </div> </div> <div style="font-family: Arial, sans-serif; font-size: 14px;"><br> </div> <div class="protonmail_quote"> On Monday, August 25th, 2025 at 14:37, Robert Cocker <a class="moz-txt-link-rfc2396E" href="mailto:rcocker87@gmail.com"><rcocker87@gmail.com></a> wrote:<br> <blockquote class="protonmail_quote" type="cite"> <div style="font-family: Aptos, -apple-system, HelveticaNeue, sans-serif;font-size: 12pt"> <div style="font-family: Aptos, Aptos_MSFontService, -apple-system, Roboto, Arial, Helvetica, sans-serif; font-size: 12pt;" dir="ltr"> Hey group has bhvye core dev baked out an update for this </div> </div> <div id="ms-outlook-mobile-body-separator-line" style="font-family: Aptos, -apple-system, HelveticaNeue, sans-serif;font-size: 12pt" dir="auto"> <br> </div> <div id="ms-outlook-mobile-signature" style="font-family: Aptos, -apple-system, HelveticaNeue, sans-serif;font-size: 12pt"> <div dir="ltr">Robert</div> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> <a class="moz-txt-link-abbreviated" href="mailto:owner-freebsd-virtualization@FreeBSD.org">owner-freebsd-virtualization@FreeBSD.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:owner-freebsd-virtualization@FreeBSD.org"><owner-freebsd-virtualization@FreeBSD.org></a> on behalf of Jonathan Vasquez <a class="moz-txt-link-rfc2396E" href="mailto:jon@xyinn.org"><jon@xyinn.org></a><br> <b>Sent:</b> Monday, August 25, 2025 11:35:31 AM<br> <b>To:</b> Jonathan Vasquez <a class="moz-txt-link-rfc2396E" href="mailto:jon@xyinn.org"><jon@xyinn.org></a><br> <b>Cc:</b> Stephan Althaus <a class="moz-txt-link-rfc2396E" href="mailto:Stephan.Althaus@Duedinghausen.eu"><Stephan.Althaus@Duedinghausen.eu></a>; <a class="moz-txt-link-abbreviated" href="mailto:virtualization@freebsd.org">virtualization@freebsd.org</a> <a class="moz-txt-link-rfc2396E" href="mailto:virtualization@freebsd.org"><virtualization@freebsd.org></a><br> <b>Subject:</b> Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro)</font> <div> </div> </div> <div> <div style="font-family:Arial,sans-serif; font-size:14px">I was able to get a fresh Win 10 Pro VM running through vm-bhyve but no progress. Still same Code 43 error as before. I wasn't able to figure out how to pass the <code>rom=bios.rom</code> through vm-bhyve, I didn't see anything about it in the config.sample file and I also tried to bypass that by adding it in bhyve_options but no go, the VM wouldn't even start if I did that. I see there is a "bhyve_devices" that contains all of the slot devices but I don't see that as an adjustable variable based on the config.sample info. I've pasted my current vm-bhyve template below. My next test is to try the same thing inside a Linux guest to see what happens.</div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div style="font-family:Arial,sans-serif; font-size:14px">----------------</div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div style="font-family:Arial,sans-serif; font-size:14px"><span>loader="uefi" </span> <div><span>cpu="4" </span></div> <div><span>cpu_sockets="1" </span></div> <div><span>cpu_cores="4" </span></div> <div><span>cpu_threads="1" </span></div> <div><span>memory="16G" </span></div> <div><span>wired_memory="yes" </span></div> <div><span> </span></div> <div><span>utctime="no" </span></div> <div><span>ahci_device_limit="8" </span></div> <div><span> </span></div> <div><span>disk0_type="nvme" </span></div> <div><span>disk0_name="disk0.img" </span></div> <div><span> </span></div> <div><span>network0_type="e1000" </span></div> <div><span>network0_switch="public" </span></div> <div><span> </span></div> <div><span>passthru0="3/0/0=2:0" </span></div> <div><span>passthru1="3/0/1=2:1" </span></div> <div><span> </span></div> <div><span>#bhyve_options="-s 2:0,passthru,3/0/0,rom=bios.rom" </span></div> <div><span><br> </span></div> <div><span># Using RDP post-install. </span></div> <div><span>graphics="no" </span></div> <div><span>graphics_port="5900" </span></div> <div><span>graphics_listen="0.0.0.0" </span></div> <div><span>graphics_res="1920x1080" </span></div> <div><span>graphics_wait="yes" </span></div> <div><span> </span></div> <div><span>xhci_mouse="yes" </span></div> <div><span>uuid="535d54b5-81cb-11f0-93a2-98b78501bda8" </span></div> <span>network0_mac="58:9c:fc:04:5a:eb" </span></div> <div style="font-family:Arial,sans-serif; font-size:14px"><span><br> </span></div> <div style="font-family:Arial,sans-serif; font-size:14px">----------------</div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div class="x_protonmail_signature_block" style="font-family:Arial,sans-serif; font-size:14px"> <div class="x_protonmail_signature_block-user"> <div>Jonathan Vasquez<br> </div> <div>PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279<br> </div> <div>Sent with ProtonMail Secure Email<br> </div> <div><br> </div> </div> <div class="x_protonmail_signature_block-proton x_protonmail_signature_block-empty"> </div> </div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div class="x_protonmail_quote">On Monday, August 25th, 2025 at 11:05, Jonathan Vasquez <a class="moz-txt-link-rfc2396E" href="mailto:jon@xyinn.org"><jon@xyinn.org></a> wrote:<br> <blockquote class="x_protonmail_quote" type="cite"> <div style="font-family:Arial,sans-serif; font-size:14px">Unfortunately I don't have an intel machine with an available PCIe slot. However I'm trying another thing which is to passthrough the card to a Linux guest. I need to run more experiments but a preliminary boot of a Linux Mint ISO did show the AMD Radeon RX 6900 XT card in <code>lspci</code> and showed that it was using the <code>amdgpu</code> driver. I then did a basic <code>glxgears</code> and it opened up in my VNC window and showed the FPS counter at over 2000 fps. The only issue I noticed was when shutting down the live CD environment there were <code>amdgpu</code> failure messages where it was failing to write something to it. I'm currently redoing my "basic bhyve script" and moving it over to "vm-bhyve" since it seems a lot of people are using that instead of the more manual approach. This should hopefully help simplify things for debugging. If the card works on Linux via passthrough this would also be useful info and can allow me to see the performance of Cyperpunk 2077, and some other games (even if I'm not getting 100% of the card's performance).</div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div class="x_protonmail_signature_block" style="font-family:Arial,sans-serif; font-size:14px"> <div class="x_protonmail_signature_block-user"> <div>Jonathan Vasquez<br> </div> <div>PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279<br> </div> <div>Sent with ProtonMail Secure Email<br> </div> <div><br> </div> </div> <div class="x_protonmail_signature_block-proton x_protonmail_signature_block-empty"> </div> </div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div class="x_protonmail_quote">On Monday, August 25th, 2025 at 11:00, Stephan Althaus <a class="moz-txt-link-rfc2396E" href="mailto:Stephan.Althaus@Duedinghausen.eu"><Stephan.Althaus@Duedinghausen.eu></a> wrote:<br> <blockquote class="x_protonmail_quote" type="cite"> <div class="x_moz-cite-prefix">On 8/25/25 16:56, Jonathan Vasquez wrote:<br> </div> <blockquote type="cite"> <div style="font-family:Arial,sans-serif; font-size:14px">Hey Stephan,</div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div style="font-family:Arial,sans-serif; font-size:14px">Yup, I've tried passing both 3/0/0 and 3/0/1 and no go, and I also make sure to reboot the host in between runs due to the amd hardware issue. I don't believe 3/0/1 is necessary though based on the examples by Corvin in his talk. He only passed in 3/0/0 (I'm guessing his card also has a 3/0/1 function that he isn't using since that's the audio channel).</div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div class="x_protonmail_signature_block" style="font-family:Arial,sans-serif; font-size:14px"> <div class="x_protonmail_signature_block-user"> <div>Jonathan Vasquez<br> </div> <div>PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279<br> </div> <div>Sent with ProtonMail Secure Email<br> </div> <div><br> </div> </div> <div class="x_protonmail_signature_block-proton x_protonmail_signature_block-empty"> </div> </div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div class="x_protonmail_quote">On Monday, August 25th, 2025 at 10:52, Stephan Althaus <a class="x_moz-txt-link-rfc2396E" href="mailto:Stephan.Althaus@Duedinghausen.eu" rel="noreferrer nofollow noopener" moz-do-not-send="true"> <Stephan.Althaus@Duedinghausen.eu></a> wrote:<br> <blockquote class="x_protonmail_quote" type="cite"> <div class="x_moz-cite-prefix">On 8/25/25 16:47, Stephan Althaus wrote:<br> </div> <blockquote type="cite"> <div class="x_moz-cite-prefix">On 8/24/25 23:58, Jonathan Vasquez wrote:<br> </div> <blockquote type="cite"> <div style="font-family:Arial,sans-serif; font-size:14px"><span>Hey all,</span></div> <div style="font-family:Arial,sans-serif; font-size:14px"><span><br> </span></div> <div style="font-family:Arial,sans-serif; font-size:14px"><span>I've been experimenting with trying to get my server's dedicated AMD Radeon RX 6900 XT working on a bhyve Windows 10 Pro client for gaming purposes. I've been reading a bunch of blog posts, sites, files, and videos (thanks Corvin) to try and piece things together. I've been able to get the VM to recognize the card, but still stuck on the Code 43 issue (with or without using the vBIOS I've extracted from my dedicated card and also ensuring any options needed to be on in the UEFI settings are enabled). This machine is running on an AMD Ryzen 9 7950X 16-Core Processor, so it's a full AMD setup, no Intel stuff. It technically should be relatively "easy" to get AMD working from what I've read (excluding the known AMD Hardware reset issue). I've documented everything I've done so far, and will be documenting more and simplifying the instructions as I make progress. Please let me know if you see any thing weird or anything I could try based on what I've rewritten. I'll try to give it a go and provide feedback. Thank you!</span></div> <div style="font-family:Arial,sans-serif; font-size:14px"><span><br> </span></div> <div style="font-family:Arial,sans-serif; font-size:14px"><span><a target="_blank" rel="noreferrer nofollow noopener" href="https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd" class="x_moz-txt-link-freetext moz-txt-link-freetext" moz-do-not-send="true">https://xyinn.org/blog/freebsd/freebsd_bhyve_gpu_passthrough_amd</a></span><br>; </div> <div style="font-family:Arial,sans-serif; font-size:14px"><br> </div> <div class="x_protonmail_signature_block" style="font-family:Arial,sans-serif; font-size:14px"> <div class="x_protonmail_signature_block-user"> <div>Jonathan Vasquez<br> </div> <div>PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279<br> </div> <div>Sent with ProtonMail Secure Email<br> </div> <div><br> </div> </div> <div class="x_protonmail_signature_block-proton x_protonmail_signature_block-empty"> </div> </div> </blockquote> <p>Hi!</p> <p>i have a AMD RX6400 GPU setup running on 14.2 with win10 guest, host is a intel system.</p> <p>i think you have to passthrough the 3/0/1 device as well!</p> <p>The bios.rom is extracted from this card in place ?</p> <p><br> </p> <p>Regards,<br> Stephan</p> <p><br> </p> </blockquote> <p>Ah, and be shure to reboot the whole setup, inclusive host, for every new test-case, <br> at least as long at is not guaranteed that we can live without..</p> <p>Regars,<br> Stephan</p> <p><br> </p> </blockquote> <br> </div> </blockquote> <p>Hi!</p> <p>My only next assumption would be that this only does not work on AMD host.</p> <p>You don't have an intel system to test, you would havedone it, if so.. (?)</p> <p>Stephan</p> <p><br> </p> </blockquote> <br> </div> </blockquote> <br> </div> </div> </blockquote> <br> </div> </blockquote> <br> </div> </blockquote> <p>Hello!</p> <p>I have a successful GPU passthrough of an AMD RX6400 to Win10 on an Intel host.</p> <p>Just want to share some Infos</p> <p>+ no problem with RDP here, that's my use case</p> <p>+ i used GPU-Z within the vm to check if the gpu is actually useable, it shows GPU frequency > 0 , and OpenGL >1.1 when it is.</p> <p>+ i have thr ROM in my bhyve start script, the vm has no problem with it. i don't use vm-bhyve so far.</p> <p>- your win 10 vm is EFI install ?</p> <p>+ yes, mesa is software rendering, sort of 'software GPU' </p> <p>It seems to me that the only noticeable difference between our systems is that mine is an intel host..</p> <p>Regards,<br> Stephan</p> </body> </html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?346a503c-e83a-4833-ac62-d7969f8728ef>