Date: Tue, 23 Sep 2025 12:50:17 +0000 From: Jonathan Vasquez <jon@xyinn.org> To: =?utf-8?Q?Goran_Meki=C4=87?= <meka@tilda.center> Cc: Matt Churchyard <matt@userve.net>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> In-Reply-To: <mOxiSfmxWrHIMEStshHuwzAKYh003kyxHWvA_NY42h2TJfzeGaYSA5ePY1WNK67MtP_quGeVkFzRTTeSf7ESI-gMBbcoxtr7DeQ24xr5Ntg=@xyinn.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <Ags2LfZGkLKZGcRQuyphkcg9wsf5q-HB1cIcphC2Eujdm0EtRkTJdHq-UHj6QLLsT53dLsfLeOib25LpGSIA-4IIqCBMMlXcfNsfCcobRJs=@xyinn.org> <MVEz5qPYM1pdpKQkFxy-FMucy9_MNUG5kn7cQGs0Czl6bU68vITWeEomJxHNqnK-qBrxo8qsdNYVD-B41qnVVz4D_QZP9E4QB75bkG5bu_s=@xyinn.org> <CAFDf7ULvsf0W8iGSYu-PWGWPgzrOC%2BoshOH1c2tDqLtkgbcw1w@mail.gmail.com> <ceec94aac0998e388eee720ad04c0e481cfc601b.camel@FreeBSD.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <l8EqFTkrrLvYOUcKLKIRbCxHHRDMjM--eRHlFwVyC0oHzp58cXM9bP9uLrbMe3wree4k9tI4ERHgFiBt3s9KB7htgR7FXaNNwmub7bq7K14=@xyinn.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <mOxiSfmxWrHIMEStshHuwzAKYh003kyxHWvA_NY42h2TJfzeGaYSA5ePY1WNK67MtP_quGeVkFzRTTeSf7ESI-gMBbcoxtr7DeQ24xr5Ntg=@xyinn.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Alright all. So a few updates: After the 22+ hour uptime two days ago, I did a manual reboot of host/guest= to continue testing the "deterministic stability" of the system (as I ment= ioned yesterday). Within a few minutes, the VM exited with a new error code= (0x88) (previously it was 0x60). After I did the PCI passthrough adjustmen= ts, I haven't gotten 0x60 anymore and the VM has become a lot more stable. = For whatever reason I do sometimes still get a 0x88, but it hasn't happened= too many times. After the initial 0x88 exit code, I rebooted the host/vm a= gain, then the VM worked for about 3-4 hours before getting another 0x88. A= fter another host/vm reboot, the VM became stable and ran for another 19 ho= urs (which is this morning). I then turned it off since I wanted to experim= ent with the pkill situation we were discussing recently. So besides figuri= ng out what's causing it to sometimes do a 0x88, we are close I think to a = stable setup. I've noticed that in situations where it stays stable, I usua= lly have my Task Manager up since I want to see the resource utilization, t= emperatures, and uptime of the system quickly, while I monitor it. Could th= ere be something with the task manager always polling different hardware fo= r their state that may cause the VM to stay alive (vs not having task manag= er and just idling the machine causing it to die?). There is no sleep suppo= rt in the VM so we don't need to worry about the VM going to sleep/hiberati= on. I've also worked on a start/stop script to integrate this into the rc.d sys= tem a little bit better. Before I was using /etc/crontab and @reboot to sta= rt the VM at boot time after everything has started up, which works fine, b= ut I couldn't find a @shutdown equivalent. So I wrote a basic rc.d script. = The script is working but the problem is that for w/e reason it blocks the = rest of the services on my machine from starting. I investigated it a bit a= nd it has to do with "rcorder". I also analyzed the order by doing "rcorder= /etc/rc.d/* /usr/local/etc/rc.d/*" which revealed where in the chain it fa= lls. Originally there was no "REQUIRE:" line so it was being thrown into th= e beginning of the chain. I then switched it to "REQUIRE: LOGIN" which thre= w it close to the end of the chain (which is good), but it still has some s= ervices that follow it, and I noticed that even though I can ping the box a= fter reboot, for w/e reason even though sshd falls before LOGIN, I still co= uldn't ssh into the box until I shutdown the Windows VM, which allows execu= tion to continue. The start script I have for bhyve is a blocking call sinc= e it runs "bhyve ..." which would end up capturing the terminal. When I do = it through /etc/crontab @reboot, this doesn't seem to be an issue, but it i= s when doing it through an rc.d script. Anyone have any recommendations abo= ut this? I'll post the three scripts I'm using at the end of this message. Also, I'm more open now to writing a new section for the handbook regarding= everything I've learned for GPU passthrough for gaming purposes. Although = I have to preface that my experience is limited and it's only optimized for= this flow, so if I write it, I would only be able to write about this part= icular path and things/caveats around it. Normally the handbook usually has= a more general knowledge approach as well which I'm not qualified to write= about at the moment. Would it be fine for me to start writing this section= and get it into the handbook, and other people more experienced than me ca= n expand that section with more general knowledge about bhyve/passthrough (= and of course Intel/NVIDIA stuff can come after)? I can start with GPU pass= through / gaming / AMD Host + AMD Dedicated Card / and I can also add the s= tart/stop scripts + rc.d script. The paths, names, and values will need to be adjusted for each person's ind= ividual setup: VM START SCRIPT: #!/bin/sh =20 VM_DIR=3D"$(dirname $(realpath $0))" =20 VM_NAME=3D"$(basename $VM_DIR)" =20 cd "$VM_DIR" =20 bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ -s 0,hostbridge \ -s 1,nvme,disk0.img \ -s 3:0,passthru,3/0/0 \ -s 3:1,passthru,3/0/1 \ -s 13:0,passthru,13/0/0 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu \ -l com1,stdio \ $VM_NAME # Exit the script here and leave some options for debugging more =20 # ergonomically after the exit line. =20 exit =20 # Only use this when installing the VM for the first time: =20 # VNC. Once you install Windows and enable RDP, you can turn this off. = =20 -s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1024,h=3D768,wait \ =20 # The Windows ISO and the VirtIO Drivers ISO. =20 # You can download the latest stable virtio drivers at: =20 # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-v= irtio/virtio-win.iso =20 -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \ =20 # If you want a network device without using virtio, you can use e1000. = =20 # You should install the virtio drivers and use virtio-net instead for = =20 # better performance. =20 -s 2,e1000,tap0 \=20 -s 2,virtio-net,tap0 \ =20 VM STOP SCRIPT: #!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=3Dgaming --destroy RC.D SCRIPT: #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable=20 start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" vm_gaming_start() { =09/atlantis/vms/gaming/start.sh } vm_gaming_stop() { =09/atlantis/vms/gaming/stop.sh } load_rc_config $name=20 : ${vm_gaming_enable:=3D"NO"}=20 run_rc_command "$1"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=>