Date: Thu, 25 Sep 2025 12:01:43 +0000 From: Jonathan Vasquez <jon@xyinn.org> To: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <INv1QRTDanPqExdkRH9g_SlRzYv6-m_Xg5SWWYjpkvkQ_RqVdS_OuVJgJvQW3ZAnhsS334kzM-0IP0rj4c6MN2WRvws2_FII5XBr1aL5Rw4=@xyinn.org> In-Reply-To: <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <l8EqFTkrrLvYOUcKLKIRbCxHHRDMjM--eRHlFwVyC0oHzp58cXM9bP9uLrbMe3wree4k9tI4ERHgFiBt3s9KB7htgR7FXaNNwmub7bq7K14=@xyinn.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <mOxiSfmxWrHIMEStshHuwzAKYh003kyxHWvA_NY42h2TJfzeGaYSA5ePY1WNK67MtP_quGeVkFzRTTeSf7ESI-gMBbcoxtr7DeQ24xr5Ntg=@xyinn.org> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hey all,
I was able to get another 21+ hours of uptime on idle with no issues, and I did some more gaming last night and it's stable. I haven't experienced any of the weird vm exits recently so that's good. There are probably cases where the vm may exit but I'm not sure what those situations are given the recent stability. I'll keep monitoring but I'm pretty happy with how everything turned out so far.
I've also debug the script a bit more to see why it wasn't shutting down. The docs aren't explicit about it (or at least I haven't found a place yet for it), but I think it may be related to the pid stuff. It doesn't seem that rc.shutdown calls the stop command if it's simply defined given there are performance optimizations to skip certain things even if KEYWORD: shutdown is defined. So after doing a "ps -aux | grep gaming" and seeing what showed up, I saw that there are 3-4 pids that exist with my current set up. daemon pid, main bhyve pid, start.sh pid, etc. The pid that was getting written by daemon previously was the pid of my start.sh script, but the pid of the bhyve process that gets started by start.sh, so that would be a "grandchild of daemon", so I think that's why originally it wasn't working for me. I now merged my start/stop script logic into the main rc.d script, so daemon is directly executing bhyve, and thus the child process recorded in the pidfile is correct. The stop command can now load up the child pidfile, and properly call kill on that pid directly. My script now works on shutdown properly. The only thing I noticed was that even though I do now see Windows displaying the "shutting down" message, it kinda seems like the system isn't necessarily obeying my "sleep 10" call afterwards, which means it probably still isn't 100% a graceful shutdown, but much better than before. There is also the bhyvectl --destroy ... command after and I'm not sure if that's running. Below is the improved and centralized rc.d script:
#!/bin/sh
# PROVIDE: vm_gaming
# REQUIRE: LOGIN
# KEYWORD: nojail shutdown
. /etc/rc.subr
name=vm_gaming
rcvar=vm_gaming_enable
vm_path="/atlantis/vms/gaming"
vm_name="gaming"
start_cmd="${name}_start"
stop_cmd="${name}_stop"
restart_cmd="${name}_restart"
pidfile="/var/run/${name}.child.pid"
vm_gaming_start()
{
daemon \
-o /var/log/${name}.log \
-p "${pidfile}" \
bhyve -AHPSw -c sockets=1,cores=16,threads=1 -m 32G \
-s 0,hostbridge \
-s 1,nvme,${vm_path}/disk0.img \
-s 3:0,passthru,3/0/0 \
-s 3:1,passthru,3/0/1 \
-s 13:0,passthru,13/0/0 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \
-o console=stdio \
${vm_name}
}
vm_gaming_stop()
{
# Send SIGTERM twice to make sure Windows listens to
# the ACPI shutdown signal.
pid="$(cat ${pidfile})"
kill ${pid}
kill ${pid}
# Wait a bit for the guest to shutdown properly before
# we continue shutting down the host.
sleep 10
bhyvectl --vm=${vm_name} --destroy
}
vm_gaming_restart()
{
# NOTE: AMD users will most likely experience the famous
# AMD Hardware Reset Bug. This means that after you reboot
# the guest, you most likely won't have video out. If this
# happens, you'll need to restart the host. Sometimes this
# command will work for AMD users though. So you can try a
# restart and see if it works.
vm_gaming_stop
vm_gaming_start
}
load_rc_config $name
: ${vm_gaming_enable:="NO"}
run_rc_command "$1"
Jonathan Vasquez
PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279
Sent with ProtonMail Secure Email
On Wednesday, September 24th, 2025 at 09:05, Jonathan Vasquez <jon@xyinn.org> wrote:
> Hello all,
>
> I was able to get another stable vm on idle (this time without leaving Task Manager
> opened) for another 22 hours. I also spent some time today improving my start/stop
> scripts. Using Corvin's suggestion about the "daemon" process, I was able to
> now successfully start the VM at boot time without blocking. I had to make some
> adjustments to my bhyve script so it doesn't use "-l com1,stdio", because during
> boot time, even with me making the script require "LOGIN syscons", it still
> would throw the following error:
>
> Unable to initialize backend 'stdio' for LPC device com1
> 100 Device emulation initialization error: Inappropriate ioctl for device
>
> Which prevented the VM from starting. I've switched to using "-o console=stdio"
> and that worked. I'm able to get regular console output, which allows the VM
> to always start up successfully, and also allows us to log to a file when we run
> our vm start script through daemon.
>
> I've also observed that there were a few times when I restarted the VM without
> restarting the host, 2-3 times, and each time the video output worked (meaning
> I wasn't affected by the AMD Hardware Reset Bug). However, after that I wasn't
> able to get that working anymore. So I would still recommend restarting the host.
>
> I added the following to the script:
>
> - Daemonized support so the VM starts up non-blocking. This also includes
> logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since the
> pid addresses looked off. We also don't seem to need it at the moment.
> - Added the restart command (with a note about the AMD issue).
> - Added some quality of life stuff like having some variables at the top
> to easily set your start/stop paths.
>
> Some other things are that if I do "service vm_gaming <some command>", the
>
> current console in tmux seems to get really laggy and the input gets weird. If
> I open another tmux window, then input is fine again. There are no other
> consequences other than laggy/wrong input in the current console. This doesn't
> seem to negatively affect anything when starting it automatically at boot time
> through rc.d.
>
> One last thing is that I noticed that when I do a "shutdown -r now", I don't
> see windows getting the ACPI shutdown signal, so I don't see Windows gracefully
> shutting down. I have read if you do "reboot" it will just send the SIGTERM signal,
> and kill everything afterwards, so the "shutdown" command should be used to
> give advance notice to processes. Given that my rc.d script has "shutdown" in
> the keywords list, I was expecting the "vm_gaming_stop" command to be ran
> automatically in order to gracefully shutdown the VM, and clean itself up. Is this
> not working because I'm not using -P / -p?
>
> I've updated my blog post to include all of these new scripts and modifications.
>
> I'll post the current scripts below. Thanks for reading.
>
> Jonathan
>
>
> ## Bhyve Scripts
>
> ### start script
>
> This is the main script that will start your vm. Save this as `start.sh`
> in a folder called `gaming` somewhere on your system:
>
> `#!/bin/sh VM_DIR="$(dirname $(realpath $0))" VM_NAME="$(basename $VM_DIR)" cd "$VM_DIR" bhyve -AHPSw -c sockets=1,cores=16,threads=1 -m 32G \\ -s 0,hostbridge \\ -s 1,nvme,disk0.img \\ -s 3:0,passthru,3/0/0 \\ -s 3:1,passthru,3/0/1 \\ -s 13:0,passthru,13/0/0 \\ -s 30,xhci,tablet \\ -s 31,lpc \\ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \\ -o console=stdio \\ $VM_NAME # Exit the script here and leave some options for debugging more # ergonomically after the exit line. exit # Only use this when installing the VM for the first time: # VNC. Once you install Windows and enable RDP, you can turn this off. -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \\ # The Windows ISO and the VirtIO Drivers ISO. # You can download the latest stable virtio drivers at: # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \\ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \\ # If you want a network device without using virtio, you can use e1000. # You should install the virtio drivers and use virtio-net instead for # better performance. -s 2,e1000,tap0 \\ -s 2,virtio-net,tap0 \\`
>
> ### stop script
>
> This is your stop script. Save this as `stop.sh` somewhere.
>
> `NOTE:` This is currently set up to kill any process with `bhyve` in
> the name. On this system I'm only running one bhyve VM, which is the gaming one.
> Adjust accordingly to restrict its scope.
>
> `#!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=gaming --destroy`
>
> ### rc.d script
>
> This script will allow you to start the VM automatically at boot time. Save this
> file in your `/usr/local/etc/rc.d/` directory with the name `vm_gaming`.
> Make sure to adjust any paths to where you saved your start/stop scripts:
>
> `NOTE:` At the moment I noticed that if I do a `shutdown -r now`, I don't
> actually see Windows gracefully shutting down (like when I send the ACPI shutdown
> signal). Currently looking into this.
>
> `#!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=vm_gaming rcvar=vm_gaming_enable start_cmd="${name}_start" stop_cmd="${name}_stop" restart_cmd="${name}_restart" path_to_start_script="/atlantis/vms/gaming/start.sh" path_to_stop_script="/atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon \\ -o /var/log/${name}.log \\ "${path_to_start_script}" } vm_gaming_stop() { "${path_to_stop_script}" } vm_gaming_restart() { # NOTE: AMD users will most likely experience the famous # AMD Hardware Reset Bug. This means that after you reboot # the guest, you most likely won't have video out. If this # happens, you'll need to restart the host. Sometimes this # command will work for AMD users though. So you can try a # restart and see if it works. vm_gaming_stop vm_gaming_start } load_rc_config $name : ${vm_gaming_enable:="NO"} run_rc_command "$1"`
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?INv1QRTDanPqExdkRH9g_SlRzYv6-m_Xg5SWWYjpkvkQ_RqVdS_OuVJgJvQW3ZAnhsS334kzM-0IP0rj4c6MN2WRvws2_FII5XBr1aL5Rw4=>