Date: Wed, 24 Sep 2025 13:05:00 +0000 From: Jonathan Vasquez <jon@xyinn.org> To: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> In-Reply-To: <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <l8EqFTkrrLvYOUcKLKIRbCxHHRDMjM--eRHlFwVyC0oHzp58cXM9bP9uLrbMe3wree4k9tI4ERHgFiBt3s9KB7htgR7FXaNNwmub7bq7K14=@xyinn.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <mOxiSfmxWrHIMEStshHuwzAKYh003kyxHWvA_NY42h2TJfzeGaYSA5ePY1WNK67MtP_quGeVkFzRTTeSf7ESI-gMBbcoxtr7DeQ24xr5Ntg=@xyinn.org> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello all, I was able to get another stable vm on idle (this time without leaving Task Manager opened) for another 22 hours. I also spent some time today improving my start/stop scripts. Using Corvin's suggestion about the "daemon" process, I was able to now successfully start the VM at boot time without blocking. I had to make some adjustments to my bhyve script so it doesn't use "-l com1,stdio", because during boot time, even with me making the script require "LOGIN syscons", it still would throw the following error: Unable to initialize backend 'stdio' for LPC device com1 100 Device emulation initialization error: Inappropriate ioctl for device Which prevented the VM from starting. I've switched to using "-o console=stdio" and that worked. I'm able to get regular console output, which allows the VM to always start up successfully, and also allows us to log to a file when we run our vm start script through daemon. I've also observed that there were a few times when I restarted the VM without restarting the host, 2-3 times, and each time the video output worked (meaning I wasn't affected by the AMD Hardware Reset Bug). However, after that I wasn't able to get that working anymore. So I would still recommend restarting the host. I added the following to the script: - Daemonized support so the VM starts up non-blocking. This also includes logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since the pid addresses looked off. We also don't seem to need it at the moment. - Added the restart command (with a note about the AMD issue). - Added some quality of life stuff like having some variables at the top to easily set your start/stop paths. Some other things are that if I do "service vm_gaming <some command>", the current console in tmux seems to get really laggy and the input gets weird. If I open another tmux window, then input is fine again. There are no other consequences other than laggy/wrong input in the current console. This doesn't seem to negatively affect anything when starting it automatically at boot time through rc.d. One last thing is that I noticed that when I do a "shutdown -r now", I don't see windows getting the ACPI shutdown signal, so I don't see Windows gracefully shutting down. I have read if you do "reboot" it will just send the SIGTERM signal, and kill everything afterwards, so the "shutdown" command should be used to give advance notice to processes. Given that my rc.d script has "shutdown" in the keywords list, I was expecting the "vm_gaming_stop" command to be ran automatically in order to gracefully shutdown the VM, and clean itself up. Is this not working because I'm not using -P / -p? I've updated my blog post to include all of these new scripts and modifications. I'll post the current scripts below. Thanks for reading. Jonathan ## Bhyve Scripts ### start script This is the main script that will start your vm. Save this as **`start.sh`** in a folder called **`gaming`** somewhere on your system: ``` #!/bin/sh VM_DIR="$(dirname $(realpath $0))" VM_NAME="$(basename $VM_DIR)" cd "$VM_DIR" bhyve -AHPSw -c sockets=1,cores=16,threads=1 -m 32G \ -s 0,hostbridge \ -s 1,nvme,disk0.img \ -s 3:0,passthru,3/0/0 \ -s 3:1,passthru,3/0/1 \ -s 13:0,passthru,13/0/0 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=qemu \ -o console=stdio \ $VM_NAME # Exit the script here and leave some options for debugging more # ergonomically after the exit line. exit # Only use this when installing the VM for the first time: # VNC. Once you install Windows and enable RDP, you can turn this off. -s 29,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \ # The Windows ISO and the VirtIO Drivers ISO. # You can download the latest stable virtio drivers at: # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \ # If you want a network device without using virtio, you can use e1000. # You should install the virtio drivers and use virtio-net instead for # better performance. -s 2,e1000,tap0 \ -s 2,virtio-net,tap0 \ ``` ### stop script This is your stop script. Save this as **`stop.sh`** somewhere. **`NOTE:`** This is currently set up to kill any process with **`bhyve`** in the name. On this system I'm only running one bhyve VM, which is the gaming one. Adjust accordingly to restrict its scope. ``` #!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=gaming --destroy ``` ### rc.d script This script will allow you to start the VM automatically at boot time. Save this file in your **`/usr/local/etc/rc.d/`** directory with the name **`vm_gaming`**. Make sure to adjust any paths to where you saved your start/stop scripts: **`NOTE:`** At the moment I noticed that if I do a **`shutdown -r now`**, I don't actually see Windows gracefully shutting down (like when I send the ACPI shutdown signal). Currently looking into this. ``` #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=vm_gaming rcvar=vm_gaming_enable start_cmd="${name}_start" stop_cmd="${name}_stop" restart_cmd="${name}_restart" path_to_start_script="/atlantis/vms/gaming/start.sh" path_to_stop_script="/atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon \ -o /var/log/${name}.log \ "${path_to_start_script}" } vm_gaming_stop() { "${path_to_stop_script}" } vm_gaming_restart() { # NOTE: AMD users will most likely experience the famous # AMD Hardware Reset Bug. This means that after you reboot # the guest, you most likely won't have video out. If this # happens, you'll need to restart the host. Sometimes this # command will work for AMD users though. So you can try a # restart and see if it works. vm_gaming_stop vm_gaming_start } load_rc_config $name : ${vm_gaming_enable:="NO"} run_rc_command "$1" ```
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=>