Date: Wed, 24 Sep 2025 13:05:00 +0000 From: Jonathan Vasquez <jon@xyinn.org> To: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> In-Reply-To: <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <l8EqFTkrrLvYOUcKLKIRbCxHHRDMjM--eRHlFwVyC0oHzp58cXM9bP9uLrbMe3wree4k9tI4ERHgFiBt3s9KB7htgR7FXaNNwmub7bq7K14=@xyinn.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <mOxiSfmxWrHIMEStshHuwzAKYh003kyxHWvA_NY42h2TJfzeGaYSA5ePY1WNK67MtP_quGeVkFzRTTeSf7ESI-gMBbcoxtr7DeQ24xr5Ntg=@xyinn.org> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello all, I was able to get another stable vm on idle (this time without leaving Task= Manager opened) for another 22 hours. I also spent some time today improving my sta= rt/stop scripts. Using Corvin's suggestion about the "daemon" process, I was able t= o now successfully start the VM at boot time without blocking. I had to make = some adjustments to my bhyve script so it doesn't use "-l com1,stdio", because d= uring boot time, even with me making the script require "LOGIN syscons", it still would throw the following error: Unable to initialize backend 'stdio' for LPC device com1 100 Device emulation initialization error: Inappropriate ioctl for device Which prevented the VM from starting. I've switched to using "-o console=3D= stdio" and that worked. I'm able to get regular console output, which allows the V= M to always start up successfully, and also allows us to log to a file when w= e run our vm start script through daemon. I've also observed that there were a few times when I restarted the VM with= out restarting the host, 2-3 times, and each time the video output worked (mean= ing I wasn't affected by the AMD Hardware Reset Bug). However, after that I was= n't able to get that working anymore. So I would still recommend restarting the= host. I added the following to the script: - Daemonized support so the VM starts up non-blocking. This also includes logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since= the pid addresses looked off. We also don't seem to need it at the moment. - Added the restart command (with a note about the AMD issue). - Added some quality of life stuff like having some variables at the top to easily set your start/stop paths. Some other things are that if I do "service vm_gaming <some command>", the current console in tmux seems to get really laggy and the input gets weird.= If I open another tmux window, then input is fine again. There are no other consequences other than laggy/wrong input in the current console. This does= n't seem to negatively affect anything when starting it automatically at boot t= ime through rc.d. One last thing is that I noticed that when I do a "shutdown -r now", I don'= t see windows getting the ACPI shutdown signal, so I don't see Windows gracef= ully shutting down. I have read if you do "reboot" it will just send the SIGTERM= signal, and kill everything afterwards, so the "shutdown" command should be used to give advance notice to processes. Given that my rc.d script has "shutdown" = in the keywords list, I was expecting the "vm_gaming_stop" command to be ran automatically in order to gracefully shutdown the VM, and clean itself up. = Is this not working because I'm not using -P / -p? I've updated my blog post to include all of these new scripts and modificat= ions. I'll post the current scripts below. Thanks for reading. Jonathan ## Bhyve Scripts ### start script This is the main script that will start your vm. Save this as **`start.sh`*= * in a folder called **`gaming`** somewhere on your system: ``` #!/bin/sh VM_DIR=3D"$(dirname $(realpath $0))" VM_NAME=3D"$(basename $VM_DIR)" cd "$VM_DIR" bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ -s 0,hostbridge \ -s 1,nvme,disk0.img \ -s 3:0,passthru,3/0/0 \ -s 3:1,passthru,3/0/1 \ -s 13:0,passthru,13/0/0 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu \ -o console=3Dstdio \ $VM_NAME # Exit the script here and leave some options for debugging more # ergonomically after the exit line. exit # Only use this when installing the VM for the first time: # VNC. Once you install Windows and enable RDP, you can turn this off. -s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1024,h=3D768,wait \ # The Windows ISO and the VirtIO Drivers ISO. # You can download the latest stable virtio drivers at: # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-v= irtio/virtio-win.iso -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \ # If you want a network device without using virtio, you can use e1000. # You should install the virtio drivers and use virtio-net instead for # better performance. -s 2,e1000,tap0 \ -s 2,virtio-net,tap0 \ ``` ### stop script This is your stop script. Save this as **`stop.sh`** somewhere. **`NOTE:`** This is currently set up to kill any process with **`bhyve`** i= n the name. On this system I'm only running one bhyve VM, which is the gaming= one. Adjust accordingly to restrict its scope. ``` #!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=3Dgaming --destroy ``` ### rc.d script This script will allow you to start the VM automatically at boot time. Save= this file in your **`/usr/local/etc/rc.d/`** directory with the name **`vm_gamin= g`**. Make sure to adjust any paths to where you saved your start/stop scripts: **`NOTE:`** At the moment I noticed that if I do a **`shutdown -r now`**, I= don't actually see Windows gracefully shutting down (like when I send the ACPI sh= utdown signal). Currently looking into this. ``` #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" path_to_start_script=3D"/atlantis/vms/gaming/start.sh" path_to_stop_script=3D"/atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon \ -o /var/log/${name}.log \ "${path_to_start_script}" } vm_gaming_stop() { "${path_to_stop_script}" } vm_gaming_restart() { # NOTE: AMD users will most likely experience the famous # AMD Hardware Reset Bug. This means that after you reboot # the guest, you most likely won't have video out. If this # happens, you'll need to restart the host. Sometimes this # command will work for AMD users though. So you can try a # restart and see if it works. vm_gaming_stop vm_gaming_start } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_command "$1" ```
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=>