From nobody Wed Sep 24 13:05:00 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWxsP05Whz67q7m for ; Wed, 24 Sep 2025 13:05:13 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10624.protonmail.ch (mail-10624.protonmail.ch [79.135.106.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWxsL1LqPz3WsK for ; Wed, 24 Sep 2025 13:05:09 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=NyRN6Vs7; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.24 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758719106; x=1758978306; bh=jWE7r/W7hQdL4vbI1QyMFk9wzDuE9a3Y38pBlsyRV1Y=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=NyRN6Vs7Pqz0UTXaOjB+HglSLuN440doWKhJoO+CSAveWW9r/VIf6USf22u3oFrp/ NSSR1guowYOlTXwrp95U8Qyubaf4F4QsYcG61j263n6La16fMBrkcJr0RI1kFDo9sF YmzOHibdwYMdBj5kuiQ5sNoz8xVejCZXQAkm7JEvPBWjmE90DShbYOcZFgvChvJuIs wQEFU1llTb/paMDlnHqRbUxQORPzPgFaotDcENRgxWQcn7wjzQN2d9C4l0AUuWT9Xn 3IKsP2CR7WPXUg2uW3qolxfXmEUZxupzlHZ2nZ5mozIy5QBw/KGSwIJZhDe6TvC4M6 XbB/f5Kec6gLA== Date: Wed, 24 Sep 2025 13:05:00 +0000 To: "freebsd-virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> In-Reply-To: <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 99973b1cf5039c86cd1b4f3f71e4ae9cbe9f14f9 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.37 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.970]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.24:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[jon]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.24:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cWxsL1LqPz3WsK Hello all, I was able to get another stable vm on idle (this time without leaving Task= Manager opened) for another 22 hours. I also spent some time today improving my sta= rt/stop scripts. Using Corvin's suggestion about the "daemon" process, I was able t= o now successfully start the VM at boot time without blocking. I had to make = some adjustments to my bhyve script so it doesn't use "-l com1,stdio", because d= uring boot time, even with me making the script require "LOGIN syscons", it still would throw the following error: Unable to initialize backend 'stdio' for LPC device com1 100 Device emulation initialization error: Inappropriate ioctl for device Which prevented the VM from starting. I've switched to using "-o console=3D= stdio" and that worked. I'm able to get regular console output, which allows the V= M to always start up successfully, and also allows us to log to a file when w= e run our vm start script through daemon. I've also observed that there were a few times when I restarted the VM with= out restarting the host, 2-3 times, and each time the video output worked (mean= ing I wasn't affected by the AMD Hardware Reset Bug). However, after that I was= n't able to get that working anymore. So I would still recommend restarting the= host. I added the following to the script: - Daemonized support so the VM starts up non-blocking. This also includes logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since= the pid addresses looked off. We also don't seem to need it at the moment. - Added the restart command (with a note about the AMD issue). - Added some quality of life stuff like having some variables at the top to easily set your start/stop paths. Some other things are that if I do "service vm_gaming ", the current console in tmux seems to get really laggy and the input gets weird.= If I open another tmux window, then input is fine again. There are no other consequences other than laggy/wrong input in the current console. This does= n't seem to negatively affect anything when starting it automatically at boot t= ime through rc.d. One last thing is that I noticed that when I do a "shutdown -r now", I don'= t see windows getting the ACPI shutdown signal, so I don't see Windows gracef= ully shutting down. I have read if you do "reboot" it will just send the SIGTERM= signal, and kill everything afterwards, so the "shutdown" command should be used to give advance notice to processes. Given that my rc.d script has "shutdown" = in the keywords list, I was expecting the "vm_gaming_stop" command to be ran automatically in order to gracefully shutdown the VM, and clean itself up. = Is this not working because I'm not using -P / -p? I've updated my blog post to include all of these new scripts and modificat= ions. I'll post the current scripts below. Thanks for reading. Jonathan ## Bhyve Scripts ### start script This is the main script that will start your vm. Save this as **`start.sh`*= * in a folder called **`gaming`** somewhere on your system: ``` #!/bin/sh VM_DIR=3D"$(dirname $(realpath $0))" VM_NAME=3D"$(basename $VM_DIR)" cd "$VM_DIR" bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ -s 0,hostbridge \ -s 1,nvme,disk0.img \ -s 3:0,passthru,3/0/0 \ -s 3:1,passthru,3/0/1 \ -s 13:0,passthru,13/0/0 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu \ -o console=3Dstdio \ $VM_NAME # Exit the script here and leave some options for debugging more # ergonomically after the exit line. exit # Only use this when installing the VM for the first time: # VNC. Once you install Windows and enable RDP, you can turn this off. -s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1024,h=3D768,wait \ # The Windows ISO and the VirtIO Drivers ISO. # You can download the latest stable virtio drivers at: # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-v= irtio/virtio-win.iso -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \ # If you want a network device without using virtio, you can use e1000. # You should install the virtio drivers and use virtio-net instead for # better performance. -s 2,e1000,tap0 \ -s 2,virtio-net,tap0 \ ``` ### stop script This is your stop script. Save this as **`stop.sh`** somewhere. **`NOTE:`** This is currently set up to kill any process with **`bhyve`** i= n the name. On this system I'm only running one bhyve VM, which is the gaming= one. Adjust accordingly to restrict its scope. ``` #!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=3Dgaming --destroy ``` ### rc.d script This script will allow you to start the VM automatically at boot time. Save= this file in your **`/usr/local/etc/rc.d/`** directory with the name **`vm_gamin= g`**. Make sure to adjust any paths to where you saved your start/stop scripts: **`NOTE:`** At the moment I noticed that if I do a **`shutdown -r now`**, I= don't actually see Windows gracefully shutting down (like when I send the ACPI sh= utdown signal). Currently looking into this. ``` #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" path_to_start_script=3D"/atlantis/vms/gaming/start.sh" path_to_stop_script=3D"/atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon \ -o /var/log/${name}.log \ "${path_to_start_script}" } vm_gaming_stop() { "${path_to_stop_script}" } vm_gaming_restart() { # NOTE: AMD users will most likely experience the famous # AMD Hardware Reset Bug. This means that after you reboot # the guest, you most likely won't have video out. If this # happens, you'll need to restart the host. Sometimes this # command will work for AMD users though. So you can try a # restart and see if it works. vm_gaming_stop vm_gaming_start } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_command "$1" ```