From nobody Thu Sep 25 12:01:43 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cXXPz29CPz68Pdv for ; Thu, 25 Sep 2025 12:01:59 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10626.protonmail.ch (mail-10626.protonmail.ch [79.135.106.26]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cXXPs6MpPz3xld for ; Thu, 25 Sep 2025 12:01:52 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=B9mvWExO; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.26 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758801708; x=1759060908; bh=ot6TObKxNERBnysgye/p4IHr6UkFppW6yCcbdNJcDTY=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=B9mvWExO/hhjHMO/VEy/8RxJMBJkzuQXRxtuGhaV8g1/TnB1yKZI4TGyPYcg6S9Tc C0jcrnvSa0RP/ONPEM5vGx2pCW8Jz+6gQY+mItwwukKPCUM2ngjvaSR+BnqSO4hBnb ZHjOW9Xl0jonrtTAgUyJYtTVqxTW/nDfbpe/VsrHIc2N0nZVCL00hnIOx37bBvw/MF jfvMLfVMVffUmN2qNz9U4Iup748n7R/suP88eyw7ZQ4No/tJ6fMziYvIm0COaV/7va Hc3Jt72VEdOhFJJ0FSzRHaafBwl3jNweri2Zy2+59qf+2bB+QfXw4rZBDcaivNjDxf BEWHwMMJ5i39w== Date: Thu, 25 Sep 2025 12:01:43 +0000 To: "freebsd-virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> <7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr_iAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=@xyinn.org> <9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org> <-rzPcOUnhEy7jFcu7cF7V_kjAVPV_2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd_MWIlH4QaMGU-8l2nXKeOY-Eo=@xyinn.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: a4562f7de2440b6a055cecd5e12ab9070c1cf1bd List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.26:from]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; MIME_GOOD(-0.10)[text/plain]; FREEFALL_USER(0.00)[jon]; MIME_TRACE(0.00)[0:+]; MISSING_XM_UA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:62371, ipnet:79.135.106.0/24, country:CH]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_EQ_ADDR_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.26:from]; RCVD_COUNT_ZERO(0.00)[0]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cXXPs6MpPz3xld Hey all, I was able to get another 21+ hours of uptime on idle with no issues, and I= did some more gaming last night and it's stable. I haven't experienced any= of the weird vm exits recently so that's good. There are probably cases wh= ere the vm may exit but I'm not sure what those situations are given the re= cent stability. I'll keep monitoring but I'm pretty happy with how everythi= ng turned out so far. I've also debug the script a bit more to see why it wasn't shutting down. T= he docs aren't explicit about it (or at least I haven't found a place yet f= or it), but I think it may be related to the pid stuff. It doesn't seem tha= t rc.shutdown calls the stop command if it's simply defined given there are= performance optimizations to skip certain things even if KEYWORD: shutdown= is defined. So after doing a "ps -aux | grep gaming" and seeing what showe= d up, I saw that there are 3-4 pids that exist with my current set up. daem= on pid, main bhyve pid, start.sh pid, etc. The pid that was getting written= by daemon previously was the pid of my start.sh script, but the pid of the= bhyve process that gets started by start.sh, so that would be a "grandchil= d of daemon", so I think that's why originally it wasn't working for me. I = now merged my start/stop script logic into the main rc.d script, so daemon = is directly executing bhyve, and thus the child process recorded in the pid= file is correct. The stop command can now load up the child pidfile, and pr= operly call kill on that pid directly. My script now works on shutdown prop= erly. The only thing I noticed was that even though I do now see Windows di= splaying the "shutting down" message, it kinda seems like the system isn't = necessarily obeying my "sleep 10" call afterwards, which means it probably = still isn't 100% a graceful shutdown, but much better than before. There is= also the bhyvectl --destroy ... command after and I'm not sure if that's r= unning. Below is the improved and centralized rc.d script: #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable vm_path=3D"/atlantis/vms/gaming" vm_name=3D"gaming" start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" pidfile=3D"/var/run/${name}.child.pid" vm_gaming_start() { =09daemon \ =09=09-o /var/log/${name}.log \ =09=09-p "${pidfile}" \ =09=09bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ =09=09-s 0,hostbridge \ =09=09-s 1,nvme,${vm_path}/disk0.img \ =09=09-s 3:0,passthru,3/0/0 \ =09=09-s 3:1,passthru,3/0/1 \ =09=09-s 13:0,passthru,13/0/0 \ =09=09-s 30,xhci,tablet \ =09=09-s 31,lpc \ =09=09-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu = \ =09=09-o console=3Dstdio \ =09=09${vm_name} } vm_gaming_stop() { =09# Send SIGTERM twice to make sure Windows listens to =09# the ACPI shutdown signal. =09pid=3D"$(cat ${pidfile})" =09kill ${pid} =09kill ${pid} =09# Wait a bit for the guest to shutdown properly before =09# we continue shutting down the host. =09sleep 10 =09bhyvectl --vm=3D${vm_name} --destroy } vm_gaming_restart() { =09# NOTE: AMD users will most likely experience the famous =09# AMD Hardware Reset Bug. This means that after you reboot =09# the guest, you most likely won't have video out. If this =09# happens, you'll need to restart the host. Sometimes this =09# command will work for AMD users though. So you can try a =09# restart and see if it works. =09vm_gaming_stop =09vm_gaming_start } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_command "$1" Jonathan Vasquez PGP: 34DA 858C 1447 509E C77A D49F FB85 90B7 C4CA 5279 Sent with ProtonMail Secure Email On Wednesday, September 24th, 2025 at 09:05, Jonathan Vasquez wrote: > Hello all, >=20 > I was able to get another stable vm on idle (this time without leaving Ta= sk Manager > opened) for another 22 hours. I also spent some time today improving my s= tart/stop > scripts. Using Corvin's suggestion about the "daemon" process, I was able= to > now successfully start the VM at boot time without blocking. I had to mak= e some > adjustments to my bhyve script so it doesn't use "-l com1,stdio", because= during > boot time, even with me making the script require "LOGIN syscons", it sti= ll > would throw the following error: >=20 > Unable to initialize backend 'stdio' for LPC device com1 > 100 Device emulation initialization error: Inappropriate ioctl for device >=20 > Which prevented the VM from starting. I've switched to using "-o console= =3Dstdio" > and that worked. I'm able to get regular console output, which allows the= VM > to always start up successfully, and also allows us to log to a file when= we run > our vm start script through daemon. >=20 > I've also observed that there were a few times when I restarted the VM wi= thout > restarting the host, 2-3 times, and each time the video output worked (me= aning > I wasn't affected by the AMD Hardware Reset Bug). However, after that I w= asn't > able to get that working anymore. So I would still recommend restarting t= he host. >=20 > I added the following to the script: >=20 > - Daemonized support so the VM starts up non-blocking. This also includes > logging to a file at /var/log/vm_gaming.log. I didn't add -P and -p since= the > pid addresses looked off. We also don't seem to need it at the moment. > - Added the restart command (with a note about the AMD issue). > - Added some quality of life stuff like having some variables at the top > to easily set your start/stop paths. >=20 > Some other things are that if I do "service vm_gaming ", th= e >=20 > current console in tmux seems to get really laggy and the input gets weir= d. If > I open another tmux window, then input is fine again. There are no other > consequences other than laggy/wrong input in the current console. This do= esn't > seem to negatively affect anything when starting it automatically at boot= time > through rc.d. >=20 > One last thing is that I noticed that when I do a "shutdown -r now", I do= n't > see windows getting the ACPI shutdown signal, so I don't see Windows grac= efully > shutting down. I have read if you do "reboot" it will just send the SIGTE= RM signal, > and kill everything afterwards, so the "shutdown" command should be used = to > give advance notice to processes. Given that my rc.d script has "shutdown= " in > the keywords list, I was expecting the "vm_gaming_stop" command to be ran > automatically in order to gracefully shutdown the VM, and clean itself up= . Is this > not working because I'm not using -P / -p? >=20 > I've updated my blog post to include all of these new scripts and modific= ations. >=20 > I'll post the current scripts below. Thanks for reading. >=20 > Jonathan >=20 >=20 > ## Bhyve Scripts >=20 > ### start script >=20 > This is the main script that will start your vm. Save this as `start.sh` > in a folder called `gaming` somewhere on your system: >=20 > `#!/bin/sh VM_DIR=3D"$(dirname $(realpath $0))" VM_NAME=3D"$(basename $VM= _DIR)" cd "$VM_DIR" bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 3= 2G \\ -s 0,hostbridge \\ -s 1,nvme,disk0.img \\ -s 3:0,passthru,3/0/0 \\ -s= 3:1,passthru,3/0/1 \\ -s 13:0,passthru,13/0/0 \\ -s 30,xhci,tablet \\ -s 3= 1,lpc \\ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqe= mu \\ -o console=3Dstdio \\ $VM_NAME # Exit the script here and leave some = options for debugging more # ergonomically after the exit line. exit # Only= use this when installing the VM for the first time: # VNC. Once you instal= l Windows and enable RDP, you can turn this off. -s 29,fbuf,tcp=3D0.0.0.0:5= 900,w=3D1024,h=3D768,wait \\ # The Windows ISO and the VirtIO Drivers ISO. = # You can download the latest stable virtio drivers at: # https://fedorapeo= ple.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.is= o -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \\ -s 5,ahci-cd,.= ./files/virtio-win-0.1.271.iso \\ # If you want a network device without us= ing virtio, you can use e1000. # You should install the virtio drivers and = use virtio-net instead for # better performance. -s 2,e1000,tap0 \\ -s 2,vi= rtio-net,tap0 \\` >=20 > ### stop script >=20 > This is your stop script. Save this as `stop.sh` somewhere. >=20 > `NOTE:` This is currently set up to kill any process with `bhyve` in > the name. On this system I'm only running one bhyve VM, which is the gami= ng one. > Adjust accordingly to restrict its scope. >=20 > `#!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACP= I shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to sh= utdown properly before # we continue shutting down the host. sleep 10 bhyve= ctl --vm=3Dgaming --destroy` >=20 > ### rc.d script >=20 > This script will allow you to start the VM automatically at boot time. Sa= ve this > file in your `/usr/local/etc/rc.d/` directory with the name `vm_gaming`. > Make sure to adjust any paths to where you saved your start/stop scripts: >=20 > `NOTE:` At the moment I noticed that if I do a `shutdown -r now`, I don't > actually see Windows gracefully shutting down (like when I send the ACPI = shutdown > signal). Currently looking into this. >=20 > `#!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdo= wn . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable start_cmd=3D"${= name}_start" stop_cmd=3D"${name}_stop" restart_cmd=3D"${name}_restart" path= _to_start_script=3D"/atlantis/vms/gaming/start.sh" path_to_stop_script=3D"/= atlantis/vms/gaming/stop.sh" vm_gaming_start() { daemon \\ -o /var/log/${na= me}.log \\ "${path_to_start_script}" } vm_gaming_stop() { "${path_to_stop_s= cript}" } vm_gaming_restart() { # NOTE: AMD users will most likely experien= ce the famous # AMD Hardware Reset Bug. This means that after you reboot # = the guest, you most likely won't have video out. If this # happens, you'll = need to restart the host. Sometimes this # command will work for AMD users = though. So you can try a # restart and see if it works. vm_gaming_stop vm_g= aming_start } load_rc_config $name : ${vm_gaming_enable:=3D"NO"} run_rc_com= mand "$1"`