From nobody Tue Sep 23 12:50:17 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cWKZq4T2fz68c8l for ; Tue, 23 Sep 2025 12:50:27 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-4317.protonmail.ch (mail-4317.protonmail.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cWKZl1Gkgz3SLS for ; Tue, 23 Sep 2025 12:50:23 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=I3OT4Szj; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 185.70.43.17 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758631820; x=1758891020; bh=KC4qe9TeDWbGUBQlqi7s30rVueNu65y8QJI2GwdHzSM=; h=Date:To:From:Cc:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=I3OT4Szj/M+bBpSKrle27cjUkB3w0fZTwG4E/MJ7uj+HYSgO5Gap/6bXs5L93nqtb tvxVpNh5+KGY5JApYexCG11FXeOIIOIJt6UE/HKqdvTsw+xtyDk1/vrVyKjwQkVCSo ZvYk9fC+iVb9/CS6d0OvgKnu5f9mi2HApfZCIFwjOtUdZzV4GIcSwm1PmvwKE10n8C hnmAGke7TCH/fv6SUhjq/BbDKDc82Q9NtIiKS+e+xnOeejFSyQaKra3dE9Wau5fyNk VSStUUR0rdGSxrljZBsSodk9/GNFcCngvTkVKV6IikdoYmNT+1ZFF0OBhLFR6w+3Cz ZO0RLwJFK4RQA== Date: Tue, 23 Sep 2025 12:50:17 +0000 To: =?utf-8?Q?Goran_Meki=C4=87?= From: Jonathan Vasquez Cc: Matt Churchyard , "freebsd-virtualization@freebsd.org" Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: <3HATaLlOogPiMZWbxSnWPCnJC11_unXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8_5xCLw=@xyinn.org> In-Reply-To: References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> <77221679b528434788d667441d1a32a2@userve.net> <44d9286c-0866-4805-827d-5cb91511800b@tilda.center> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: e6bc22d34daea356800598f4574d8012def05c2d List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[185.70.43.17:from]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[185.70.43.17:from]; MIME_TRACE(0.00)[0:+]; FREEFALL_USER(0.00)[jon]; TO_DN_EQ_ADDR_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; MISSING_XM_UA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_COUNT_ZERO(0.00)[0]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cWKZl1Gkgz3SLS Alright all. So a few updates: After the 22+ hour uptime two days ago, I did a manual reboot of host/guest= to continue testing the "deterministic stability" of the system (as I ment= ioned yesterday). Within a few minutes, the VM exited with a new error code= (0x88) (previously it was 0x60). After I did the PCI passthrough adjustmen= ts, I haven't gotten 0x60 anymore and the VM has become a lot more stable. = For whatever reason I do sometimes still get a 0x88, but it hasn't happened= too many times. After the initial 0x88 exit code, I rebooted the host/vm a= gain, then the VM worked for about 3-4 hours before getting another 0x88. A= fter another host/vm reboot, the VM became stable and ran for another 19 ho= urs (which is this morning). I then turned it off since I wanted to experim= ent with the pkill situation we were discussing recently. So besides figuri= ng out what's causing it to sometimes do a 0x88, we are close I think to a = stable setup. I've noticed that in situations where it stays stable, I usua= lly have my Task Manager up since I want to see the resource utilization, t= emperatures, and uptime of the system quickly, while I monitor it. Could th= ere be something with the task manager always polling different hardware fo= r their state that may cause the VM to stay alive (vs not having task manag= er and just idling the machine causing it to die?). There is no sleep suppo= rt in the VM so we don't need to worry about the VM going to sleep/hiberati= on. I've also worked on a start/stop script to integrate this into the rc.d sys= tem a little bit better. Before I was using /etc/crontab and @reboot to sta= rt the VM at boot time after everything has started up, which works fine, b= ut I couldn't find a @shutdown equivalent. So I wrote a basic rc.d script. = The script is working but the problem is that for w/e reason it blocks the = rest of the services on my machine from starting. I investigated it a bit a= nd it has to do with "rcorder". I also analyzed the order by doing "rcorder= /etc/rc.d/* /usr/local/etc/rc.d/*" which revealed where in the chain it fa= lls. Originally there was no "REQUIRE:" line so it was being thrown into th= e beginning of the chain. I then switched it to "REQUIRE: LOGIN" which thre= w it close to the end of the chain (which is good), but it still has some s= ervices that follow it, and I noticed that even though I can ping the box a= fter reboot, for w/e reason even though sshd falls before LOGIN, I still co= uldn't ssh into the box until I shutdown the Windows VM, which allows execu= tion to continue. The start script I have for bhyve is a blocking call sinc= e it runs "bhyve ..." which would end up capturing the terminal. When I do = it through /etc/crontab @reboot, this doesn't seem to be an issue, but it i= s when doing it through an rc.d script. Anyone have any recommendations abo= ut this? I'll post the three scripts I'm using at the end of this message. Also, I'm more open now to writing a new section for the handbook regarding= everything I've learned for GPU passthrough for gaming purposes. Although = I have to preface that my experience is limited and it's only optimized for= this flow, so if I write it, I would only be able to write about this part= icular path and things/caveats around it. Normally the handbook usually has= a more general knowledge approach as well which I'm not qualified to write= about at the moment. Would it be fine for me to start writing this section= and get it into the handbook, and other people more experienced than me ca= n expand that section with more general knowledge about bhyve/passthrough (= and of course Intel/NVIDIA stuff can come after)? I can start with GPU pass= through / gaming / AMD Host + AMD Dedicated Card / and I can also add the s= tart/stop scripts + rc.d script. The paths, names, and values will need to be adjusted for each person's ind= ividual setup: VM START SCRIPT: #!/bin/sh =20 VM_DIR=3D"$(dirname $(realpath $0))" =20 VM_NAME=3D"$(basename $VM_DIR)" =20 cd "$VM_DIR" =20 bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ -s 0,hostbridge \ -s 1,nvme,disk0.img \ -s 3:0,passthru,3/0/0 \ -s 3:1,passthru,3/0/1 \ -s 13:0,passthru,13/0/0 \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu \ -l com1,stdio \ $VM_NAME # Exit the script here and leave some options for debugging more =20 # ergonomically after the exit line. =20 exit =20 # Only use this when installing the VM for the first time: =20 # VNC. Once you install Windows and enable RDP, you can turn this off. = =20 -s 29,fbuf,tcp=3D0.0.0.0:5900,w=3D1024,h=3D768,wait \ =20 # The Windows ISO and the VirtIO Drivers ISO. =20 # You can download the latest stable virtio drivers at: =20 # https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-v= irtio/virtio-win.iso =20 -s 4,ahci-cd,../files/Win10_22H2_English_x64v1_2023.iso \ -s 5,ahci-cd,../files/virtio-win-0.1.271.iso \ =20 # If you want a network device without using virtio, you can use e1000. = =20 # You should install the virtio drivers and use virtio-net instead for = =20 # better performance. =20 -s 2,e1000,tap0 \=20 -s 2,virtio-net,tap0 \ =20 VM STOP SCRIPT: #!/bin/sh # Send SIGTERM twice to make sure Windows listens to # the ACPI shutdown signal. pkill bhyve pkill bhyve # Wait a bit for the guest to shutdown properly before # we continue shutting down the host. sleep 10 bhyvectl --vm=3Dgaming --destroy RC.D SCRIPT: #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN # KEYWORD: nojail shutdown . /etc/rc.subr name=3Dvm_gaming rcvar=3Dvm_gaming_enable=20 start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" vm_gaming_start() { =09/atlantis/vms/gaming/start.sh } vm_gaming_stop() { =09/atlantis/vms/gaming/stop.sh } load_rc_config $name=20 : ${vm_gaming_enable:=3D"NO"}=20 run_rc_command "$1"