From nobody Sun Sep 28 15:46:30 2025 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cZTFr1clYz68sSZ for ; Sun, 28 Sep 2025 15:46:40 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-24420.protonmail.ch (mail-24420.protonmail.ch [109.224.244.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cZTFn2tKCz3tN5 for ; Sun, 28 Sep 2025 15:46:36 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b=fi2mW70G; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 109.224.244.20 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1759074392; x=1759333592; bh=sTAcPxgTmv5oKZMvb6MdOQrqii3YdavefsoIWkpkS2s=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=fi2mW70GetEeBvf7tAgKPzgToBrCrWil9QLI38RVkslLb6TjTGq/A7YfL1yPRCoVa Nl8KqYOOfsUoRcNrVJd4hSTSpp8SH+peDcEfXRNiQsw174pAQTDx7GgWyvRCEMTVbe 1FW/EkHkAfqbgXFqcaeg2dKFg6P24duG5GJ12ENksC2n5djcRG74mVZ3ZDfdsQMIja 0ku6b5BMOwJkjtg5S7117xXuF1EqGKPnjJXI889QhswgNfXfJXWdIeI9QoWbLSvMsf r1KBvjRIMeHBXrhRp1gJpycZRNkdbQ24kbnAI3BjGfkuXdmqgSN9tf8KcXekSlAIQ+ YXeqNHl60tL4A== Date: Sun, 28 Sep 2025 15:46:30 +0000 To: "freebsd-virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: =?us-ascii?Q?<6CV-OY6BcErrWRit9jSpi6fWsYBG3E=5FZ3u6eTLPcz6foPAZV1gQpZYaZTR7JA=5F1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=3D@xyinn.org>_<77221679b528434788d667441d1a32a2@userve.net>_<44d9286c-0866-4805-827d-5cb91511800b@tilda.center>__<3HATaLlOogPiMZWbxSnWPCnJC11=5FunXloaoQzyF8Ek01s9gZGVqnKKruZgFTrqHQBCi5xWNEzLzK37xiNdn3yCwXG8U7NfXKRmwY8=5F5xCLw=3D@xyinn.org>_<7CNs3oS07TqJcWsTrJCZmDlAypBuy9yh4xr=5FiAgGnUOGjqAP5iRl1JMkuna8LwFSNIvphQLojKi214B4w6ef2q2IRgvjdolLwUD8KCnXaSQ=3D@xyinn.org>_<9c53f1805e22fcf2c9eb878e0c7cc9a723c30dd8.camel@FreeBSD.org>_<-rzPcOUnhEy7jFcu7cF7V=5FkjAVPV=5F2deuhv7EY-eJl536fUllXK-3SSnWq13HStABMZuphkRAysMMhDXvd=5FMWIlH4QaMGU-8l2nXKeOY-Eo=3D@xyinn.org>__?= Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 40800ea8fc35fc60338a4f58e2ef16c30dea4936 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[109.224.244.20:from]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:109.224.244.0/24]; MIME_GOOD(-0.10)[text/plain]; DKIM_TRACE(0.00)[xyinn.org:+]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; TO_DN_EQ_ADDR_ALL(0.00)[]; ARC_NA(0.00)[]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[jon]; ASN(0.00)[asn:62371, ipnet:109.224.244.0/24, country:CH]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[109.224.244.20:from] X-Rspamd-Queue-Id: 4cZTFn2tKCz3tN5 Hey all, I hope everyone is doing well. I wanted to give some final updates= since I feel I'm basically done with my research, set up, and testing. I'm= pleased to say that I'm officially calling my setup as "stable". For the p= ast week, I've been testing the VM by either playing some games on it for a= few hours, or leaving the VM on idle, and so far I haven't really experien= ced any VM crashes. The games seem to be working smoothly when I'm playing = them, and I don't get any sudden VM crashes. I left the VM on idle for the = past 1d20h and no issues. I wanted to finish working on my rc.d start up/sh= utdown script so I had to bring the VM down to do that :P. There were some = instances where the VM wouldn't really display any output and would effecti= vely be frozen when I rebooted the machine multiple consecutive times durin= g my testing, so much so that doing a "service vm_gaming restart" (or stop)= would just leave the script at the "Waiting for PIDs" message. In this cas= e I had to do a "bhyvectl --vm=3Dgaming --force-poweroff" to kill it. The s= ubsequent start of the VM then seemed to work correctly and video was worki= ng again. This doesn't happen all the time but it could happen so I wanted = to mention that for the record. Once the VM is up, it's pretty stable. I do= see the windows start up screen always say "Getting devices ready" and the= n it proceeds to complete the boot with no issues. Not sure what that's abo= ut but there are no negative consequences. In regards to my research regarding the graceful shutdown, given that throu= ghout all of many days of testing, my VM hasn't gotten any disk corruption,= I believe that the KEYWORD: shutdown (or KEYWORD: nojail shutdown) have be= en working correctly. When doing my testing, I do see that if I do a "servi= ce vm_gaming stop" explicitly, I see Windows 10 showing the light blue scre= en with the "Shutting down" message. When I do "shutdown -r now" I sometime= s see the same light blue screen, but usually I don't see the screen, what = I do see though is that if I move my mouse cursor to the middle of the scre= en, once I trigger the shutdown, I would see the light blue circle/loop aro= und the mouse cursor start to spin. This happens every single time I run th= e "shutdown -r now" command. So this means that the shutdown signal is prop= erly being sent, and my theory is that the GPU/OS is just not reflecting th= at screen update due to the speed at which things are happening. I also do = see that the time it takes to reboot after the shutdown -r now is called ta= kes a little bit of time, which I believe is also partly the time it takes = for the system to wait for the bhyve vm to shutdown and for resource to pos= sibly be cleaned (bhyvectl ... destroy). So I believe I'm done with the mai= n research, experimentation, and testing. I've updated my blog post as well= , and below you can find my updated rc.d script. I saw Corvin recently pus= hed some updates to CURRENT to get NVIDIA passthrough working (PIN related = work). I do have my Razer Blade 15" (2020) that has an Intel proc and NVIDI= A GTX 2060. In the future I may try and install FreeBSD on there to test th= e Intel/NVIDIA side of things, but that's not something I'm planning on doi= ng any time soon. Take care all and I hope all of this helps y'all have some fun and relax. S= tay safe, Jonathan! ---- updated script #!/bin/sh # PROVIDE: vm_gaming # REQUIRE: LOGIN FILESYSTEMS # KEYWORD: shutdown . /etc/rc.subr name=3D"vm_gaming" rcvar=3D"vm_gaming_enable" start_cmd=3D"${name}_start" stop_cmd=3D"${name}_stop" stop_postcmd=3D"${name}_poststop" restart_cmd=3D"${name}_restart" status_cmd=3D"${name}_status" pidfile=3D"/var/run/${name}.pid" vm_path=3D"/atlantis/vms/gaming" vm_name=3D"gaming" load_rc_config $name : ${vm_gaming_enable:=3D"NO"} vm_gaming_start() { =09daemon \ =09=09-o /var/log/${name}.log \ =09=09-p "${pidfile}" \ =09=09bhyve -AHPSw -c sockets=3D1,cores=3D16,threads=3D1 -m 32G \ =09=09-s 0,hostbridge \ =09=09-s 1,nvme,${vm_path}/disk0.img \ =09=09-s 3:0,passthru,3/0/0 \ =09=09-s 3:1,passthru,3/0/1 \ =09=09-s 13:0,passthru,13/0/0 \ =09=09-s 30,xhci,tablet \ =09=09-s 31,lpc \ =09=09-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,fwcfg=3Dqemu = \ =09=09-o console=3Dstdio \ =09=09${vm_name} } vm_gaming_stop() { =09if [ ! -f "${pidfile}" ]; then =09=09echo "the gaming vm isn't running!" =09=09return =09fi =09# Send SIGTERM twice to make sure Windows listens to =09# the ACPI shutdown signal. =09pid=3D"$(cat ${pidfile})" =09kill "${pid}" =09kill "${pid}" =09# Wait a bit for the guest to shutdown properly before =09# we continue shutting down the host. =09wait_for_pids "${pid}" } vm_gaming_poststop() { =09bhyvectl --vm=3D"${vm_name}" --destroy } vm_gaming_restart() { =09# NOTE: AMD users will most likely experience the famous =09# AMD Hardware Reset Bug. This means that after you reboot =09# the guest, you most likely won't have video out. If this =09# happens, you'll need to restart the host. Sometimes this =09# command will work for AMD users though. So you can try a =09# restart and see if it works. =09vm_gaming_stop =09vm_gaming_start } vm_gaming_status() { =09if [ -f "${pidfile}" ]; then =09=09echo "$(cat ${pidfile})" =09fi } run_rc_command "$1"