From nobody Mon Sep 22 16:28:32 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4cVpT40zxmz686nG for ; Mon, 22 Sep 2025 16:28:40 +0000 (UTC) (envelope-from jon@xyinn.org) Received: from mail-10624.protonmail.ch (mail-10624.protonmail.ch [79.135.106.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4cVpT32SFdz3jN1 for ; Mon, 22 Sep 2025 16:28:39 +0000 (UTC) (envelope-from jon@xyinn.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=xyinn.org header.s=protonmail3 header.b="Hv8E/+sD"; dmarc=pass (policy=none) header.from=xyinn.org; spf=pass (mx1.freebsd.org: domain of jon@xyinn.org designates 79.135.106.24 as permitted sender) smtp.mailfrom=jon@xyinn.org DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xyinn.org; s=protonmail3; t=1758558516; x=1758817716; bh=GWsGoLgsTPjBvhM9rXSRxhCQBzv1mkSHIYVYKPanNiA=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=Hv8E/+sDQIaKMr3sCzElJ9tlcKAPKMKGHZiG62jk0ByHVkHv9LTZ+0klVgBrhp4bk gbY86dmhkmNclSGjtGK3NYCxvxRhTEjAD9DiwMYENPPaj5HYtZ6b15f1tf0MyMupRm cKYHg0Sc9qZ+51FQOuxHZemJOa9M/FOoSeaPl27cNkov7LMLrzWv0tfFVTmP2C9rJq kCSJGtj/mFSlcQ+U2lhhDJF+eVC12ygPN5Hpqef2xN4fHu/1iH91P4ISO4B72MAo6e SvPL9bTP5+AAf6ErhhuWJieVHofjCSRZcsUD6/F3ey1QYzbJvHv7+dqSpZFs7t7XHt K0jtlkKZws07w== Date: Mon, 22 Sep 2025 16:28:32 +0000 To: "virtualization@freebsd.org" From: Jonathan Vasquez Subject: Re: GPU Passthrough on FreeBSD 14.3 (AMD Radeon RX 6900 XT and Windows 10 Pro) Message-ID: In-Reply-To: References: <6CV-OY6BcErrWRit9jSpi6fWsYBG3E_Z3u6eTLPcz6foPAZV1gQpZYaZTR7JA_1ot5RQVqrWQaLxJFySXjspIhSbBJGxmckcDQyzxhALNus=@xyinn.org> <21e892ba-bea5-4e65-91cf-409e5e927f67@FreeBSD.org> Feedback-ID: 12351801:user:proton X-Pm-Message-ID: 5bf75b701fbbb47f224ea147ad6f744019c809a2 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[xyinn.org,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[79.135.106.24:from]; R_SPF_ALLOW(-0.20)[+ip4:79.135.106.0/24]; R_DKIM_ALLOW(-0.20)[xyinn.org:s=protonmail3]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; MISSING_XM_UA(0.00)[]; FREEFALL_USER(0.00)[jon]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_ZERO(0.00)[0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[79.135.106.24:from]; TO_DN_EQ_ADDR_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; DKIM_TRACE(0.00)[xyinn.org:+] X-Rspamd-Queue-Id: 4cVpT32SFdz3jN1 I've also started to upload some of these crash files (like the 0x88.0.txt = file) to Lumo for analysis. The AI info is interesting but it tends to have= errors on invalid bhyve command line flags or sometimes even what code it = thinks 0x88 belongs to. I'll post some of it below. This was its response t= o the 0x88.0.txt data. If we look at what it says for the 0x88 code, it doe= sn't match what I see in the freebsd code (#define=09VMCB_EXIT_ICEBP=09=09= =090x88): https://github.com/freebsd/freebsd-src/blob/4c3a868d13c053ef17326= 8cdfe1365978a282178/sys/amd64/vmm/amd/vmcb.h#L94 is the AI wrong about this? ---- Lumo Response: I can see you=E2=80=99ve uploaded a text dump named 0x88.0.txt that looks l= ike a snapshot of a bhyve/AMD=E2=80=91SVM virtual=E2=80=91machine (VM) stat= e and statistics. Here=E2=80=99s a concise walkthrough of what the file con= tains and what it tells us about the VM=E2=80=99s execution. 1. High=E2=80=91level picture Item=09Value / Meaning VM exit reason=090x88 =E2=80=93 this is the exit code for VM_EXIT_REP_INSB = (a repeated string=E2=80=91input operation, i.e., rep insb/insw/insd). Current exception vector=096 =E2=80=93 a #UD (invalid opcode) or #GP depend= ing on context; the dump shows the VM is presently handling an exception. Number of CPUs=0916 vCPUs (IDs=E2=80=AF0=E2=80=9115) are active and suspend= ed; the host has a 16=E2=80=91core topology. Total VM exits=093=E2=80=AF259=E2=80=AF037 =E2=80=93 a fairly large number,= indicating heavy interaction between guest and hypervisor. Instruction=E2=80=91emulation exits=091=E2=80=AF465=E2=80=AF913 =E2=80= =93 most of the exits are for instructions that bhyve had to emulate (e.g.,= I/O, privileged ops). Pause=E2=80=91instruction exits=091=E2=80=AF138=E2=80=AF635 =E2=80=93 the g= uest frequently executed pause, which bhyve intercepts for power=E2=80= =91saving. HLT exits=0975=E2=80=AF539 =E2=80=93 the guest entered the halt state many = times (often seen in idle loops). In/out exits=09492=E2=80=AF316 =E2=80=93 the guest performed port=E2=80= =91I/O operations that required hypervisor handling. CPUID exits=093=E2=80=AF735 =E2=80=93 the guest queried CPU features. MSR reads/writes=091=E2=80=AF783 reads, 95 writes =E2=80=93 typical for a m= odern OS that accesses model=E2=80=91specific registers. Nested page=E2=80=91fault exits=09610 =E2=80=93 indicates the guest= =E2=80=99s second=E2=80=91level address translation (NPT/EPT) missed some p= ages. Memory layout=09Two huge sysmem regions (=E2=89=8833=E2=80=AFGB and = =E2=89=8829=E2=80=AFGB) plus a 16=E2=80=AFKB boot ROM region. Most of the R= AM is =E2=80=9Cwired=E2=80=9D (pinned) for the VM. RTC time=09Mon Sep 22 10:35:00 2025 =E2=80=93 matches today=E2=80=99s date,= confirming the snapshot is recent. 2. What the 0x88 exit really means Opcode involved: rep insb/insw/insd (string input from an I/O port). Why bhyve exits: AMD=E2=80=91SVM (and Intel VT=E2=80=91x) cannot direct= ly execute this instruction in guest mode; they force a VM exit so the hype= rvisor can either emulate the I/O or inject a #DB (debug) exception. Typical causes: Guest reading from a legacy serial port, keyboard controller, or ot= her device that uses bulk input. Anti=E2=80=91debugging tricks where malware repeatedly executes rep= insb to detect a hypervisor. Impact: Each such exit costs a few microseconds of latency. A high freq= uency can degrade performance, especially if the guest is in a tight loop d= oing bulk reads. 3. Other notable fields Field=09Interpretation cr_intercept =3D 0xfee2fee2=09The hypervisor is set to intercept many contr= ol=E2=80=91register accesses (CR0, CR3, CR4, etc.). exc_intercept =3D 0x00040000=09Exception interception is enabled for the #P= F (page=E2=80=91fault) vector. inst1_intercept =3D 0xddc4000f=09Intercepts a specific instruction pattern = (likely hlt and pause). exception_bitmap =3D 0x40000=09Only the page=E2=80=91fault exception (bit= =E2=80=AF18) is being intercepted. io_bitmap =3D 0x2b053b000=09Defines which I/O ports cause exits; the presen= ce of many in/out exits suggests a fairly wide bitmap. msr_bitmap =3D 0x2b0539000=09Controls which Model=E2=80=91Specific Register= accesses trigger exits. capability flags=09The VM supports hlt_exit, pause_exit, unrestricted_guest= , ipi_exit; lacks mtrap_exit, enable_invpcid, etc. guest_pat =3D 0x0007010600070106=09PAT (Page Attribute Table) settings = =E2=80=93 standard write=E2=80=91combining / write=E2=80=91through configur= ation. asid =3D 0x07fd=09Address=E2=80=91Space Identifier used for NPT/EPT = =E2=80=93 helps reduce TLB flushes. rflags =3D 0x40193=09Typical user=E2=80=91mode flag state (IF=3D1, ZF=3D0, = etc.). rip =3D 0xffffd001b2ce151d=09Current instruction pointer; the address falls= within the first sysmem region, suggesting the guest is executing normal c= ode when the dump was taken. rsp =3D 0xffffe3018da8e7b8=09Stack pointer =E2=80=93 also in the wired sysm= em area. 4. Performance clues High pause=E2=80=91exit count (>1=E2=80=AFM) + high hlt=E2=80=91exit co= unt (~75=E2=80=AFk) indicate the guest spends a lot of time in idle loops. = That=E2=80=99s typical for a server VM that sleeps when idle. Large number of instruction=E2=80=91emulation exits (~1.46=E2=80=AFM) s= uggests the guest is doing a fair amount of privileged I/O (e.g., disk, net= work, or device drivers) that the hypervisor must emulate. Relatively low NMI and ExtINT counts (0) imply the VM isn=E2=80=99t rec= eiving asynchronous hardware interrupts, which aligns with a mostly self= =E2=80=91contained workload. 5. What you might want to investigate next Why so many rep insb exits? Look at the guest=E2=80=99s I/O port map (e.g., serial console, leg= acy floppy). If the guest is repeatedly reading from a port that isn= =E2=80=99t needed, you could disable that device in the bhyve config. If the exits are caused by malicious anti=E2=80=91debug code, consi= der adding a pass=E2=80=91through for that port or using a more permissive = I/O bitmap. Tune the I/O bitmap If you know which ports the guest actually needs, narrow the bitmap= to reduce unnecessary exits. Check pause=E2=80=91exit handling Modern kernels already use pause for spin=E2=80=91waits; the high c= ount is normal, but you can enable the pause_exit capability (already set) = to let the hypervisor handle them efficiently. Memory wiring Almost all RAM is marked =E2=80=9Cwired/iommu=E2=80=9D. If you don= =E2=80=99t need that much pinned memory, you could relax the wiring to allo= w the host to reclaim pages under pressure. Look at the instruction pointer (rip) Mapping 0xffffd001b2ce151d back to the guest=E2=80=99s binary (via = objdump/gdb on the guest image) could reveal exactly which instruction caus= ed the 0x88 exit. That can confirm whether it=E2=80=99s a legitimate driver= read or something suspicious. TL;DR Your dump shows a bhyve VM that is currently handling a VM_EXIT_REP_INSB (0= x88) =E2=80=93 a bulk input operation from an I/O port. The VM has been qui= te busy: over three million exits, many of which are for instruction emulat= ion, pause, and I/O. Memory is largely wired, and the guest is idle= =E2=80=91heavy (lots of hlt/pause). If you want to improve performance or d= iagnose a possible anti=E2=80=91debugging pattern, focus on the ports gener= ating the rep insb exits and consider tightening the I/O bitmap or providin= g a proper device backend.