From nobody Sat Oct 11 19:58:15 2025 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ckZD73prRz6CPG0 for ; Sat, 11 Oct 2025 19:58:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R12" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ckZD73F7Qz3s0K for ; Sat, 11 Oct 2025 19:58:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760212695; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DaU4g9ocS2lnMXyU4nr7RNbNi+u1j856WTHPDduhu5U=; b=yV2s2xCxuYdkDGAFAZr3Idb78VLCd5CR2Ynt7+CcW4WiH5IUlf9KDK2ts1yfZKkG6ffUTf YXYQcLhzp9DBnNZaBkEG9SkhGqUgm+DOqSfvJXNaRuWSTN51zixV4u5ITJssuOD0mOz2WQ W6pqAQEd8jO26WbPow1Jx/uoETFjZr12z2L8ELmxRqW25X3fFgKP0pDG0sct2meQpJ9IIK YvEYVytLjDW7qW6GAM3oidwCLV6Z+94zBsF3MPv0HKlwu+/ECgA8POOijw4ZheZuCgx3PS gxnz7Y7SuHG8wzExSlQf+xKosNKei2zuztRZ1Iyyg+OpGoNizeGL3k9zeX+lyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1760212695; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=DaU4g9ocS2lnMXyU4nr7RNbNi+u1j856WTHPDduhu5U=; b=BLQfOdsdIdZUmlZNfCGH+V78eK5LawL3qtTyjxZRQFxmRMl+eaxe4gkXTg2kk4nwHS+pFL 6I0XKV4I32nbdToXSDcLrJWME7aQqoRg4iOBfWp0K+GoL2xSi0zJ83ciip6fkhgIX1fNd2 +HUgUPT9AxWsW/khan4zHXsLgLIh725nDyCb1DbNyvIpp5xnnDHH50bPOKPZm0glgK/bFg WRrU//ODxJeS3rtP0e/Duv3J0a5Z0iLfXLTVY0yrQbrYIyV0ow1IwuX1DA46nlcR1LCymX evhAWnpBP/lIusyWpUDiUeHEBnP/hTDDiqS5SJ+4ZNLJFgdeJBowGdbUr/Y7oQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1760212695; a=rsa-sha256; cv=none; b=a06qSCxd1N6tMx3NbjR4F7oVVgLAU0CgImtWpYZ7wp8IeCgqO57bz4aFsSALqye25CDxfX 6t+hk4qm1wVVUs2aytIgcvjCz7WfX5mT+MI3CkUVsdVt5B2oMrVGJmij4pf8/DXFnTS7+4 Rgh2eM6Nv4Lf2AVxr8GubSWagCK1TKhQhqsyvmO6xIi70UUIF/trTbt28NBlb9xfzL3daK dNozOiPJLxZAk7Res/5OCKwAex1eyL4TYUaCdwTEdTuc188NHuR6NXcOU5l0o2B3f6/IPH Q99YRuZN+fG8LYXjS45EDh644Gj02NI0QBQp2IvyECBkSyTHXpVDCgfH2SecUA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4ckZD72rnMzp0Z for ; Sat, 11 Oct 2025 19:58:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 59BJwFZF038909 for ; Sat, 11 Oct 2025 19:58:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 59BJwFMl038907 for virtualization@FreeBSD.org; Sat, 11 Oct 2025 19:58:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 290155] bhyve PCI passthrough regression in 14.3-RELEASE: VMM IOMMU fails to initialize on Intel systems Date: Sat, 11 Oct 2025 19:58:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 14.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: luc+freebsd@2113.ch X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D290155 Bug ID: 290155 Summary: bhyve PCI passthrough regression in 14.3-RELEASE: VMM IOMMU fails to initialize on Intel systems Product: Base System Version: 14.3-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: luc+freebsd@2113.ch Created attachment 264491 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D264491&action= =3Dedit dmesg output on workin host 14.2 Host Kernel: 14.2-RELEASE-p1 FreeBSD app-01 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64 =3D=3D=3D HARDWARE =3D=3D=3D hostb0@pci0:0:0:0: class=3D0x060000 rev=3D0x0d hdr=3D0x00 vendor=3D0x8= 086 device=3D0x3e30 subvendor=3D0x1043 subdevice=3D0x8694 vendor =3D 'Intel Corporation' device =3D '8th/9th Gen Core 8-core Desktop Processor Host Bridge/D= RAM Registers [Coffee Lake S]' class =3D bridge CPU: Intel Coffee Lake (8th/9th Gen Core) Chipset: Intel Z370 Motherboard: ASUS (VT-d enabled in BIOS) =3D=3D=3D CONFIGURATION =3D=3D=3D Loader Configuration (/boot/loader.conf): kern.geom.label.disk_ident.enable=3D"0" kern.geom.label.gptid.enable=3D"0" cryptodev_load=3D"YES" zfs_load=3D"YES" # # bhyve # vmm_load=3D"YES" nmdm_load=3D"YES" if_bridge_load=3D"YES" if_tap_load=3D"YES" # if_cxgbe_load=3D"YES" # Mellanox ConnectX-4 Lx # mlx5_core0@pci0:1:0:0: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0= x15b3 device=3D0x1015 subvendor=3D0x1590 subdevice=3D0x00d4 # mlx5_core0@pci0:1:0:1: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0= x15b3 device=3D0x1015 subvendor=3D0x1590 subdevice=3D0x00d4 # Samsung 990 PRO # nvme3@pci0:7:0:0: class=3D0x010802 rev=3D0x00 hdr=3D0x00 vendor=3D0= x144d device=3D0xa80c subvendor=3D0x144d subdevice=3D0xa801 # vendor =3D 'Samsung Electronics Co Ltd' # device =3D 'NVMe SSD Controller S4LV008[Pascal]' # class =3D mass storage # subclass =3D NVM # Add to existing pptdevs or create new line pptdevs=3D"1/0/0 7/0/0" Devices for Passthrough: - Mellanox ConnectX-4 Lx (dual-port 10/25GbE): PCI 1:0:0 and 1:0:1 - Samsung NVMe SSD: PCI 7:0:0 =3D=3D=3D WORKING STATE (14.2-RELEASE-p7 as host) =3D=3D=3D VT-d Status: DMAR: Length=3D176, Revision=3D1, Checksum=3D255, DMAR Detection: PPT Device Attachment: ppt0@pci0:1:0:0: class=3D0x020000 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 5b3 device=3D0x1015 subvendor=3D0x1590 subdevice=3D0x00d4 ppt1@pci0:7:0:0: class=3D0x010802 rev=3D0x00 hdr=3D0x00 vendor=3D0x1= 44d device=3D0xa80c subvendor=3D0x144d subdevice=3D0xa801 ppt0 mem 0x2ffc000000-0x2ffdffffff irq 16 at device 0.0 on pci1 ppt1 mem 0xdfa00000-0xdfa03fff irq 16 at device 0.0 on pci7 VMM Status: hw.vmm.iommu.enable: 1 hw.vmm.iommu.initialized: 1 hw.vmm.ppt.devices: 2 VM Status: store-01 default bhyveload 8 16384M - Yes [1] Running (37745) Result: VM boots successfully, devices are accessible inside guest Guest OS pciconf output (showing passed-through devices): - mlx5_core0@pci0:0:5:0: Mellanox ConnectX-4 Lx (working) - nvme2@pci0:0:6:0: Samsung NVMe SSD (working) Guest is running FreeBSD 14.3-RELEASE-p4 successfully. =3D=3D=3D BROKEN STATE (14.3-RELEASE-p3/p4 as host) =3D=3D=3D Configuration: Identical to 14.2 (same loader.conf, same hardware) VT-d Status: ENABLED (same DMAR table present) DMAR Detection:=20 - dmar0 and dmar1 detected - Regular devices being re-mapped successfully - DMAR is functional PPT Device Attachment: - ppt0, ppt1, ppt2 show in pciconf - dmesg shows successful hardware attachment - hw.vmm.ppt.devices =3D 2 (VMM sees devices) VMM Status: - hw.vmm.iommu.enable =3D 1 - hw.vmm.iommu.initialized =3D 0 _ CRITICAL FAILURE - hw.vmm.vmx.initialized =3D 1 (Intel VMX working) Result:=20 - VM crashes immediately when started with passthrough devices - Boot gets to kernel load, then crashes during hardware initialization - Exit code 4 (reboot/crash) Workaround Attempts on 14.3: 1. Setting hw.vmm.iommu.enable=3D"0" in loader.conf: - Result: hw.vmm.ppt.devices becomes 0, devices not detected at all 2. Setting hw.vmm.iommu.enable=3D"1" in loader.conf: - Result: hw.vmm.iommu.initialized stays 0, VM crashes 3. Setting hw.dmar.enable=3D"1" in loader.conf: - Result: No change, IOMMU still not initialized No configuration allows VMM IOMMU to initialize on 14.3-RELEASE. =3D=3D=3D ROOT CAUSE ANALYSIS =3D=3D=3D The AMD IOMMU driver integration work that was merged into stable/14=20 for 14.3-RELEASE appears to have introduced a regression affecting=20 Intel DMAR system ? Key observations: 1. DMAR hardware is functional (regular device remapping works) 2. ppt driver attaches successfully to hardware 3. VMM recognizes ppt devices 4. BUT: VMM IOMMU subsystem fails to initialize 5. Without VMM IOMMU init, bhyve cannot safely use passthrough 6. Result: VM crashes when passthrough devices are configured This appears to be a change in initialization order or requirements=20 where 14.3 now requires VMM IOMMU to be initialized before allowing=20 passthrough, but the Intel DMAR integration with VMM IOMMU is broken. References from FreeBSD Status Reports: - Q2 2024: "AMD IOMMU driver... will include bhyve integration" - Q4 2024: "AMD IOMMU driver... was pushed to src tree in early November" - The driver is in stable/14 branch (14.3-RELEASE) =3D=3D=3D REGRESSION CONFIRMATION =3D=3D=3D Tested using FreeBSD boot environments: - Host: 14.2-RELEASE-p7 (Oct 11, 2025) =3D WORKS - Host: 14.3-RELEASE-p3 (Oct 11, 2025) =3D FAILS Same hardware, same BIOS settings, same loader.conf configuration. The only variable is the FreeBSD version on the HOST system. Reproducibility: 100% reproducible on Intel Z370 chipset =3D=3D=3D IMPACT =3D=3D=3D CRITICAL for users upgrading from 14.2 to 14.3: - All bhyve PCI passthrough configurations will break on Intel systems - Silent failure: appears to configure correctly but VMs crash at boot - No error messages indicating IOMMU initialization failure - No workaround exists within 14.3-RELEASE Affected users: - Anyone using bhyve with PCI passthrough on Intel systems - GPU passthrough configurations - Network card passthrough (SR-IOV, dedicated NICs) - NVMe drive passthrough - Any PCI device passthrough scenario =3D=3D=3D EXPECTED BEHAVIOR =3D=3D=3D When VT-d is enabled and DMAR is functional: 1. hw.vmm.iommu.initialized should equal 1 2. PCI passthrough should work as it does in 14.2-RELEASE 3. OR: System should gracefully degrade to 14.2 behavior if IOMMU init fails =3D=3D=3D ACTUAL BEHAVIOR =3D=3D=3D On 14.3-RELEASE with Intel DMAR: 1. hw.vmm.iommu.initialized always equals 0 2. VM crashes when passthrough devices are configured 3. No error messages explaining the failure 4. No graceful fallback to non-IOMMU passthrough mode =3D=3D=3D WORKAROUND =3D=3D=3D Stay on FreeBSD 14.2-RELEASE as host OS. Boot environments make this easy to maintain while waiting for fix. =3D=3D=3D RECOMMENDED FIX =3D=3D=3D 1. Debug why VMM IOMMU initialization fails on Intel DMAR systems 2. Fix initialization to properly detect and use Intel DMAR 3. OR: Allow passthrough to work without strict IOMMU init (14.2 behavior) 4. Add error messages when IOMMU init fails to aid debugging =3D=3D=3D ADDITIONAL NOTES =3D=3D=3D - Guest OS version doesn't matter (can run 14.3 guest on 14.2 host) - Only affects 14.3-RELEASE as HOST, not as guest - Problem is in VMM kernel module's IOMMU integration - May affect other Intel chipsets beyond Z370 =3D=3D=3D FILES ATTACHED =3D=3D=3D 1. Full dmesg from 14.2-RELEASE-p7 (working) 2. Full sysctl hw.vmm output from 14.2 3. pciconf -lv output showing device configuration 4. Guest pciconf output showing successful passthrough =3D=3D=3D Bug report created at /tmp/freebsd-ppt-bug-14.3.txt =3D=3D=3D =3D=3D=3D Supporting files: /tmp/dmesg-14.2-working.txt /tmp/sysctl-vmm-14.2.txt /tmp/pciconf-14.2.txt --=20 You are receiving this mail because: You are the assignee for the bug.=