Date: Sat, 11 Oct 2025 19:58:15 +0000 From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 290155] bhyve PCI passthrough regression in 14.3-RELEASE: VMM IOMMU fails to initialize on Intel systems Message-ID: <bug-290155-27103@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=290155 Bug ID: 290155 Summary: bhyve PCI passthrough regression in 14.3-RELEASE: VMM IOMMU fails to initialize on Intel systems Product: Base System Version: 14.3-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: luc+freebsd@2113.ch Created attachment 264491 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=264491&action=edit dmesg output on workin host 14.2 Host Kernel: 14.2-RELEASE-p1 FreeBSD app-01 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64 === HARDWARE === hostb0@pci0:0:0:0: class=0x060000 rev=0x0d hdr=0x00 vendor=0x8086 device=0x3e30 subvendor=0x1043 subdevice=0x8694 vendor = 'Intel Corporation' device = '8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S]' class = bridge CPU: Intel Coffee Lake (8th/9th Gen Core) Chipset: Intel Z370 Motherboard: ASUS (VT-d enabled in BIOS) === CONFIGURATION === Loader Configuration (/boot/loader.conf): kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" cryptodev_load="YES" zfs_load="YES" # # bhyve # vmm_load="YES" nmdm_load="YES" if_bridge_load="YES" if_tap_load="YES" # if_cxgbe_load="YES" # Mellanox ConnectX-4 Lx # mlx5_core0@pci0:1:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4 # mlx5_core0@pci0:1:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4 # Samsung 990 PRO # nvme3@pci0:7:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa80c subvendor=0x144d subdevice=0xa801 # vendor = 'Samsung Electronics Co Ltd' # device = 'NVMe SSD Controller S4LV008[Pascal]' # class = mass storage # subclass = NVM # Add to existing pptdevs or create new line pptdevs="1/0/0 7/0/0" Devices for Passthrough: - Mellanox ConnectX-4 Lx (dual-port 10/25GbE): PCI 1:0:0 and 1:0:1 - Samsung NVMe SSD: PCI 7:0:0 === WORKING STATE (14.2-RELEASE-p7 as host) === VT-d Status: DMAR: Length=176, Revision=1, Checksum=255, DMAR Detection: PPT Device Attachment: ppt0@pci0:1:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x15b3 device=0x1015 subvendor=0x1590 subdevice=0x00d4 ppt1@pci0:7:0:0: class=0x010802 rev=0x00 hdr=0x00 vendor=0x144d device=0xa80c subvendor=0x144d subdevice=0xa801 ppt0 mem 0x2ffc000000-0x2ffdffffff irq 16 at device 0.0 on pci1 ppt1 mem 0xdfa00000-0xdfa03fff irq 16 at device 0.0 on pci7 VMM Status: hw.vmm.iommu.enable: 1 hw.vmm.iommu.initialized: 1 hw.vmm.ppt.devices: 2 VM Status: store-01 default bhyveload 8 16384M - Yes [1] Running (37745) Result: VM boots successfully, devices are accessible inside guest Guest OS pciconf output (showing passed-through devices): - mlx5_core0@pci0:0:5:0: Mellanox ConnectX-4 Lx (working) - nvme2@pci0:0:6:0: Samsung NVMe SSD (working) Guest is running FreeBSD 14.3-RELEASE-p4 successfully. === BROKEN STATE (14.3-RELEASE-p3/p4 as host) === Configuration: Identical to 14.2 (same loader.conf, same hardware) VT-d Status: ENABLED (same DMAR table present) DMAR Detection: - dmar0 and dmar1 detected - Regular devices being re-mapped successfully - DMAR is functional PPT Device Attachment: - ppt0, ppt1, ppt2 show in pciconf - dmesg shows successful hardware attachment - hw.vmm.ppt.devices = 2 (VMM sees devices) VMM Status: - hw.vmm.iommu.enable = 1 - hw.vmm.iommu.initialized = 0 _ CRITICAL FAILURE - hw.vmm.vmx.initialized = 1 (Intel VMX working) Result: - VM crashes immediately when started with passthrough devices - Boot gets to kernel load, then crashes during hardware initialization - Exit code 4 (reboot/crash) Workaround Attempts on 14.3: 1. Setting hw.vmm.iommu.enable="0" in loader.conf: - Result: hw.vmm.ppt.devices becomes 0, devices not detected at all 2. Setting hw.vmm.iommu.enable="1" in loader.conf: - Result: hw.vmm.iommu.initialized stays 0, VM crashes 3. Setting hw.dmar.enable="1" in loader.conf: - Result: No change, IOMMU still not initialized No configuration allows VMM IOMMU to initialize on 14.3-RELEASE. === ROOT CAUSE ANALYSIS === The AMD IOMMU driver integration work that was merged into stable/14 for 14.3-RELEASE appears to have introduced a regression affecting Intel DMAR system ? Key observations: 1. DMAR hardware is functional (regular device remapping works) 2. ppt driver attaches successfully to hardware 3. VMM recognizes ppt devices 4. BUT: VMM IOMMU subsystem fails to initialize 5. Without VMM IOMMU init, bhyve cannot safely use passthrough 6. Result: VM crashes when passthrough devices are configured This appears to be a change in initialization order or requirements where 14.3 now requires VMM IOMMU to be initialized before allowing passthrough, but the Intel DMAR integration with VMM IOMMU is broken. References from FreeBSD Status Reports: - Q2 2024: "AMD IOMMU driver... will include bhyve integration" - Q4 2024: "AMD IOMMU driver... was pushed to src tree in early November" - The driver is in stable/14 branch (14.3-RELEASE) === REGRESSION CONFIRMATION === Tested using FreeBSD boot environments: - Host: 14.2-RELEASE-p7 (Oct 11, 2025) = WORKS - Host: 14.3-RELEASE-p3 (Oct 11, 2025) = FAILS Same hardware, same BIOS settings, same loader.conf configuration. The only variable is the FreeBSD version on the HOST system. Reproducibility: 100% reproducible on Intel Z370 chipset === IMPACT === CRITICAL for users upgrading from 14.2 to 14.3: - All bhyve PCI passthrough configurations will break on Intel systems - Silent failure: appears to configure correctly but VMs crash at boot - No error messages indicating IOMMU initialization failure - No workaround exists within 14.3-RELEASE Affected users: - Anyone using bhyve with PCI passthrough on Intel systems - GPU passthrough configurations - Network card passthrough (SR-IOV, dedicated NICs) - NVMe drive passthrough - Any PCI device passthrough scenario === EXPECTED BEHAVIOR === When VT-d is enabled and DMAR is functional: 1. hw.vmm.iommu.initialized should equal 1 2. PCI passthrough should work as it does in 14.2-RELEASE 3. OR: System should gracefully degrade to 14.2 behavior if IOMMU init fails === ACTUAL BEHAVIOR === On 14.3-RELEASE with Intel DMAR: 1. hw.vmm.iommu.initialized always equals 0 2. VM crashes when passthrough devices are configured 3. No error messages explaining the failure 4. No graceful fallback to non-IOMMU passthrough mode === WORKAROUND === Stay on FreeBSD 14.2-RELEASE as host OS. Boot environments make this easy to maintain while waiting for fix. === RECOMMENDED FIX === 1. Debug why VMM IOMMU initialization fails on Intel DMAR systems 2. Fix initialization to properly detect and use Intel DMAR 3. OR: Allow passthrough to work without strict IOMMU init (14.2 behavior) 4. Add error messages when IOMMU init fails to aid debugging === ADDITIONAL NOTES === - Guest OS version doesn't matter (can run 14.3 guest on 14.2 host) - Only affects 14.3-RELEASE as HOST, not as guest - Problem is in VMM kernel module's IOMMU integration - May affect other Intel chipsets beyond Z370 === FILES ATTACHED === 1. Full dmesg from 14.2-RELEASE-p7 (working) 2. Full sysctl hw.vmm output from 14.2 3. pciconf -lv output showing device configuration 4. Guest pciconf output showing successful passthrough === Bug report created at /tmp/freebsd-ppt-bug-14.3.txt === === Supporting files: /tmp/dmesg-14.2-working.txt /tmp/sysctl-vmm-14.2.txt /tmp/pciconf-14.2.txt -- You are receiving this mail because: You are the assignee for the bug.help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-290155-27103>