Date: Fri, 10 Jul 2026 09:22:06 +0000 From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 296662] bhyve: passthru caches PCI config header while device is still recovering from FLR; physical BARs left zeroed Message-ID: <bug-296662-27103@https.bugs.freebsd.org/bugzilla/>
index | next in thread | raw e-mail
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=296662 Bug ID: 296662 Summary: bhyve: passthru caches PCI config header while device is still recovering from FLR; physical BARs left zeroed Product: Base System Version: 15.1-RELEASE Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: bhyve Assignee: virtualization@FreeBSD.org Reporter: truegrom@gmail.com Created attachment 272687 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=272687&action=edit Proof of concept, not for commit: readiness wait + BAR restore in cfginit() ### Environment - Host: FreeBSD 15.1-RELEASE (releng/15.1-n283562-96841ea08dcf) GENERIC amd64 - CPU: AMD Ryzen (Cezanne), AMD-Vi IOMMU (ivhd0), hw.vmm.amdvi.enable=1 - Device: Intel Arc A380 (DG2), 8086:56a5 rev 05, subvendor 1849:6006 (ASRock), plus its HDA audio function 8086:4f92; both attached to ppt(4) via pptdevs="3/0/0 4/0/0" - Guest: Ubuntu 24.04.4, kernel 6.17.0-35-generic, booted via grub-bhyve (vm-bhyve 1.7.3), devices: -s 0:6:0,passthru,3/0/0 -s 0:7:0,passthru,4/0/0 - IOMMU initializes fine (hw.vmm.iommu.initialized: 1); the EN-25:20.vmm fix is included in 15.1. ### Overview Intel DG2 GPUs do not complete FLR cleanly: the Transactions Pending bit never clears, and after the FLR the device does not respond to config space accesses (all reads return 0xff) for up to ~1 second. Measured on this card with `devctl reset pci0:3:0:0`: reads return 0xffffffff immediately after the reset and recover within ~1 s. When such a device is assigned to a VM, two things go wrong in sequence: 1. ppt(4) resets the device (ppt_pci_reset() -> pcie_flr(dev, ..., force); "pci0:3:0:0: Transactions pending after FLR!" is logged on every start). pcie_flr() waits a fixed delay and then pci_cfg_restore() writes the saved config into a device that is not yet responding, so the restore is lost: afterwards the physical BARs read back as zero (only the type flag bits remain, e.g. 0x00000004 / 0x0000000c) and PCIM_CMD_MEMEN / PCIM_CMD_BUSMASTEREN are clear in the command register. 2. Immediately afterwards, bhyve's passthru_init() -> cfginit() copies the physical PCI header into the emulated config space: for (int i = 0; i <= PCIR_MAXLAT; i += 4) pci_set_cfgdata32(pi, i, passthru_read_config(...)); Because the device is still unresponsive, vendor/device are cached as 0xffff. Since the standard header range is served to the guest from this cache ("Emulate most PCI header register", set_pcir_handler(sc, 0, PCIR_MAXLAT + 1, passthru_cfgread_emulate, ...)), the guest permanently sees an empty slot. A PCI rescan in the guest does not help even after the device has recovered, because the poisoned cache is never refreshed. Fast-resetting functions are unaffected — the audio function 4f92 of the same card resets instantly and is always visible in the guest — which is what makes the failure look so puzzling: one function of the card appears, the other silently does not, with no error from bhyve. Even when the header is read correctly (see PoC below), the guest driver then fails with "Device is non-operational; MMIO access returns 0xFFFFFFFF!" because of the zeroed physical BARs from step 1. ### Steps to Reproduce 1. Attach an Intel Arc GPU function to ppt(4) via pptdevs. 2. Start any bhyve VM with `-s N,passthru,B/S/F` for that function. 3. Observe "Transactions pending after FLR!" on the host console. 4. In the guest: `lspci` — the slot is empty. `echo 1 > /sys/bus/pci/rescan` does not help. 5. On the host, `pciconf -r pciB:S:F 0x10` etc. show zeroed BARs and a cleared command register, while `pciconf -r ... 0x00` returns the correct vendor/device ID (the card itself has recovered). ### Actual Results Guest sees no device; bhyve reports no error. After manually restoring the BAR values and command register with pciconf(8) on the host and re-scanning the PCI bus in the guest, the i915 driver initializes the GPU completely (GuC/HuC load and authenticate, /dev/dri/renderD128 works, VA-API transcoding functional) — confirming that nothing else is missing. ### Expected Results The passthru device is visible and functional in the guest, or at minimum bhyve reports a diagnostic that the device did not respond. ### Root cause confirmation (proof of concept) To confirm the analysis, a proof-of-concept change to cfginit() was tested on the affected system (see attachment): - poll PCIR_VENDOR (50 ms x 40, i.e. up to 2 s) until the device responds before caching the config space header; - then compare each BAR register of the physical device against the kernel's cached value (PCIOCGETBAR, which survives the reset), rewrite it if it was lost, and re-enable memory decoding and bus mastering. With the PoC applied, a plain `vm stop` / `vm start` cycle produces a fully working GPU in the guest with no manual intervention; bhyve logs "PCI 3/0/0: restoring BAR0 after reset" / "...BAR2..." on each start. This confirms both halves of the analysis (poisoned header cache; lost config restore). The PoC is provided for reproduction and as a mitigation for affected users — not as a ready-to-commit fix. Suggested directions for a proper fix (left to the maintainers): - Kernel: pcie_flr() in sys/dev/pci/pci.c could poll for device readiness (vendor ID != 0xffff, as Linux does in pci_dev_wait()) after issuing the FLR and before pci_cfg_restore(). That would fix the lost-restore problem for all consumers of pcie_flr(), not just bhyve, and is probably the right layer for it. - bhyve: cfginit() should not cache a config header read from a device that is still returning 0xff, and could verify/restore the physical BARs against the kernel's cached values before use. ### Observed behaviour with and without the PoC - Without the PoC: on every VM start, "Transactions pending after FLR!" on the host; guest sees an empty slot; physical BARs read back zeroed (0x00000004 / 0x0000000c) with MEMEN/BUSMASTEREN cleared; manually rewriting the BARs and command register with pciconf(8) plus a PCI rescan in the guest brings the device fully up — confirming the mechanism. - With the PoC: plain `vm stop` / `vm start` cycles produce a fully functional GPU in the guest with no manual intervention. In the Ubuntu 24.04 guest (kernel 6.17), i915 initializes completely (DMC, GuC 70.36.0 loaded, HuC authenticated), /dev/dri/renderD128 is created, and VA-API reports full decode/encode profiles (H.264/HEVC/VP9/AV1); hardware transcoding is functional. - The audio function (8086:4f92) of the same card, which resets quickly, behaves identically before and after the PoC — the readiness wait is effectively a no-op for devices that recover promptly, and the BAR restore writes nothing when registers already match the kernel's cached values. usr.sbin/bhyve/pci_passthru.c is currently identical in main and releng/15.1, so the PoC applies unmodified to both; all testing was done on 15.1-RELEASE. ### Provenance disclosure The root-cause analysis was performed interactively on the affected hardware; all measurements, register dumps and logs in this report are from that system. The analysis and the attached proof-of-concept diff were developed with the assistance of an LLM (Anthropic Claude). I am not a C developer and have not performed a full code review of the diff, so I am explicitly NOT proposing it for commit; it is attached only to document how the root cause was confirmed and as a temporary mitigation for affected users. Please treat it as a reproduction aid — the root cause description should be sufficient for a committer to implement a proper fix independently. What I can vouch for personally is the empirical side: the reproduction steps, the register dumps, the before/after behaviour, and the fact that the described mechanism (manual BAR restore via pciconf(8) + guest PCI rescan) brings the device fully up without any code changes. I am happy to test any proposed fix on the affected hardware and can provide any additional diagnostics on request. A related but independent bug (bhyve's gvt-d probe claiming discrete Intel GPUs and failing VM startup) is filed separately. -- You are receiving this mail because: You are the assignee for the bug.home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-296662-27103>