Date: Thu, 12 Mar 2020 01:18:18 +0100 From: Alex Troussov <erleya@gmail.com> To: freebsd-virtualization@freebsd.org Subject: [GPU pass-through] no compatible bridge window for claimed BAR Message-ID: <CAONCVozTcKP_=8AdOCfFNiRQnQ254%2BFVn0ZDRK-V8Zo%2BFFd_qQ@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hello all,
While trying to pass-through GPU card from Bhyve host (FreeBSD 12.1)
to Debian Buster VM, I get error during VM boot like:
> dmesg | grep "no compatible bridge window"
pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff 64bit
pref]: no compatible bridge window
but GPU appears in VM:
> lspci -v
00:01.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX
1060 6GB] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB]
Flags: fast devsel, IRQ 17
Memory at c1000000 (32-bit, non-prefetchable) [disabled] [size=16M]
Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at c2000000 (64-bit, prefetchable) [disabled] [size=32M]
I/O ports at 2080 [disabled] [size=128]
[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
Kernel driver in use: nvidia
Kernel modules: nvidia
00:01.1 Audio device: NVIDIA Corporation GP106 High Definition Audio
Controller (rev a1)
Subsystem: NVIDIA Corporation GP106 High Definition Audio Controller
Flags: fast devsel, IRQ 18
Memory at c4000000 (32-bit, non-prefetchable) [disabled] [size=16K]
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
and its nvidia driver is partially loaded ("nvidia-smi -L" doesn't detect
it).
On VM complete device tree is:
> lspci -tv
-[0000:00]-+-00.0 Network Appliance Corporation Device 1275
+-01.0 NVIDIA Corporation GP106 [GeForce GTX 1060 6GB]
+-01.1 NVIDIA Corporation GP106 High Definition Audio Controller
+-02.0 Red Hat, Inc Virtio network device
+-03.0 Red Hat, Inc Virtio block device
+-1d.0 Device fb5d:40fb
+-1e.0 Intel Corporation 7 Series/C210 Series Chipset Family
USB xHCI Host Controller
\-1f.0 Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
And my BHyve command-line parameters are:
> bhyve -AHPS -c 4 -m 16G -u \
-s 0,hostbridge \
-s 1:0,passthru,1/0/0 -s 1:1,passthru,1/0/1 \
-s 2,virtio-net,tap0,mac=00:a0:98:fa:5c:8e \
-s 3,virtio-blk,/dev/zvol/attic/tuxdisk0 \
-s 29,fbuf,tcp=192.168.0.2:5900,w=640,h=480,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
-l com1,stdio \
tux
I tried to remove/rescan GPU card on VM with no success.
On host system card is properly masked:
> pciconv -lv
ppt0@pci0:1:0:0: class=0x030000 card=0x11d710de chip=0x1c0310de
rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GP106 [GeForce GTX 1060 6GB]'
class = display
subclass = VGA
ppt1@pci0:1:0:1: class=0x040300 card=0x11d710de chip=0x10f110de
rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GP106 High Definition Audio Controller'
class = multimedia
subclass = HDA
What would be a good strategy to find a solution?
>From what I can read from all the info above,
somehow the requested memory space for BAR1 for that device is
0xd000000000-0xd00fffffff which is out of addressable space on the system.
In the same time device reports prefetchable memory slot of 256 Mb starting
from address 0xd0000000.
Why it is different? Bhyve or UEFI bug? VM Linux config? Please help...
--
Regards,
Alex
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAONCVozTcKP_=8AdOCfFNiRQnQ254%2BFVn0ZDRK-V8Zo%2BFFd_qQ>