Date: Thu, 12 Mar 2020 15:25:41 +0100 From: Alex Erley <erleya@gmail.com> To: Peter Grehan <grehan@freebsd.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: [GPU pass-through] no compatible bridge window for claimed BAR Message-ID: <9b26dee7-8656-b5ba-5d72-9a01638ee438@gmail.com> In-Reply-To: <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org> References: <CAONCVozTcKP_=8AdOCfFNiRQnQ254%2BFVn0ZDRK-V8Zo%2BFFd_qQ@mail.gmail.com> <07921dcf-11d5-f440-a42f-d7ec950cab10@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hello Peter, Many thanks for your insight! Changing PCI_EMUL_MEMBASE64 from 0xD000000000 to 0x0440000000 (16Gb guest memory limit) and PCI_EMUL_MEMLIMIT64 from 0xFD00000000 to 0x07ffffffff (32Gb host memory limit) still fails. From guest dmesg: ... [mem 0xc0000000-0xffffffff] available for PCI devices ... pci 0000:00:01.0: [10de:1c03] type 00 class 0x030000 pci 0000:00:01.0: reg 0x10: [mem 0xc0000000-0xc0ffffff] pci 0000:00:01.0: reg 0x14: [mem 0x440000000-0x44fffffff 64bit pref] pci 0000:00:01.0: reg 0x1c: [mem 0xc2000000-0xc3ffffff 64bit pref] pci 0000:00:01.0: reg 0x24: [io 0x2000-0x207f] pci 0000:00:01.0: reg 0x30: [mem 0xd3000000-0xd307ffff pref] pci 0000:00:01.1: [10de:10f1] type 00 class 0x040300 pci 0000:00:01.1: reg 0x10: [mem 0xc4000000-0xc4003fff] ... pci 0000:00:01.0: can't claim BAR 1 [mem 0x440000000-0x44fffffff 64bit pref]: no compatible bridge window ... pci 0000:00:01.0: BAR 1: no space for [mem size 0x10000000 64bit pref] pci 0000:00:01.0: BAR 1: trying firmware assignment [mem 0x440000000-0x44fffffff 64bit pref] pci 0000:00:01.0: BAR 1: assigned [mem 0x440000000-0x44fffffff 64bit pref] It seems guest requires BAR to be within its addressable space. Changing PCI_EMUL_MEMBASE64 to 0x00c0000000 (as guest suggests in its dmesg) didn't work either: bhyve: failed to initialize BARs for PCI 1/0/0 device emulation initialization error: Cannot allocate memory From the BHyve sources it becomes evident that when claimed BAR size is <= 32Mb, it is mandatory allocated as 32-bit BAR. The only case BAR goes for 64-bit allocation is when its size > 32Mb. In my case it is 256 Mb and it fails only for this BAR. I think the 64-bit BAR allocation still has to be fixed in BHyve code. Anyway, we are getting closer :) Any ideas are really appreciated. Have a nice day, Alex On 3/12/20 1:31 AM, Peter Grehan wrote: > Hi Alex, > >>> dmesg | grep "no compatible bridge window" >> pci 0000:00:01.0: can't claim BAR 1 [mem 0xd000000000-0xd00fffffff >> 64bit pref]: no compatible bridge window > ...> From what I can read from all the info above, >> somehow the requested memory space for BAR1 for that device is >> 0xd000000000-0xd00fffffff which is out of addressable space on the >> system. > > Yep, that's the issue, and it's a bhyve bug - there is no check to > see if the 64-bit window is within the addressable range of the > processor. > > A quick fix is to change the constant for that range in pci_emul.c > > #define PCI_EMUL_MEMBASE64 0xD000000000UL #define > PCI_EMUL_MEMLIMIT64 0xFD00000000UL > > .. to a value that is within the address bits of the CPU, but also > above guest DRAM. > > later, > > Peter.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9b26dee7-8656-b5ba-5d72-9a01638ee438>