Date: Tue, 10 Jan 2017 18:01:17 -0800 From: <soralx@cydem.org> To: <misc-freebsd@talk2dom.com> Cc: <freebsd-virtualization@freebsd.org> Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170110180117.7f246b5a@mscad14> In-Reply-To: <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com>
next in thread | previous in thread | raw e-mail | index | archive | help
> Found my original attempt by modifying /usr/src/sys/amd64/vmm/x86.c
> Unified diff follows, but this didn't work for me.
> ("bhyve_id[]" commented out to prevent compiler complaints)
Who knows what sort of trickery nVidia's driver is up to besides
CPUID when determining the presence of virtualization.
Regardless of that, VGA PCIe passthrough does not work in bhyve
even with Quadro 2000 (which Xen people have had success with).
The problem appears to be in the area of assigning memory-mapped
I/O ranges by bhyve for the VGA card to a region outside of the
CPU's addressable space; i.e., bhyve does not check CPUID's
0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while
bhyve assigns 0xd000000000 & above for the large Prefetch Memory
chunks, which requires 40 address bits). At least this is my
understanding of why VGA passthrough does not work.
This seems easy to fix. Could someone who knows better have a look?
Unlike Linux, FreeBSD has no problem assigning BAR range outside
addressable range, and then panics when trying to write to these
virtual memory addresses. See [0] below.
> There doesn't seem to be support for CPUID 0x40000001 in bhyve either.
What is it supposed to do?
[0]
Linux dmesg:
[ 0.204799] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
[ 0.205474] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources
[ 0.206080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[ 0.207306] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00])
[ 0.207724] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]
[ 0.208291] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[ 0.208759] acpi PNP0A03:00: host bridge window [mem 0xd000000000-0xd00c0fffff window] (ignored, not CPU addressable)
[ 0.209517] PCI host bridge to bus 0000:00
[ 0.209808] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[ 0.210281] pci_bus 0000:00: root bus resource [io 0x0d00-0x1fff window]
[ 0.210752] pci_bus 0000:00: root bus resource [io 0x2000-0x211f window]
[ 0.211224] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xc40fffff window]
[ 0.211743] pci_bus 0000:00: root bus resource [bus 00]
[...]
[ 0.223902] PCI: Using ACPI for IRQ routing
[ 0.265987] pci 0000:00:03.0: can't claim BAR 1 [mem 0xd000000000-0xd007ffffff 64bit pref]: no compatible bridge window
[ 0.266735] pci 0000:00:03.0: can't claim BAR 3 [mem 0xd008000000-0xd00bffffff 64bit pref]: no compatible bridge window
[ 0.284717] pci 0000:00:03.0: can't claim BAR 6 [mem 0xf6000000-0xf607ffff pref]: no compatible bridge window
[...]
[ 0.285407] pci 0000:00:03.0: BAR 1: no space for [mem size 0x08000000 64bit pref]
[ 0.285933] pci 0000:00:03.0: BAR 1: trying firmware assignment [mem 0xd000000000-0xd007ffffff 64bit pref]
[ 0.286599] pci 0000:00:03.0: BAR 1: [mem 0xd000000000-0xd007ffffff 64bit pref] conflicts with PCI mem [mem 0x00000000-0x7fffffffff]
[ 0.287419] pci 0000:00:03.0: BAR 1: failed to assign [mem size 0x08000000 64bit pref]
[ 0.287968] pci 0000:00:03.0: BAR 3: no space for [mem size 0x04000000 64bit pref]
[ 0.288506] pci 0000:00:03.0: BAR 3: trying firmware assignment [mem 0xd008000000-0xd00bffffff 64bit pref]
[ 0.289173] pci 0000:00:03.0: BAR 3: [mem 0xd008000000-0xd00bffffff 64bit pref] conflicts with PCI mem [mem 0x00000000-0x7fffffffff]
[ 0.289992] pci 0000:00:03.0: BAR 3: failed to assign [mem size 0x04000000 64bit pref]
[ 0.290539] pci 0000:00:03.0: BAR 6: assigned [mem 0xc0080000-0xc00fffff pref]
[ 0.291039] pci 0000:00:01.0: BAR 6: assigned [mem 0xc0002000-0xc00027ff pref]
[ 0.291540] pci 0000:00:02.0: BAR 6: assigned [mem 0xc0002800-0xc0002fff pref]
Cannot get output from Linux's `lspci -vvn` booted with "pci=nocrs" kernel option,
as it hangs now close to the end of boot process (not sure why, was able to finish
booting before).
Another machine:
vgapci0@pci0:1:0:0: class=0x030000 card=0x083510de chip=0x0df810de rev=0xa1 hdr=0x00
vendor = 'nVidia Corporation'
device = 'GF108 [Quadro 600]'
class = display
subclass = VGA
bar [10] = type Memory, range 32, base 0xfa000000, size 16777216, enabled
bar [14] = type Prefetchable Memory, range 64, base 0xe8000000, size 134217728, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xf0000000, size 33554432, enabled
bar [24] = type I/O Port, range 32, base 0xe000, size 128, enabled
hdac0@pci0:1:0:1: class=0x040300 card=0x083510de chip=0x0bea10de rev=0xa1 hdr=0x00
vendor = 'nVidia Corporation'
device = 'GF108 High Definition Audio Controller'
class = multimedia
subclass = HDA
bar [10] = type Memory, range 32, base 0xfb080000, size 16384, enabled
Host:
ppt0@pci0:1:0:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GF106GL [Quadro 2000]'
class = display
subclass = VGA
bar [10] = type Memory, range 32, base 0xf4000000, size 33554432, enabled
bar [14] = type Prefetchable Memory, range 64, base 0xe0000000, size 134217728, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xe8000000, size 67108864, enabled
bar [24] = type I/O Port, range 32, base 0xe000, size 128, enabled
ppt1@pci0:1:0:1: class=0x040300 card=0x084a10de chip=0x0be910de rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GF106 High Definition Audio Controller'
class = multimedia
subclass = HDA
bar [10] = type Memory, range 32, base 0xf6080000, size 16384, enabled
Guest:
vgapci0@pci0:0:4:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GF106GL [Quadro 2000]'
class = display
subclass = VGA
bar [10] = type Memory, range 32, base 0xc2000000, size 33554432, enabled
bar [14] = type Prefetchable Memory, range 64, base 0xd000000000, size 134217728, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xd008000000, size 67108864, enabled
bar [24] = type I/O Port, range 32, base 0x2080, size 128, enabled
Guest dmesg:
found-> vendor=0x10de, dev=0x0dd8, revid=0xa1
domain=0, bus=0, slot=4, func=0
class=03-00-00, hdrtype=0x00, mfdev=0
cmdreg=0x0403, statreg=0x0010, cachelnsz=16 (dwords)
lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns)
intpin=a, irq=16
powerspec 3 supports D0 D3 current D0
MSI supports 1 message, 64 bit
map[10]: type Memory, range 32, base 0xc2000000, size 25, enabled
pcib0: allocated type 3 (0xc2000000-0xc3ffffff) for rid 10 of pci0:0:4:0
map[14]: type Prefetchable Memory, range 64, base 0xd000000000, size 27, enabled
pcib0: allocated type 3 (0xd000000000-0xd007ffffff) for rid 14 of pci0:0:4:0
map[1c]: type Prefetchable Memory, range 64, base 0xd008000000, size 26, enabled
pcib0: allocated type 3 (0xd008000000-0xd00bffffff) for rid 1c of pci0:0:4:0
map[24]: type I/O Port, range 32, base 0x2080, size 7, enabled
pcib0: allocated type 4 (0x2080-0x20ff) for rid 24 of pci0:0:4:0
pcib0: no PRT entry for 0.4.INTA
--
[SorAlx] ridin' VN2000 Classic LT
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170110180117.7f246b5a>