From owner-freebsd-virtualization@freebsd.org Wed Jan 11 02:01:20 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E047CA9849 for ; Wed, 11 Jan 2017 02:01:20 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 2C1B51C4C for ; Wed, 11 Jan 2017 02:01:19 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id BCC47F802; Tue, 10 Jan 2017 18:01:18 -0800 (PST) Date: Tue, 10 Jan 2017 18:01:17 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170110180117.7f246b5a@mscad14> In-Reply-To: <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 02:01:20 -0000 > Found my original attempt by modifying /usr/src/sys/amd64/vmm/x86.c > Unified diff follows, but this didn't work for me. > ("bhyve_id[]" commented out to prevent compiler complaints) Who knows what sort of trickery nVidia's driver is up to besides CPUID when determining the presence of virtualization. Regardless of that, VGA PCIe passthrough does not work in bhyve even with Quadro 2000 (which Xen people have had success with). The problem appears to be in the area of assigning memory-mapped I/O ranges by bhyve for the VGA card to a region outside of the CPU's addressable space; i.e., bhyve does not check CPUID's 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while bhyve assigns 0xd000000000 & above for the large Prefetch Memory chunks, which requires 40 address bits). At least this is my understanding of why VGA passthrough does not work. This seems easy to fix. Could someone who knows better have a look? Unlike Linux, FreeBSD has no problem assigning BAR range outside addressable range, and then panics when trying to write to these virtual memory addresses. See [0] below. > There doesn't seem to be support for CPUID 0x40000001 in bhyve either. What is it supposed to do? [0] Linux dmesg: [ 0.204799] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.205474] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.206080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207306] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00]) [ 0.207724] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.208291] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.208759] acpi PNP0A03:00: host bridge window [mem 0xd000000000-0xd00c0fffff window] (ignored, not CPU addressable) [ 0.209517] PCI host bridge to bus 0000:00 [ 0.209808] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.210281] pci_bus 0000:00: root bus resource [io 0x0d00-0x1fff window] [ 0.210752] pci_bus 0000:00: root bus resource [io 0x2000-0x211f window] [ 0.211224] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xc40fffff window] [ 0.211743] pci_bus 0000:00: root bus resource [bus 00] [...] [ 0.223902] PCI: Using ACPI for IRQ routing [ 0.265987] pci 0000:00:03.0: can't claim BAR 1 [mem 0xd000000000-0xd007ffffff 64bit pref]: no compatible bridge window [ 0.266735] pci 0000:00:03.0: can't claim BAR 3 [mem 0xd008000000-0xd00bffffff 64bit pref]: no compatible bridge window [ 0.284717] pci 0000:00:03.0: can't claim BAR 6 [mem 0xf6000000-0xf607ffff pref]: no compatible bridge window [...] [ 0.285407] pci 0000:00:03.0: BAR 1: no space for [mem size 0x08000000 64bit pref] [ 0.285933] pci 0000:00:03.0: BAR 1: trying firmware assignment [mem 0xd000000000-0xd007ffffff 64bit pref] [ 0.286599] pci 0000:00:03.0: BAR 1: [mem 0xd000000000-0xd007ffffff 64bit pref] conflicts with PCI mem [mem 0x00000000-0x7fffffffff] [ 0.287419] pci 0000:00:03.0: BAR 1: failed to assign [mem size 0x08000000 64bit pref] [ 0.287968] pci 0000:00:03.0: BAR 3: no space for [mem size 0x04000000 64bit pref] [ 0.288506] pci 0000:00:03.0: BAR 3: trying firmware assignment [mem 0xd008000000-0xd00bffffff 64bit pref] [ 0.289173] pci 0000:00:03.0: BAR 3: [mem 0xd008000000-0xd00bffffff 64bit pref] conflicts with PCI mem [mem 0x00000000-0x7fffffffff] [ 0.289992] pci 0000:00:03.0: BAR 3: failed to assign [mem size 0x04000000 64bit pref] [ 0.290539] pci 0000:00:03.0: BAR 6: assigned [mem 0xc0080000-0xc00fffff pref] [ 0.291039] pci 0000:00:01.0: BAR 6: assigned [mem 0xc0002000-0xc00027ff pref] [ 0.291540] pci 0000:00:02.0: BAR 6: assigned [mem 0xc0002800-0xc0002fff pref] Cannot get output from Linux's `lspci -vvn` booted with "pci=nocrs" kernel option, as it hangs now close to the end of boot process (not sure why, was able to finish booting before). Another machine: vgapci0@pci0:1:0:0: class=0x030000 card=0x083510de chip=0x0df810de rev=0xa1 hdr=0x00 vendor = 'nVidia Corporation' device = 'GF108 [Quadro 600]' class = display subclass = VGA bar [10] = type Memory, range 32, base 0xfa000000, size 16777216, enabled bar [14] = type Prefetchable Memory, range 64, base 0xe8000000, size 134217728, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xf0000000, size 33554432, enabled bar [24] = type I/O Port, range 32, base 0xe000, size 128, enabled hdac0@pci0:1:0:1: class=0x040300 card=0x083510de chip=0x0bea10de rev=0xa1 hdr=0x00 vendor = 'nVidia Corporation' device = 'GF108 High Definition Audio Controller' class = multimedia subclass = HDA bar [10] = type Memory, range 32, base 0xfb080000, size 16384, enabled Host: ppt0@pci0:1:0:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GF106GL [Quadro 2000]' class = display subclass = VGA bar [10] = type Memory, range 32, base 0xf4000000, size 33554432, enabled bar [14] = type Prefetchable Memory, range 64, base 0xe0000000, size 134217728, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xe8000000, size 67108864, enabled bar [24] = type I/O Port, range 32, base 0xe000, size 128, enabled ppt1@pci0:1:0:1: class=0x040300 card=0x084a10de chip=0x0be910de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GF106 High Definition Audio Controller' class = multimedia subclass = HDA bar [10] = type Memory, range 32, base 0xf6080000, size 16384, enabled Guest: vgapci0@pci0:0:4:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GF106GL [Quadro 2000]' class = display subclass = VGA bar [10] = type Memory, range 32, base 0xc2000000, size 33554432, enabled bar [14] = type Prefetchable Memory, range 64, base 0xd000000000, size 134217728, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xd008000000, size 67108864, enabled bar [24] = type I/O Port, range 32, base 0x2080, size 128, enabled Guest dmesg: found-> vendor=0x10de, dev=0x0dd8, revid=0xa1 domain=0, bus=0, slot=4, func=0 class=03-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0403, statreg=0x0010, cachelnsz=16 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=16 powerspec 3 supports D0 D3 current D0 MSI supports 1 message, 64 bit map[10]: type Memory, range 32, base 0xc2000000, size 25, enabled pcib0: allocated type 3 (0xc2000000-0xc3ffffff) for rid 10 of pci0:0:4:0 map[14]: type Prefetchable Memory, range 64, base 0xd000000000, size 27, enabled pcib0: allocated type 3 (0xd000000000-0xd007ffffff) for rid 14 of pci0:0:4:0 map[1c]: type Prefetchable Memory, range 64, base 0xd008000000, size 26, enabled pcib0: allocated type 3 (0xd008000000-0xd00bffffff) for rid 1c of pci0:0:4:0 map[24]: type I/O Port, range 32, base 0x2080, size 7, enabled pcib0: allocated type 4 (0x2080-0x20ff) for rid 24 of pci0:0:4:0 pcib0: no PRT entry for 0.4.INTA -- [SorAlx] ridin' VN2000 Classic LT