Date: Fri, 2 Sep 2016 13:54:11 +0100 From: Dom <misc-freebsd@talk2dom.com> To: freebsd-virtualization@freebsd.org Subject: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <727839e2-4be1-5661-8c19-6d13f73ad36f@talk2dom.com>
next in thread | raw e-mail | index | archive | help
Hello,
Setup:
nvidia GTX960 in PCIe slot
intel i7-4790K CPU
FreeBSD 11-RC2 host
CentOS 7 guest with kernel 3.10.0-327.28.3.el7.x86_64
Using vm-bhyve port
I've hit two issues:
1. BAR allocation
Workaround (for me) is adding "pci=nocrs" to linux guest's kernel
command line.
Without "pci=nocrs" (or with "pci=use_crs"), the GTX960 doesn't get its
256MB block allocated.
Bhyve's ACPI table produces this error linux-side regardless of "pci="
setting:
[ 0.163085] acpi PNP0A03:00: host bridge window
[0xd000000000-0xd0100fffff] (ignored, not CPU addressable)
which then leads to this:
[ 0.215369] pci 0000:00:06.0: can't claim BAR 1 [mem
0xd000000000-0xd00fffffff 64bit pref]: no compatible bridge window
and then, with "pci=use_crs" (i.e. use ACPI host bridge windows):
[ 0.164030] pci_bus 0000:00: root bus resource [bus 00]
[ 0.164379] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7]
[ 0.164799] pci_bus 0000:00: root bus resource [io 0x0d00-0x1fff]
[ 0.165206] pci_bus 0000:00: root bus resource [io 0x2000-0x211f]
[ 0.165623] pci_bus 0000:00: root bus resource [mem
0xc0000000-0xc41fffff]
...
[ 0.231762] pci 0000:00:06.0: BAR 1: no space for [mem size
0x10000000 64bit pref]
[ 0.232263] pci 0000:00:06.0: BAR 1: trying firmware assignment [mem
size 0x10000000 64bit pref]
[ 0.232855] pci 0000:00:06.0: BAR 1: [mem size 0x10000000 64bit pref]
conflicts with PCI mem [mem 0x00000000-0x7fffffffff]
[ 0.233579] pci 0000:00:06.0: BAR 1: failed to assign [mem size
0x10000000 64bit pref]
but with "pci=nocrs" (i.e. ignore ACPI host bridge windows):
[ 0.163967] pci_bus 0000:00: root bus resource [bus 00]
[ 0.164323] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]
[ 0.164745] pci_bus 0000:00: root bus resource [mem
0x00000000-0x7fffffffff]
...
[ 0.230203] pci 0000:00:06.0: BAR 1: assigned [mem
0x140000000-0x14fffffff 64bit pref]
2. "can't derive routing for PCI INT"
Linux-side dmesg related output:
[ 1.677168] nvidia 0000:00:06.0: can't derive routing for PCI INT A
[ 1.677600] nvidia 0000:00:06.0: PCI INT A: no GSI
Host-side info (when GTX960 is NOT configured as a pass-thru dev):
vgapci0@pci0:1:0:0: class=0x030000 card=0x19623842 chip=0x140110de
rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GM206 [GeForce GTX 960]'
class = display
subclass = VGA
bar [10] = type Memory, range 32, base 0xf6000000, size 16777216,
enabled
bar [14] = type Prefetchable Memory, range 64, base 0xe0000000,
size 268435456, enabled
bar [1c] = type Prefetchable Memory, range 64, base 0xf0000000,
size 33554432, enabled
bar [24] = type I/O Port, range 32, base 0xe000, size 128, enabled
cap 01[60] = powerspec 3 supports D0 D3 current D0
cap 05[68] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[78] = PCI-Express 2 legacy endpoint max data 128(256) RO
link x4(x16) speed 2.5(8.0)
ecap 0002[100] = VC 1 max VC0
ecap 001e[258] = unknown 1
ecap 0004[128] = Power Budgeting 1
ecap 0001[420] = AER 2 0 fatal 0 non-fatal 4 corrected
ecap 000b[600] = Vendor 1 ID 1
ecap 0019[900] = PCIe Sec 1 lane errors 0xf
Linux-side info (when GTX960 IS configured as a pass-thru dev, also with
"pci=nocrs"):
00:06.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX
960] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device 1962
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at c1000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 140000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at c2000000 (64-bit, prefetchable) [size=32M]
Region 5: I/O ports at 2080 [size=128]
Expansion ROM at f7000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [78] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
unlimited, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+
AuxPwr- TransPend-
LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM not
supported, Exit Latency L0s <1us, L1 <4us
ClockPM+ Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range AB, TimeoutDis+,
LTR+, OBFF Via message
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR+, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete+, EqualizationPhase1+
EqualizationPhase2+, EqualizationPhase3+,
LinkEqualizationRequest-
Kernel driver in use: nvidia
Any ideas?
Thanks,
Dom
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?727839e2-4be1-5661-8c19-6d13f73ad36f>