Date: Fri, 9 Sep 2016 10:20:41 +0100 From: Dom <misc-freebsd@talk2dom.com> To: freebsd-virtualization@freebsd.org Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <22df7de9-1dfb-561c-125d-99af2b3da70b@talk2dom.com> In-Reply-To: <62b2a72a-3662-5b3e-5029-f284a625d05b@freebsd.org> References: <727839e2-4be1-5661-8c19-6d13f73ad36f@talk2dom.com> <62b2a72a-3662-5b3e-5029-f284a625d05b@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Peter,
Thanks for getting back to me. Here's the info you requested:
>> [ 0.163085] acpi PNP0A03:00: host bridge window
>> [0xd000000000-0xd0100fffff] (ignored, not CPU addressable)
>
> That one is most likely a bug in bhyve, where the space used for 64-bit
> BAR placement isn't tested against the max physaddr width of the host
> CPU.
>
> To confirm, would you be able to report on this value on your system ?
>
> # sudo pkg install cpuid
> # cpuid | grep ^80000008
On my Intel i7-4790K CPU:
# cpuid | grep ^80000008
80000008 00003027 00000000 00000000 00000000
> The device has an MSI capability, but the nvidia driver may not use it.
> bhyve PCI passthrough requires the use of MSI/MSI-x interrupts, and
> doesn't support using legacy interrupts.
>
> This could be confirmed from the output of /proc/interrupts when
> booting Linux on the system.
Output of /proc/interrupts:
CPU0
0: 137 IO-APIC-edge timer
1: 9 IO-APIC-edge i8042
4: 965 IO-APIC-edge serial
8: 0 IO-APIC-edge rtc0
9: 0 IO-APIC-fasteoi acpi
12: 138 IO-APIC-edge i8042
17: 0 IO-APIC-fasteoi snd_hda_intel
24: 0 PCI-MSI-edge virtio0-config
25: 8535 PCI-MSI-edge virtio0-req.0
26: 0 PCI-MSI-edge virtio1-config
27: 123 PCI-MSI-edge virtio1-input.0
28: 1 PCI-MSI-edge virtio1-output.0
NMI: 0 Non-maskable interrupts
LOC: 6050 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
IWI: 2484 IRQ work interrupts
RTR: 0 APIC ICR read retries
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 1 Machine check polls
ERR: 0
MIS: 0
I guess the lack of a line containing PCI-MSI-* here indicates the
nvidia driver isn't using an MSI/MSI-x interrupt?
However, searching the web suggests the Linux nvidia driver does use MSI
interrupts. This taken from a working non-VM Linux dmesg:
[ 4.330536] nvidia 0000:05:00.0: PCI INT A -> GSI 16 (level, low) ->
IRQ 16
[ 4.330542] nvidia 0000:05:00.0: setting latency timer to 64
Source: https://bugzilla.kernel.org/show_bug.cgi?id=20432#c2
(Thread also mentions disabling MSI)
I'll try some Linux boot options and reordering the devices when calling
bhyve to see if that changes anything.
Thanks,
Dom
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?22df7de9-1dfb-561c-125d-99af2b3da70b>