Date: Mon, 7 Nov 2016 08:12:55 -0600 From: Rajil Saraswat <rajil.s@gmail.com> To: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve, ixgbe and debian 8 Message-ID: <c368eb48-e3c5-fc74-f761-13d5e56da4b9@gmail.com> In-Reply-To: <8c44e3c4-fcc8-d288-26f0-df742b1c79a7@gmail.com> References: <124ae089-6215-7632-44cf-319b5f88c0b5@gmail.com> <8c44e3c4-fcc8-d288-26f0-df742b1c79a7@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 11/07/2016 07:41 AM, Rajil Saraswat wrote:
> On 11/07/2016 07:08 AM, Rajil Saraswat wrote:
>> Hello,
>>
>> I have PCI passthrough working now for the Intel X540-AT2 NIC and is
>> seen inside debian 8 vm.
>>
>> # lspci
>>
>> 00:00.0 Host bridge: Network Appliance Corporation Device 1275
>> 00:04.0 SCSI storage controller: Red Hat, Inc Virtio block device
>> 00:05.0 Ethernet controller: Red Hat, Inc Virtio network device
>> 00:06.0 Ethernet controller: Intel Corporation Ethernet Controller
>> 10-Gigabit X540-AT2 (rev 01)
>> 00:1f.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
>>
>>
>> However, on loading the module the kernel gives the error
>>
>> #modprobe ixgbe
>>
>> [ 34.929546] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
>> version 3.19.1-k
>> [ 34.929549] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
>> [ 34.929706] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
>> [ 34.929708] ixgbe 0000:00:06.0: PCI INT A: no GSI
>> [ 34.930354] ixgbe 0000:00:06.0: Adapter removed
>> [ 34.931710] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
>> [ 34.931717] ixgbe: probe of 0000:00:06.0 failed with error -5
>>
>> Any idea what could be the issue?
>>
>> Thanks,
>>
>> Rajil
>>
> I tried with the second onboard NIC and it worked the first time. iperf
> showed 10g speeds. After restarting the VM, i get an error for both
> NICs. This a dual CPU motherboard Supermicro X10DriT with a pair of
> onboard NICs.
>
> [ 0.628758] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver -
> version 3.19.1-k
> [ 0.628760] ixgbe: Copyright (c) 1999-2014 Intel Corporation.
> [ 0.628899] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
> [ 0.628901] ixgbe 0000:00:06.0: PCI INT A: no GSI
> [ 0.629519] ixgbe 0000:00:06.0: Adapter removed
> [ 0.630143] ixgbe 0000:00:06.0: can't derive routing for PCI INT A
> [ 0.630147] ixgbe: probe of 0000:00:06.0 failed with error -5
> [ 0.630199] ixgbe 0000:00:06.1: can't derive routing for PCI INT B
> [ 0.630200] ixgbe 0000:00:06.1: PCI INT B: no GSI
> [ 0.630789] ixgbe 0000:00:06.1: Adapter removed
> [ 0.639469] ixgbe 0000:00:06.1: can't derive routing for PCI INT B
> [ 0.639473] ixgbe: probe of 0000:00:06.1 failed with error -5
>
After a cold reboot of the server, the devices are again seen in debian.
I guess somehow the devices become unusable when the vm is restarted.
[ 1.146946] ixgbe 0000:00:06.0: PCI Express bandwidth of 32GT/s available
[ 1.146949] ixgbe 0000:00:06.0: (Speed:5.0GT/s, Width: x8, Encoding
Loss:20%)
[ 1.146953] ixgbe 0000:00:06.0: This is not sufficient for optimal
performance of this card.
[ 1.146955] ixgbe 0000:00:06.0: For optimal performance, at least
60GT/s of bandwidth is required.
[ 1.146956] ixgbe 0000:00:06.0: A slot with more lanes and/or higher
speed is suggested.
[ 1.189458] ixgbe 0000:00:06.0: MAC: 3, PHY: 3, PBA No: 030C00-000
[ 1.189461] ixgbe 0000:00:06.0: 0c:c4:7a:de:cc:c0
[ 1.339231] ixgbe 0000:00:06.0: Intel(R) 10 Gigabit Network Connection
[ 1.339364] ixgbe 0000:00:06.1: can't derive routing for PCI INT B
[ 1.339366] ixgbe 0000:00:06.1: PCI INT B: no GSI
[ 1.348140] input: PS/2 Generic Mouse as
/devices/platform/i8042/serio1/input/input2
[ 1.566630] tsc: Refined TSC clocksource calibration: 2198.264 MHz
[ 1.793929] ixgbe 0000:00:06.1: irq 47 for MSI/MSI-X
[ 1.793950] ixgbe 0000:00:06.1: irq 48 for MSI/MSI-X
[ 1.795079] ixgbe 0000:00:06.1: Multiqueue Disabled: Rx Queue count =
1, Tx Queue count = 1
[ 1.859258] ixgbe 0000:00:06.1: PCI Express bandwidth of 32GT/s available
[ 1.859261] ixgbe 0000:00:06.1: (Speed:5.0GT/s, Width: x8, Encoding
Loss:20%)
[ 1.859262] ixgbe 0000:00:06.1: This is not sufficient for optimal
performance of this card.
[ 1.859264] ixgbe 0000:00:06.1: For optimal performance, at least
60GT/s of bandwidth is required.
[ 1.859265] ixgbe 0000:00:06.1: A slot with more lanes and/or higher
speed is suggested.
[ 1.902549] ixgbe 0000:00:06.1: MAC: 3, PHY: 3, PBA No: 030C00-000
[ 1.902554] ixgbe 0000:00:06.1: 0c:c4:7a:de:cc:c1
[ 2.055631] ixgbe 0000:00:06.1: Intel(R) 10 Gigabit Network Connection
# cat /proc/interrupts
CPU0
0: 45 IO-APIC-edge timer
1: 9 IO-APIC-edge i8042
4: 3756 IO-APIC-edge serial
8: 0 IO-APIC-edge rtc0
9: 0 IO-APIC-fasteoi acpi
12: 138 IO-APIC-edge i8042
40: 0 PCI-MSI-edge virtio0-config
41: 5667 PCI-MSI-edge virtio0-requests
42: 0 PCI-MSI-edge virtio1-config
43: 139 PCI-MSI-edge virtio1-input.0
44: 1 PCI-MSI-edge virtio1-output.0
45: 120 PCI-MSI-edge eth1-TxRx-0
46: 0 PCI-MSI-edge eth1
47: 1170 PCI-MSI-edge eth2-TxRx-0
48: 3 PCI-MSI-edge eth2
NMI: 0 Non-maskable interrupts
LOC: 8265 Local timer interrupts
SPU: 0 Spurious interrupts
PMI: 0 Performance monitoring interrupts
IWI: 0 IRQ work interrupts
RTR: 0 APIC ICR read retries
RES: 0 Rescheduling interrupts
CAL: 0 Function call interrupts
TLB: 0 TLB shootdowns
TRM: 0 Thermal event interrupts
THR: 0 Threshold APIC interrupts
MCE: 0 Machine check exceptions
MCP: 1 Machine check polls
HYP: 0 Hypervisor callback interrupts
ERR: 0
MIS: 0
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c368eb48-e3c5-fc74-f761-13d5e56da4b9>