Date: Thu, 09 Apr 2015 16:59:08 +0000 From: bugzilla-noreply@freebsd.org To: freebsd-bugs@FreeBSD.org Subject: [Bug 199321] Total MSI-X vector allocation limited to 191 vectors Message-ID: <bug-199321-8@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199321 Bug ID: 199321 Summary: Total MSI-X vector allocation limited to 191 vectors Product: Base System Version: 11.0-CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: freebsd-bugs@FreeBSD.org Reporter: jimharris@FreeBSD.org System Configuration: 2x 18C E5-v3 CPU (SMT enabled) 2x X520 Intel Ethernet 10Gb Adapter (4 ports total) 11.0-CURRENT (r281283) ixgbe driver tries to allocate full range of 64 MSI-X vectors per port - first two ports succeed, but third and fourth ports fail and fall back to a single MSI vector. msix_alloc() tries to use intr_next_cpu() to round-robin MSI-X vectors across all cores. But driver initialization happens before APs are started, so intr_next_cpu() just returns the BSP's apic ID 0. Each local APIC is limited to APIC_NUM_IOINTS==191 vectors, so a single local APIC cannot handle the 64 vectors for each of the 4 ports. intr_shuffle_irqs runs in SI_SUB_SMP phase and is intended to shuffle iras off of core 0 to APs, but at this point it is too late - pci_alloc_msix() already reported to ixgbe(4) that it could not allocate all 64 vectors. x86 defines NUM_MSI_INTS as 512, but this bug really restricts number of MSI-X vectors to APIC_NUM_IOINTS (191). Impact is that some drivers/devices which can take advantage of multiple queues and one interrupt per queue must fall back to a single interrupt. There may also be inconsistency across platforms based on device enumeration - whichever devices enumerate first will get its allotment of MSI-X vectors. This problem will become exacerbated as NVMe SSDs become more prevalent, which also try to allocate 32-64 MSIx vectors each. There are systems available already with 8 NVMe SSDs in a single system. -- You are receiving this mail because: You are the assignee for the bug.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-199321-8>