Date: Thu, 21 Jun 2012 15:17:48 +0530 From: Venkat Duvvuru <venkatduvvuru.ml@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: MSI-X limitation in freebsd 8.2 Message-ID: <CAGdae7anhqdzacVt-BMv3e7GvKFO4QbYEVxd9k-agGjLfPi_iQ@mail.gmail.com> In-Reply-To: <201206201054.40824.jhb@freebsd.org> References: <CAGdae7YW0jt4gTmHufHFTrcdtFGhngu_Wi%2BMDg1nY_T0J1C_SQ@mail.gmail.com> <201206201054.40824.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
John - Thanks for the reply. All the CPUs are ~100% idle. I don't see any interrupt storm on any of the irqs (vmstat -i). One observation I made is that I see messages like these in dmesg ===> mem 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff irq 40 at device 0.1 on pci6 Looking at the irq value I think it is the INTx irq range which shouldn't have probably got allocated as the device is msix capable and there are vectors allocated for these devices in the range (256-380). Could this be a problem? The scenario where I am hitting this problem is a setup with 4 NICs, each NIC with two ports and each port using up 4 msix vectors. The system is fine till some ports are up but once I ifup the 5th port, the system becomes sluggish. I'm not sure whether all the 30 vectors are from a single cpu..I don't know how to get that information. /Venkat On Wed, Jun 20, 2012 at 8:24 PM, John Baldwin <jhb@freebsd.org> wrote: > On Wednesday, June 20, 2012 6:47:34 am Venkat Duvvuru wrote: > > Hi, > > MSI-x supports upto 2048 vectors but what I see in freebsd 8.2 is that > when > > I use more than ~30 vectors, system becomes dead slow. > > Is there a limitation on number of msi vectors that can be used in 8.2? > > FreeBSD/x86 supports roughly 191 distinct interrupts on each CPU, and each > MSI-X interrupt takes up a single vector on the CPU it resides on. If you > exhaust that set of vectors, then pci_alloc_msix() will fail (or return a > smaller count). However, I haven't seen any issues with the system being > slow > with 30 vectors. Are these 30 vectors all tied to a single device, and are > they evenly distributed across the available CPUs? Have you run top when > the > system was sluggish to see what each CPU was doing? > > -- > John Baldwin >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGdae7anhqdzacVt-BMv3e7GvKFO4QbYEVxd9k-agGjLfPi_iQ>