From owner-freebsd-net@FreeBSD.ORG  Wed Jun 20 14:55:05 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id D719C1065672
	for <freebsd-net@freebsd.org>; Wed, 20 Jun 2012 14:55:05 +0000 (UTC)
	(envelope-from jhb@freebsd.org)
Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net
	[IPv6:2001:470:1f10:75::2])
	by mx1.freebsd.org (Postfix) with ESMTP id A965D8FC19
	for <freebsd-net@freebsd.org>; Wed, 20 Jun 2012 14:55:05 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
	by bigwig.baldwin.cx (Postfix) with ESMTPSA id AB7A5B97E;
	Wed, 20 Jun 2012 10:55:02 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: freebsd-net@freebsd.org
Date: Wed, 20 Jun 2012 10:54:40 -0400
User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; )
References: <CAGdae7YW0jt4gTmHufHFTrcdtFGhngu_Wi+MDg1nY_T0J1C_SQ@mail.gmail.com>
In-Reply-To: <CAGdae7YW0jt4gTmHufHFTrcdtFGhngu_Wi+MDg1nY_T0J1C_SQ@mail.gmail.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Message-Id: <201206201054.40824.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
	(bigwig.baldwin.cx); Wed, 20 Jun 2012 10:55:02 -0400 (EDT)
Cc: Venkat Duvvuru <venkatduvvuru.ml@gmail.com>
Subject: Re: MSI-X limitation in freebsd 8.2
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 20 Jun 2012 14:55:05 -0000

On Wednesday, June 20, 2012 6:47:34 am Venkat Duvvuru wrote:
> Hi,
> MSI-x supports upto 2048 vectors but what I see in freebsd 8.2 is that when
> I use more than ~30 vectors, system becomes dead slow.
> Is there a limitation on number of msi vectors that can be used in 8.2?

FreeBSD/x86 supports roughly 191 distinct interrupts on each CPU, and each 
MSI-X interrupt takes up a single vector on the CPU it resides on.  If you
exhaust that set of vectors, then pci_alloc_msix() will fail (or return a
smaller count).  However, I haven't seen any issues with the system being slow 
with 30 vectors.  Are these 30 vectors all tied to a single device, and are 
they evenly distributed across the available CPUs?  Have you run top when the 
system was sluggish to see what each CPU was doing?

-- 
John Baldwin