From owner-freebsd-net@FreeBSD.ORG Thu Jun 21 09:47:50 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 66EAC106566C; Thu, 21 Jun 2012 09:47:50 +0000 (UTC) (envelope-from venkatduvvuru.ml@gmail.com) Received: from mail-gh0-f182.google.com (mail-gh0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0BBAE8FC12; Thu, 21 Jun 2012 09:47:49 +0000 (UTC) Received: by ghbz22 with SMTP id z22so320745ghb.13 for ; Thu, 21 Jun 2012 02:47:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=E41q0VC+DzYfrIKGI22vyrmbFChzOMq9bV+lgc9Da5o=; b=ce4z8MVOj94CIOyRmCo0c8Hvuq1r3rTeyLsG1eoIHaxjLmecQbFSeTSx2mQYN0Ze+7 EPIlHJripWtJdO6MmXx1PKczh3NHCZzeJRnxkGDsZnKFu3Zy6w9OcYZ8NVs99sSryKZJ 9AoqCB8K+vhjbixNiqdaPdbZr+XcGI8kuQPcGKyiXTdntAGVDgCIWPg7Uez5FQksk+Nt PjiEjPHregK5XLnB20kzTKa53Lkt3nmCewFhPT6T94+6Mbq32wkGXZN4yTbqjbPWw8gZ g9Dkvsp0PSV3hCrmb4AW79Q5u8mS0g1/Suw5Q/XV3kVnnpQWEoePIV7t8FRgQIP7H4Nx +CvA== MIME-Version: 1.0 Received: by 10.50.10.164 with SMTP id j4mr6923587igb.13.1340272068404; Thu, 21 Jun 2012 02:47:48 -0700 (PDT) Received: by 10.64.77.72 with HTTP; Thu, 21 Jun 2012 02:47:48 -0700 (PDT) In-Reply-To: <201206201054.40824.jhb@freebsd.org> References: <201206201054.40824.jhb@freebsd.org> Date: Thu, 21 Jun 2012 15:17:48 +0530 Message-ID: From: Venkat Duvvuru To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: MSI-X limitation in freebsd 8.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jun 2012 09:47:50 -0000 John - Thanks for the reply. All the CPUs are ~100% idle. I don't see any interrupt storm on any of the irqs (vmstat -i). One observation I made is that I see messages like these in dmesg ===> mem 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff irq 40 at device 0.1 on pci6 Looking at the irq value I think it is the INTx irq range which shouldn't have probably got allocated as the device is msix capable and there are vectors allocated for these devices in the range (256-380). Could this be a problem? The scenario where I am hitting this problem is a setup with 4 NICs, each NIC with two ports and each port using up 4 msix vectors. The system is fine till some ports are up but once I ifup the 5th port, the system becomes sluggish. I'm not sure whether all the 30 vectors are from a single cpu..I don't know how to get that information. /Venkat On Wed, Jun 20, 2012 at 8:24 PM, John Baldwin wrote: > On Wednesday, June 20, 2012 6:47:34 am Venkat Duvvuru wrote: > > Hi, > > MSI-x supports upto 2048 vectors but what I see in freebsd 8.2 is that > when > > I use more than ~30 vectors, system becomes dead slow. > > Is there a limitation on number of msi vectors that can be used in 8.2? > > FreeBSD/x86 supports roughly 191 distinct interrupts on each CPU, and each > MSI-X interrupt takes up a single vector on the CPU it resides on. If you > exhaust that set of vectors, then pci_alloc_msix() will fail (or return a > smaller count). However, I haven't seen any issues with the system being > slow > with 30 vectors. Are these 30 vectors all tied to a single device, and are > they evenly distributed across the available CPUs? Have you run top when > the > system was sluggish to see what each CPU was doing? > > -- > John Baldwin >