From owner-freebsd-net@FreeBSD.ORG Fri Jun 22 14:24:23 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D410D106564A for ; Fri, 22 Jun 2012 14:24:23 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 7D71B8FC15 for ; Fri, 22 Jun 2012 14:24:23 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id CED10B99F; Fri, 22 Jun 2012 10:24:22 -0400 (EDT) From: John Baldwin To: Venkat Duvvuru Date: Fri, 22 Jun 2012 10:04:08 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; ) References: In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201206221004.08538.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 22 Jun 2012 10:24:22 -0400 (EDT) Cc: freebsd-net@freebsd.org Subject: Re: MSI-X limitation in freebsd 8.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jun 2012 14:24:23 -0000 On Friday, June 22, 2012 6:25:23 am Venkat Duvvuru wrote: > and no matter in which order I bring up the interfaces, the system becomes > sluggish after the 5th interface is brought up. So doesn't look like a > specific driver issue. Humm. Would it be possible to use KTR_SCHED and capture a ktr dump while it is sluggish? > On Fri, Jun 22, 2012 at 2:25 PM, Venkat Duvvuru > wrote: > > > This setup has variety of NICs, Intel's, Emulex's and all are stock > > drivers. > > > > /Venkat > > > > On Thu, Jun 21, 2012 at 7:17 PM, John Baldwin wrote: > > > >> On Thursday, June 21, 2012 8:50:06 am Venkat Duvvuru wrote: > >> > While I'm able to ping to the machine..every command on the current > >> active > >> > session is sluggish infact doesn't complete, unable to open another ssh > >> > session..an already opened ssh session doesn't react which is running > >> > "systat -vmstat"... > >> > > >> > The only thing I could do is to successfully complete "top -P" command > >> > once..even top hung for the second time. > >> > >> Hmm, is this a stock FreeBSD driver or an out-of-tree NIC driver? > >> > >> > /Venkat > >> > > >> > On Thu, Jun 21, 2012 at 5:46 PM, John Baldwin wrote: > >> > > >> > > On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote: > >> > > > John - Thanks for the reply. > >> > > > > >> > > > All the CPUs are ~100% idle. I don't see any interrupt storm on any > >> of > >> > > the > >> > > > irqs (vmstat -i). > >> > > > > >> > > > One observation I made is that I see messages like these in dmesg > >> > > > > >> > > > ===> mem > >> > > 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff > >> > > > irq 40 at device 0.1 on pci6 > >> > > > > >> > > > Looking at the irq value I think it is the INTx irq range which > >> shouldn't > >> > > > have probably got allocated as the device is msix capable and there > >> are > >> > > > vectors allocated for these devices in the range (256-380). > >> > > > > >> > > > Could this be a problem? > >> > > > >> > > No, that line is output before the driver's attach routine is run, so > >> it > >> > > will > >> > > always show INTx IRQ value even if it isn't used. > >> > > > >> > > > The scenario where I am hitting this problem is a setup with 4 > >> NICs, each > >> > > > NIC with two ports and each port using up 4 msix vectors. The > >> system is > >> > > > fine till some ports are up but once I ifup the 5th port, the system > >> > > > becomes sluggish. > >> > > > > >> > > > I'm not sure whether all the 30 vectors are from a single cpu..I > >> don't > >> > > know > >> > > > how to get that information. > >> > > > >> > > Unfortunately there isn't an easy way. I have this gdb script which > >> can > >> > > display it from kgdb on x86: > >> > > > >> > > define irqs > >> > > set $e = event_list->tqh_first > >> > > while ($e != 0) > >> > > if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0) > >> > > set $src = (struct intsrc *)$e->ie_source > >> > > if ($src->is_pic->pic_enable_source == > >> &ioapic_enable_source) > >> > > set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu > >> > > else > >> > > if ($src->is_pic->pic_enable_source == > >> &msi_enable_source) > >> > > set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu > >> > > else > >> > > set $_cpu = 0 > >> > > end > >> > > end > >> > > printf "CPU %d: %s\n", $_cpu, $e->ie_fullname > >> > > end > >> > > set $e = $e->ie_list.tqe_next > >> > > end > >> > > end > >> > > > >> > > document irqs > >> > > Dump list of IRQs with associated CPU. > >> > > end > >> > > > >> > > However, unless the driver is using BUS_BIND_IRQ() or you are using > >> cpuset > >> > > -x, > >> > > the interrupts should be round-robin assigned among CPUs. > >> > > > >> > > What exactly do you mean by sluggish? Trying to interact with the > >> box over > >> > > SSH is sluggish? Is there a change in RTT if you are pinging the > >> box, is > >> > > there a change in performance of TCP or UDP streams to/from the box? > >> > > > >> > > -- > >> > > John Baldwin > >> > > > >> > > >> > >> -- > >> John Baldwin > >> > > > > > -- John Baldwin