From owner-freebsd-net@FreeBSD.ORG Thu Jun 21 12:50:07 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 935B4106564A; Thu, 21 Jun 2012 12:50:07 +0000 (UTC) (envelope-from venkatduvvuru.ml@gmail.com) Received: from mail-gh0-f182.google.com (mail-gh0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2CADA8FC12; Thu, 21 Jun 2012 12:50:07 +0000 (UTC) Received: by ghbz22 with SMTP id z22so468652ghb.13 for ; Thu, 21 Jun 2012 05:50:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=nGF/DbKO67rayK6Gc+XZqykH2hiNoEXQ1Bc6uYsQtJM=; b=pH+EtTh1Tp6pCDRPCyDz/7zXUkPjB90HUGugq5Khsz0yTNldHKS3pCgc632WEsaZ9y m75FVaRzq1VzlWWhS1c/yA16r/eOFdNS0Zk4Fqpj2+N08oWhcQC+ivgI9CtN2UF0dVpr g0rjIsCOiOPmtidWeNDxwZcjYJUKuAnv6I3ovyAG8W2jHCid8y9OPlhXCTKrQy4ECSh6 npy7NFohrqgCzET/x0UobZR0wFHyo6KV9i45FT2uIfBTae10ez0edaOvW/oF7WZa4tS4 XSuTuHBnbqYKaoLw0deKqH2GeumVDOITfyQ40nJqDOC7B+l4Et9r30GwNkjvnZRT7QmX pk0A== MIME-Version: 1.0 Received: by 10.50.36.227 with SMTP id t3mr7329726igj.13.1340283006350; Thu, 21 Jun 2012 05:50:06 -0700 (PDT) Received: by 10.64.77.72 with HTTP; Thu, 21 Jun 2012 05:50:06 -0700 (PDT) In-Reply-To: <201206210816.22774.jhb@freebsd.org> References: <201206201054.40824.jhb@freebsd.org> <201206210816.22774.jhb@freebsd.org> Date: Thu, 21 Jun 2012 18:20:06 +0530 Message-ID: From: Venkat Duvvuru To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: MSI-X limitation in freebsd 8.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jun 2012 12:50:07 -0000 While I'm able to ping to the machine..every command on the current active session is sluggish infact doesn't complete, unable to open another ssh session..an already opened ssh session doesn't react which is running "systat -vmstat"... The only thing I could do is to successfully complete "top -P" command once..even top hung for the second time. /Venkat On Thu, Jun 21, 2012 at 5:46 PM, John Baldwin wrote: > On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote: > > John - Thanks for the reply. > > > > All the CPUs are ~100% idle. I don't see any interrupt storm on any of > the > > irqs (vmstat -i). > > > > One observation I made is that I see messages like these in dmesg > > > > ===> mem > 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff > > irq 40 at device 0.1 on pci6 > > > > Looking at the irq value I think it is the INTx irq range which shouldn't > > have probably got allocated as the device is msix capable and there are > > vectors allocated for these devices in the range (256-380). > > > > Could this be a problem? > > No, that line is output before the driver's attach routine is run, so it > will > always show INTx IRQ value even if it isn't used. > > > The scenario where I am hitting this problem is a setup with 4 NICs, each > > NIC with two ports and each port using up 4 msix vectors. The system is > > fine till some ports are up but once I ifup the 5th port, the system > > becomes sluggish. > > > > I'm not sure whether all the 30 vectors are from a single cpu..I don't > know > > how to get that information. > > Unfortunately there isn't an easy way. I have this gdb script which can > display it from kgdb on x86: > > define irqs > set $e = event_list->tqh_first > while ($e != 0) > if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0) > set $src = (struct intsrc *)$e->ie_source > if ($src->is_pic->pic_enable_source == &ioapic_enable_source) > set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu > else > if ($src->is_pic->pic_enable_source == &msi_enable_source) > set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu > else > set $_cpu = 0 > end > end > printf "CPU %d: %s\n", $_cpu, $e->ie_fullname > end > set $e = $e->ie_list.tqe_next > end > end > > document irqs > Dump list of IRQs with associated CPU. > end > > However, unless the driver is using BUS_BIND_IRQ() or you are using cpuset > -x, > the interrupts should be round-robin assigned among CPUs. > > What exactly do you mean by sluggish? Trying to interact with the box over > SSH is sluggish? Is there a change in RTT if you are pinging the box, is > there a change in performance of TCP or UDP streams to/from the box? > > -- > John Baldwin >