From owner-freebsd-net@FreeBSD.ORG Thu Jun 21 13:00:09 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B07A11065673; Thu, 21 Jun 2012 13:00:09 +0000 (UTC) (envelope-from venkatduvvuru.ml@gmail.com) Received: from mail-gh0-f182.google.com (mail-gh0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id 38B138FC15; Thu, 21 Jun 2012 13:00:09 +0000 (UTC) Received: by ghbz22 with SMTP id z22so479681ghb.13 for ; Thu, 21 Jun 2012 06:00:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=sKtBRth/hAaIJM0D3IPHx8bD41VGC0meDhk4EPmlSis=; b=YpM8WHTY588h3/7rjRra0qcPWzLHUTMKc/SCBtl7IHPzb652Jo2OGcl6OVpmdSAHCG mZAHWNjTUMxh5SIe+JgtVvKVwWpQigHcX8L3tHiQIB8Sv/1wwRVaXA3twVQ6B2xc9vaO Z9RLOmMRKxcRV2MAYVIW57HLYKI5/uyda3BwKKZtFpsSNrTd9DMCANCGe7TYBrHSVYrA 940HZF1lqfl2PUeSlaroXUTAmy2/B95b0R4ep/+qst+pRtvS8scyb1mei9uf68KHmksG nMK/wRVDJGR2GX22KIYtSjL7wxATgSq43PvVkeD1D/rSm5DV9jl9xqy8ksamRhhjv+Hj 4GaA== MIME-Version: 1.0 Received: by 10.50.183.230 with SMTP id ep6mr7155238igc.52.1340283608505; Thu, 21 Jun 2012 06:00:08 -0700 (PDT) Received: by 10.64.77.72 with HTTP; Thu, 21 Jun 2012 06:00:08 -0700 (PDT) In-Reply-To: References: <201206201054.40824.jhb@freebsd.org> <201206210816.22774.jhb@freebsd.org> Date: Thu, 21 Jun 2012 18:30:08 +0530 Message-ID: From: Venkat Duvvuru To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: MSI-X limitation in freebsd 8.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jun 2012 13:00:09 -0000 I suspected the hardware hence installed Linux and cross verified that Linux is working fine. /Venkat On Thu, Jun 21, 2012 at 6:20 PM, Venkat Duvvuru wrote: > While I'm able to ping to the machine..every command on the current active > session is sluggish infact doesn't complete, unable to open another ssh > session..an already opened ssh session doesn't react which is running > "systat -vmstat"... > > The only thing I could do is to successfully complete "top -P" command > once..even top hung for the second time. > > /Venkat > > > On Thu, Jun 21, 2012 at 5:46 PM, John Baldwin wrote: > >> On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote: >> > John - Thanks for the reply. >> > >> > All the CPUs are ~100% idle. I don't see any interrupt storm on any of >> the >> > irqs (vmstat -i). >> > >> > One observation I made is that I see messages like these in dmesg >> > >> > ===> mem >> 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff >> > irq 40 at device 0.1 on pci6 >> > >> > Looking at the irq value I think it is the INTx irq range which >> shouldn't >> > have probably got allocated as the device is msix capable and there are >> > vectors allocated for these devices in the range (256-380). >> > >> > Could this be a problem? >> >> No, that line is output before the driver's attach routine is run, so it >> will >> always show INTx IRQ value even if it isn't used. >> >> > The scenario where I am hitting this problem is a setup with 4 NICs, >> each >> > NIC with two ports and each port using up 4 msix vectors. The system is >> > fine till some ports are up but once I ifup the 5th port, the system >> > becomes sluggish. >> > >> > I'm not sure whether all the 30 vectors are from a single cpu..I don't >> know >> > how to get that information. >> >> Unfortunately there isn't an easy way. I have this gdb script which can >> display it from kgdb on x86: >> >> define irqs >> set $e = event_list->tqh_first >> while ($e != 0) >> if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0) >> set $src = (struct intsrc *)$e->ie_source >> if ($src->is_pic->pic_enable_source == &ioapic_enable_source) >> set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu >> else >> if ($src->is_pic->pic_enable_source == &msi_enable_source) >> set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu >> else >> set $_cpu = 0 >> end >> end >> printf "CPU %d: %s\n", $_cpu, $e->ie_fullname >> end >> set $e = $e->ie_list.tqe_next >> end >> end >> >> document irqs >> Dump list of IRQs with associated CPU. >> end >> >> However, unless the driver is using BUS_BIND_IRQ() or you are using >> cpuset -x, >> the interrupts should be round-robin assigned among CPUs. >> >> What exactly do you mean by sluggish? Trying to interact with the box >> over >> SSH is sluggish? Is there a change in RTT if you are pinging the box, is >> there a change in performance of TCP or UDP streams to/from the box? >> >> -- >> John Baldwin >> > >