From owner-freebsd-net@FreeBSD.ORG Tue Nov 1 17:42:18 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0CEE106564A for ; Tue, 1 Nov 2011 17:42:18 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 43CB38FC15 for ; Tue, 1 Nov 2011 17:42:17 +0000 (UTC) Received: by wyg36 with SMTP id 36so776928wyg.13 for ; Tue, 01 Nov 2011 10:42:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=SaH2to4KCGYhPT14oKKWKb2N5bviXOfk+Waaaf1EfT8=; b=SrCVKUjkYlCMql0E/MmfB1l3YOMDJgZ16QhSvHDcZlbqgMJDDpJeJ2r7MCQmWud9LH jwbq7rVme7qDi8Dm8rWIgwKQdXi1jykKUh5x2hCIszjzwoNOBq/L3hw0QZr835QZPbrr RJHBB3/fJOfoxPrVXb5DHa+hU3VgYDZUiwkUk= Received: by 10.216.137.86 with SMTP id x64mr182114wei.2.1320169337100; Tue, 01 Nov 2011 10:42:17 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id et20sm17153546wbb.15.2011.11.01.10.42.12 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 01 Nov 2011 10:42:15 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Tue, 01 Nov 2011 10:40:41 -0700 From: YongHyeon PYUN Date: Tue, 1 Nov 2011 10:40:41 -0700 To: "Paul A. Procacci" Message-ID: <20111101174041.GC6914@michelle.cdnetworks.com> References: <20111101015746.GA96508@nat.myhome> <20111101051637.GC2445@nat.myhome> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20111101051637.GC2445@nat.myhome> User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: [High Interrupt Count] Networking Difficulties X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Nov 2011 17:42:18 -0000 On Tue, Nov 01, 2011 at 12:16:37AM -0500, Paul A. Procacci wrote: > On Mon, Oct 31, 2011 at 08:57:46PM -0500, Paul A. Procacci wrote: > > Gents, > > > > I'm having quite an aweful problem that I need a bit of help with. > > > > I have an HPDL360 G3 ( http://h18000.www1.hp.com/products/quickspecs/11504_na/11504_na.HTML ) which acts as a NAT (via PF) for several (600+) class C's amongst 24+ machines sitting behind it. > > It's running FPSense (FreeBSD 8.1-RELEASE-p4). > > > > The important guts are: > > > > 2 x 2.8 GHz Cpus > > 2 BGE interfaces on a PCI-X bus. > > > > During peak times this machine is only able to handle between 500Mbps - 600Mbps before running out of cpu capacity. (300Mbps(ish) on the LAN, 300Mbps(ish) on the WAN) It's due to the high number of interrupts. > > I was speaking with a networking engineer here and he mentioned that I should look at "Interrupt Coalescing" to increase throughput. > > The only information I found online regarding this was a post from 2 years ago here: http://lists.freebsd.org/pipermail/freebsd-net/2009-June/022227.html > > > > The tunables mentioned in the above post aren't present in my system, so I imagine this never made it into the bge driver. Assuming this to be the case, I started looking at DEVICE_POLLING as a solution. > > I did try implementing device polling, but the results were worse than I expected. netisr was using 100% of a single cpu while the other cpu remained mostly idle. > > Not knowing exactly what netisr is, I reverted the changes. > > > > This leads me to this list. Given the scenario above, I'm nearly certain I need to use device polling instead of the standard interrupt driven setup. > > The two sysctl's that I've come across thus far that I think are what I need are: > > > > net.isr.maxthreads > > hern.hz > > > > I would assume setting net.isr.maxthreads to 2 given my dual core machine is advisable, but I'm not 100% sure. > > What are the caveats in setting this higher? Given the output of `sysctl -d net.isr.maxthreads` I would expect anything higher than the number of cores to be detrimental. Is this correct? > > > > kern.hz I'm more unsure of. I understand what the sysctl is, but I'm not sure how to come up with a reasonable number. > > Generally speaking, and in your experience, would a setting of 2000 achive close to the theoritical meximum of the cards? Is there an upper limit that I would be worried about? > > > > Random Question: > > - is device polling really the answer? I am missing something in the bge driver that I've overlooked? > > - what tunables directly effect processing high volumes of packets. > > > > > > After some more coffee, and source code reading, I've now learned that having device polling enabled forces netisr to limit the number of threads it creates to 1. > This kinda defeats the purpose of enabling device polling. This makes me believe that device polling isn't going to be a great solution afterall. > > A snippet from dmesg: > > bge0: mem 0xf7ef0000-0xf7efffff irq 30 at device 2.0 on pci1 > brgphy0: PHY 1 on miibus0 > bge1: mem 0xf7ff0000-0xf7ffffff irq 29 at device 2.0 on pci4 > brgphy1: PHY 1 on miibus1 > > > Any help/advice is appreciated, and sorry for following up to myself with this information. > In most cases there is *NO* need to use DEVICE_POLLING on advanced controllers like bge(4). How many interrupts do you see on your box? Are you seeing more than 50K interrupts per second? bge(4) already supports interrupt coalescing but its configuration is not tunable yet. So you may have to patch driver to change that. I guess there were a couple of fixes for BCM5703 that sits on PCI-X bus since 8.1-RELEASE. Do you see similar problem on 8.2-RELEASE?