From owner-freebsd-net@FreeBSD.ORG Tue Nov 1 02:08:17 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2605E1065672 for ; Tue, 1 Nov 2011 02:08:17 +0000 (UTC) (envelope-from pprocacci@datapipe.com) Received: from EXFESMQ04.datapipe-corp.net (exfesmq04.datapipe.com [64.27.120.68]) by mx1.freebsd.org (Postfix) with ESMTP id D09758FC1E for ; Tue, 1 Nov 2011 02:08:16 +0000 (UTC) Received: from nat.myhome (192.168.128.103) by EXFESMQ04.datapipe-corp.net (192.168.128.29) with Microsoft SMTP Server (TLS) id 14.1.339.1; Mon, 31 Oct 2011 21:57:24 -0400 Date: Mon, 31 Oct 2011 20:57:46 -0500 From: "Paul A. Procacci" To: Message-ID: <20111101015746.GA96508@nat.myhome> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-Originating-IP: [192.168.128.103] Content-Transfer-Encoding: quoted-printable Subject: [High Interrupt Count] Networking Difficulties X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Nov 2011 02:08:17 -0000 Gents, I'm having quite an aweful problem that I need a bit of help with. I have an HPDL360 G3 ( http://h18000.www1.hp.com/products/quickspecs/11504_= na/11504_na.HTML ) which acts as a NAT (via PF) for several (600+) class C'= s amongst 24+ machines sitting behind it. It's running FPSense (FreeBSD 8.1-RELEASE-p4). The important guts are: 2 x 2.8 GHz Cpus 2 BGE interfaces on a PCI-X bus. During peak times this machine is only able to handle between 500Mbps - 600= Mbps before running out of cpu capacity. (300Mbps(ish) on the LAN, 300Mbps= (ish) on the WAN) It's due to the high number of interrupts. I was speaking with a networking engineer here and he mentioned that I shou= ld look at "Interrupt Coalescing" to increase throughput. The only information I found online regarding this was a post from 2 years = ago here: http://lists.freebsd.org/pipermail/freebsd-net/2009-June/022227.h= tml The tunables mentioned in the above post aren't present in my system, so I = imagine this never made it into the bge driver. Assuming this to be the ca= se, I started looking at DEVICE_POLLING as a solution. I did try implementing device polling, but the results were worse than I ex= pected. netisr was using 100% of a single cpu while the other cpu remained= mostly idle. Not knowing exactly what netisr is, I reverted the changes. This leads me to this list. Given the scenario above, I'm nearly certain I= need to use device polling instead of the standard interrupt driven setup. The two sysctl's that I've come across thus far that I think are what I nee= d are: net.isr.maxthreads hern.hz I would assume setting net.isr.maxthreads to 2 given my dual core machine i= s advisable, but I'm not 100% sure. What are the caveats in setting this higher? Given the output of `sysctl -= d net.isr.maxthreads` I would expect anything higher than the number of cor= es to be detrimental. Is this correct? kern.hz I'm more unsure of. I understand what the sysctl is, but I'm not s= ure how to come up with a reasonable number. Generally speaking, and in your experience, would a setting of 2000 achive = close to the theoritical meximum of the cards? Is there an upper limit tha= t I would be worried about? Random Question: - is device polling really the answer? I am missing something in the bge d= river that I've overlooked? - what tunables directly effect processing high volumes of packets. Network Interfaces: ###########################################################################= ####### bge0: flags=3D8943 metric 0= mtu 1500 options=3D8009b ether 00:0b:cd:ca:1d:1a inet 209.18.70.211 netmask 0xffffff00 broadcast 209.18.70.255 inet6 fe80::20b:cdff:feca:1d1a%bge0 prefixlen 64 scopeid 0x1 nd6 options=3D3 media: Ethernet autoselect (1000baseT ) status: active bge1: flags=3D8943 metric 0= mtu 1500 options=3D8009b ether 00:0b:cd:ca:1a:74 inet 172.16.0.3 netmask 0xfffc0000 broadcast 172.19.255.255 inet6 fe80::20b:cdff:feca:1a74%bge1 prefixlen 64 scopeid 0x2 nd6 options=3D3 media: Ethernet autoselect (1000baseT ) status: active ###########################################################################= ####### I appreciate the help in advance. Thanks, Paul ________________________________ This message may contain confidential or privileged information. If you are= not the intended recipient, please advise us immediately and delete this m= essage. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for= further information on confidentiality and the risks of non-secure electro= nic communication. If you cannot access these links, please notify us by re= ply message and we will send the contents to you.