Date: Mon, 31 Oct 2011 20:57:46 -0500 From: "Paul A. Procacci" <pprocacci@datapipe.com> To: <freebsd-net@freebsd.org> Subject: [High Interrupt Count] Networking Difficulties Message-ID: <20111101015746.GA96508@nat.myhome>
next in thread | raw e-mail | index | archive | help
Gents, I'm having quite an aweful problem that I need a bit of help with. I have an HPDL360 G3 ( http://h18000.www1.hp.com/products/quickspecs/11504_= na/11504_na.HTML ) which acts as a NAT (via PF) for several (600+) class C'= s amongst 24+ machines sitting behind it. It's running FPSense (FreeBSD 8.1-RELEASE-p4). The important guts are: 2 x 2.8 GHz Cpus 2 BGE interfaces on a PCI-X bus. During peak times this machine is only able to handle between 500Mbps - 600= Mbps before running out of cpu capacity. (300Mbps(ish) on the LAN, 300Mbps= (ish) on the WAN) It's due to the high number of interrupts. I was speaking with a networking engineer here and he mentioned that I shou= ld look at "Interrupt Coalescing" to increase throughput. The only information I found online regarding this was a post from 2 years = ago here: http://lists.freebsd.org/pipermail/freebsd-net/2009-June/022227.h= tml The tunables mentioned in the above post aren't present in my system, so I = imagine this never made it into the bge driver. Assuming this to be the ca= se, I started looking at DEVICE_POLLING as a solution. I did try implementing device polling, but the results were worse than I ex= pected. netisr was using 100% of a single cpu while the other cpu remained= mostly idle. Not knowing exactly what netisr is, I reverted the changes. This leads me to this list. Given the scenario above, I'm nearly certain I= need to use device polling instead of the standard interrupt driven setup. The two sysctl's that I've come across thus far that I think are what I nee= d are: net.isr.maxthreads hern.hz I would assume setting net.isr.maxthreads to 2 given my dual core machine i= s advisable, but I'm not 100% sure. What are the caveats in setting this higher? Given the output of `sysctl -= d net.isr.maxthreads` I would expect anything higher than the number of cor= es to be detrimental. Is this correct? kern.hz I'm more unsure of. I understand what the sysctl is, but I'm not s= ure how to come up with a reasonable number. Generally speaking, and in your experience, would a setting of 2000 achive = close to the theoritical meximum of the cards? Is there an upper limit tha= t I would be worried about? Random Question: - is device polling really the answer? I am missing something in the bge d= river that I've overlooked? - what tunables directly effect processing high volumes of packets. Network Interfaces: ###########################################################################= ####### bge0: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0= mtu 1500 options=3D8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,L= INKSTATE> ether 00:0b:cd:ca:1d:1a inet 209.18.70.211 netmask 0xffffff00 broadcast 209.18.70.255 inet6 fe80::20b:cdff:feca:1d1a%bge0 prefixlen 64 scopeid 0x1 nd6 options=3D3<PERFORMNUD,ACCEPT_RTADV> media: Ethernet autoselect (1000baseT <full-duplex>) status: active bge1: flags=3D8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0= mtu 1500 options=3D8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,L= INKSTATE> ether 00:0b:cd:ca:1a:74 inet 172.16.0.3 netmask 0xfffc0000 broadcast 172.19.255.255 inet6 fe80::20b:cdff:feca:1a74%bge1 prefixlen 64 scopeid 0x2 nd6 options=3D3<PERFORMNUD,ACCEPT_RTADV> media: Ethernet autoselect (1000baseT <full-duplex>) status: active ###########################################################################= ####### I appreciate the help in advance. Thanks, Paul ________________________________ This message may contain confidential or privileged information. If you are= not the intended recipient, please advise us immediately and delete this m= essage. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for= further information on confidentiality and the risks of non-secure electro= nic communication. If you cannot access these links, please notify us by re= ply message and we will send the contents to you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111101015746.GA96508>