Date: Mon, 31 Oct 2011 20:57:46 -0500 From: "Paul A. Procacci" <pprocacci@datapipe.com> To: <freebsd-net@freebsd.org> Subject: [High Interrupt Count] Networking Difficulties Message-ID: <20111101015746.GA96508@nat.myhome>
next in thread | raw e-mail | index | archive | help
Gents, I'm having quite an aweful problem that I need a bit of help with. I have an HPDL360 G3 ( http://h18000.www1.hp.com/products/quickspecs/11504_na/11504_na.HTML ) which acts as a NAT (via PF) for several (600+) class C's amongst 24+ machines sitting behind it. It's running FPSense (FreeBSD 8.1-RELEASE-p4). The important guts are: 2 x 2.8 GHz Cpus 2 BGE interfaces on a PCI-X bus. During peak times this machine is only able to handle between 500Mbps - 600Mbps before running out of cpu capacity. (300Mbps(ish) on the LAN, 300Mbps(ish) on the WAN) It's due to the high number of interrupts. I was speaking with a networking engineer here and he mentioned that I should look at "Interrupt Coalescing" to increase throughput. The only information I found online regarding this was a post from 2 years ago here: http://lists.freebsd.org/pipermail/freebsd-net/2009-June/022227.html The tunables mentioned in the above post aren't present in my system, so I imagine this never made it into the bge driver. Assuming this to be the case, I started looking at DEVICE_POLLING as a solution. I did try implementing device polling, but the results were worse than I expected. netisr was using 100% of a single cpu while the other cpu remained mostly idle. Not knowing exactly what netisr is, I reverted the changes. This leads me to this list. Given the scenario above, I'm nearly certain I need to use device polling instead of the standard interrupt driven setup. The two sysctl's that I've come across thus far that I think are what I need are: net.isr.maxthreads hern.hz I would assume setting net.isr.maxthreads to 2 given my dual core machine is advisable, but I'm not 100% sure. What are the caveats in setting this higher? Given the output of `sysctl -d net.isr.maxthreads` I would expect anything higher than the number of cores to be detrimental. Is this correct? kern.hz I'm more unsure of. I understand what the sysctl is, but I'm not sure how to come up with a reasonable number. Generally speaking, and in your experience, would a setting of 2000 achive close to the theoritical meximum of the cards? Is there an upper limit that I would be worried about? Random Question: - is device polling really the answer? I am missing something in the bge driver that I've overlooked? - what tunables directly effect processing high volumes of packets. Network Interfaces: ################################################################################## bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 00:0b:cd:ca:1d:1a inet 209.18.70.211 netmask 0xffffff00 broadcast 209.18.70.255 inet6 fe80::20b:cdff:feca:1d1a%bge0 prefixlen 64 scopeid 0x1 nd6 options=3<PERFORMNUD,ACCEPT_RTADV> media: Ethernet autoselect (1000baseT <full-duplex>) status: active bge1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE> ether 00:0b:cd:ca:1a:74 inet 172.16.0.3 netmask 0xfffc0000 broadcast 172.19.255.255 inet6 fe80::20b:cdff:feca:1a74%bge1 prefixlen 64 scopeid 0x2 nd6 options=3<PERFORMNUD,ACCEPT_RTADV> media: Ethernet autoselect (1000baseT <full-duplex>) status: active ################################################################################## I appreciate the help in advance. Thanks, Paul ________________________________ This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20111101015746.GA96508>