From owner-freebsd-net Wed Oct 9 18:19: 3 2002 Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C57637B401; Wed, 9 Oct 2002 18:19:00 -0700 (PDT) Received: from bunyip.cc.uq.edu.au (bunyip.cc.uq.edu.au [130.102.2.1]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2C98A43E42; Wed, 9 Oct 2002 18:18:59 -0700 (PDT) (envelope-from csmith@its.uq.edu.au) Received: from [130.102.152.68] (tobermory.its.uq.edu.au [130.102.152.68]) by bunyip.cc.uq.edu.au (8.9.3/8.9.3) with ESMTP id LAA06621; Thu, 10 Oct 2002 11:18:51 +1000 (GMT+1000) User-Agent: Microsoft-Entourage/10.0.0.1309 Date: Thu, 10 Oct 2002 11:18:42 +1000 Subject: Re: High interrupt load on firewalls From: Christopher Smith To: Luigi Rizzo Cc: , Message-ID: In-Reply-To: <20021009170002.A54675@carp.icir.org> Mime-version: 1.0 Content-type: text/plain; charset="US-ASCII" Content-transfer-encoding: 7bit Sender: owner-freebsd-net@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org On 10/10/02 10:00 AM, "Luigi Rizzo" wrote: > On Thu, Oct 10, 2002 at 09:38:40AM +1000, Christopher Smith wrote: > ... >> With the 2.4GHz 2650 we have currently, er, "borrowed" to do some testing >> with, the load is down to 35% or so (highest I've seen it is 40%) and the >> packet loss is less than 1 packet every 5 or 10 minutes. > > quite reasonable then, i wouldn't spend too much time in trying to > improve things, engineer's time is way more expensive than a new > box -- it adds up easily. True. We need to redo the whole ruleset though, simply for manageability reasons. May as well do some performance optimising at the same time. >> The existing firewall ruleset is not particularly optimised at all, and >> rewriting the whole thing is an ongoing project of mine. I might up the >> priority of this, given your comments. I get the impression that what >> appears from top to be a high interrupt load may simply be a high "system" >> load from ipfilter processing the packets. Is this correct ? Is there any >> (reasonably easy) way of checking ? > > you might have both problems -- the network stack operates as a soft > interrupt, whether this is accounted as intr or system activity i > am not sure. I tried a quick and dirty way by just dropping the whole ruleset for ~10 seconds. The interrupt load (in top) was about a third of "normal" for that time period. So, it appears the problem is definitely in the ruleset. >> What tools do you you use to test and measure performance for things like >> this ? > > nothing special, i have a trivial c program which loops around a sendto() > trying to push udp packets out as fast as possible, i disable icmp replies > with > net.inet.udp.blackhole=1 Ok, so any of the network benching products that can spit out a stream of UDP traffic should suffice ? > and then measure the card stats with > > ns -i -w 1 -d > > (ns is picobsd's version of netstat, in /usr/src/release/picobsd/tinyware/ns) > Be warned that some cards update the stats everi 1 or 2 seconds, so you might > have to up the refresh time from 1 second ('-w 1' above) to 2 or more seconds > ( -w 2 ...) Ok. Will normal netstat do ? I tried it on one of our machines and got these results: mr2fw2# netstat -w1 -i -d input (Total) output packets errs bytes packets errs bytes colls drops 39518 0 14415117 39509 0 28791210 0 0 50231 0 18969196 50117 0 37782558 0 0 50494 0 17912692 50212 0 35471676 0 0 53685 0 20592345 53547 0 40987500 0 0 44862 0 17009437 44897 0 33974566 0 0 mr2fw2# netstat -w10 -i -d input (Total) output packets errs bytes packets errs bytes colls drops 346558 0 112637886 346876 0 224797074 0 0 392474 0 132204783 392437 0 263929848 0 0 431339 0 139024315 431086 0 277224060 0 0 428132 0 135837843 427895 0 271022908 0 0 390021 0 131117003 389502 0 261278610 0 0 This only seems to indicate ca. 80kpps, which doesn't seem to agree with the numbers I see in 'systat -ip'. Is there a counter rolling over somewhere ? >> Also, are there any cards/chipsets you would recommend for firewalling high >> speed links like this ? > > no idea, i have only tried the intel pro/1000 which seems to work > reasonably well for what i have done so far (which is very little, > basically speed testing and porting the polling support to it), > but i have not tried any other GigE card. Ok then. Apparently there's a machine coming in soon with a Broadcom 5700-based 3Com card in it. Might try and purloin that for a day or two to do some experimenting on. -- +- Christopher Smith, Systems Administrator ------------------------------+ | Server & Security Group, Information Technology Services | | The University of Queensland, Brisbane, Australia, 4072 | +- Ph +61 7 3365 4046 | email csmith@its.uq.edu.au | Fax +61 7 3365 4065 -+ To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message