Date: Sun, 10 Nov 2002 02:37:33 -0800 From: Luigi Rizzo <rizzo@icir.org> To: Julian Elischer <julian@FreeBSD.ORG> Cc: freebsd-net@FreeBSD.ORG, toasty@dragondata.com Subject: Re: Packet forwarding overhead - with ipfw counting Message-ID: <20021110023733.A86928@xorpc.icir.org> In-Reply-To: <20021110094150.E655837B401@hub.freebsd.org>; from julian@FreeBSD.ORG on Sun, Nov 10, 2002 at 01:41:50AM -0800 References: <5.1.1.5.2.20021109202725.00b61a10@127.0.0.1> <20021110094150.E655837B401@hub.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
another way to do the count efficiently is to use dummynet dynamic pipes: ipfw add 100 pipe 1 ip from 10.0.0.0/24 to any ipfw add 100 pipe 2 ip from any to 10.0.0.0/24 ipfw pipe 1 config mask src-ip 0xffffffff ipfw pipe 2 config mask dst-ip 0xffffffff sysctl net.inet.ip.dummynet.expire=0 and then do ipfw pipe show to display the stats (and read the ipfw manpage for details). cheers luigi On Sun, Nov 10, 2002 at 01:41:50AM -0800, Julian Elischer wrote: > > > I have a server acting as a router. Dual bge gigabit network interfaces > > (PCI-X), one is the WAN side the other is the LAN side. > > > When we're pushing 250-300mbits through, we're using about 15% of its > > 2.4Ghz P4 Xeon CPU. All of it is in "interrupt" time... that seems a bit > > high, but that'll still let us max things out at 1gbit so we're ok. > > > However, we wanted to do some MRTG style traffic charts per ip. I added > > about 30 sets of ipfw rules like this: > > > count ip from 10.0.0.160 to any > > count ip from any to 10.0.0.160 > > > Having these in place more than tripled the CPU usage. Am I just hitting a > > non-optimized codepath in ipfw, or is this normal for these kind of rules? > > > make sure you minimise the rules each packet passes through.. > for a start, start with a rule like: > > # send packets through the externa linteface for counting > 100 skipto 1000 ip from any to any in recv fxp0 > 110 skipto 2000 ip from any to any out xmit fxp0 > # let packets going through the other (internal) interfaces pass. > 120 accept ip from any to any > > Ok, so now only count incoming packets at rule 1000 > and outgoing packets at 2000 You have imediatly halved the > number of rules each packet traverses. > > Next, try use a binary tree structure of some sort (using skipto) > on each set of these rules to ensure each packet sees the minimum > number of rules. > > for example assuming you are couning 30 addresses 10.0.0.1 through > 10.0.0.31 > > > > you can do as follows for counting the incoming packets. > Each packet traverses on average 5 rules. > > # don't count packets outside the range we are interested in. > 1000 skipto 1900 ip from any to not 10.0.0.0/27 > # make a binary tree down to 4 addresses > 1010 skipto 1500 ip from any to 10.0.0.16/28 > # 0..15 > 1020 skipto 1300 ip from any to 10.0.0.8/29 > # 0..7 > 1100 skipto 1204 ip from any to 10.0.0.4/30 > # 0..3 > 1200 skipto 3000 count ip from any to 10.0.0.0 > 1201 skipto 3000 count ip from any to 10.0.0.1 > 1202 skipto 3000 count ip from any to 10.0.0.2 > 1203 skipto 3000 count ip from any to 10.0.0.3 > # 4..7 > 1204 skipto 3000 count ip from any to 10.0.0.4 > 1205 skipto 3000 count ip from any to 10.0.0.5 > 1206 skipto 3000 count ip from any to 10.0.0.6 > 1207 skipto 3000 count ip from any to 10.0.0.7 > # 8..16 > 1300 skipto 1312 ip from any to 10.0.0.12/30 > # 8..15 > 1308 skipto 3000 count ip from any to 10.0.0.8 > 1309 skipto 3000 count ip from any to 10.0.0.9 > 1310 skipto 3000 count ip from any to 10.0.0.10 > 1311 skipto 3000 count ip from any to 10.0.0.11 > # 12..15 > 1312 skipto 3000 count ip from any to 10.0.0.12 > 1313 skipto 3000 count ip from any to 10.0.0.13 > 1314 skipto 3000 count ip from any to 10.0.0.14 > 1315 skipto 3000 count ip from any to 10.0.0.15 > # 16..31 > 1500 skipto 1600 ip from any to 10.0.0.24/29 > # 16..23 > 1500 skipto 1520 ip from any to 10.0.0.20/30 > # 16..19 > 1516 skipto 3000 count ip from any to 10.0.0.16 > 1517 skipto 3000 count ip from any to 10.0.0.17 > 1518 skipto 3000 count ip from any to 10.0.0.18 > 1519 skipto 3000 count ip from any to 10.0.0.19 > # 20..23 > 1520 skipto 3000 count ip from any to 10.0.0.20 > 1521 skipto 3000 count ip from any to 10.0.0.21 > 1522 skipto 3000 count ip from any to 10.0.0.22 > 1523 skipto 3000 count ip from any to 10.0.0.23 > # 24..31 > 1600 skipto 1628 ip from any to 10.0.0.28/30 > # 24..27 > 1624 skipto 3000 count ip from any to 10.0.0.24 > 1625 skipto 3000 count ip from any to 10.0.0.25 > 1626 skipto 3000 count ip from any to 10.0.0.26 > 1627 skipto 3000 count ip from any to 10.0.0.27 > # 28..31 > 1628 skipto 3000 count ip from any to 10.0.0.28 > 1629 skipto 3000 count ip from any to 10.0.0.29 > 1630 skipto 3000 count ip from any to 10.0.0.30 > 1631 skipto 3000 count ip from any to 10.0.0.31 > > > Obviously a similar rule set can be created for outgoing packets. > A shell script could be written to write this ruleset.. > [note I have not tested it but I have done similar in the past.] > > > This reduces the number of rules tested per packet from 64 > to 6 > > julian > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-net" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021110023733.A86928>