Date: Sun, 10 Nov 2002 01:41:50 -0800 (PST) From: julian@FreeBSD.ORG (Julian Elischer) To: freebsd-net@freebsd.org, toasty@dragondata.com Subject: Re: Packet forwarding overhead - with ipfw counting Message-ID: <20021110094150.E655837B401@hub.freebsd.org> In-Reply-To: <5.1.1.5.2.20021109202725.00b61a10@127.0.0.1>
next in thread | previous in thread | raw e-mail | index | archive | help
> I have a server acting as a router. Dual bge gigabit network interfaces > (PCI-X), one is the WAN side the other is the LAN side. > When we're pushing 250-300mbits through, we're using about 15% of its > 2.4Ghz P4 Xeon CPU. All of it is in "interrupt" time... that seems a bit > high, but that'll still let us max things out at 1gbit so we're ok. > However, we wanted to do some MRTG style traffic charts per ip. I added > about 30 sets of ipfw rules like this: > count ip from 10.0.0.160 to any > count ip from any to 10.0.0.160 > Having these in place more than tripled the CPU usage. Am I just hitting a > non-optimized codepath in ipfw, or is this normal for these kind of rules? make sure you minimise the rules each packet passes through.. for a start, start with a rule like: # send packets through the externa linteface for counting 100 skipto 1000 ip from any to any in recv fxp0 110 skipto 2000 ip from any to any out xmit fxp0 # let packets going through the other (internal) interfaces pass. 120 accept ip from any to any Ok, so now only count incoming packets at rule 1000 and outgoing packets at 2000 You have imediatly halved the number of rules each packet traverses. Next, try use a binary tree structure of some sort (using skipto) on each set of these rules to ensure each packet sees the minimum number of rules. for example assuming you are couning 30 addresses 10.0.0.1 through 10.0.0.31 you can do as follows for counting the incoming packets. Each packet traverses on average 5 rules. # don't count packets outside the range we are interested in. 1000 skipto 1900 ip from any to not 10.0.0.0/27 # make a binary tree down to 4 addresses 1010 skipto 1500 ip from any to 10.0.0.16/28 # 0..15 1020 skipto 1300 ip from any to 10.0.0.8/29 # 0..7 1100 skipto 1204 ip from any to 10.0.0.4/30 # 0..3 1200 skipto 3000 count ip from any to 10.0.0.0 1201 skipto 3000 count ip from any to 10.0.0.1 1202 skipto 3000 count ip from any to 10.0.0.2 1203 skipto 3000 count ip from any to 10.0.0.3 # 4..7 1204 skipto 3000 count ip from any to 10.0.0.4 1205 skipto 3000 count ip from any to 10.0.0.5 1206 skipto 3000 count ip from any to 10.0.0.6 1207 skipto 3000 count ip from any to 10.0.0.7 # 8..16 1300 skipto 1312 ip from any to 10.0.0.12/30 # 8..15 1308 skipto 3000 count ip from any to 10.0.0.8 1309 skipto 3000 count ip from any to 10.0.0.9 1310 skipto 3000 count ip from any to 10.0.0.10 1311 skipto 3000 count ip from any to 10.0.0.11 # 12..15 1312 skipto 3000 count ip from any to 10.0.0.12 1313 skipto 3000 count ip from any to 10.0.0.13 1314 skipto 3000 count ip from any to 10.0.0.14 1315 skipto 3000 count ip from any to 10.0.0.15 # 16..31 1500 skipto 1600 ip from any to 10.0.0.24/29 # 16..23 1500 skipto 1520 ip from any to 10.0.0.20/30 # 16..19 1516 skipto 3000 count ip from any to 10.0.0.16 1517 skipto 3000 count ip from any to 10.0.0.17 1518 skipto 3000 count ip from any to 10.0.0.18 1519 skipto 3000 count ip from any to 10.0.0.19 # 20..23 1520 skipto 3000 count ip from any to 10.0.0.20 1521 skipto 3000 count ip from any to 10.0.0.21 1522 skipto 3000 count ip from any to 10.0.0.22 1523 skipto 3000 count ip from any to 10.0.0.23 # 24..31 1600 skipto 1628 ip from any to 10.0.0.28/30 # 24..27 1624 skipto 3000 count ip from any to 10.0.0.24 1625 skipto 3000 count ip from any to 10.0.0.25 1626 skipto 3000 count ip from any to 10.0.0.26 1627 skipto 3000 count ip from any to 10.0.0.27 # 28..31 1628 skipto 3000 count ip from any to 10.0.0.28 1629 skipto 3000 count ip from any to 10.0.0.29 1630 skipto 3000 count ip from any to 10.0.0.30 1631 skipto 3000 count ip from any to 10.0.0.31 Obviously a similar rule set can be created for outgoing packets. A shell script could be written to write this ruleset.. [note I have not tested it but I have done similar in the past.] This reduces the number of rules tested per packet from 64 to 6 julian To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-net" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20021110094150.E655837B401>