From owner-freebsd-net@FreeBSD.ORG Wed May 3 00:52:23 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 92F3A16A403 for ; Wed, 3 May 2006 00:52:23 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8BD6943D4C for ; Wed, 3 May 2006 00:52:19 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (3jypbmpi9qs7981d@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.4/8.13.3) with ESMTP id k430qCXo015255; Tue, 2 May 2006 17:52:12 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.4/8.13.3/Submit) id k430q7b8015254; Tue, 2 May 2006 17:52:07 -0700 (PDT) (envelope-from jmg) Date: Tue, 2 May 2006 17:52:07 -0700 From: John-Mark Gurney To: Julian Elischer Message-ID: <20060503005207.GT728@funkthat.com> Mail-Followup-To: Julian Elischer , tpeixoto@widesoft.com.br, Lee Johnston , freebsd-net@freebsd.org, mihai@duras.ro References: <4456AD8E.2060703@widesoft.com.br> <4456B415.3080901@elischer.org> <4456BF4A.7050107@widesoft.com.br> <4456D19F.7030101@elischer.org> <4456D553.30202@elischer.org> <4456D6A3.8080503@elischer.org> <59701.200.230.201.250.1146589752.squirrel@www.widemail.com.br> <44579F89.6020703@elischer.org> <4457F905.4050503@widesoft.com.br> <4457FB74.1010202@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4457FB74.1010202@elischer.org> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: Lee Johnston , freebsd-net@freebsd.org, mihai@duras.ro Subject: Re: Packet loss with traffic shaper and routing X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 May 2006 00:52:23 -0000 Julian Elischer wrote this message on Tue, May 02, 2006 at 17:38 -0700: > tpeixoto@widesoft.com.br wrote: > > ok gotcha > > > probably the pipes are stored in a list or something. > > check the code and see if a hash table woudl be better.. Looks like it's already a hash, though a bit small: #define HASHSIZE 16 #define HASH(num) ((((num) >> 8) ^ ((num) >> 4) ^ (num)) & 0x0f) static __inline struct dn_pipe * locate_pipe(int pipe_nr) { struct dn_pipe *pipe; SLIST_FOREACH(pipe, &pipehash[HASH(pipe_nr)], next) if (pipe->pipe_nr == pipe_nr) return (pipe); return (NULL); } Look at increasing HASHSIZE and changing the hash function... > >Julian Elischer wrote: > > > >>tpeixoto@widesoft.com.br wrote: > >> > >>>Hello. > >>>I think I should give some 'real world' examples. > >>> > >>> > >>>/etc/rc.firewall: > >>> > >>>[Ss][Hh][Aa][Pp][Ee][Rr]) > >>>setup_loopback > >>> > >>>. /etc/rc.shaper > >>> > >>>${fwcmd} add 65000 pass all from any to any > >>>;; > >>> > >>> > >>>/etc/rc.shaper: > >>> > >>>${fwcmd} pipe 1 config bw 512Kbit/s > >>>${fwcmd} pipe 2 config bw 512Kbit/s > >>>${fwcmd} add pipe 1 all from any to any MAC any 00:11:22:33:44:55 in > >>>${fwcmd} add pipe 2 all from any to any MAC 00:11:22:33:44:55 any out > >>>${fwcmd} pipe 3 config bw 256Kbit/s > >>>${fwcmd} pipe 4 config bw 256Kbit/s > >>>${fwcmd} add pipe 3 all from any to any MAC any 66:77:88:99:aa:bb in > >>>${fwcmd} add pipe 4 all from any to any MAC 66:77:88:99:aa:bb any out > >>>${fwcmd} pipe 5 config bw 128Kbit/s > >>>${fwcmd} pipe 6 config bw 128Kbit/s > >>>${fwcmd} add pipe 5 all from any to any MAC any 00:01:02:03:04:05 in > >>>${fwcmd} add pipe 6 all from any to any MAC 00:01:02:03:04:05 any out > >>>${fwcmd} pipe 7 config bw 512Kbit/s > >>>${fwcmd} pipe 8 config bw 1024Kbit/s > >>>${fwcmd} add pipe 7 all from any to any MAC any 06:07:08:09:0a:0b in > >>>${fwcmd} add pipe 8 all from any to any MAC 06:07:08:09:0a:0b any out > >>>${fwcmd} pipe 9 config bw 64Kbit/s > >>>${fwcmd} pipe 10 config bw 64Kbit/s > >>>${fwcmd} add pipe 9 all from any to any MAC any ab:cd:ef:00:11:22 in > >>>${fwcmd} add pipe 10 all from any to any MAC ab:cd:ef:00:11:22 any out > >>> > >>> > >>OK, so, put the MACs in numerical order: > >> > >>00:01:02:03:04:05 > >>00:11:22:33:44:55 > >>06:07:08:09:0a:0b > >>66:77:88:99:aa:bb > >>ab:cd:ef:00:11:22 > >> > >> > >>work out MASKS that divide them into a binary set. > >> > >>e.g. > >>1 skipto 10 all from any to not MAC 00:00:00:00:00:00/8 > >>2 skipto 5 all from any to not MAC 00:01:00:00:00:00/16 > >>3 pipe 1 ip from any to any > >>5 pipe 2 ip from any to any > >> > >>10 skipto 12 all from any to not MAC 06:00:00:00:00:00/8 > >>11 pipe 3 all from any to any > >>12 skipto 14 all from any to not MAC 66:00:00:00:00:00/8 > >>13 pipe 4 all from any to any > >>14 pipe 5 all from any to any > >> > >>now, if you continue this on, you will run 16 rules to divide the > >>1600 rules up to find the right pipe. > >> > > > >I got your point. > >But what I am telling is that it's not the search or it's not _only_ > >the search in the firewall rules that is making the interrupts go high. > >Please, see below. > > > > > >>> > >>>This example is for 5 clients. We have 1600. > >>>As you can see, there are 2 rules and 2 pipes per host, not 1600. > >>> > >>> > >>>If we try rc.firewall like this... > >>> > >>>setup_loopback > >>>${fwcmd} add 65000 pass all from any to any > >>> > >>>... we are ok. Interrupts are low. > >>> > >>>So, following your line of thought, I tried a simple test... > >>> > >>>setup_loopback > >>>${fwcmd} skipto 65000 ip from any to any MAC any any > >>>. /etc/rc.shaper > >>>${fwcmd} add 65000 pass all from any to any > >>> > >>>This way, the packets will never pass through shaper rules, but > >>>interrupts > >>>still get very high. > >>> > >>> > >> > >>I don't see how that proves anything > >> > > > >See, if we have just 4 rules in the kernel (3 from setup_loopback + > >allow any to any), we don't have problems with interrupts. They are > >low, about 15~20% with the same traffic. > >But, if we have a 'full' set of rules, let's say 3205 (3 from > >setup_loopback + skipto 65000 + 3200 pipes + allow any to any), where > >only 5 of them are being matched (setup_loopback, 'skipto 65000' and > >'allow any to any' - the skipto 65000 rule prevents any packet to > >search through my 3200 pipes, right?), we still see interrupts go to > >70~90%. > >So, what I am saying is that even if we use skipto rules to create > >'shortcuts' in the firewall stack, the system still uses lots of > >interrupts. It seems that no matter whether the packets are being > >checked against the rules or not, as long there are so many rules, the > >interrupts will be generated. > > > >Let me know if you got my point. > >I'll do some more tests reducing the number of pipes while keeping the > >same amount of rules to see whether this has some effect in the > >interrupts. > > > >BTW: I tested your other suggestion about splitting 'in' and 'out' > >rules but it made no difference regarding system interrupts. > > > >Thanks again! > > > > > >>>Basically, we need a solution to shape each MAC address with its > >>>specifics > >>>download e upload speeds. > >>>Given the tests, I don't see how skipto can help, but if you believe > >>>that > >>>tablearg (which I am not familiar with) might help, we can try it with > >>>7.x. > >>> > >>> > >> > >>Tablearg only works with IP addresses. > >> > >>>Thanks. > >>> > >>> > >>> > >>> > >>>>oops, forgot to fix my cut-n- pastes.. corrected triage below.. > >>>> > >>>> > >>>>Julian Elischer wrote: > >>>> > >>>> > >>>> > >>>>>Julian Elischer wrote: > >>>>> > >>>>> > >>>>> > >>>>>>tpeixoto@widesoft.com.br wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>>>>That would do it.. > >>>>>>>> > >>>>>>>>In all versions of FreeBSD > >>>>>>>>you can use the skipto rule to make sure that only a few rules are > >>>>>>>>run for any > >>>>>>>>address. Use it to to a binary search for the right pipe.' > >>>>>>>>carefully using 'skipto' and 'table' can make it efficient to do > >>>>>>>>very complex > >>>>>>>>filters like that. > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>>Sorry, but I didn't realized how to use that as we have to shape > >>>>>>>each user individually, i.e., each MAC address on the LAN has its > >>>>>>>own download and upload speeds. > >>>>>>> > >>>>>>>Could you clarify how to improve the situation with the tools you > >>>>>>>mentioned? > >>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>Assuming you can not use "tablearg" yet (it will make this REALLY > >>>>>>EASY) > >>>>>>then if you have 30 IPs you want to shape from 1.1.1.1 to 1.1.1.30 > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>>then, consider the following example using IP addresses. > >>>>> > >>>>> > >>>>> > >>>>>> > >>>>>> > >>>>> > >>>>>ipfw add 1000 skipto 1110 ip from any to 1.1.1.16/28 > >>>>>ipfw add 1010 skipto 1032 ip from any to 1.1.1.8/29 > >>>>>ipfw add 1012 skipto 1021 ip from any to 1.1.1.4./30 > >>>>> ipfw add 1013 [anything] ip from any to 1.1.1.0 > >>>>> ipfw add 1014 [anything] ip from any to 1.1.1.1 > >>>>>ipfw add 1015 [anything] ip from any to 1.1.1.2 > >>>>>ipfw add 1016 [anything] ip from any to 1.1.1.3 > >>>>> > >>>>> > >>>>>ipfw add 1021 anything] ip from any to 1.1.1.4 > >>>>>ipfw add 1022 [anything] ip from any to 1.1.1.5 > >>>>>ipfw add 1023 [anything] ip from any to 1.1.1.6 > >>>>>ipfw add 1024 [anything] ip from any to 1.1.1.7 > >>>>> > >>>>> > >>>>>ipfw add 1032 skipto 1051 ip from any to 1.1.1.12./30 > >>>>> > >>>>>ipfw add 1040 [anything] ip from any to 1.1.1.8 > >>>>>ipfw add 1041 [anything] ip from any to 1.1.1.9 > >>>>>ipfw add 1042 [anything] ip from any to 1.1.1.10 > >>>>>ipfw add 1043 [anything] ip from any to 1.1.1.11 > >>>>> > >>>>> > >>>>>ipfw add 1051 [anything] ip from any to 1.1.1.12 > >>>>>ipfw add 1052 [anything] ip from any to 1.1.1.13 > >>>>>ipfw add 1053 [anything] ip from any to 1.1.1.14 > >>>>>ipfw add 1054 [anything] ip from any to 1.1.1.15 > >>>>> > >>>>> > >>>>>ipfw add 1110 skipto 1132 ip from any to 1.1.1.24/29 > >>>>>ipfw add 1112 skipto 1121 ip from any to 1.1.1.20./30 > >>>>>ipfw add 1113 [anything] ip from any to 1.1.1.16 > >>>>>ipfw add 1114 [anything] ip from any to 1.1.1.17 > >>>>>ipfw add 1115 [anything] ip from any to 1.1.1.18 > >>>>> ipfw add 1116 [anything] ip from any to 1.1.1.19 > >>>>> ipfw add 1121 anything] ip from any to 1.1.1.20 > >>>>>ipfw add 1122 [anything] ip from any to 1.1.1.21 > >>>>>ipfw add 1123 [anything] ip from any to 1.1.1.22 > >>>>>ipfw add 1124 [anything] ip from any to 1.1.1.23 > >>>>> > >>>>> > >>>>>ipfw add 1132 skipto 1151 ip from any to 1.1.1.28./30 > >>>>> > >>>>>ipfw add 1140 [anything] ip from any to 1.1.1.24 > >>>>>ipfw add 1141 [anything] ip from any to 1.1.1.25 > >>>>>ipfw add 1142 [anything] ip from any to 1.1.1.26 > >>>>>ipfw add 1143 [anything] ip from any to 1.1.1.27 > >>>>> > >>>>> > >>>>>ipfw add 1151 [anything] ip from any to 1.1.1.28 > >>>>>ipfw add 1152 [anything] ip from any to 1.1.1.29 > >>>>>ipfw add 1153 [anything] ip from any to 1.1.1.30 > >>>>>ipfw add 1154 [anything] ip from any to 1.1.1.31 > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>now this example shows a binary search in IP space, written > >>>>>(including > >>>>>bugs) by hand > >>>>>but if you are willing to write a suitable perl script, you can > >>>>>generate a binary search in MAC address space > >>>>>just as easily. just sort them into order and search.. > >>>>> > >>>>>I'm not going to try it by had, but for 1600 hosts you should only > >>>>>need to go through > >>>>>15 rules per host on average, instead of 1600 rules per host. > >>>>>that should cut down your ipfw cpu usage by 1/100 > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>freebsd.org" > >>>>>> > >>>>> > >>> > >>> > >>> > >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."