Date: Tue, 02 May 2006 17:38:12 -0700 From: Julian Elischer <julian@elischer.org> To: tpeixoto@widesoft.com.br Cc: Lee Johnston <lee@wildcard.net.uk>, freebsd-net@freebsd.org, mihai@duras.ro Subject: Re: Packet loss with traffic shaper and routing Message-ID: <4457FB74.1010202@elischer.org> In-Reply-To: <4457F905.4050503@widesoft.com.br> References: <49594.200.230.201.250.1146063341.squirrel@www.widemail.com.br> <444F8E89.2050905@wildcard.net.uk> <56286.200.230.201.250.1146067775.squirrel@www.widemail.com.br> <1146073590.1089.80.camel@sky.mediasat.ro> <59615.200.230.201.250.1146083577.squirrel@www.widemail.com.br> <445038CA.2050008@pacific.net.sg> <4456AD8E.2060703@widesoft.com.br> <4456B415.3080901@elischer.org> <4456BF4A.7050107@widesoft.com.br> <4456D19F.7030101@elischer.org> <4456D553.30202@elischer.org> <4456D6A3.8080503@elischer.org> <59701.200.230.201.250.1146589752.squirrel@www.widemail.com.br> <44579F89.6020703@elischer.org> <4457F905.4050503@widesoft.com.br>
next in thread | previous in thread | raw e-mail | index | archive | help
tpeixoto@widesoft.com.br wrote: ok gotcha probably the pipes are stored in a list or something. check the code and see if a hash table woudl be better.. > > Julian Elischer wrote: > >> tpeixoto@widesoft.com.br wrote: >> >>> Hello. >>> I think I should give some 'real world' examples. >>> >>> >>> /etc/rc.firewall: >>> >>> [Ss][Hh][Aa][Pp][Ee][Rr]) >>> setup_loopback >>> >>> . /etc/rc.shaper >>> >>> ${fwcmd} add 65000 pass all from any to any >>> ;; >>> >>> >>> /etc/rc.shaper: >>> >>> ${fwcmd} pipe 1 config bw 512Kbit/s >>> ${fwcmd} pipe 2 config bw 512Kbit/s >>> ${fwcmd} add pipe 1 all from any to any MAC any 00:11:22:33:44:55 in >>> ${fwcmd} add pipe 2 all from any to any MAC 00:11:22:33:44:55 any out >>> ${fwcmd} pipe 3 config bw 256Kbit/s >>> ${fwcmd} pipe 4 config bw 256Kbit/s >>> ${fwcmd} add pipe 3 all from any to any MAC any 66:77:88:99:aa:bb in >>> ${fwcmd} add pipe 4 all from any to any MAC 66:77:88:99:aa:bb any out >>> ${fwcmd} pipe 5 config bw 128Kbit/s >>> ${fwcmd} pipe 6 config bw 128Kbit/s >>> ${fwcmd} add pipe 5 all from any to any MAC any 00:01:02:03:04:05 in >>> ${fwcmd} add pipe 6 all from any to any MAC 00:01:02:03:04:05 any out >>> ${fwcmd} pipe 7 config bw 512Kbit/s >>> ${fwcmd} pipe 8 config bw 1024Kbit/s >>> ${fwcmd} add pipe 7 all from any to any MAC any 06:07:08:09:0a:0b in >>> ${fwcmd} add pipe 8 all from any to any MAC 06:07:08:09:0a:0b any out >>> ${fwcmd} pipe 9 config bw 64Kbit/s >>> ${fwcmd} pipe 10 config bw 64Kbit/s >>> ${fwcmd} add pipe 9 all from any to any MAC any ab:cd:ef:00:11:22 in >>> ${fwcmd} add pipe 10 all from any to any MAC ab:cd:ef:00:11:22 any out >>> >>> >> OK, so, put the MACs in numerical order: >> >> 00:01:02:03:04:05 >> 00:11:22:33:44:55 >> 06:07:08:09:0a:0b >> 66:77:88:99:aa:bb >> ab:cd:ef:00:11:22 >> >> >> work out MASKS that divide them into a binary set. >> >> e.g. >> 1 skipto 10 all from any to not MAC 00:00:00:00:00:00/8 >> 2 skipto 5 all from any to not MAC 00:01:00:00:00:00/16 >> 3 pipe 1 ip from any to any >> 5 pipe 2 ip from any to any >> >> 10 skipto 12 all from any to not MAC 06:00:00:00:00:00/8 >> 11 pipe 3 all from any to any >> 12 skipto 14 all from any to not MAC 66:00:00:00:00:00/8 >> 13 pipe 4 all from any to any >> 14 pipe 5 all from any to any >> >> now, if you continue this on, you will run 16 rules to divide the >> 1600 rules up to find the right pipe. >> > > I got your point. > But what I am telling is that it's not the search or it's not _only_ > the search in the firewall rules that is making the interrupts go high. > Please, see below. > > >>> >>> This example is for 5 clients. We have 1600. >>> As you can see, there are 2 rules and 2 pipes per host, not 1600. >>> >>> >>> If we try rc.firewall like this... >>> >>> setup_loopback >>> ${fwcmd} add 65000 pass all from any to any >>> >>> ... we are ok. Interrupts are low. >>> >>> So, following your line of thought, I tried a simple test... >>> >>> setup_loopback >>> ${fwcmd} skipto 65000 ip from any to any MAC any any >>> . /etc/rc.shaper >>> ${fwcmd} add 65000 pass all from any to any >>> >>> This way, the packets will never pass through shaper rules, but >>> interrupts >>> still get very high. >>> >>> >> >> I don't see how that proves anything >> > > See, if we have just 4 rules in the kernel (3 from setup_loopback + > allow any to any), we don't have problems with interrupts. They are > low, about 15~20% with the same traffic. > But, if we have a 'full' set of rules, let's say 3205 (3 from > setup_loopback + skipto 65000 + 3200 pipes + allow any to any), where > only 5 of them are being matched (setup_loopback, 'skipto 65000' and > 'allow any to any' - the skipto 65000 rule prevents any packet to > search through my 3200 pipes, right?), we still see interrupts go to > 70~90%. > So, what I am saying is that even if we use skipto rules to create > 'shortcuts' in the firewall stack, the system still uses lots of > interrupts. It seems that no matter whether the packets are being > checked against the rules or not, as long there are so many rules, the > interrupts will be generated. > > Let me know if you got my point. > I'll do some more tests reducing the number of pipes while keeping the > same amount of rules to see whether this has some effect in the > interrupts. > > BTW: I tested your other suggestion about splitting 'in' and 'out' > rules but it made no difference regarding system interrupts. > > Thanks again! > > >>> Basically, we need a solution to shape each MAC address with its >>> specifics >>> download e upload speeds. >>> Given the tests, I don't see how skipto can help, but if you believe >>> that >>> tablearg (which I am not familiar with) might help, we can try it with >>> 7.x. >>> >>> >> >> Tablearg only works with IP addresses. >> >>> Thanks. >>> >>> >>> >>> >>>> oops, forgot to fix my cut-n- pastes.. corrected triage below.. >>>> >>>> >>>> Julian Elischer wrote: >>>> >>>> >>>> >>>>> Julian Elischer wrote: >>>>> >>>>> >>>>> >>>>>> tpeixoto@widesoft.com.br wrote: >>>>>> >>>>>> >>>>>> >>>>>>>> That would do it.. >>>>>>>> >>>>>>>> In all versions of FreeBSD >>>>>>>> you can use the skipto rule to make sure that only a few rules are >>>>>>>> run for any >>>>>>>> address. Use it to to a binary search for the right pipe.' >>>>>>>> carefully using 'skipto' and 'table' can make it efficient to do >>>>>>>> very complex >>>>>>>> filters like that. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> Sorry, but I didn't realized how to use that as we have to shape >>>>>>> each user individually, i.e., each MAC address on the LAN has its >>>>>>> own download and upload speeds. >>>>>>> >>>>>>> Could you clarify how to improve the situation with the tools you >>>>>>> mentioned? >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Assuming you can not use "tablearg" yet (it will make this REALLY >>>>>> EASY) >>>>>> then if you have 30 IPs you want to shape from 1.1.1.1 to 1.1.1.30 >>>>>> >>>>> >>>>> >>>>> >>>>> then, consider the following example using IP addresses. >>>>> >>>>> >>>>> >>>>>> >>>>>> >>>>> >>>>> ipfw add 1000 skipto 1110 ip from any to 1.1.1.16/28 >>>>> ipfw add 1010 skipto 1032 ip from any to 1.1.1.8/29 >>>>> ipfw add 1012 skipto 1021 ip from any to 1.1.1.4./30 >>>>> ipfw add 1013 [anything] ip from any to 1.1.1.0 >>>>> ipfw add 1014 [anything] ip from any to 1.1.1.1 >>>>> ipfw add 1015 [anything] ip from any to 1.1.1.2 >>>>> ipfw add 1016 [anything] ip from any to 1.1.1.3 >>>>> >>>>> >>>>> ipfw add 1021 anything] ip from any to 1.1.1.4 >>>>> ipfw add 1022 [anything] ip from any to 1.1.1.5 >>>>> ipfw add 1023 [anything] ip from any to 1.1.1.6 >>>>> ipfw add 1024 [anything] ip from any to 1.1.1.7 >>>>> >>>>> >>>>> ipfw add 1032 skipto 1051 ip from any to 1.1.1.12./30 >>>>> >>>>> ipfw add 1040 [anything] ip from any to 1.1.1.8 >>>>> ipfw add 1041 [anything] ip from any to 1.1.1.9 >>>>> ipfw add 1042 [anything] ip from any to 1.1.1.10 >>>>> ipfw add 1043 [anything] ip from any to 1.1.1.11 >>>>> >>>>> >>>>> ipfw add 1051 [anything] ip from any to 1.1.1.12 >>>>> ipfw add 1052 [anything] ip from any to 1.1.1.13 >>>>> ipfw add 1053 [anything] ip from any to 1.1.1.14 >>>>> ipfw add 1054 [anything] ip from any to 1.1.1.15 >>>>> >>>>> >>>>> ipfw add 1110 skipto 1132 ip from any to 1.1.1.24/29 >>>>> ipfw add 1112 skipto 1121 ip from any to 1.1.1.20./30 >>>>> ipfw add 1113 [anything] ip from any to 1.1.1.16 >>>>> ipfw add 1114 [anything] ip from any to 1.1.1.17 >>>>> ipfw add 1115 [anything] ip from any to 1.1.1.18 >>>>> ipfw add 1116 [anything] ip from any to 1.1.1.19 >>>>> ipfw add 1121 anything] ip from any to 1.1.1.20 >>>>> ipfw add 1122 [anything] ip from any to 1.1.1.21 >>>>> ipfw add 1123 [anything] ip from any to 1.1.1.22 >>>>> ipfw add 1124 [anything] ip from any to 1.1.1.23 >>>>> >>>>> >>>>> ipfw add 1132 skipto 1151 ip from any to 1.1.1.28./30 >>>>> >>>>> ipfw add 1140 [anything] ip from any to 1.1.1.24 >>>>> ipfw add 1141 [anything] ip from any to 1.1.1.25 >>>>> ipfw add 1142 [anything] ip from any to 1.1.1.26 >>>>> ipfw add 1143 [anything] ip from any to 1.1.1.27 >>>>> >>>>> >>>>> ipfw add 1151 [anything] ip from any to 1.1.1.28 >>>>> ipfw add 1152 [anything] ip from any to 1.1.1.29 >>>>> ipfw add 1153 [anything] ip from any to 1.1.1.30 >>>>> ipfw add 1154 [anything] ip from any to 1.1.1.31 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> now this example shows a binary search in IP space, written >>>>> (including >>>>> bugs) by hand >>>>> but if you are willing to write a suitable perl script, you can >>>>> generate a binary search in MAC address space >>>>> just as easily. just sort them into order and search.. >>>>> >>>>> I'm not going to try it by had, but for 1600 hosts you should only >>>>> need to go through >>>>> 15 rules per host on average, instead of 1600 rules per host. >>>>> that should cut down your ipfw cpu usage by 1/100 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> freebsd.org" >>>>>> >>>>> >>> >>> >>> >>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4457FB74.1010202>