Date: Tue, 02 May 2006 21:27:49 -0300 From: tpeixoto@widesoft.com.br To: Julian Elischer <julian@elischer.org> Cc: Lee Johnston <lee@wildcard.net.uk>, freebsd-net@freebsd.org, mihai@duras.ro Subject: Re: Packet loss with traffic shaper and routing Message-ID: <4457F905.4050503@widesoft.com.br> In-Reply-To: <44579F89.6020703@elischer.org> References: <49594.200.230.201.250.1146063341.squirrel@www.widemail.com.br> <444F8E89.2050905@wildcard.net.uk> <56286.200.230.201.250.1146067775.squirrel@www.widemail.com.br> <1146073590.1089.80.camel@sky.mediasat.ro> <59615.200.230.201.250.1146083577.squirrel@www.widemail.com.br> <445038CA.2050008@pacific.net.sg> <4456AD8E.2060703@widesoft.com.br> <4456B415.3080901@elischer.org> <4456BF4A.7050107@widesoft.com.br> <4456D19F.7030101@elischer.org> <4456D553.30202@elischer.org> <4456D6A3.8080503@elischer.org> <59701.200.230.201.250.1146589752.squirrel@www.widemail.com.br> <44579F89.6020703@elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Julian Elischer wrote:
> tpeixoto@widesoft.com.br wrote:
>
>> Hello.
>> I think I should give some 'real world' examples.
>>
>>
>> /etc/rc.firewall:
>>
>> [Ss][Hh][Aa][Pp][Ee][Rr])
>> setup_loopback
>>
>> . /etc/rc.shaper
>>
>> ${fwcmd} add 65000 pass all from any to any
>> ;;
>>
>>
>> /etc/rc.shaper:
>>
>> ${fwcmd} pipe 1 config bw 512Kbit/s
>> ${fwcmd} pipe 2 config bw 512Kbit/s
>> ${fwcmd} add pipe 1 all from any to any MAC any 00:11:22:33:44:55 in
>> ${fwcmd} add pipe 2 all from any to any MAC 00:11:22:33:44:55 any out
>> ${fwcmd} pipe 3 config bw 256Kbit/s
>> ${fwcmd} pipe 4 config bw 256Kbit/s
>> ${fwcmd} add pipe 3 all from any to any MAC any 66:77:88:99:aa:bb in
>> ${fwcmd} add pipe 4 all from any to any MAC 66:77:88:99:aa:bb any out
>> ${fwcmd} pipe 5 config bw 128Kbit/s
>> ${fwcmd} pipe 6 config bw 128Kbit/s
>> ${fwcmd} add pipe 5 all from any to any MAC any 00:01:02:03:04:05 in
>> ${fwcmd} add pipe 6 all from any to any MAC 00:01:02:03:04:05 any out
>> ${fwcmd} pipe 7 config bw 512Kbit/s
>> ${fwcmd} pipe 8 config bw 1024Kbit/s
>> ${fwcmd} add pipe 7 all from any to any MAC any 06:07:08:09:0a:0b in
>> ${fwcmd} add pipe 8 all from any to any MAC 06:07:08:09:0a:0b any out
>> ${fwcmd} pipe 9 config bw 64Kbit/s
>> ${fwcmd} pipe 10 config bw 64Kbit/s
>> ${fwcmd} add pipe 9 all from any to any MAC any ab:cd:ef:00:11:22 in
>> ${fwcmd} add pipe 10 all from any to any MAC ab:cd:ef:00:11:22 any out
>>
>>
> OK, so, put the MACs in numerical order:
>
> 00:01:02:03:04:05
> 00:11:22:33:44:55
> 06:07:08:09:0a:0b
> 66:77:88:99:aa:bb
> ab:cd:ef:00:11:22
>
>
> work out MASKS that divide them into a binary set.
>
> e.g.
> 1 skipto 10 all from any to not MAC 00:00:00:00:00:00/8
> 2 skipto 5 all from any to not MAC 00:01:00:00:00:00/16
> 3 pipe 1 ip from any to any
> 5 pipe 2 ip from any to any
>
> 10 skipto 12 all from any to not MAC 06:00:00:00:00:00/8
> 11 pipe 3 all from any to any
> 12 skipto 14 all from any to not MAC 66:00:00:00:00:00/8
> 13 pipe 4 all from any to any
> 14 pipe 5 all from any to any
>
> now, if you continue this on, you will run 16 rules to divide the 1600
> rules up to find the right pipe.
>
I got your point.
But what I am telling is that it's not the search or it's not _only_ the
search in the firewall rules that is making the interrupts go high.
Please, see below.
>>
>> This example is for 5 clients. We have 1600.
>> As you can see, there are 2 rules and 2 pipes per host, not 1600.
>>
>>
>> If we try rc.firewall like this...
>>
>> setup_loopback
>> ${fwcmd} add 65000 pass all from any to any
>>
>> ... we are ok. Interrupts are low.
>>
>> So, following your line of thought, I tried a simple test...
>>
>> setup_loopback
>> ${fwcmd} skipto 65000 ip from any to any MAC any any
>> . /etc/rc.shaper
>> ${fwcmd} add 65000 pass all from any to any
>>
>> This way, the packets will never pass through shaper rules, but
>> interrupts
>> still get very high.
>>
>>
>
> I don't see how that proves anything
>
See, if we have just 4 rules in the kernel (3 from setup_loopback +
allow any to any), we don't have problems with interrupts. They are low,
about 15~20% with the same traffic.
But, if we have a 'full' set of rules, let's say 3205 (3 from
setup_loopback + skipto 65000 + 3200 pipes + allow any to any), where
only 5 of them are being matched (setup_loopback, 'skipto 65000' and
'allow any to any' - the skipto 65000 rule prevents any packet to search
through my 3200 pipes, right?), we still see interrupts go to 70~90%.
So, what I am saying is that even if we use skipto rules to create
'shortcuts' in the firewall stack, the system still uses lots of
interrupts. It seems that no matter whether the packets are being
checked against the rules or not, as long there are so many rules, the
interrupts will be generated.
Let me know if you got my point.
I'll do some more tests reducing the number of pipes while keeping the
same amount of rules to see whether this has some effect in the interrupts.
BTW: I tested your other suggestion about splitting 'in' and 'out' rules
but it made no difference regarding system interrupts.
Thanks again!
>> Basically, we need a solution to shape each MAC address with its
>> specifics
>> download e upload speeds.
>> Given the tests, I don't see how skipto can help, but if you believe that
>> tablearg (which I am not familiar with) might help, we can try it with
>> 7.x.
>>
>>
>
> Tablearg only works with IP addresses.
>
>> Thanks.
>>
>>
>>
>>
>>> oops, forgot to fix my cut-n- pastes.. corrected triage below..
>>>
>>>
>>> Julian Elischer wrote:
>>>
>>>
>>>> Julian Elischer wrote:
>>>>
>>>>
>>>>> tpeixoto@widesoft.com.br wrote:
>>>>>
>>>>>
>>>>>>> That would do it..
>>>>>>>
>>>>>>> In all versions of FreeBSD
>>>>>>> you can use the skipto rule to make sure that only a few rules are
>>>>>>> run for any
>>>>>>> address. Use it to to a binary search for the right pipe.'
>>>>>>> carefully using 'skipto' and 'table' can make it efficient to do
>>>>>>> very complex
>>>>>>> filters like that.
>>>>>>>
>>>>>>>
>>>>>> Sorry, but I didn't realized how to use that as we have to shape
>>>>>> each user individually, i.e., each MAC address on the LAN has its
>>>>>> own download and upload speeds.
>>>>>>
>>>>>> Could you clarify how to improve the situation with the tools you
>>>>>> mentioned?
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Assuming you can not use "tablearg" yet (it will make this REALLY
>>>>> EASY)
>>>>> then if you have 30 IPs you want to shape from 1.1.1.1 to 1.1.1.30
>>>>>
>>>>
>>>>
>>>> then, consider the following example using IP addresses.
>>>>
>>>>
>>>>>
>>>>>
>>>> ipfw add 1000 skipto 1110 ip from any to 1.1.1.16/28
>>>> ipfw add 1010 skipto 1032 ip from any to 1.1.1.8/29
>>>> ipfw add 1012 skipto 1021 ip from any to 1.1.1.4./30
>>>>
>>>> ipfw add 1013 [anything] ip from any to 1.1.1.0
>>>>
>>>> ipfw add 1014 [anything] ip from any to 1.1.1.1
>>>> ipfw add 1015 [anything] ip from any to 1.1.1.2
>>>> ipfw add 1016 [anything] ip from any to 1.1.1.3
>>>>
>>>>
>>>> ipfw add 1021 anything] ip from any to 1.1.1.4
>>>> ipfw add 1022 [anything] ip from any to 1.1.1.5
>>>> ipfw add 1023 [anything] ip from any to 1.1.1.6
>>>> ipfw add 1024 [anything] ip from any to 1.1.1.7
>>>>
>>>>
>>>> ipfw add 1032 skipto 1051 ip from any to 1.1.1.12./30
>>>>
>>>> ipfw add 1040 [anything] ip from any to 1.1.1.8
>>>> ipfw add 1041 [anything] ip from any to 1.1.1.9
>>>> ipfw add 1042 [anything] ip from any to 1.1.1.10
>>>> ipfw add 1043 [anything] ip from any to 1.1.1.11
>>>>
>>>>
>>>> ipfw add 1051 [anything] ip from any to 1.1.1.12
>>>> ipfw add 1052 [anything] ip from any to 1.1.1.13
>>>> ipfw add 1053 [anything] ip from any to 1.1.1.14
>>>> ipfw add 1054 [anything] ip from any to 1.1.1.15
>>>>
>>>>
>>>> ipfw add 1110 skipto 1132 ip from any to 1.1.1.24/29
>>>> ipfw add 1112 skipto 1121 ip from any to 1.1.1.20./30
>>>> ipfw add 1113 [anything] ip from any to 1.1.1.16
>>>> ipfw add 1114 [anything] ip from any to 1.1.1.17
>>>> ipfw add 1115 [anything] ip from any to 1.1.1.18
>>>>
>>>> ipfw add 1116 [anything] ip from any to 1.1.1.19
>>>>
>>>> ipfw add 1121 anything] ip from any to 1.1.1.20
>>>> ipfw add 1122 [anything] ip from any to 1.1.1.21
>>>> ipfw add 1123 [anything] ip from any to 1.1.1.22
>>>> ipfw add 1124 [anything] ip from any to 1.1.1.23
>>>>
>>>>
>>>> ipfw add 1132 skipto 1151 ip from any to 1.1.1.28./30
>>>>
>>>> ipfw add 1140 [anything] ip from any to 1.1.1.24
>>>> ipfw add 1141 [anything] ip from any to 1.1.1.25
>>>> ipfw add 1142 [anything] ip from any to 1.1.1.26
>>>> ipfw add 1143 [anything] ip from any to 1.1.1.27
>>>>
>>>>
>>>> ipfw add 1151 [anything] ip from any to 1.1.1.28
>>>> ipfw add 1152 [anything] ip from any to 1.1.1.29
>>>> ipfw add 1153 [anything] ip from any to 1.1.1.30
>>>> ipfw add 1154 [anything] ip from any to 1.1.1.31
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> now this example shows a binary search in IP space, written (including
>>>> bugs) by hand
>>>> but if you are willing to write a suitable perl script, you can
>>>> generate a binary search in MAC address space
>>>> just as easily. just sort them into order and search..
>>>>
>>>> I'm not going to try it by had, but for 1600 hosts you should only
>>>> need to go through
>>>> 15 rules per host on average, instead of 1600 rules per host.
>>>> that should cut down your ipfw cpu usage by 1/100
>>>>
>>>>
>>>>
>>>>
>>>>> freebsd.org"
>>>>>
>>
>>
>>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4457F905.4050503>