Date: Tue, 01 Jul 2008 20:09:01 -0400 From: Paul <paul@gtcomm.net> To: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] Message-ID: <486AC71D.2080804@gtcomm.net> In-Reply-To: <486AA299.7090904@elischer.org> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010257570.19444@filebunker.xip.at> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010308320.19444@filebunker.xip.at> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <ea7b9c170806302005n2a66f592h2127f87a0ba2c6d2@mail.gmail.com> <20080701033117.GH83626@cdnetworks.co.kr> <ea7b9c170806302050p2a3a5480t29923a4ac2d7c852@mail.gmail.com> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486A9BBF.1060308@gtcomm.net> <486AA299.7090904@elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Apparently lagg hasn't been giant fixed :/ Can we do something about
this quickly?
with adaptive giant i get more performance on lagg but the cpu usage is
smashed 100%
I get about 50k more pps per interface (so 150kpps total which STILL is
less than a single gigabit port)
Check it out
68 processes: 9 running, 41 sleeping, 18 waiting
CPU: 0.0% user, 0.0% nice, 89.5% system, 0.0% interrupt, 10.5% idle
Mem: 8016K Active, 6192K Inact, 47M Wired, 108K Cache, 9056K Buf, 1919M Free
Swap: 8192M Total, 8192M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
38 root -68 - 0K 16K CPU1 1 3:29 100.00% em2 taskq
37 root -68 - 0K 16K CPU0 0 3:31 98.78% em1 taskq
36 root -68 - 0K 16K CPU3 3 2:53 82.42% em0 taskq
11 root 171 ki31 0K 16K RUN 2 22:48 79.00% idle: cpu2
10 root 171 ki31 0K 16K RUN 3 20:51 22.90% idle: cpu3
39 root -68 - 0K 16K RUN 2 0:32 16.60% em3 taskq
12 root 171 ki31 0K 16K RUN 1 20:16 2.05% idle: cpu1
13 root 171 ki31 0K 16K RUN 0 20:25 1.90% idle: cpu0
input (em0) output
packets errs bytes packets errs bytes colls
122588 0 7355280 0 0 0 0
123057 0 7383420 0 0 0 0
input (em1) output
packets errs bytes packets errs bytes colls
174917 11899 10495032 2 0 178 0
173967 11697 10438038 2 0 356 0
174630 10603 10477806 2 0 268 0
input (em2) output
packets errs bytes packets errs bytes colls
175843 3928 10550580 0 0 0 0
175952 5750 10557120 0 0 0 0
Still less performance than single gig-e.. that giant lock really sucks
, and why on earth would LAGG require that.. It seems so simple to fix :/
Anyone up for it:) I wish I was a programmer sometimes, but network
engineering will have to do. :D
Julian Elischer wrote:
> Paul wrote:
>> Is PF better than ipfw? iptables almost has no impact on routing
>> performance unless I add a swath of rules to it and then it bombs
>> I need maybe 10 rules max and I don't want 20% performance drop for
>> that.. :P
>
> well lots of people have wanted to fix it, and I've investigated
> quite a lot but it takes someone with 2 weeks of free time and
> all the right clue. It's not inherrent in ipfw but it needs some
> TLC from someone who cares :-).
>
>
>
>> Ouch! :) Is this going to be fixed any time soon? We have some
>> money that can be used for development costs to fix things like this
>> because
>> we use linux and freebsd machines as firewalls for a lot of customers
>> and with the increasing bandwidth and pps the customers are demanding
>> more and I
>> can't give them better performance with a brand new dual xeon or
>> opteron machine vs the old p4 machines I have them running on now :/
>> The only difference
>> in the new machine vs old machine is that the new one can take in
>> more pps and drop it but it can't route a whole lot more.
>> Routing/firewalling must still not be lock free, ugh.. :P
>>
>> Thanks
>>
>>
>>
>> Julian Elischer wrote:
>>> Paul wrote:
>>>> ULE without PREEMPTION is now yeilding better results.
>>>> input (em0) output
>>>> packets errs bytes packets errs bytes colls
>>>> 571595 40639 34564108 1 0 226 0
>>>> 577892 48865 34941908 1 0 178 0
>>>> 545240 84744 32966404 1 0 178 0
>>>> 587661 44691 35534512 1 0 178 0
>>>> 587839 38073 35544904 1 0 178 0
>>>> 587787 43556 35540360 1 0 178 0
>>>> 540786 39492 32712746 1 0 178 0
>>>> 572071 55797 34595650 1 0 178 0
>>>>
>>>> *OUCH, IPFW HURTS..
>>>> loading ipfw, and adding one ipfw rule allow ip from any to any
>>>> drops 100Kpps off :/ what's up with THAT?
>>>> unloaded ipfw module and back 100kpps more again, that's not right
>>>> with ONE rule.. :/
>>>
>>> ipfw need sto gain a lock on hte firewall before running,
>>> and is quite complex.. I can believe it..
>>>
>>> in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two
>>> interfaces (bridged) but I think it has slowed down since then due
>>> to the SMP locking.
>>>
>>>
>>>>
>>>> em0 taskq is still jumping cpus.. is there any way to lock it to
>>>> one cpu or is this just a function of ULE
>>>>
>>>> running a tar czpvf all.tgz * and seeing if pps changes..
>>>> negligible.. guess scheduler is doing it's job at least..
>>>>
>>>> Hmm. even when it's getting 50-60k errors per second on the
>>>> interface I can still SCP a file through that interface although
>>>> it's not fast.. 3-4MB/s..
>>>>
>>>> You know, I wouldn't care if it added 5ms latency to the packets
>>>> when it was doing 1mpps as long as it didn't drop any.. Why can't
>>>> it do that? Queue them up and do them in bigggg chunks so none are
>>>> dropped........hmm?
>>>>
>>>> 32 bit system is compiling now.. won't do > 400kpps with GENERIC
>>>> kernel, as with 64 bit did 450k with GENERIC, although that could be
>>>> the difference between opteron 270 and opteron 2212..
>>>>
>>>> Paul
>>>>
>>>> _______________________________________________
>>>> freebsd-net@freebsd.org mailing list
>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>>>
>>>
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?486AC71D.2080804>