Date: Tue, 01 Jul 2008 18:49:31 -0400 From: Paul <paul@gtcomm.net> To: Julian Elischer <julian@elischer.org> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] Message-ID: <486AB47B.2050200@gtcomm.net> In-Reply-To: <486A9A0E.6060308@elischer.org> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010257570.19444@filebunker.xip.at> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010308320.19444@filebunker.xip.at> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <ea7b9c170806302005n2a66f592h2127f87a0ba2c6d2@mail.gmail.com> <20080701033117.GH83626@cdnetworks.co.kr> <ea7b9c170806302050p2a3a5480t29923a4ac2d7c852@mail.gmail.com> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Ok, now THIS is absoultely a whole bunch of ridiculousness..
I set up etherchannel, and I'm evenly distributing packets over em0 em1
and em2 to lagg0
and i get WORSE performance than with a single interface.. Can anyone
explain this one? This is horrible.
I got em0-em2 taskq's using 80% cpu EACH and they are only doing 100kpps
EACH
looks:
packets errs bytes packets errs bytes colls
105050 11066 6303000 0 0 0 0
104952 13969 6297120 0 0 0 0
104331 12121 6259860 0 0 0 0
input (em1) output
packets errs bytes packets errs bytes colls
103734 70658 6223998 0 0 0 0
103483 75703 6209046 0 0 0 0
103848 76195 6230886 0 0 0 0
input (em2) output
packets errs bytes packets errs bytes colls
103299 62957 6197940 1 0 226 0
106388 73071 6383280 1 0 178 0
104503 70573 6270180 4 0 712 0
last pid: 1378; load averages: 2.31, 1.28,
0.57 up
0+00:06:27 17:42:32
68 processes: 8 running, 42 sleeping, 18 waiting
CPU: 0.0% user, 0.0% nice, 58.9% system, 0.0% interrupt, 41.1% idle
Mem: 7980K Active, 5932K Inact, 47M Wired, 16K Cache, 8512K Buf, 1920M Free
Swap: 8192M Total, 8192M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 171 ki31 0K 16K RUN 2 5:18 80.47% idle: cpu2
38 root -68 - 0K 16K CPU3 3 2:30 80.18% em2 taskq
37 root -68 - 0K 16K CPU1 1 2:28 76.90% em1 taskq
36 root -68 - 0K 16K CPU2 2 2:28 72.56% em0 taskq
13 root 171 ki31 0K 16K RUN 0 3:32 29.20% idle: cpu0
12 root 171 ki31 0K 16K RUN 1 3:29 27.88% idle: cpu1
10 root 171 ki31 0K 16K RUN 3 3:21 25.63% idle: cpu3
39 root -68 - 0K 16K - 3 0:32 17.68% em3 taskq
See that's total wrongness.. something is very wrong here. Does anyone
have any ideas? I really need to get this working.
I figured if I evenly distributed the packets over 3 interfaces it
simulates having 3 rx queues because it has a separate process for each
interface
and the result is WAY more CPU usage and a little over half the pps
throughput with a single port ..
If anyone is interested in tackling some these issues please e-mail me.
It would be greatly appreciated.
Paul
Julian Elischer wrote:
> Paul wrote:
>> ULE without PREEMPTION is now yeilding better results.
>> input (em0) output
>> packets errs bytes packets errs bytes colls
>> 571595 40639 34564108 1 0 226 0
>> 577892 48865 34941908 1 0 178 0
>> 545240 84744 32966404 1 0 178 0
>> 587661 44691 35534512 1 0 178 0
>> 587839 38073 35544904 1 0 178 0
>> 587787 43556 35540360 1 0 178 0
>> 540786 39492 32712746 1 0 178 0
>> 572071 55797 34595650 1 0 178 0
>>
>> *OUCH, IPFW HURTS..
>> loading ipfw, and adding one ipfw rule allow ip from any to any drops
>> 100Kpps off :/ what's up with THAT?
>> unloaded ipfw module and back 100kpps more again, that's not right
>> with ONE rule.. :/
>
> ipfw need sto gain a lock on hte firewall before running,
> and is quite complex.. I can believe it..
>
> in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two
> interfaces (bridged) but I think it has slowed down since then due to
> the SMP locking.
>
>
>>
>> em0 taskq is still jumping cpus.. is there any way to lock it to one
>> cpu or is this just a function of ULE
>>
>> running a tar czpvf all.tgz * and seeing if pps changes..
>> negligible.. guess scheduler is doing it's job at least..
>>
>> Hmm. even when it's getting 50-60k errors per second on the interface
>> I can still SCP a file through that interface although it's not
>> fast.. 3-4MB/s..
>>
>> You know, I wouldn't care if it added 5ms latency to the packets when
>> it was doing 1mpps as long as it didn't drop any.. Why can't it do
>> that? Queue them up and do them in bigggg chunks so none are
>> dropped........hmm?
>>
>> 32 bit system is compiling now.. won't do > 400kpps with GENERIC
>> kernel, as with 64 bit did 450k with GENERIC, although that could be
>> the difference between opteron 270 and opteron 2212..
>>
>> Paul
>>
>> _______________________________________________
>> freebsd-net@freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?486AB47B.2050200>