From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 00:06:59 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A4531065678 for ; Wed, 2 Jul 2008 00:06:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 0B7E58FC12 for ; Wed, 2 Jul 2008 00:06:58 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDpok-0004XK-AO for freebsd-net@freebsd.org; Tue, 01 Jul 2008 20:03:26 -0400 Message-ID: <486AC71D.2080804@gtcomm.net> Date: Tue, 01 Jul 2008 20:09:01 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486A9BBF.1060308@gtcomm.net> <486AA299.7090904@elischer.org> In-Reply-To: <486AA299.7090904@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 00:06:59 -0000 Apparently lagg hasn't been giant fixed :/ Can we do something about this quickly? with adaptive giant i get more performance on lagg but the cpu usage is smashed 100% I get about 50k more pps per interface (so 150kpps total which STILL is less than a single gigabit port) Check it out 68 processes: 9 running, 41 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 89.5% system, 0.0% interrupt, 10.5% idle Mem: 8016K Active, 6192K Inact, 47M Wired, 108K Cache, 9056K Buf, 1919M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 38 root -68 - 0K 16K CPU1 1 3:29 100.00% em2 taskq 37 root -68 - 0K 16K CPU0 0 3:31 98.78% em1 taskq 36 root -68 - 0K 16K CPU3 3 2:53 82.42% em0 taskq 11 root 171 ki31 0K 16K RUN 2 22:48 79.00% idle: cpu2 10 root 171 ki31 0K 16K RUN 3 20:51 22.90% idle: cpu3 39 root -68 - 0K 16K RUN 2 0:32 16.60% em3 taskq 12 root 171 ki31 0K 16K RUN 1 20:16 2.05% idle: cpu1 13 root 171 ki31 0K 16K RUN 0 20:25 1.90% idle: cpu0 input (em0) output packets errs bytes packets errs bytes colls 122588 0 7355280 0 0 0 0 123057 0 7383420 0 0 0 0 input (em1) output packets errs bytes packets errs bytes colls 174917 11899 10495032 2 0 178 0 173967 11697 10438038 2 0 356 0 174630 10603 10477806 2 0 268 0 input (em2) output packets errs bytes packets errs bytes colls 175843 3928 10550580 0 0 0 0 175952 5750 10557120 0 0 0 0 Still less performance than single gig-e.. that giant lock really sucks , and why on earth would LAGG require that.. It seems so simple to fix :/ Anyone up for it:) I wish I was a programmer sometimes, but network engineering will have to do. :D Julian Elischer wrote: > Paul wrote: >> Is PF better than ipfw? iptables almost has no impact on routing >> performance unless I add a swath of rules to it and then it bombs >> I need maybe 10 rules max and I don't want 20% performance drop for >> that.. :P > > well lots of people have wanted to fix it, and I've investigated > quite a lot but it takes someone with 2 weeks of free time and > all the right clue. It's not inherrent in ipfw but it needs some > TLC from someone who cares :-). > > > >> Ouch! :) Is this going to be fixed any time soon? We have some >> money that can be used for development costs to fix things like this >> because >> we use linux and freebsd machines as firewalls for a lot of customers >> and with the increasing bandwidth and pps the customers are demanding >> more and I >> can't give them better performance with a brand new dual xeon or >> opteron machine vs the old p4 machines I have them running on now :/ >> The only difference >> in the new machine vs old machine is that the new one can take in >> more pps and drop it but it can't route a whole lot more. >> Routing/firewalling must still not be lock free, ugh.. :P >> >> Thanks >> >> >> >> Julian Elischer wrote: >>> Paul wrote: >>>> ULE without PREEMPTION is now yeilding better results. >>>> input (em0) output >>>> packets errs bytes packets errs bytes colls >>>> 571595 40639 34564108 1 0 226 0 >>>> 577892 48865 34941908 1 0 178 0 >>>> 545240 84744 32966404 1 0 178 0 >>>> 587661 44691 35534512 1 0 178 0 >>>> 587839 38073 35544904 1 0 178 0 >>>> 587787 43556 35540360 1 0 178 0 >>>> 540786 39492 32712746 1 0 178 0 >>>> 572071 55797 34595650 1 0 178 0 >>>> >>>> *OUCH, IPFW HURTS.. >>>> loading ipfw, and adding one ipfw rule allow ip from any to any >>>> drops 100Kpps off :/ what's up with THAT? >>>> unloaded ipfw module and back 100kpps more again, that's not right >>>> with ONE rule.. :/ >>> >>> ipfw need sto gain a lock on hte firewall before running, >>> and is quite complex.. I can believe it.. >>> >>> in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two >>> interfaces (bridged) but I think it has slowed down since then due >>> to the SMP locking. >>> >>> >>>> >>>> em0 taskq is still jumping cpus.. is there any way to lock it to >>>> one cpu or is this just a function of ULE >>>> >>>> running a tar czpvf all.tgz * and seeing if pps changes.. >>>> negligible.. guess scheduler is doing it's job at least.. >>>> >>>> Hmm. even when it's getting 50-60k errors per second on the >>>> interface I can still SCP a file through that interface although >>>> it's not fast.. 3-4MB/s.. >>>> >>>> You know, I wouldn't care if it added 5ms latency to the packets >>>> when it was doing 1mpps as long as it didn't drop any.. Why can't >>>> it do that? Queue them up and do them in bigggg chunks so none are >>>> dropped........hmm? >>>> >>>> 32 bit system is compiling now.. won't do > 400kpps with GENERIC >>>> kernel, as with 64 bit did 450k with GENERIC, although that could be >>>> the difference between opteron 270 and opteron 2212.. >>>> >>>> Paul >>>> >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >>> > >