From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 06:45:59 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 800771065672 for ; Thu, 3 Jul 2008 06:45:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 3B3378FC14 for ; Thu, 3 Jul 2008 06:45:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEIWA-0002nV-H2; Thu, 03 Jul 2008 02:42:10 -0400 Message-ID: <486C7611.9030905@gtcomm.net> Date: Thu, 03 Jul 2008 02:47:45 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 06:45:59 -0000 Preliminary 32 bit results... When I started out it looked like 32 bit was worse than 64 bit, but it's just the timers are different. For instance, 4000 hz in 64 bit gives better results than 4000hz in 32 bit. Low HZ gives better result with polling on in 32 bit Bottom line, so far I'm not able to get any better performance out of 32 bit at all. In fact I think it might be even a tad slower. I didn't see as high of bursts like I did on 64 bit so far but I'm still testing. Tomorrow comes opteron 2222 so it's 1ghz faster than this one, and I can see if it scales directly with cpu speed or what happens. I did another SMP test with an interesting results. I took one of the cpus out of the machine, so it was just left with a single 2212 (dual core) and it performed better. Less contention I suppose? some results: kern.hz=4000 hw.em.rxd=512 hw.em.txd=512 polling on, idle polling on (only way I can get a reliable netstat output) input (em0) output packets errs bytes packets errs bytes colls 681961 117612 42281586 1 0 226 0 655095 83418 40615892 2 0 220 0 683881 93559 42400626 1 0 178 0 683637 90452 42385498 1 0 178 0 683345 87471 42367394 1 0 178 0 682737 81483 42329696 2 0 220 0 683154 95413 42355552 1 0 178 0 684556 111013 42442476 1 0 178 0 684365 110960 42430634 1 0 178 0 679089 116440 42103518 3 0 534 0 684328 122713 42428340 1 0 178 0 684852 121387 42460828 1 0 178 0 685358 113256 42492200 1 0 178 0 685060 123110 42473724 1 0 178 0 684463 118335 42436710 1 0 178 0 677182 127788 41985300 2 0 356 0 685920 126144 42527044 1 0 178 0 684946 107034 42466656 1 0 178 0 (reboot) kern.hz=1000 input (em0) output packets errs bytes packets errs bytes colls 679611 97394 42136046 5 0 762 0 663939 104714 41164254 5 0 1322 0 685538 91102 42503412 4 0 536 0 676704 94629 41955668 2 0 404 0 685323 115060 42490030 1 0 178 0 675954 105506 41909164 2 0 356 0 655321 92118 40629906 1 0 178 0 686826 85674 42583228 2 0 356 0 686378 89983 42555440 1 0 178 0 685539 80180 42503422 1 0 178 0 686704 88626 42575652 1 0 178 0 686567 88596 42567158 1 0 178 0 687031 82640 42595936 3 0 398 0 sysctl -w kern.polling.each_burst=50 kern.polling.each_burst: 256 -> 50 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 693036 39992 42968315 3 0 400 0 695538 58189 43123360 1 0 178 0 692670 62765 42945544 1 0 178 0 693219 60755 42979580 2 0 220 0 692637 64761 42943498 1 sysctl -w kern.polling.each_burst=33 kern.polling.each_burst: 50 -> 33 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 690530 63359 42812868 1 0 226 0 689748 57670 42764380 1 0 178 0 690489 57874 42810322 1 0 178 0 689655 60606 42758614 1 0 178 0 ^C [root@ircrouter ~]# sysctl -w kern.polling.each_burst=3 kern.polling.each_burst: 33 -> 3 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 612234 110896 37958512 1 0 226 0 614391 112506 38092246 1 0 178 0 ^C [root@ircrouter ~]# sysctl -w kern.polling.each_burst=800 kern.polling.each_burst: 3 -> 800 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 668057 76496 41419538 1 0 226 0 667689 88674 41396720 2 0 220 0 670526 106654 41572616 1 0 178 0 667326 97832 41374216 1 0 178 0 ^C [root@ircrouter ~]# sysctl -w kern.polling.each_burst=66 kern.polling.each_burst: 800 -> 66 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 690164 89290 42790172 1 0 226 0 688886 74360 42710936 1 0 178 0 674079 77027 41792902 1 0 178 0 kern.hz=2000 input (em0) output packets errs bytes packets errs bytes colls 699116 238016 43345196 1 0 178 0 698263 225244 43292310 1 0 290 0 697246 222395 43229256 1 0 178 0 696749 207766 43198442 1 0 178 0 697304 217384 43232852 1 0 178 0 696401 209901 43176866 1 0 178 0 696508 207757 43183500 1 0 178 0 ^C hz=2000 with 1024/1024 descriptors input (em0) output packets errs bytes packets errs bytes colls 670315 235780 41559534 1 0 226 0 683218 225838 42359520 1 0 178 0 682998 242551 42345880 1 0 178 0 681777 239649 42270178 1 0 178 0 hz=1000 with 256/256 descriptors netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 740584 160355 45916212 2 0 0 0 746027 165198 46253678 1 0 178 0 746068 165921 46256220 1 0 178 0 746505 167527 46283314 1 0 178 0 743902 175019 46121928 1 0 178 0 746130 179795 46260064 1 0 178 0 744457 166448 46156338 1 0 178 0 746169 176137 46262482 1 0 178 0 hz=667 with 256/256 input (em0) output packets errs bytes packets errs bytes colls 742614 91687 46042072 1 0 226 0 739746 85695 45864256 1 0 178 0 733723 85162 45490840 3 0 398 0 737561 102207 45728786 1 0 178 0 739618 127597 45856320 1 0 178 0 ^C Hrm finally same pps as 64 bit..... Now I wonder what happens if I go back to the 64 bit and try 1000 256/256 ?? I don't think I tried that one.. Guess another reinstall :> Installing 64 bit.. (again) Just to be sure.. Paul Ingo Flaschberger wrote: > Dear Paul, > >> SMP DISABLED on my Opteron 2212 (ULE, Preemption on) >> Yields ~750kpps in em0 and out em1 (one direction) >> I am miffed why this yields more pps than >> a) with all 4 cpus running and b) 4 cpus with lagg load balanced over >> 3 incoming connections so 3 taskq threads > > because less locking, less synchronisation, .... > >> I would be willing to set up test equipment (several servers plugged >> into a switch) with ipkvm and power port access >> if someone or a group of people want to figure out ways to improve >> the routing process, ipfw, and lagg. >> >> Maximum PPS with one ipfw rule on UP: >> tops out about 570Kpps.. almost 200kpps lower ? (frown) > > can you post the rule here? > >> I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's >> in here and see how that scales, using UP same kernel etc I have now. > > really, please try 32bit and 1 cpu. > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >