From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 18:22:55 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CE691065674 for ; Wed, 2 Jul 2008 18:22:55 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id B93358FC0C for ; Wed, 2 Jul 2008 18:22:54 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KE6vB-0004AV-TS; Wed, 02 Jul 2008 14:19:14 -0400 Message-ID: <486BC7F5.5070604@gtcomm.net> Date: Wed, 02 Jul 2008 14:24:53 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 18:22:55 -0000 Fastforward works with lagg , lagg just has some issues that need to be fixed, even on UP system. It has the same issue as IPFW. kern.polling.idlepoll_sleeping: 1 kern.polling.stalled: 806 kern.polling.suspect: 97861 kern.polling.phase: 0 kern.polling.enable: 0 kern.polling.handlers: 2 kern.polling.residual_burst: 0 kern.polling.pending_polls: 0 kern.polling.lost_polls: 128535 kern.polling.short_ticks: 1455 kern.polling.reg_frac: 1200 kern.polling.user_frac: 0 kern.polling.idle_poll: 0 kern.polling.each_burst: 50 kern.polling.burst_max: 1440 kern.polling.burst: 377 It's doing 720kpps right now, and it's having a lot of errors, but the cpu is 40% idle! I don't understand? Is it reporting the wrong cpu usage in TOP? Is this a bug? input (em0) output packets errs bytes packets errs bytes colls 722012 42861 44764748 1 0 178 0 704432 52679 43674800 2 0 580 0 693297 53536 42984418 1 0 178 0 704046 42525 43650854 2 0 220 0 714959 37876 44327462 1 0 178 0 744923 24202 46185230 1 0 178 0 726069 34699 45016282 1 0 178 0 681837 78581 42273898 1 0 178 0 663106 85699 41112576 1 0 178 0 708274 55414 43912992 1 0 178 0 659659 94430 40898862 1 0 178 0 669235 100248 41492574 1 0 178 0 676510 100102 41943624 1 0 178 0 679847 98972 42150518 1 0 178 0 677700 92586 42017416 2 0 356 0 672639 86454 41703622 1 0 178 0 675841 72821 41902146 1 0 178 0 679522 86423 42130368 1 0 178 0 660737 72883 40965698 1 0 178 0 637085 81303 39499274 1 0 178 0 655463 98183 40638710 1 0 178 0 input (em0) output packets errs bytes packets errs bytes colls 683650 66140 42386304 1 0 178 0 654910 110089 40604424 1 0 290 0 647969 120709 40174082 1 0 178 0 666260 67037 41308124 1 0 178 0 671570 68276 41637344 1 0 178 0 691683 60819 42884350 1 0 178 0 663656 79528 41146728 2 0 244 0 703917 47860 43642870 2 0 356 0 710988 55792 44081258 2 0 220 0 697062 77661 43217848 1 0 178 0 65 processes: 2 running, 46 sleeping, 17 waiting CPU: 0.0% user, 0.0% nice, 10.8% system, 45.7% interrupt, 43.5% idle Mem: 7968K Active, 6028K Inact, 42M Wired, 84K Cache, 8768K Buf, 1925M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 4:06 86.43% idle 36 root -68 - 0K 16K - 0:57 3.81% em3 taskq 13 root -44 - 0K 16K WAIT 9:22 1.42% swi1: net 1429 root 44 0 8084K 2052K RUN 0:00 0.29% top 11 root -32 - 0K 16K WAIT 0:01 0.05% swi4: clock sio net.isr.swi_count: 39442306 net.isr.drop: 0 net.isr.queued: 8 net.isr.deferred: 0 net.isr.directed: 2189 net.isr.count: 2189 net.isr.direct: 1 net.route.netisr_maxqlen: 16384 net.inet.ip.intr_queue_maxlen: 16384 net.inet.ip.intr_queue_drops: 0 em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 22574958 em0: Receive No Buffers = 65713041 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 52679 em0: watchdog timeouts = 0 em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 547984791 em0: Good Packets Xmtd = 5000 em0: TSO Contexts Xmtd = 18 em0: TSO Contexts Failed = 0 kern.hz=2000 hw.em.rxd=512 hw.em.txd=512 -----------Reboot with 4096/4096........(my guess is that it will be a lot worse, more errors..) ........ Without polling, 4096 is horrible, about 200kpps less ... :/ Turning on polling.. polling on, 4096 is bad, input (em0) output packets errs bytes packets errs bytes colls 622379 307753 38587506 1 0 178 0 635689 277303 39412718 1 0 178 0 625552 291235 38784244 2 0 580 0 630143 287872 39068870 1 0 178 0 620225 292071 38453954 1 0 178 0 627499 295329 38904942 1 0 178 0 623854 288086 38678952 1 0 178 0 632433 267698 39210850 1 0 178 0 619177 279541 38388978 1 0 178 0 618049 265926 38319038 2 0 356 0 627026 263882 38875616 1 0 178 0 em0: Missed Packets = 16570461 em0: Receive No Buffers = 9220592 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 40539 ------Rebooting with 256/256 descriptors.......... .......... No polling: 843762 25337 52313248 1 0 178 0 763555 0 47340414 1 0 178 0 830189 0 51471722 1 0 178 0 838724 0 52000892 1 0 178 0 813594 939 50442832 1 0 178 0 807303 763 50052790 1 0 178 0 791024 0 49043492 1 0 178 0 768316 1106 47635596 1 0 178 0 Machine is maxed and is unresponsive.. Polling ON: input (em0) output packets errs bytes packets errs bytes colls 784138 179079 48616564 1 0 226 0 788815 129608 48906530 2 0 356 0 755555 142997 46844426 2 0 468 0 803670 144459 49827544 1 0 178 0 777649 147120 48214242 1 0 178 0 779539 146820 48331422 1 0 178 0 786201 148215 48744478 2 0 356 0 776013 101660 48112810 1 0 178 0 774239 145041 48002834 2 0 356 0 771774 102969 47850004 1 0 178 0 Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm really mistified by this.. Every time it maxes out and gets errors, top reports: CPU: 0.0% user, 0.0% nice, 10.1% system, 45.3% interrupt, 44.6% idle pretty much the same line every time 256/256 blows away 4096 , probably fits the descriptors into the cache lines on the cpu and 4096 has too many cache misses and causes worse performance. This is probably just some nasty programming and they could optimize it for 4096 by taking larger chunks, we don't need the latency to be 0.08ms for each packet, i don't care if it's 0.3ms as long as it doesn't drop any. Setting HZ=100 and 256/256 gets a maximum higher than 2000 polling with 256/256 but the box is unresponsive around 850kpps If this only worked with SMP and was optimized it could do millions of pps :/ Ingo Flaschberger wrote: > Dear Paul, > >> I still don't like the huge hit ipfw and lagg take :/ > > I think, you can't use fastforward with with lagg. > >> ** I tried polling in UP mode and I got some VERY interesting results.. >> CPU is 44% idle (idle polling isn't on) but I'm getting errors! >> It's doing 530kpps with ipfw loaded, which without polling uses 100% >> cpu but now it says my cpu is 44% idle? that makes no sense.. If it >> was idle why am I getting errors? I only get errors when em taskq >> was eating 100% cpu.. >> Idle polling on/off makes no difference. >> user_frac is set to 5 .. > > what are your values: > kern.polling.reg_frac= > kern.polling.user_frac= > kern.polling.burst_max= > > I use: > kern.polling.reg_frac=20 > kern.polling.user_frac=20 > kern.polling.burst_max=512 > > if you need more than 1000, you need to change the code: > src/sys/kern/kern_poll.c > #define MAX_POLL_BURST_MAX 1000 > >> So my maximum without polling is close to 800kpps but if I push that >> it starts locking me from doing things, or > > how many kpps do you want to achieve? > >> HZ=2000 for this test (512/512 descriptors) > > you mean: > hw.em.rxd=512 > hw.em.txd=512 > ? > > can you try with polling: > hw.em.rxd=4096 > hw.em.txd=4096 > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >