From owner-freebsd-performance@FreeBSD.ORG Thu Nov 23 22:14:13 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 14D3116A416 for ; Thu, 23 Nov 2006 22:14:13 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 88B4D43D62 for ; Thu, 23 Nov 2006 22:13:34 +0000 (GMT) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.13.6/8.13.6) with ESMTP id kANME5VC017630; Thu, 23 Nov 2006 17:14:05 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.6/8.13.3) with ESMTP id kANMDt6C006356 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 23 Nov 2006 17:14:05 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200611232214.kANMDt6C006356@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Thu, 23 Nov 2006 17:13:59 -0500 To: freebsd-performance@freebsd.org From: Mike Tancsa In-Reply-To: References: <4557CECD.2000609@samsco.org> <200611132054.kADKsFvK045726@lava.sentex.ca> <4558E3DC.6080800@samsco.org> <200611200454.kAK4sdat083568@lava.sentex.ca> <7.1.0.9.0.20061120160757.14d4a728@sentex.net> <200611220247.kAM2l9JP095066@lava.sentex.ca> <20061122130947.GM20405@obiwan.tataz.chchile.org> <200611231652.kANGqJsr005016@lava.sentex.ca> <200611231803.kANI3KBf005315@lava.sentex.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: ClamAV version 0.88.3, clamav-milter version 0.88.3 on clamscanner2 X-Virus-Status: Clean Subject: Re: Broadcom bge forwarding performance (was em forwarding performance ) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Nov 2006 22:14:13 -0000 >>test setup description at >>http://www.tancsa.com/blast.html More testing :) This time with a pair of PCIe 1x bge nics, as well as using the patch at http://lists.freebsd.org/pipermail/freebsd-net/2006-November/012389.html As I switched to bge, I could test against Dragonfly as well since its em driver is broken with the dual port PT1000 Couple of interesting notes. Unlike the em nics, I dont see the box being overwhelmed with packets to the point of netstat -ni showing input errors. I dont know if this is just a measurement issue, or the driver is smarter about throttling back the sender rate with the switch ? I also noticed that the initial blast, as seen by the routing box, sees pps rates in around 500k, but then slowly goes down to about 460k and settles in around that rate but then eventually go back up towards 500kpps. 26479 0 1588740 2 0 332 0 511592 0 30695562 3 0 386 0 508043 0 30482580 2 0 332 0 508026 0 30481560 2 0 332 0 input (Total) output packets errs bytes packets errs bytes colls 506675 0 30400628 2 0 332 0 511805 0 30708402 3 0 498 0 477792 0 28667520 2 0 332 0 454527 0 27271620 2 0 332 0 454815 0 27288900 2 0 332 0 453920 0 27235200 2 0 444 0 449027 0 26941620 2 0 332 0 449383 0 26962980 2 0 332 0 461838 0 27710280 2 0 332 0 455068 0 27304080 2 0 332 0 455014 0 27300840 2 0 332 0 411416 0 24684960 4 0 664 0 3 0 180 2 0 332 0 2 0 120 2 0 332 0 4 0 293 2 0 332 0 2 0 120 2 0 332 0 443378 0 26602680 2 0 332 0 455782 0 27346920 2 0 332 0 453432 0 27205920 2 0 332 0 453816 0 27228960 2 0 332 0 450561 0 27033660 2 0 332 0 input (Total) output packets errs bytes packets errs bytes colls 450317 0 27019020 2 0 332 0 454149 0 27248940 2 0 444 0 With the above patch and from current as of today (including http://lists.freebsd.org/pipermail/cvs-all/2006-November/197096.html) the box is *very* responsive even when two blasts of packets are going in opposite directions! Here initially is one blast coming from 192.168.88.176 to 192.168.44.1, then I start up another blast from 192.168.44.244 to 192.168.88.206. The box is totally responsive from the management interface [r2-current]# vmstat -i interrupt total rate irq4: sio0 929 0 irq14: ata0 1489 0 irq15: ata1 67 0 irq16: bge1 3158402 1980 irq17: bge0 4918 3 irq19: bge2 2358601 1478 cpu0: timer 3187013 1998 irq256: em0 4 0 irq257: em1 4 0 cpu1: timer 3186657 1997 Total 11898084 7459 [r2-current]# [r2-current]# ifstat -b bge0 bge1 bge2 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 0.47 1.67 222466.8 0.00 0.00 174265.6 0.94 1.17 233668.8 0.00 0.00 183040.6 0.47 1.17 212013.7 0.00 0.00 166077.4 0.94 1.17 220571.9 0.00 0.00 172781.3 0.47 1.17 240600.0 0.00 0.00 188470.0 0.94 1.17 239261.9 0.00 0.00 187421.8 0.47 1.17 210286.3 0.00 0.00 164724.2 32.20 46.60 236021.0 0.00 0.00 184883.1 9.18 8.46 240039.5 0.00 0.00 188030.9 7.31 7.54 237144.6 0.00 0.00 185763.3 7.59 7.62 231464.9 0.00 0.00 181314.2 0.47 2.47 210767.9 0.00 0.00 165101.5 1.40 2.47 216543.6 0.00 0.00 169625.8 0.94 2.47 213539.4 0.00 0.00 167272.6 1.40 2.47 214488.8 0.00 0.00 168016.2 0.94 2.47 197169.2 21056.33 24296.24 154449.2 1.40 2.47 117003.6 101840.6 117507.9 91652.81 2.20 2.89 119036.2 100521.9 115987.3 93245.06 1.87 2.47 117262.4 101812.5 117475.5 91855.57 0.94 2.47 118018.5 101145.1 116706.3 92447.80 1.40 2.47 116557.3 102412.8 118168.2 91303.23 0.94 2.47 117745.3 101736.3 117388.0 92233.84 1.40 2.47 117146.3 102174.3 117893.4 91764.60 0.94 2.47 116071.5 102631.6 118421.1 90922.66 0.94 2.47 116262.2 102233.0 117961.1 91072.04 0.94 2.47 116043.0 102018.6 117713.2 90899.94 1.87 2.47 115890.5 101801.2 117463.4 90780.86 0.94 2.47 117120.8 101347.2 116939.1 91744.60 1.40 2.47 118051.4 100909.8 116434.4 92473.56 0.94 2.47 118767.3 102499.5 118268.6 93034.42 1.40 2.47 116915.7 101836.6 117503.8 91583.94 1.40 3.34 116517.4 102566.9 118346.0 91271.97 1.73 2.89 116316.7 102750.2 118558.4 91114.77 0.94 2.47 117356.1 101465.1 117074.6 91928.93 1.40 2.47 117906.8 101133.9 116693.0 92360.30 1.82 2.47 116012.6 102530.9 118305.3 90876.51 1.40 2.47 115859.8 103404.1 119311.9 90756.86 0.94 2.47 117364.7 102034.7 117732.3 91935.72 1.40 2.47 117721.1 100830.2 116342.5 92214.86 1.40 2.47 116521.8 102289.6 118026.9 91275.45 0.94 2.47 117061.8 101941.4 117624.2 91698.38 bge0 bge1 bge2 Kbps in Kbps out Kbps in Kbps out Kbps in Kbps out 1.87 2.47 91718.83 119571.1 137967.1 71846.42 0.94 3.47 0.00 182065.9 210075.6 0.00 1.40 2.47 0.00 150718.8 173906.3 0.00 0.47 2.47 0.00 0.00 0.00 0.00 1.40 2.47 0.00 0.00 0.00 0.00 0.94 2.47 0.00 0.00 0.00 0.00 1.40 2.47 0.00 0.00 0.00 0.00 [r2-current]# netstat -ni 1 input (Total) output packets errs bytes packets errs bytes colls 449655 0 26979300 449622 0 21134055 0 462036 0 27722160 462018 0 21715820 0 457761 0 27465660 457814 0 21514989 0 454539 0 27272340 454446 0 21363555 0 469598 0 28175880 469277 0 22175141 0 501175 0 30070500 501315 0 24809563 0 500932 0 30055920 500938 0 24777688 0 502038 0 30122322 501827 0 24855812 0 501276 0 30076560 501521 0 24804540 0 501378 0 30082680 501418 0 24821941 0 501099 0 30065940 501067 0 24806099 0 502164 0 30129840 501890 0 24862704 0 501911 0 30114660 502006 0 24856474 0 499672 0 29980320 499854 0 24742438 0 499575 0 29974502 499586 0 24738495 0 498042 0 29882520 498027 0 24659622 0 500104 0 30006240 500065 0 24751670 0 500569 0 30034140 500653 0 24768087 0 506240 0 30374400 506113 0 25066903 0 500648 0 30038880 500563 0 24777197 0 input (Total) output packets errs bytes packets errs bytes colls 501650 0 30099000 501554 0 24845586 0 501332 0 30079962 501550 0 24825352 0 500664 0 30039840 500713 0 24778844 0 501305 0 30078300 501235 0 24812561 0 500147 0 30008873 499945 0 24769644 0 501150 0 30069000 501284 0 24827402 0 503208 0 30192420 503357 0 24909495 0 499578 0 29974740 499528 0 24717105 0 500846 0 30050760 500855 0 24799047 0 501295 0 30077700 501261 0 24825246 0 496749 0 29804940 496919 0 24691116 0 449463 0 26967780 449513 0 23372416 0 427765 0 25665900 427857 0 22244044 0 2 0 120 2 0 316 0 Results along with ipfw rule tests at http://www.tancsa.com/blast.html I also compared Dragonfly using the bge nics and their behaviour is quite different as compared to FreeBSD. Out of the box its "slower" but it keeps its forwarding speed with or without ipfw loaded. Development kernel results are up there as well and it behaves much like the other BSDs but is notably faster on the firewall rule processing. However, FreeBSD HEAD when in UP mode will live lock with 2 blasts of packets going in opposite directions. The management interface as well as the serial console is not responsive at all. [r2-dragonfly]# netstat -ni Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll bge0 1500 00:10:18:14:15:12 600 0 232 0 0 bge0 1500 fe80:1::210 fe80:1::210:18ff: 0 - 0 - - bge0 1500 192.168.43 192.168.43.224 280 - 223 - - bge1 1500 00:10:18:14:27:d5 76475440 1271 8606533 0 0 bge1 1500 192.168.88 192.168.88.223 4 - 4 - - bge1 1500 fe80:2::210 fe80:2::210:18ff: 0 - 0 - - bge2 1500 00:10:18:14:38:d2 8631578 2465 76061495 0 0 bge2 1500 192.168.44 192.168.44.223 1 - 1 - - bge2 1500 fe80:3::210 fe80:3::210:18ff: 0 - 0 - -