From owner-freebsd-net@FreeBSD.ORG Tue Nov 21 05:50:40 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BBF9616A403; Tue, 21 Nov 2006 05:50:40 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0C9C243D4C; Tue, 21 Nov 2006 05:50:19 +0000 (GMT) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.13.6/8.13.6) with ESMTP id kAL5ocmL011146; Tue, 21 Nov 2006 00:50:38 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.6/8.13.3) with ESMTP id kAL5obPR089912 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 21 Nov 2006 00:50:37 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200611210550.kAL5obPR089912@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 21 Nov 2006 00:50:50 -0500 To: freebsd-performance@freebsd.org From: Mike Tancsa In-Reply-To: <200611200454.kAK4sdat083568@lava.sentex.ca> References: <2a41acea0611081719h31be096eu614d2f2325aff511@mail.gmail.com> <200611091536.kA9FaltD018819@lava.sentex.ca> <45534E76.6020906@samsco.org> <200611092200.kA9M0q1E020473@lava.sentex.ca> <200611102004.kAAK4iO9027778@lava.sentex.ca> <2a41acea0611101400w5b8cef40ob84ed6de181f3e2c@mail.gmail.com> <200611102221.kAAML6ol028630@lava.sentex.ca> <455570D8.6070000@samsco.org> <200611120412.kAC4CuIB035746@lava.sentex.ca> <45574ECA.4080207@samsco.org> <200611130040.kAD0etbp040637@lava.sentex.ca> <4557CECD.2000609@samsco.org> <200611130158.kAD1wdKE040908@lava.sentex.ca> <4557EF13.9060305@samsco.org> <200611130454.kAD4sZwe041556@lava.sentex.ca> <4557FF7A.8020704@samsco.org> <200611132054.kADKsFvK045726@lava.sentex.ca> <4558E3DC.6080800@samsco.org> <200611200454.kAK4sdat083568@lava.sentex.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Virus-Scanned: ClamAV version 0.88.3, clamav-milter version 0.88.3 on clamscanner2 X-Virus-Status: Clean Cc: freebsd-net Subject: Re: em forwarding performance (was Proposed 6.2 em RELEASE patch X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Nov 2006 05:50:40 -0000 I am moving this thread over to performance after this post as it makes more sense to continue there. At 11:54 PM 11/19/2006, Mike Tancsa wrote: >At 04:30 PM 11/13/2006, Scott Long wrote: >>Mike Tancsa wrote: >>>At 12:15 AM 11/13/2006, Scott Long wrote: >>> >>>>Is this with EM_INTR_FAST enabled also? >>> >>>Without it, the 2 streams are definitely lossy on the management interface >>> >>> ---Mike >> >>Ok, and would you be able to test the polling options as well? > > >Note about platforms. The HEAD w Patch is a patch >glebius@freebsd.org asked me to test. FastFWD is with >net.inet.ip.fastforwarding on. Also with FastFWD set to one, I >always used the kernel options ADAPTIVE_GIANT commented out and >added NO_ADAPTIVE_MUTEXES. INET6 was removed from all kernels as >well. With these kernel changes, and fast forwarding on, I was able >to keep the box r2 responsive from the console as while blasting >packets across its 2 interfaces. Otherwise, the box seemingly >livelocked. For the linux kernel config, it was pretty well the >default, except I removed INET6, IPSEC and disabled iptables. The >LINUX kernel was 2.6.18.2 on FC5. > >The first test is with UDP netperf. >/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t >UDP_STREAM -- -m 10 -s 32768 -S 32768 >/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t >UDP_STREAM -- -m 64 -s 32768 -S 32768 >/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t >UDP_STREAM -- -m 128 -s 32768 -S 32768 >/usr/local/netperf/netperf -l 60 -H 192.168.44.1 -i 10,2 -I 99,10 -t >UDP_STREAM -- -m 200 -s 32768 -S 32768 Again, this is the same setup as described at http://www.tancsa.com/blast.jpg My goals of this testing was to understand the following : 1) the new em driver to make sure it works well for me and give it a good shake out under load for RELENG_6 2) understand the implications of various configs on forwarding performance of SMP vs UP vs Polling vs Fast Interrupts and to avoid livelock when there is a lot of pps In this round of testing, I tried of RELENG_6 i386 in UP config as well. Although raw packets / second (pps) forwarding was faster, the box was pretty unresponsive and userland apps (i.e. routing) and made the config pretty unusable with fast_forwarding enabled. Once ipfw was loaded, the box would totally lock up and routing daemons was start to spin out of control as hold timers expired. Bottom line, there is slightly less raw pps performance out of an SMP config, but the box seems to be far more resilient against a high pps attack. RELENG_6 SMP with #define EM_FAST_INTR 1 in if_em.c net.inet.ip.fastforwarding=1 with ADAPTIVE_GIANT removed from the kernel and NO_ADAPTIVE_MUTEXES added gives decent pps forwarding rates, without the box becoming live locked. Note, in the real world, given an average packet size of ~600 bytes, a box in this config should be able to route and firewall gigabit speeds without much issue and can sustain moderate DDoS pps attacks over the net. For the routing test, I used 2 peers each sending ~ 195K routes. I re-tested the single UDP stream with 194k routes loaded in the kernel routing table and 2 bgp peers. Then, while blasting across the box, I cleared the session which had all the routes installed in the kernel so that the daemon would have to reinstall all the routes to point to the other peer. While this was happening (10 seconds on SMP boxes, MUCH longer ~ 1min on UP, sometimes total failure) I was measuring throughput. On UP it didnt drop too much, a bit more on SMP, but convergence was quite fast, about 10 seconds. Similarly, installing ipfw rules on the UP version made the box totally live lock in non polling mode, but seemed to perform well enough in polling mode. pf lagged quite far behind The biggest difference seems to be in the efficiency of the firewall rules in LINUX vs FreeBSD. Granted, the rules I inserted, are poorly written, but adding rules did seem to have a linearly negative impact on performance, where as rules via iptables did not significantly impact forwarding rates. However, in the LINUX test, I seemed to trigger some race in bgpd when doing the clear that took a little more proding with FreeBSD, but is there as well :( So it might be back to version .98 The table is also up at http://www.tancsa.com/blast.html which might be easier to read Straight Routing test One Stream 194K Routes bgp clear and single ipfw 5 ipfw ruipfw 10 pf 1 pf 5 Linux 581,310 590,584 579,833 582,963 575,718 579,442 FreeBSD HEAD Nov 20 +FastFWD 540,695 529,425 439,980 398,283 370,458 FreeBSD HEAD Nov 11 441,560 RELENG6 i386 407,403 RELENG6 i386 FastFWD 557,580 562,547 484,250 425,791 386,644 353,856 333,293 FreeBSD HEAD w Patch 422,294 FreeBSD HEAD w Patch FastFWD 567,290 564,340 482,093 436,205 381,459 359,248 361,458 AMD64 RELENG6 w FastFWD 574,590 549,233 486,737 400,050 320,129 296,760 273,824 AMD64 RELENG6 polling 285,917 AMD64 RELENG6 polling FastFWD 512,042 RELENG6 i386 polling FastFWD 558,600 550,041 RELENG6 i386 polling FastFWD HZ=2000 565,520 563,068 373,904 RELENG_6 UP i386 400,188 RELENG_6 UP i386 FastFWD 584,900 582,534 560,033 560,323 RELENG_6 UP i386 FastFWD Polling 585,934 476,629 422,633 393,301 Straight Routing test 2 streams opposite direction Linux 473,810 452,205 408,932 FreeBSD HEAD Nov 11 204,043 FreeBSD HEAD nov 20 + fastFWD 312,140 250,277 223,289 208,551 RELENG6 i386 165,461 RELENG6 i386 FastFWD 368,960 353,437 216,597 206,077 194,47669,50067,385 FreeBSD HEAD w Patch 127,832 FreeBSD HEAD w Patch FastFWD 346,220 404,785 249,617 234,047157,603 AMD64 RELENG6 w Polling 155,659 AMD64 RELENG6 w Polling FastFWD 231,541 RELENG6 i386 polling FastFWD 319,350 312,621 RELENG6 i386 polling FastFWD HZ=2000 300,280 299,018 RELENG_6 UP i386 FastFWD Polling 342,551 229,652 205,322