From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:29:34 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49840106564A for ; Tue, 1 Jul 2008 01:29:34 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 071CE8FC0C for ; Tue, 1 Jul 2008 01:29:33 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m611TVev042493; Mon, 30 Jun 2008 21:29:31 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m611TUV5083067 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 30 Jun 2008 21:29:31 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807010129.m611TUV5083067@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 30 Jun 2008 21:29:38 -0400 To: Paul From: Mike Tancsa In-Reply-To: <48694A9D.1030001@gtcomm.net> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <48694A9D.1030001@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:29:34 -0000 At 05:05 PM 6/30/2008, Paul wrote: >With hours and days of tweaking i can't even get 500k pps :/ no >firewall no anything else.. >What is your kernel config? Sysctl configs? The only thing that makes a difference is net.inet.ip.fastforwarding=1 >My machine i'm testing on is dual opteron 2212 , with intel 2 port >82571 nic.. xeon dual core on a supermicro MB. I am using one NIC on the MB and one on the dual port. em0@pci0:10:1:0: class=0x020000 card=0x11798086 chip=0x10798086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller' class = network subclass = ethernet cap 01[dc] = powerspec 2 supports D0 D3 current D0 cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction cap 05[f0] = MSI supports 1 message, 64 bit em1@pci0:10:1:1: class=0x020000 card=0x11798086 chip=0x10798086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller' class = network subclass = ethernet cap 01[dc] = powerspec 2 supports D0 D3 current D0 cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction cap 05[f0] = MSI supports 1 message, 64 bit em2@pci0:13:0:0: class=0x020000 card=0x108c15d9 chip=0x108c8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82573E Intel Corporation 82573E Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint em3@pci0:14:0:0: class=0x020000 card=0x109a15d9 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82573L Intel PRO/1000 PL Network Adaptor' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint >Using 7-STABLE and I tried 6-stable and -current >I get the RTM_MISS with 7 and current but only with certain types of >packets at a certain rate.. :/ I wonder if its a bug with the em driver ? I dont have any other dual port cards handy right now to test with >I can not get more than 500kpps.. i tried everything I could think >of... lowering the rx descriptors on EM to 512 instead of 2048 gave >me some more.. I was stuck at 400kpps until i changed those and i >lowered the rx processing limit. >My tests are going incoming em0 and outgoing em1 in one direction >only and it has major errors when em0 taskq gets close to 80% cpu.. I now have 3 boxes now generating traffic through the box acting as a router. I will try some other operating systems as well to see how they compare when back at the office on Wednesday >I am pretty disappointed that it maxes out a little over 400kpps and >even then it gets some errors here and there , mainly missed packets >due to no buffer and rx overruns (dev.em.0.stats=1) Something about the MB you are using perhaps ? Just for rough comparison, how long does # time make -j4 buildkernel > /var/log/build.out.k 670.485u 66.061s 8:29.54 144.5% 5962+1087k 9185+7419io 380pf+0w take on your machine ? The above value is with inet6 and sctp commented out from the kernel. ---Mike >Mike Tancsa wrote: >>At 04:04 AM 6/29/2008, Paul wrote: >>>This is just a question but who can get more than 400k pps >>>forwarding performance ? >> >> >>OK, I setup 2 boxes on either end of a RELENG_7 box from about May >>7th just now, to see with 2 boxes blasting across it how it would work. >>*However*, this is with no firewall loaded and, I must enable ip >>fast forwarding. Without that enabled, the box just falls over. >> >>even at 20Kpps, I start seeing all sorts of messages spewing to >>route -n monitor >> >> >>got message of size 96 on Mon Jun 30 15:39:10 2008 >>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, >>errno 0, flags: >>locks: inits: >>sockaddrs: >> default >> >>I am starting to wonder if those messages are the results of >>corrupted packets the machine just cant keep up with ? >> >> >>CPU is >> >>CPU: Intel(R) Xeon(R) CPU 3070 @ 2.66GHz (2660.01-MHz >>686-class CPU) >> >> >> input (Total) output >> packets errs bytes packets errs bytes colls >> 611945 0 77892098 611955 0 77013002 0 >> 616727 0 78215508 616742 0 77303454 0 >> 617066 0 78162130 617082 0 77238434 0 >> 618238 0 78302314 618225 0 77377582 0 >> 617035 0 78141000 617038 0 77215672 0 >> 617625 0 78225600 617588 0 77301734 0 >> 616190 0 78017320 616165 0 77091774 0 >> 615583 0 78064130 615628 0 77152800 0 >> 617662 0 78254388 617658 0 77332340 0 >> 618000 0 78269912 617950 0 77344554 0 >> 617248 0 78183136 617315 0 77259588 0 >> 617325 0 78204566 617289 0 77282094 0 >> 618391 0 78337734 618357 0 77413756 0 >> 616025 0 78116070 616082 0 77203116 0 >> >> >>To generate the packets, I am just using >>/usr/src/tools/tools/netblast on 2 endpoints starting at about the same time >> >># ./netblast 10.10.1.2 500 100 40 >> >>start: 1214854131.083679919 >>finish: 1214854171.084668592 >>send calls: 20139141 >>send errors: 0 >>approx send rate: 503478 >>approx error rate: 0 >> >> >># ./netblast 10.10.1.3 500 10 40 >> >>start: 1214854273.882202815 >>finish: 1214854313.882319031 >>send calls: 23354971 >>send errors: 18757223 >>approx send rate: 114943 >>approx error rate: 0 >> >>The box in the middle doing the forwarding >> >>1[spare-r7]# ifconfig -u >>em0: flags=8843 metric 0 mtu 1500 >> >>options=19b >> ether 00:1b:21:08:32:a8 >> inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255 >> media: Ethernet autoselect (1000baseTX ) >> status: active >>em1: flags=8843 metric 0 mtu 1500 >> options=9b >> ether 00:1b:21:08:32:a9 >> inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255 >> media: Ethernet autoselect (100baseTX ) >> status: active >>em3: flags=8843 metric 0 mtu 1500 >> >>options=19b >> ether 00:30:48:90:4c:ff >> inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255 >> media: Ethernet autoselect (1000baseTX ) >> status: active >>lo0: flags=8049 metric 0 mtu 16384 >> inet 127.0.0.1 netmask 0xff000000 >> >> >>I am going to try a few more tests with and without, firewall rules >>etc as well as an updated kernel to RELENG_7 as of today and see how that goes. >> >> ---Mike >>