Date: Mon, 30 Jun 2008 17:05:33 -0400 From: Paul <paul@gtcomm.net> To: Mike Tancsa <mike@sentex.net> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] Message-ID: <48694A9D.1030001@gtcomm.net> In-Reply-To: <200806301944.m5UJifJD081781@lava.sentex.ca> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
With hours and days of tweaking i can't even get 500k pps :/ no firewall no anything else.. What is your kernel config? Sysctl configs? My machine i'm testing on is dual opteron 2212 , with intel 2 port 82571 nic.. Using 7-STABLE and I tried 6-stable and -current I get the RTM_MISS with 7 and current but only with certain types of packets at a certain rate.. :/ I can not get more than 500kpps.. i tried everything I could think of... lowering the rx descriptors on EM to 512 instead of 2048 gave me some more.. I was stuck at 400kpps until i changed those and i lowered the rx processing limit. My tests are going incoming em0 and outgoing em1 in one direction only and it has major errors when em0 taskq gets close to 80% cpu.. I am pretty disappointed that it maxes out a little over 400kpps and even then it gets some errors here and there , mainly missed packets due to no buffer and rx overruns (dev.em.0.stats=1) Mike Tancsa wrote: > At 04:04 AM 6/29/2008, Paul wrote: >> This is just a question but who can get more than 400k pps forwarding >> performance ? > > > OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th > just now, to see with 2 boxes blasting across it how it would work. > *However*, this is with no firewall loaded and, I must enable ip fast > forwarding. Without that enabled, the box just falls over. > > even at 20Kpps, I start seeing all sorts of messages spewing to route > -n monitor > > > got message of size 96 on Mon Jun 30 15:39:10 2008 > RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno > 0, flags:<DONE> > locks: inits: > sockaddrs: <DST> > default > > I am starting to wonder if those messages are the results of corrupted > packets the machine just cant keep up with ? > > > CPU is > > CPU: Intel(R) Xeon(R) CPU 3070 @ 2.66GHz (2660.01-MHz > 686-class CPU) > > > input (Total) output > packets errs bytes packets errs bytes colls > 611945 0 77892098 611955 0 77013002 0 > 616727 0 78215508 616742 0 77303454 0 > 617066 0 78162130 617082 0 77238434 0 > 618238 0 78302314 618225 0 77377582 0 > 617035 0 78141000 617038 0 77215672 0 > 617625 0 78225600 617588 0 77301734 0 > 616190 0 78017320 616165 0 77091774 0 > 615583 0 78064130 615628 0 77152800 0 > 617662 0 78254388 617658 0 77332340 0 > 618000 0 78269912 617950 0 77344554 0 > 617248 0 78183136 617315 0 77259588 0 > 617325 0 78204566 617289 0 77282094 0 > 618391 0 78337734 618357 0 77413756 0 > 616025 0 78116070 616082 0 77203116 0 > > > To generate the packets, I am just using > /usr/src/tools/tools/netblast on 2 endpoints starting at about the > same time > > # ./netblast 10.10.1.2 500 100 40 > > start: 1214854131.083679919 > finish: 1214854171.084668592 > send calls: 20139141 > send errors: 0 > approx send rate: 503478 > approx error rate: 0 > > > # ./netblast 10.10.1.3 500 10 40 > > start: 1214854273.882202815 > finish: 1214854313.882319031 > send calls: 23354971 > send errors: 18757223 > approx send rate: 114943 > approx error rate: 0 > > The box in the middle doing the forwarding > > 1[spare-r7]# ifconfig -u > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4> > ether 00:1b:21:08:32:a8 > inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255 > media: Ethernet autoselect (1000baseTX <full-duplex>) > status: active > em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM> > ether 00:1b:21:08:32:a9 > inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255 > media: Ethernet autoselect (100baseTX <full-duplex>) > status: active > em3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > > options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4> > ether 00:30:48:90:4c:ff > inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255 > media: Ethernet autoselect (1000baseTX <full-duplex>) > status: active > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > inet 127.0.0.1 netmask 0xff000000 > > > I am going to try a few more tests with and without, firewall rules > etc as well as an updated kernel to RELENG_7 as of today and see how > that goes. > > ---Mike > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?48694A9D.1030001>