From owner-freebsd-stable@FreeBSD.ORG Fri Oct 8 15:20:08 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DDB2716A4CE; Fri, 8 Oct 2004 15:20:07 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 61CBB43D1F; Fri, 8 Oct 2004 15:20:07 +0000 (GMT) (envelope-from scottl@FreeBSD.org) Received: from [192.168.254.11] (junior-wifi.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.12.11/8.12.10) with ESMTP id i98FJx5h030928; Fri, 8 Oct 2004 09:19:59 -0600 (MDT) (envelope-from scottl@FreeBSD.org) Message-ID: <4166AFE0.5060000@FreeBSD.org> Date: Fri, 08 Oct 2004 09:18:56 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.2) Gecko/20040831 X-Accept-Language: en-us, en MIME-Version: 1.0 To: David Gilbert References: <16742.40802.187425.402461@canoe.dclg.ca> In-Reply-To: <16742.40802.187425.402461@canoe.dclg.ca> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, hits=0.0 required=3.8 tests=none autolearn=no version=2.63 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on pooker.samsco.org cc: freebsd-current@FreeBSD.org cc: freebsd-stable@FreeBSD.org Subject: Re: Packet passing performance study on exotic hardware. X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Oct 2004 15:20:08 -0000 David Gilbert wrote: > The opportunity presented itelf for me to test packet passing ability > on some fairly exotic hardware. The motherboard I really wanted to > test not only had separate memory busses for each cpu, but also had > two separate PCI-X busses (one slot each). To this, I added two > intel pro/1000 gigabit ethernet cards (PCI-X versions). > > I had two sets of processors to test: two 246's and two 240's. > > The packets in this test are all minimal 64 byte UDP packets. > > My first goal was to determine the DDOS stability of FreeBSD 5.3, and > Linux on this hardware. I was using amd64 binaries for both FreeBSD > and linux. > > Right out of the box (with polling), Linux passed 550 kpps (kilo > packets wer second). Full data rate would be 1.9 mpps. On linux, the > 240 processors passed only 450 kppps (which is somewhat expected). > > Right out of the box, FreeBSD 5.3 (with polling) passed about 200 > kpps. net.isr.enable=1 increased that without polling to about 220 > kpps (although livelock ensued without polling as packet load > increased). With excessive tuning, we got FreeBSD 5.3 to pass 270 > kpps. This included polling, nmbclusters, net.isr, and some em > patches. I can't see where to get more performance. > > To compare, we loaded FreeBSD-5.3 ia32 and achieved almost identical > performance. > > Then, also to compare, we loaded FreeBSD-4.10 ia32 and it promptly > passed 550 kpps (almost identical to the linux performance) (with > polling). > > Some interesting things about 5.3(-BETA4) in this environment: > > - without polling, it definately livelocks. > > - with polling and excessive packets, it doesn't "receive" the full > load of packets. In netstat -w, they show as input "errors" > although the number of "errors" isn't strictly related to the > number of dropped packets. It's just some large number that > generally increases with the number of dropped packets. > > - With net.isr and not polling, both cpus are used (220 kpps) > > - With net.isr and polling, one cpu is used (270 kpps, one cpu free > for other tasks) > > - It's worth noting that only FreeBSD 5.3 used two cpus to pass > packets at any time. Neither linux nor 4.10 used the other cpu. > > - hz and polling tuning options didn't really change packets passed > significantly. > > During the next week, I will continue testing with full simulated > routing tables, random packets and packets between 350 and 550 bytes > (average ISP out/in packet sizes). I will add to this report then. > If anyone has tuning advice for FreeBSD 5.3, I'd like to hear it. > > Dave. > Interesting results. One thing to note is that a severe bug in the if_em driver was fixed for BETA7. The symptoms of this bug include apparent livelock of the machine during heavy xmit load. You might want to update and re-run your tests. Scott