From owner-freebsd-performance@FreeBSD.ORG Tue Dec 27 21:19:54 2005 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 916C816A41F for ; Tue, 27 Dec 2005 21:19:54 +0000 (GMT) (envelope-from mikej@rogers.com) Received: from smtp109.rog.mail.re2.yahoo.com (smtp109.rog.mail.re2.yahoo.com [68.142.225.207]) by mx1.FreeBSD.org (Postfix) with SMTP id 6904D43D75 for ; Tue, 27 Dec 2005 21:19:47 +0000 (GMT) (envelope-from mikej@rogers.com) Received: (qmail 49942 invoked from network); 27 Dec 2005 21:19:46 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=rogers.com; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:CC:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding; b=hfkjbm9cTmZkh5jBYyhqtX5g1In0IUlz4UJG6F/VOMZcSWcnXyc4YuvlNqcP61U5WjLTevLPgMAIpQxhskORPgictN2dUv4y4fvYqdqd7AzFu2BfGHd2KduixjZBH3PRChi0M6ECxQh22qml/qoujLRSP04+JW7q8oaK1kMmPFI= ; Received: from unknown (HELO ?70.30.133.190?) (mikej@rogers.com@70.30.133.190 with plain) by smtp109.rog.mail.re2.yahoo.com with SMTP; 27 Dec 2005 21:19:46 -0000 Message-ID: <43B1B001.2060007@rogers.com> Date: Tue, 27 Dec 2005 16:20:01 -0500 From: Mike Jakubik User-Agent: Thunderbird 1.5 (Windows/20051201) MIME-Version: 1.0 To: David Gilbert References: <43B05F1E.3060908@duras.ro> <17329.35594.458303.75695@canoe.dclg.ca> In-Reply-To: <17329.35594.458303.75695@canoe.dclg.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org, Mihai Tanasescu Subject: Re: Device polling heavy traffic X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Dec 2005 21:19:54 -0000 David Gilbert wrote: >>>>>> "Mihai" == Mihai Tanasescu writes: >>>>>> > > Mihai> Hello, I'm running the following setup: > > Mihai> Freebsd Dual Xeon 3Ghz machine (SMP enabled) > > Mihai> 3 x 100 Mbits/s links (used at 80% capacity) - 3 x Intel 100 > Mihai> fxp > > Mihai> 1 x 1000 Gbit link to a cisco router (transfers downstream the > Mihai> other 3 links) - 1 x Intel em > > Mihai> I'm getting something arround 100k pkt/sec input and 100k > Mihai> pkt/sec output as "systat -ip 1" shows. > > Mihai> Kernel polling is enabled. I have tried options HZ=1000, > Mihai> options HZ=2500 to see if anything changes. > > Mihai> The problem: If I ping this machine or anything that is routed > Mihai> through it I get response times of 10-15-30 ms and once in 30 > Mihai> seconds a packet is lost. > > Mihai> If I disable kernel.polling.enable then I get response times of > Mihai> 1-2-3 ms but I also get a lot of interrupts and a kernel panic > Mihai> after about 20 min. > > We've done a bunch of different experiments on various hardware and > various operating systems. 300 kpps of very small packets is about > the forwarding limit of FreeBSD ... with any hardware we can find. So > if your packets are non-trivial ... there is a lesser cost to packet > size than packet number ... then 200kpps will likely show some loss > due to FreeBSD's inability to forward more packets. > > Keep in mind that I've had two engineers spend months on this with > some guidance from me. > > Now a stock linux on the same hardware can handle about 500 kpps, but > there's a caveat. Linux hashes packet streams (key of sce ip, > src port, dest ip, dest port) for both routed and terminated traffic. > This hash has some advantages, but has a huge drawback. If you spray > more than than the hash size worth of streams at the linux box (even > if it's not routing), then it basically falls over. > > (not quite... packet performance goes from 500 kpps to less than > 10kpps and everything is hosed until it stops. Profiling shows it > spends 99% of it's time in the hash emptying and allocaing code) > > This means that a few megabit of fully random small packets will knock > over an arbitrary linux box. Every time. > > Not even SCO boxes are that lame (hah... :) > > My current thinking is that the hardware model has to change to get > better performance. I'm not really a hardware guy (hardware is evil), > but as I understand it, we're approaching limits of PCI busses. > Post the message enough times? Have you tried on a PCI Express bus?