From owner-freebsd-isp@FreeBSD.ORG Wed Nov 21 18:00:08 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 349A3130 for ; Wed, 21 Nov 2012 18:00:08 +0000 (UTC) (envelope-from vandj@securenet.net) Received: from ms.securenet.net (ms.securenet.net [205.236.147.20]) by mx1.freebsd.org (Postfix) with ESMTP id DDA298FC12 for ; Wed, 21 Nov 2012 18:00:07 +0000 (UTC) Received: from ms.securenet.net (localhost [127.0.0.1]) by ms.securenet.net (Postfix) with ESMTP id 66FAFBA9F for ; Wed, 21 Nov 2012 12:52:48 -0500 (EST) X-Virus-Scanned: by f-prot anti-virus, and clamav anti-virus at SecureNet Inc. Received: from ms.securenet.net ([127.0.0.1]) by ms.securenet.net (ms.securenet.net [127.0.0.1]) (amavisd-new, port 10025) with ESMTP id oh4MX13AjB2j for ; Wed, 21 Nov 2012 12:52:48 -0500 (EST) Received: from HomePC (mtl-pppoe-adsl245.securenet.net [66.38.236.245]) by ms.securenet.net (Postfix) with ESMTP id 04028B9E1 for ; Wed, 21 Nov 2012 12:52:48 -0500 (EST) From: "Jean M Vandette" To: References: <20121120174554.GC50873@equilibrium.bsdes.net> <20121121145758.GD50873@equilibrium.bsdes.net> In-Reply-To: Subject: RE: FreeBSD boxes as a 'router'... Date: Wed, 21 Nov 2012 12:52:47 -0500 Message-ID: <04f401cdc811$04207e50$0c617af0$@net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac3IBvwwsX6k/2THT3+fa+eaIQN8qAAAOKZA Content-Language: en-us X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:00:08 -0000 Greetings all. We did allot of testing a few years ago with and without polling at that time what we had found it was not the nic's it was the bus that locked and caused a problem. The best we were able to get was 80~90MB throughput before the box would lock and reboot. We also found a single CPU ran circles around a dual CPU for this purpose. At that time we were running 3 BGP sessions so lots of memory was needed to hold the 240K routing table entries (at the time). CPU load was never an issue. We were unsuccessful and moved to a Juniper routing platform, we just could not get fast enough I/O to the bus. I know allot of the hardware out there is much faster now a days and cheaper. The best advice I can give you is build a current state-of-the-art machine bus speed is the biggest issue, faster is better, I/O is critical, you can have the fastest bus out there but if all you have is a PCI slot for example your dead in the water, it's just not fast enough. You cannot have a bottle neck. You need to check with the mfg for the bus speed of the nic. Just because the nic is built in many times it is just to save a expansion slot, the speed to the bus is not as fast as you might expect. Once you get the fastest of I/O to a fast bus put a current release and then hammer it with tests to tweak it. Expect high interrupt rates if not using polling. Once your results are what you expect and stable put it into production. You have had allot of good tweaks and info given to you over the last few days so I won't repeat them. Good luck Jean M. Vandette SecureNet Information Services -----Original Message----- From: owner-freebsd-isp@freebsd.org [mailto:owner-freebsd-isp@freebsd.org] On Behalf Of John Fretby Sent: Wednesday, November 21, 2012 11:41 AM To: Victor Balada Diaz Cc: freebsd-isp@freebsd.org Subject: Re: FreeBSD boxes as a 'router'... On 21 November 2012 14:57, Victor Balada Diaz wrote: > I think you forgot to CC the list. I'll add it so you can get more > answers. > I did forget, thanks for that! :) > em(4) and igb(4) are both drivers for Intel NICs. They just have > different capabilities. The sysctl you're asking for controls behavior > of adaptive interrupt moderation. It's a recommended tuning for end > hosts more than routers. You can read more about interrupt moderation > on this > document: > > http://www.intel.com/design/network/applnots/ap450.htm > > em(4) NICs don't have all the capabilities of igb(4) ones. Some em(4) > NICs have interrupt moderation (eg: 82574L) but not all of them do. If > your em(4) card does have interrupt moderation you can tune it with: > > hw.em.rx_int_delay > hw.em.rx_abs_int_delay > hw.em.tx_int_delay > hw.em.tx_abs_int_delay > > Exchanging latency to get more throughput. > > You can take a look at this document explaining capabilities of > different > NICs: > > > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phy > s-brochure.pdf > > You should ask supermicro what's the exact model they'll put on your > server and then decide if it's OK for you. They are apparently: em0: port 0xf020-0xf03f mem 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt ... em0: flags=8c02 metric 0 mtu 1500 options=4219b > About the interrupt storm: We've had various interrupt storms that > were caused by different problems. The most common was a software bug > with interrupts. > After > reporting on the lists it was fixed and we didn't have problems again. > > If you have a problem with high interrupts because too many small > packets (eg a DoS), getting a card with interrupt moderation should > help a lot. Most probably your problem with interrupt storms was > caused by something else like a shared interrupt with other device or > software bug. Without more analysis it's impossible to really say. > I have some details from when it happened - it doesn't look like it was a shared interrupt issue - it just literally looks like the host came up, with a stampeding hurd of "other" hosts hitting it for services that weren't yet running, and it folded :( That's why I was wondering if there was a similar sysctl for the em driver - in order to raise the number of interrupts the system allows, before declaring it "a storm". > > Keep in mind that i'm not an expert on this area, so you might get > better answers on frebsd-net@ :) > > Hope it helps. > It has - half the problem is there are *so* many options, combinations - and no matter what you pick, if you look them up enough you'll find someone finding fault with them, or casting doubts on their performance. Doesn't really help when all you want is something that has a good chance of "working" :) -Jon _______________________________________________ freebsd-isp@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org"