From owner-freebsd-isp@FreeBSD.ORG Wed Nov 21 16:40:44 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 53B639C5 for ; Wed, 21 Nov 2012 16:40:44 +0000 (UTC) (envelope-from jfretby@googlemail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 02F6C8FC15 for ; Wed, 21 Nov 2012 16:40:43 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so6333529qcs.13 for ; Wed, 21 Nov 2012 08:40:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=C6sFvCkT6RZzTxEJcMYk6fb9tO/YGu6WQU+uTsV8+tQ=; b=Pkaqrdrrfm+x7QXwvApUufV91GFFG3JAMGZudxuOGESgXczVdK71WSXfUtVQvfxsLe 55adFlRudC1dSHcYKmmRycasjNt2TaZ3SctuEUq1/bZvNusmPWWOg4XbE/TyTqteMPCJ uGLNvqCiyIfnECNRsqDKbqfKCddIEUWCkP2TJZKM0eXdzN545p4b9oh7Njbve6BDapoW GHqGWZhNZjH8BVogaXfpFfTonU5i0dm+JHmOZys6pdUXKNK1YA1g98sEdOsCGP4O8F+3 mBYuapt38/UQ/gimkhCy3SKouxeWOb8hkSuGSFnYkXULns5C9s+Pz+nCy570ObaROLDj Pnag== MIME-Version: 1.0 Received: by 10.224.42.15 with SMTP id q15mr18745775qae.68.1353516043204; Wed, 21 Nov 2012 08:40:43 -0800 (PST) Received: by 10.49.25.13 with HTTP; Wed, 21 Nov 2012 08:40:42 -0800 (PST) In-Reply-To: <20121121145758.GD50873@equilibrium.bsdes.net> References: <20121120174554.GC50873@equilibrium.bsdes.net> <20121121145758.GD50873@equilibrium.bsdes.net> Date: Wed, 21 Nov 2012 16:40:42 +0000 Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: John Fretby To: Victor Balada Diaz Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-isp@freebsd.org X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 16:40:44 -0000 On 21 November 2012 14:57, Victor Balada Diaz wrote: > I think you forgot to CC the list. I'll add it so you can get > more answers. > I did forget, thanks for that! :) > em(4) and igb(4) are both drivers for Intel NICs. They just have > different capabilities. The sysctl you're asking for controls behavior > of adaptive interrupt moderation. It's a recommended tuning for end hosts > more than routers. You can read more about interrupt moderation on this > document: > > http://www.intel.com/design/network/applnots/ap450.htm > > em(4) NICs don't have all the capabilities of igb(4) ones. Some em(4) NICs > have > interrupt moderation (eg: 82574L) but not all of them do. If your em(4) > card does > have interrupt moderation you can tune it with: > > hw.em.rx_int_delay > hw.em.rx_abs_int_delay > hw.em.tx_int_delay > hw.em.tx_abs_int_delay > > Exchanging latency to get more throughput. > > You can take a look at this document explaining capabilities of different > NICs: > > > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phys-brochure.pdf > > You should ask supermicro what's the exact model they'll put on your server > and then decide if it's OK for you. They are apparently: em0: port 0xf020-0xf03f mem 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt ... em0: flags=8c02 metric 0 mtu 1500 options=4219b > About the interrupt storm: We've had various interrupt storms that were > caused by > different problems. The most common was a software bug with interrupts. > After > reporting on the lists it was fixed and we didn't have problems again. > > If you have a problem with high interrupts because too many small packets > (eg a DoS), > getting a card with interrupt moderation should help a lot. Most probably > your problem > with interrupt storms was caused by something else like a shared interrupt > with other > device or software bug. Without more analysis it's impossible to really > say. > I have some details from when it happened - it doesn't look like it was a shared interrupt issue - it just literally looks like the host came up, with a stampeding hurd of "other" hosts hitting it for services that weren't yet running, and it folded :( That's why I was wondering if there was a similar sysctl for the em driver - in order to raise the number of interrupts the system allows, before declaring it "a storm". > > Keep in mind that i'm not an expert on this area, so you might get better > answers > on frebsd-net@ :) > > Hope it helps. > It has - half the problem is there are *so* many options, combinations - and no matter what you pick, if you look them up enough you'll find someone finding fault with them, or casting doubts on their performance. Doesn't really help when all you want is something that has a good chance of "working" :) -Jon