From owner-freebsd-net@FreeBSD.ORG Wed Nov 21 17:32:44 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C9E5668F for ; Wed, 21 Nov 2012 17:32:44 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm16-vm1.bullet.mail.ne1.yahoo.com (nm16-vm1.bullet.mail.ne1.yahoo.com [98.138.91.47]) by mx1.freebsd.org (Postfix) with ESMTP id 76B708FC1D for ; Wed, 21 Nov 2012 17:32:44 +0000 (UTC) Received: from [98.138.90.51] by nm16.bullet.mail.ne1.yahoo.com with NNFMP; 21 Nov 2012 17:32:37 -0000 Received: from [98.138.89.244] by tm4.bullet.mail.ne1.yahoo.com with NNFMP; 21 Nov 2012 17:32:37 -0000 Received: from [127.0.0.1] by omp1058.mail.ne1.yahoo.com with NNFMP; 21 Nov 2012 17:32:37 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 670976.77800.bm@omp1058.mail.ne1.yahoo.com Received: (qmail 84192 invoked by uid 60001); 21 Nov 2012 17:32:37 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353519157; bh=GJ05ix2XNfPpal05Aq29cVwSlO6n7csQDUEOSmwxHMc=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=06KDAD4ZDexfLQN6bhA8ens66/gwgzvmz3vn9qeHF6k7R90Zt6uhN2Li1VtLt6WtXVTIWV7cs5V3RxuJ+0YgTQ8tLhrIAO434r8IxZ9ElnHBMODRcAhBrMeMkiwm1H8g7X0XGYeVZPM3rGMVr3iHPTTqEcMl4Xf9ZHbbdrc14vo= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=hP8x0iGmniNHWr02a+ZfRiZgdbtzWGTX8fPPXdx0l6nWL+FHFzGhE5cAEoZMISrpqijWxc1RHDhoRlrzxl81yilom6k58zJ/oFiJ6wdflvtv1mqbPNyBbLdhzFWBnJT/1aDyUsXjgzD3dpDcel+G+RK/1lsLEOCx9pJ/ZX/Ff6I=; X-YMail-OSG: 1j.5n5UVM1lskrkSclSNxZwz3fNXnQZGC0zRl5UXkFm8NtI xynSDKzL0MIzNQ3Vt5yD29DHE.goe5t5NCGNhepr0SR6YYAxAhXd7xZC8ThO 5SuW2tUad_NfV5Yn1_lCLjez79U2YNUf6IMchxi6ABDQJce9tTqfbkWF.5Bb L5isJxaO2pq2VFTyMRDWwjnuPdFQGhrFv5X8UQH_VcEpmEtLQUPsutLRI6FM 3qtqu90vdS7fh7kQaa_tvD4pGdP96vbosNl4l7f1GN1uv.F6pJGRDmXHn6YG flphEP73fJmxjomGUO3ijsAN8lYKCS7O849Ya7KE08Lwy4bVXvdcH6Pq1WDU YelktISsctdjXr1pHUop6ywsr71zqWUCUBnmMnzc39PV_C7d6entD1AX0K3x OR28Vy_tY4eJZFHu5ccCR9VNKVNimheZ3HqXEB3.R.NQZ76TpHX_3U_1K7yj yn5q1vMbU0a81ORiqJJiO4Xzs7BkPuSM0.PgUXAQYkVUDs2DrsENZuk_jZn8 E1B0muHb0.Dyl6p6A4RrirZrw7qdZNyV4czupTZV9pmeIhvDIMv_M45dTInE UfN1SSCtr5aMiLPEdhmH8NIJ.bE6rbddRpTHrud9ClLLm4MNeOwVlWREqao0 pHssx Received: from [174.48.128.27] by web121605.mail.ne1.yahoo.com via HTTP; Wed, 21 Nov 2012 09:32:37 PST X-Rocket-MIMEInfo: 001.001, CgotLS0gT24gV2VkLCAxMS8yMS8xMiwgSm9obiBGcmV0YnkgPGpmcmV0YnlAZ29vZ2xlbWFpbC5jb20.IHdyb3RlOgoKPiBGcm9tOiBKb2huIEZyZXRieSA8amZyZXRieUBnb29nbGVtYWlsLmNvbT4KPiBTdWJqZWN0OiBSZTogRnJlZUJTRCBib3hlcyBhcyBhICdyb3V0ZXInLi4uCj4gVG86ICJWaWN0b3IgQmFsYWRhIERpYXoiIDx2aWN0b3JAYnNkZXMubmV0Pgo.IENjOiBmcmVlYnNkLWlzcEBmcmVlYnNkLm9yZwo.IERhdGU6IFdlZG5lc2RheSwgTm92ZW1iZXIgMjEsIDIwMTIsIDExOjQwIEFNCj4gT24gMjEBMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.123.460 Message-ID: <1353519157.76595.YahooMailClassic@web121605.mail.ne1.yahoo.com> Date: Wed, 21 Nov 2012 09:32:37 -0800 (PST) From: Barney Cordoba Subject: Re: FreeBSD boxes as a 'router'... To: John Fretby In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:32:44 -0000 --- On Wed, 11/21/12, John Fretby wrote: > From: John Fretby > Subject: Re: FreeBSD boxes as a 'router'... > To: "Victor Balada Diaz" > Cc: freebsd-isp@freebsd.org > Date: Wednesday, November 21, 2012, 11:40 AM > On 21 November 2012 14:57, Victor > Balada Diaz > wrote: > > > > I think you forgot to CC the list. I'll add it so you > can get > > more answers. > > > > I did forget, thanks for that! :) > > > > em(4) and igb(4) are both drivers for Intel NICs. They > just have > > different capabilities. The sysctl you're asking for > controls behavior > > of adaptive interrupt moderation. It's a recommended > tuning for end hosts > > more than routers. You can read more about interrupt > moderation on this > > document: > > > > http://www.intel.com/design/network/applnots/ap450.htm > > > > em(4) NICs don't have all the capabilities of igb(4) > ones. Some em(4) NICs > > have > > interrupt moderation (eg: 82574L) but not all of them > do. If your em(4) > > card does > > have interrupt moderation you can tune it with: > > > > hw.em.rx_int_delay > > hw.em.rx_abs_int_delay > > hw.em.tx_int_delay > > hw.em.tx_abs_int_delay > > > > Exchanging latency to get more throughput. > > > > You can take a look at this document explaining > capabilities of different > > NICs: > > > > > > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phys-brochure.pdf > > > > You should ask supermicro what's the exact model > they'll put on your server > > and then decide if it's OK for you. > > > They are apparently: > > em0: port > 0xf020-0xf03f mem > 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device > 25.0 on pci0 > em0: Using an MSI interrupt > ... > em0: flags=8c02 > metric 0 mtu 1500 > options=4219b > > > > About the interrupt storm: We've had various interrupt > storms that were > > caused by > > different problems. The most common was a software bug > with interrupts. > > After > > reporting on the lists it was fixed and we didn't have > problems again. > > > > If you have a problem with high interrupts because too > many small packets > > (eg a DoS), > > getting a card with interrupt moderation should help a > lot. Most probably > > your problem > > with interrupt storms was caused by something else like > a shared interrupt > > with other > > device or software bug. Without more analysis it's > impossible to really > > say. > > > > I have some details from when it happened - it doesn't look > like it was a > shared interrupt issue - it just literally looks like the > host came up, > with a stampeding hurd of "other" hosts hitting it for > services that > weren't yet running, and it folded :( > > That's why I was wondering if there was a similar sysctl for > the em driver > - in order to raise the number of interrupts the system > allows, before > declaring it "a storm". > > > > > > Keep in mind that i'm not an expert on this area, so > you might get better > > answers > > on frebsd-net@ :) > > > > Hope it helps. > > > > It has - half the problem is there are *so* many options, > combinations - > and no matter what you pick, if you look them up enough > you'll find someone > finding fault with them, or casting doubts on their > performance. > > Doesn't really help when all you want is something that has > a good chance > of "working" :) > The road to mediocrity is listening to people who are not experts. The admission that you dont understand something is a good disclosure. But the argument than understanding how to tune a system is too complicated only means that I don't want to listen to what you have to say. Polling implies that there are unnatural intervals between processing packets. So you're introducing delay into your network. The more often you poll, the less the gap. Suppose you poll every 1ms. Each received packet will have from 0-1ms of delay. Packets that are arriving as you ended you last poll will be delayed 1ms. The more knobs you have, the more you can determine what the system can do. An interrupt for every packet would have the least delay, but that's not practical for systems managing 100s of 1000s of pps. You make trade offs between delays and cpu usage. On a small network with a fast cpu, you can process every packet. On a very large network you cant. The idea that there is one way to tune a system for every environment is simply wrong. A bridge/filter has completely different requirements than a "router". And a "server" which only uses 1 NIC has different requirements than a system that forwards traffic and has to manage more locks at the hardware level. BC