From owner-freebsd-isp@FreeBSD.ORG Tue Nov 20 15:35:14 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A63FCF0C for ; Tue, 20 Nov 2012 15:35:14 +0000 (UTC) (envelope-from jfretby@googlemail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 530D08FC12 for ; Tue, 20 Nov 2012 15:35:13 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so5154938qcs.13 for ; Tue, 20 Nov 2012 07:35:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=vzkdWNqkn5VpU6VCMBCvWSnt2v2WQdvo/5MjeMOQ6yw=; b=f7IbwCsVfUVIaLRXHHx1FCix9G+jB79owyhiOifHJum2Nl0R6sEzITqyJaLPpYHpAm C1mzd14I/0wAPUa3jc6rc1HUE91keFHFK1wpuJi2xB9TulhA9FkpCpQLqv1j6laZykNy W8ZUhBd+Zs43bOlmGLEcAMEn+Gdp55sFr9i96YIFFpTIP+R+mzWFyQdhCqcGf4LkBhwQ 94B1E+XAkPXZqUyxaNqcv4EMWdgDM8ZtjpQ00TU7JETk9NPFwrPEZIFjEC3YCd40NG+m Vh62UUAaynQi7AfOtPVzL8GgXzmjnrh0kjuJyfAWW8c8iz+8N0tKGr837Js0fsD0QGa0 GvZA== MIME-Version: 1.0 Received: by 10.229.114.201 with SMTP id f9mr268147qcq.33.1353425713459; Tue, 20 Nov 2012 07:35:13 -0800 (PST) Received: by 10.49.25.13 with HTTP; Tue, 20 Nov 2012 07:35:13 -0800 (PST) Date: Tue, 20 Nov 2012 15:35:13 +0000 Message-ID: Subject: FreeBSD boxes as a 'router'... From: John Fretby To: freebsd-isp@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 15:35:14 -0000 Howdy all, We've currently got an ageing HP DL360 running as a 'router' - it has 100Mbit in/out onto our network, and has two 'bce' NIC's providing in/out. It's running quite an old version of FreeBSD (6 I think) - but works. As the network gets busier we've noticed the amount of interrupt time on it is climbing (as you'd expect - i.e. esp. if many small packets are being forwarded). Many moons ago we did experiment with this box - and enabled device polling (inc. upping the HZ on the box and recompiling the kernel etc). This didn't work very well at the time (probably because it was in it's infancy) so we left it off in the end. If we were to replace this box, with something new - say a SuperMicro based system with two: Intel 82574L's (em Driver Based) And enable polling - is it likely to "just work" these days? The current upstream is 100Mbit, we're looking to upgrade this to 1Gbit in, but with say 200Mbit comitted on it (so shouldn't go above 200Mbit). Is there anything that has to be done to enable polling - other than recompiling GENERIC to support it? - i.e. no HZ hacks or anything needed on 'modern' machines (it's a quad core Xeon). Cheers, Jon. From owner-freebsd-isp@FreeBSD.ORG Tue Nov 20 16:45:32 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 21D9CE5A for ; Tue, 20 Nov 2012 16:45:32 +0000 (UTC) (envelope-from edinilson@atinet.com.br) Received: from nolver.com.br (ntserver3.atinet.com.br [187.0.230.10]) by mx1.freebsd.org (Postfix) with ESMTP id 6E9088FC08 for ; Tue, 20 Nov 2012 16:45:29 +0000 (UTC) Received: from suporte9 ([187.0.230.48]:2622) by ntserver3.nolver.com.br with [XMail 1.27 ESMTP Server] id for from ; Tue, 20 Nov 2012 14:45:05 -0200 Message-ID: <7D5F47372DCD43F3B766C27844695A94@suporte9> From: "Edinilson - ATINET" To: References: Subject: Re: FreeBSD boxes as a 'router'... Date: Tue, 20 Nov 2012 14:45:05 -0200 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: John Fretby X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 16:45:32 -0000 Hi, I´m using Freebsd as a 'router' since version 4.3 (after this, 5.x, 6.x, 7.x). Now we are using FreeBSD 9.0-STABLE 64 bits on a Intel Server Boards S3200SH/S3210SH with 16gb RAM, 500gb SATA disk. In these years, we always use SuperMicro´s OR Intel´s Motherboards and Lan Cards from Intel (fxp and em) together with Freebsd. But, even using Intel Lan cards, we *NEVER* would be able to use polling without any kind of problem. Our upstream is 140Mbit, and even in our very busy ISP environment (BGP Full Routing+several ipfw filters+several graphs being generated), this machine + FreeBSD 9 can acomplish the task without problems. ps: You could try CPU AFFINITY in the new FreeBSD kernel version to get best results (instead of polling): http://segfault.in/2010/09/how-to-set-cpu-affinity-for-a-process-in-freebsd/ Good luck! Regards Edinilson ------------------------------------------ ATINET Tel Voz: (0xx11) 4412-0876 http://www.atinet.com.br ----- Original Message ----- From: "John Fretby" To: Sent: Tuesday, November 20, 2012 1:35 PM Subject: FreeBSD boxes as a 'router'... > Howdy all, > > We've currently got an ageing HP DL360 running as a 'router' - it has > 100Mbit in/out onto our network, and has two 'bce' NIC's providing in/out. > It's running quite an old version of FreeBSD (6 I think) - but works. > > As the network gets busier we've noticed the amount of interrupt time on > it > is climbing (as you'd expect - i.e. esp. if many small packets are being > forwarded). Many moons ago we did experiment with this box - and enabled > device polling (inc. upping the HZ on the box and recompiling the kernel > etc). This didn't work very well at the time (probably because it was in > it's infancy) so we left it off in the end. > > If we were to replace this box, with something new - say a SuperMicro > based > system with two: > > Intel 82574L's (em Driver Based) > > And enable polling - is it likely to "just work" these days? The current > upstream is 100Mbit, we're looking to upgrade this to 1Gbit in, but with > say 200Mbit comitted on it (so shouldn't go above 200Mbit). > > Is there anything that has to be done to enable polling - other than > recompiling GENERIC to support it? - i.e. no HZ hacks or anything needed > on > 'modern' machines (it's a quad core Xeon). > > Cheers, > > Jon. > _______________________________________________ > freebsd-isp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-isp > To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" > From owner-freebsd-isp@FreeBSD.ORG Tue Nov 20 17:46:03 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BA533B2B for ; Tue, 20 Nov 2012 17:46:03 +0000 (UTC) (envelope-from victor@bsdes.net) Received: from equilibrium.bsdes.net (244.Red-217-126-240.staticIP.rima-tde.net [217.126.240.244]) by mx1.freebsd.org (Postfix) with ESMTP id 581158FC12 for ; Tue, 20 Nov 2012 17:46:02 +0000 (UTC) Received: by equilibrium.bsdes.net (Postfix, from userid 1001) id ED15539832; Tue, 20 Nov 2012 18:45:54 +0100 (CET) Date: Tue, 20 Nov 2012 18:45:54 +0100 From: Victor Balada Diaz To: John Fretby Subject: Re: FreeBSD boxes as a 'router'... Message-ID: <20121120174554.GC50873@equilibrium.bsdes.net> References: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-isp@freebsd.org X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 17:46:03 -0000 On Tue, Nov 20, 2012 at 03:35:13PM +0000, John Fretby wrote: > Howdy all, > > We've currently got an ageing HP DL360 running as a 'router' - it has > 100Mbit in/out onto our network, and has two 'bce' NIC's providing in/out. > It's running quite an old version of FreeBSD (6 I think) - but works. > > As the network gets busier we've noticed the amount of interrupt time on it > is climbing (as you'd expect - i.e. esp. if many small packets are being > forwarded). Many moons ago we did experiment with this box - and enabled > device polling (inc. upping the HZ on the box and recompiling the kernel > etc). This didn't work very well at the time (probably because it was in > it's infancy) so we left it off in the end. > > If we were to replace this box, with something new - say a SuperMicro based > system with two: > > Intel 82574L's (em Driver Based) > > And enable polling - is it likely to "just work" these days? The current > upstream is 100Mbit, we're looking to upgrade this to 1Gbit in, but with > say 200Mbit comitted on it (so shouldn't go above 200Mbit). > > Is there anything that has to be done to enable polling - other than > recompiling GENERIC to support it? - i.e. no HZ hacks or anything needed on > 'modern' machines (it's a quad core Xeon). Hello John, You might find interesting to read this thread: http://lists.freebsd.org/pipermail/freebsd-current/2012-November/037590.html In short: device polling can decrease performance on modern hardware. You might want to try upgrading to a new FreeBSD version and tuning it somehow before buying a new server. More info on tuning the network stack: http://wiki.freebsd.org/NetworkPerformanceTuning Regards. Victor. -- La prueba más fehaciente de que existe vida inteligente en otros planetas, es que no han intentado contactar con nosotros. From owner-freebsd-isp@FreeBSD.ORG Tue Nov 20 18:06:41 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id DA56F4A0 for ; Tue, 20 Nov 2012 18:06:41 +0000 (UTC) (envelope-from khatfield@socllc.net) Received: from smtp126.dfw.emailsrvr.com (smtp126.dfw.emailsrvr.com [67.192.241.126]) by mx1.freebsd.org (Postfix) with ESMTP id 9EBCD8FC08 for ; Tue, 20 Nov 2012 18:06:41 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp2.relay.dfw1a.emailsrvr.com (SMTP Server) with ESMTP id D8EBB78191; Tue, 20 Nov 2012 12:57:08 -0500 (EST) X-Virus-Scanned: OK Received: by smtp2.relay.dfw1a.emailsrvr.com (Authenticated sender: khatfield-AT-socllc.net) with ESMTPSA id 6EFFC783EE; Tue, 20 Nov 2012 12:57:08 -0500 (EST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 Subject: Re: FreeBSD boxes as a 'router'... References: <20121120174554.GC50873@equilibrium.bsdes.net> From: khatfield@socllc.net Mime-Version: 1.0 In-Reply-To: <20121120174554.GC50873@equilibrium.bsdes.net> Message-Id: <1281530059.17550.1353434227450@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 11:57:05 -0600 To: Victor Balada Diaz X-NS-Received: from Apple-iPhone5C2/1001.525(khatfield@socllc.net) SECURED(HTTPS); Tue, 20 Nov 2012 17:57:05 +0000 (UTC) Cc: "freebsd-isp@freebsd.org" , John Fretby X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 18:06:42 -0000 T25lIHRoaW5nIEkgaGF2ZSBub3RpY2VkIGlzIG1peGVkIHJlc3VsdHMgd2l0aCBwb2xsaW5nIGRl cGVuZGluZyBvbiB0aGUgdmVyc2lvbi4NCg0KTXkgZXhwZXJpZW5jZSB3aXRoIHNpbWlsYXIgTklD cyB3YXMgdGhhdCBwb2xsaW5nIGluY3JlYXNlZCB0aGUgUFBTIGNhcGFiaWxpdGllcyB1cCB0byA3 LjQgYnV0IHBvc3QgNy40IHdlIGhhdmUgc2VlbiBtb3N0IGNhc2VzIHdoZXJlIHBvbGxpbmcgY2F1 c2VkIGVpdGhlciBjb25uZWN0aXZpdHkgaXNzdWVzIG9yIGRlY3JlYXNlZCBvdmVyYWxsIHBlcmZv cm1hbmNlLg0KDQpOb3cgd2Ugd2VyZSBydW5uaW5nIGZ1bGwgMUdicHMgaW4gb3VyIHRlc3RzLiBX aXRoIG9ubHkgMTQwTWJwcyB5b3Ugc2hvdWxkIGJlIGFibGUgdG8gaGFuZGxlIHRoaXMgYW1vdW50 IHdpdGhvdXQgcG9sbGluZyBvciBhZGRpdGlvbmFsIGtlcm5lbCB0d2Vha3MuIFNwZWNpZmljYWxs eSB3aXRoIDkgLSBJIHdvdWxkIHJlY29tbWVuZCBkb2luZyBuZWVkZWQgc3lzY3RsIHR3ZWFrcyB3 aXRob3V0IHBvbGxpbmcgYW5kIGFzIGxvbmcgYXMgeW91IGFyZSBub3QgcmVjZWl2aW5nIEREb1Mg dHJhZmZpYyB0aGVuIGl0IHNob3VsZCBwcm92ZSBwZXJmZWN0bHkgc3RhYmxlLg0KDQoNCg0KT24g Tm92IDIwLCAyMDEyLCBhdCAxMTo0NiBBTSwgIlZpY3RvciBCYWxhZGEgRGlheiIgPHZpY3RvckBi c2Rlcy5uZXQ+IHdyb3RlOg0KDQo+IE9uIFR1ZSwgTm92IDIwLCAyMDEyIGF0IDAzOjM1OjEzUE0g KzAwMDAsIEpvaG4gRnJldGJ5IHdyb3RlOg0KPj4gSG93ZHkgYWxsLA0KPj4gDQo+PiBXZSd2ZSBj dXJyZW50bHkgZ290IGFuIGFnZWluZyBIUCBETDM2MCBydW5uaW5nIGFzIGEgJ3JvdXRlcicgLSBp dCBoYXMNCj4+IDEwME1iaXQgaW4vb3V0IG9udG8gb3VyIG5ldHdvcmssIGFuZCBoYXMgdHdvICdi Y2UnIE5JQydzIHByb3ZpZGluZyBpbi9vdXQuDQo+PiBJdCdzIHJ1bm5pbmcgcXVpdGUgYW4gb2xk IHZlcnNpb24gb2YgRnJlZUJTRCAoNiBJIHRoaW5rKSAtIGJ1dCB3b3Jrcy4NCj4+IA0KPj4gQXMg dGhlIG5ldHdvcmsgZ2V0cyBidXNpZXIgd2UndmUgbm90aWNlZCB0aGUgYW1vdW50IG9mIGludGVy cnVwdCB0aW1lIG9uIGl0DQo+PiBpcyBjbGltYmluZyAoYXMgeW91J2QgZXhwZWN0IC0gaS5lLiBl c3AuIGlmIG1hbnkgc21hbGwgcGFja2V0cyBhcmUgYmVpbmcNCj4+IGZvcndhcmRlZCkuIE1hbnkg bW9vbnMgYWdvIHdlIGRpZCBleHBlcmltZW50IHdpdGggdGhpcyBib3ggLSBhbmQgZW5hYmxlZA0K Pj4gZGV2aWNlIHBvbGxpbmcgKGluYy4gdXBwaW5nIHRoZSBIWiBvbiB0aGUgYm94IGFuZCByZWNv bXBpbGluZyB0aGUga2VybmVsDQo+PiBldGMpLiBUaGlzIGRpZG4ndCB3b3JrIHZlcnkgd2VsbCBh dCB0aGUgdGltZSAocHJvYmFibHkgYmVjYXVzZSBpdCB3YXMgaW4NCj4+IGl0J3MgaW5mYW5jeSkg c28gd2UgbGVmdCBpdCBvZmYgaW4gdGhlIGVuZC4NCj4+IA0KPj4gSWYgd2Ugd2VyZSB0byByZXBs YWNlIHRoaXMgYm94LCB3aXRoIHNvbWV0aGluZyBuZXcgLSBzYXkgYSBTdXBlck1pY3JvIGJhc2Vk DQo+PiBzeXN0ZW0gd2l0aCB0d286DQo+PiANCj4+ICAgSW50ZWwgODI1NzRMJ3MgKGVtIERyaXZl ciBCYXNlZCkNCj4+IA0KPj4gQW5kIGVuYWJsZSBwb2xsaW5nIC0gaXMgaXQgbGlrZWx5IHRvICJq dXN0IHdvcmsiIHRoZXNlIGRheXM/IFRoZSBjdXJyZW50DQo+PiB1cHN0cmVhbSBpcyAxMDBNYml0 LCB3ZSdyZSBsb29raW5nIHRvIHVwZ3JhZGUgdGhpcyB0byAxR2JpdCBpbiwgYnV0IHdpdGgNCj4+ IHNheSAyMDBNYml0IGNvbWl0dGVkIG9uIGl0IChzbyBzaG91bGRuJ3QgZ28gYWJvdmUgMjAwTWJp dCkuDQo+PiANCj4+IElzIHRoZXJlIGFueXRoaW5nIHRoYXQgaGFzIHRvIGJlIGRvbmUgdG8gZW5h YmxlIHBvbGxpbmcgLSBvdGhlciB0aGFuDQo+PiByZWNvbXBpbGluZyBHRU5FUklDIHRvIHN1cHBv cnQgaXQ/IC0gaS5lLiBubyBIWiBoYWNrcyBvciBhbnl0aGluZyBuZWVkZWQgb24NCj4+ICdtb2Rl cm4nIG1hY2hpbmVzIChpdCdzIGEgcXVhZCBjb3JlIFhlb24pLg0KPiANCj4gSGVsbG8gSm9obiwN Cj4gDQo+IFlvdSBtaWdodCBmaW5kIGludGVyZXN0aW5nIHRvIHJlYWQgdGhpcyB0aHJlYWQ6DQo+ IA0KPiBodHRwOi8vbGlzdHMuZnJlZWJzZC5vcmcvcGlwZXJtYWlsL2ZyZWVic2QtY3VycmVudC8y MDEyLU5vdmVtYmVyLzAzNzU5MC5odG1sDQo+IA0KPiBJbiBzaG9ydDogZGV2aWNlIHBvbGxpbmcg Y2FuIGRlY3JlYXNlIHBlcmZvcm1hbmNlIG9uIG1vZGVybiBoYXJkd2FyZS4NCj4gDQo+IFlvdSBt aWdodCB3YW50IHRvIHRyeSB1cGdyYWRpbmcgdG8gYSBuZXcgRnJlZUJTRCB2ZXJzaW9uIGFuZCB0 dW5pbmcgaXQgc29tZWhvdw0KPiBiZWZvcmUgYnV5aW5nIGEgbmV3IHNlcnZlci4gTW9yZSBpbmZv IG9uIHR1bmluZyB0aGUgbmV0d29yayBzdGFjazoNCj4gDQo+IGh0dHA6Ly93aWtpLmZyZWVic2Qu b3JnL05ldHdvcmtQZXJmb3JtYW5jZVR1bmluZw0KPiANCj4gUmVnYXJkcy4NCj4gVmljdG9yLg0K PiAtLSANCj4gTGEgcHJ1ZWJhIG3DoXMgZmVoYWNpZW50ZSBkZSBxdWUgZXhpc3RlIHZpZGEgaW50 ZWxpZ2VudGUgZW4gb3Ryb3MNCj4gcGxhbmV0YXMsIGVzIHF1ZSBubyBoYW4gaW50ZW50YWRvIGNv bnRhY3RhciBjb24gbm9zb3Ryb3MuIA0KPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXw0KPiBmcmVlYnNkLWlzcEBmcmVlYnNkLm9yZyBtYWlsaW5nIGxpc3QN Cj4gaHR0cDovL2xpc3RzLmZyZWVic2Qub3JnL21haWxtYW4vbGlzdGluZm8vZnJlZWJzZC1pc3AN Cj4gVG8gdW5zdWJzY3JpYmUsIHNlbmQgYW55IG1haWwgdG8gImZyZWVic2QtaXNwLXVuc3Vic2Ny aWJlQGZyZWVic2Qub3JnIg0K From owner-freebsd-isp@FreeBSD.ORG Tue Nov 20 19:21:37 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6990F1AE for ; Tue, 20 Nov 2012 19:21:37 +0000 (UTC) (envelope-from outbackdingo@gmail.com) Received: from mail-ie0-f182.google.com (mail-ie0-f182.google.com [209.85.223.182]) by mx1.freebsd.org (Postfix) with ESMTP id 25ED38FC14 for ; Tue, 20 Nov 2012 19:21:37 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id s9so3298491iec.13 for ; Tue, 20 Nov 2012 11:21:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9MnT68T03oXjXZQFQJ5MFyN2vyuUr8V6zfQXvk4EPNM=; b=zLsPvYhQLiseoyKnYVUaS81XyNhTCYa7AZqb1R76YNjVJJmmj6W+UJ9WcHliGWvvPa w5ipjzv0PmURdXpPB/lIykNX2125WiN5SCu7GfFblHCbCCb8cr9frRdnYPdbR8t/5w2r CZXhrYRGtoYMTOTKFkXO2Ulr+/4yuHw0i+uXi1Sy0JFh9mT8K3UM2xKd74DywEmt/hrD RsN05YObuE7Ky0Voq0dz2BRN04SpOpHzA3CXiz7P+eW3/CqC2L5wL6X680Xir7i0MLVw C2sreryVH/Re1wjzX3XV0NNOTHZp9tZlVGESthHeoliYDgiSJalD3KWl1g5Mxy4Av/Tn Aceg== MIME-Version: 1.0 Received: by 10.43.69.206 with SMTP id yd14mr14938268icb.57.1353439296485; Tue, 20 Nov 2012 11:21:36 -0800 (PST) Received: by 10.64.27.225 with HTTP; Tue, 20 Nov 2012 11:21:36 -0800 (PST) In-Reply-To: <1281530059.17550.1353434227450@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> References: <20121120174554.GC50873@equilibrium.bsdes.net> <1281530059.17550.1353434227450@238ae4dab3b4454b88aea4d9f7c372c1.nuevasync.com> Date: Tue, 20 Nov 2012 14:21:36 -0500 Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: Outback Dingo To: khatfield@socllc.net Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-isp@freebsd.org" , Victor Balada Diaz , John Fretby X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Nov 2012 19:21:37 -0000 you can also use luigis netmap with certain cards and a patched open-vswitch if its purely a routing platform to potentially increase performance, we are currently working on a prototype in our lab to replace an older cisco On Tue, Nov 20, 2012 at 12:57 PM, wrote: > One thing I have noticed is mixed results with polling depending on the > version. > > My experience with similar NICs was that polling increased the PPS > capabilities up to 7.4 but post 7.4 we have seen most cases where polling > caused either connectivity issues or decreased overall performance. > > Now we were running full 1Gbps in our tests. With only 140Mbps you should > be able to handle this amount without polling or additional kernel tweaks= . > Specifically with 9 - I would recommend doing needed sysctl tweaks withou= t > polling and as long as you are not receiving DDoS traffic then it should > prove perfectly stable. > > > > On Nov 20, 2012, at 11:46 AM, "Victor Balada Diaz" > wrote: > > > On Tue, Nov 20, 2012 at 03:35:13PM +0000, John Fretby wrote: > >> Howdy all, > >> > >> We've currently got an ageing HP DL360 running as a 'router' - it has > >> 100Mbit in/out onto our network, and has two 'bce' NIC's providing > in/out. > >> It's running quite an old version of FreeBSD (6 I think) - but works. > >> > >> As the network gets busier we've noticed the amount of interrupt time > on it > >> is climbing (as you'd expect - i.e. esp. if many small packets are bei= ng > >> forwarded). Many moons ago we did experiment with this box - and enabl= ed > >> device polling (inc. upping the HZ on the box and recompiling the kern= el > >> etc). This didn't work very well at the time (probably because it was = in > >> it's infancy) so we left it off in the end. > >> > >> If we were to replace this box, with something new - say a SuperMicro > based > >> system with two: > >> > >> Intel 82574L's (em Driver Based) > >> > >> And enable polling - is it likely to "just work" these days? The curre= nt > >> upstream is 100Mbit, we're looking to upgrade this to 1Gbit in, but wi= th > >> say 200Mbit comitted on it (so shouldn't go above 200Mbit). > >> > >> Is there anything that has to be done to enable polling - other than > >> recompiling GENERIC to support it? - i.e. no HZ hacks or anything > needed on > >> 'modern' machines (it's a quad core Xeon). > > > > Hello John, > > > > You might find interesting to read this thread: > > > > > http://lists.freebsd.org/pipermail/freebsd-current/2012-November/037590.h= tml > > > > In short: device polling can decrease performance on modern hardware. > > > > You might want to try upgrading to a new FreeBSD version and tuning it > somehow > > before buying a new server. More info on tuning the network stack: > > > > http://wiki.freebsd.org/NetworkPerformanceTuning > > > > Regards. > > Victor. > > -- > > La prueba m=E1s fehaciente de que existe vida inteligente en otros > > planetas, es que no han intentado contactar con nosotros. > > _______________________________________________ > > freebsd-isp@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-isp > > To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-isp@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-isp > To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" > From owner-freebsd-isp@FreeBSD.ORG Wed Nov 21 14:58:01 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A3C30CEF for ; Wed, 21 Nov 2012 14:58:01 +0000 (UTC) (envelope-from victor@bsdes.net) Received: from equilibrium.bsdes.net (244.Red-217-126-240.staticIP.rima-tde.net [217.126.240.244]) by mx1.freebsd.org (Postfix) with ESMTP id E9AFE8FC17 for ; Wed, 21 Nov 2012 14:57:59 +0000 (UTC) Received: by equilibrium.bsdes.net (Postfix, from userid 1001) id B082D39832; Wed, 21 Nov 2012 15:57:58 +0100 (CET) Date: Wed, 21 Nov 2012 15:57:58 +0100 From: Victor Balada Diaz To: John Fretby Subject: Re: FreeBSD boxes as a 'router'... Message-ID: <20121121145758.GD50873@equilibrium.bsdes.net> References: <20121120174554.GC50873@equilibrium.bsdes.net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-isp@freebsd.org X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 14:58:01 -0000 On Wed, Nov 21, 2012 at 09:40:37AM +0000, John Fretby wrote: > On 20 November 2012 17:45, Victor Balada Diaz wrote: > > > > Hello John, > > > > You might find interesting to read this thread: > > > > > > http://lists.freebsd.org/pipermail/freebsd-current/2012-November/037590.html > > > > In short: device polling can decrease performance on modern hardware. > > > > You might want to try upgrading to a new FreeBSD version and tuning it > > somehow > > before buying a new server. More info on tuning the network stack: > > > > http://wiki.freebsd.org/NetworkPerformanceTuning > > > > Regards. > > Victor. > > -- > > La prueba más fehaciente de que existe vida inteligente en otros > > planetas, es que no han intentado contactar con nosotros. > > > > Ok, general consensus appears to be: > > > - Update the O/S (we will change the hardware anyway, it's pretty old) - > try to use -STABLE not -RELEASE. > - *Don't* use polling. > - Do consider some things like 'net.inet.fastforwarding=1' > - Keep IPFW to a minimum, use tables where possible. > - Don't use network to feed /dev/random > > The tuning guide mentions: > > hw.igb.max_interrupt_rate=32000 (or more) > > Is there anything for em / Intel NIC's similar to this? - A while ago we > had a server that kept killing itself while being restarted - logging > something similar to: > > interrupt storm detected on "irq256:"; throttling interrupt source > (Which on checking, irq256 = em0) That was on an 8.1 box. I think as the > network came up it was hit with so many repeat requests for services that > were not yet running - it generated a storm of SYN/RST packets etc. > > So you can see why I ask if there's a similar sysctl for em0/Intel NIC :) > > What's likely to happen to a system like this when it gets hit by DoS / > masses of small packets? - Same as existing? (which appears to spiral to > 100% interrupt time, and become completely unresponsive)? > > Thanks all for your time, > > -Jon I think you forgot to CC the list. I'll add it so you can get more answers. em(4) and igb(4) are both drivers for Intel NICs. They just have different capabilities. The sysctl you're asking for controls behavior of adaptive interrupt moderation. It's a recommended tuning for end hosts more than routers. You can read more about interrupt moderation on this document: http://www.intel.com/design/network/applnots/ap450.htm em(4) NICs don't have all the capabilities of igb(4) ones. Some em(4) NICs have interrupt moderation (eg: 82574L) but not all of them do. If your em(4) card does have interrupt moderation you can tune it with: hw.em.rx_int_delay hw.em.rx_abs_int_delay hw.em.tx_int_delay hw.em.tx_abs_int_delay Exchanging latency to get more throughput. You can take a look at this document explaining capabilities of different NICs: http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phys-brochure.pdf You should ask supermicro what's the exact model they'll put on your server and then decide if it's OK for you. I would just try to buy an igb(4) card. The networking tunning wiki recomends intel 82576. About the interrupt storm: We've had various interrupt storms that were caused by different problems. The most common was a software bug with interrupts. After reporting on the lists it was fixed and we didn't have problems again. If you have a problem with high interrupts because too many small packets (eg a DoS), getting a card with interrupt moderation should help a lot. Most probably your problem with interrupt storms was caused by something else like a shared interrupt with other device or software bug. Without more analysis it's impossible to really say. Keep in mind that i'm not an expert on this area, so you might get better answers on frebsd-net@ :) Hope it helps. Regards. Victor. -- La prueba más fehaciente de que existe vida inteligente en otros planetas, es que no han intentado contactar con nosotros. From owner-freebsd-isp@FreeBSD.ORG Wed Nov 21 16:40:44 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 53B639C5 for ; Wed, 21 Nov 2012 16:40:44 +0000 (UTC) (envelope-from jfretby@googlemail.com) Received: from mail-qc0-f182.google.com (mail-qc0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id 02F6C8FC15 for ; Wed, 21 Nov 2012 16:40:43 +0000 (UTC) Received: by mail-qc0-f182.google.com with SMTP id k19so6333529qcs.13 for ; Wed, 21 Nov 2012 08:40:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=C6sFvCkT6RZzTxEJcMYk6fb9tO/YGu6WQU+uTsV8+tQ=; b=Pkaqrdrrfm+x7QXwvApUufV91GFFG3JAMGZudxuOGESgXczVdK71WSXfUtVQvfxsLe 55adFlRudC1dSHcYKmmRycasjNt2TaZ3SctuEUq1/bZvNusmPWWOg4XbE/TyTqteMPCJ uGLNvqCiyIfnECNRsqDKbqfKCddIEUWCkP2TJZKM0eXdzN545p4b9oh7Njbve6BDapoW GHqGWZhNZjH8BVogaXfpFfTonU5i0dm+JHmOZys6pdUXKNK1YA1g98sEdOsCGP4O8F+3 mBYuapt38/UQ/gimkhCy3SKouxeWOb8hkSuGSFnYkXULns5C9s+Pz+nCy570ObaROLDj Pnag== MIME-Version: 1.0 Received: by 10.224.42.15 with SMTP id q15mr18745775qae.68.1353516043204; Wed, 21 Nov 2012 08:40:43 -0800 (PST) Received: by 10.49.25.13 with HTTP; Wed, 21 Nov 2012 08:40:42 -0800 (PST) In-Reply-To: <20121121145758.GD50873@equilibrium.bsdes.net> References: <20121120174554.GC50873@equilibrium.bsdes.net> <20121121145758.GD50873@equilibrium.bsdes.net> Date: Wed, 21 Nov 2012 16:40:42 +0000 Message-ID: Subject: Re: FreeBSD boxes as a 'router'... From: John Fretby To: Victor Balada Diaz Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-isp@freebsd.org X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 16:40:44 -0000 On 21 November 2012 14:57, Victor Balada Diaz wrote: > I think you forgot to CC the list. I'll add it so you can get > more answers. > I did forget, thanks for that! :) > em(4) and igb(4) are both drivers for Intel NICs. They just have > different capabilities. The sysctl you're asking for controls behavior > of adaptive interrupt moderation. It's a recommended tuning for end hosts > more than routers. You can read more about interrupt moderation on this > document: > > http://www.intel.com/design/network/applnots/ap450.htm > > em(4) NICs don't have all the capabilities of igb(4) ones. Some em(4) NICs > have > interrupt moderation (eg: 82574L) but not all of them do. If your em(4) > card does > have interrupt moderation you can tune it with: > > hw.em.rx_int_delay > hw.em.rx_abs_int_delay > hw.em.tx_int_delay > hw.em.tx_abs_int_delay > > Exchanging latency to get more throughput. > > You can take a look at this document explaining capabilities of different > NICs: > > > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phys-brochure.pdf > > You should ask supermicro what's the exact model they'll put on your server > and then decide if it's OK for you. They are apparently: em0: port 0xf020-0xf03f mem 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt ... em0: flags=8c02 metric 0 mtu 1500 options=4219b > About the interrupt storm: We've had various interrupt storms that were > caused by > different problems. The most common was a software bug with interrupts. > After > reporting on the lists it was fixed and we didn't have problems again. > > If you have a problem with high interrupts because too many small packets > (eg a DoS), > getting a card with interrupt moderation should help a lot. Most probably > your problem > with interrupt storms was caused by something else like a shared interrupt > with other > device or software bug. Without more analysis it's impossible to really > say. > I have some details from when it happened - it doesn't look like it was a shared interrupt issue - it just literally looks like the host came up, with a stampeding hurd of "other" hosts hitting it for services that weren't yet running, and it folded :( That's why I was wondering if there was a similar sysctl for the em driver - in order to raise the number of interrupts the system allows, before declaring it "a storm". > > Keep in mind that i'm not an expert on this area, so you might get better > answers > on frebsd-net@ :) > > Hope it helps. > It has - half the problem is there are *so* many options, combinations - and no matter what you pick, if you look them up enough you'll find someone finding fault with them, or casting doubts on their performance. Doesn't really help when all you want is something that has a good chance of "working" :) -Jon From owner-freebsd-isp@FreeBSD.ORG Wed Nov 21 18:00:08 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 349A3130 for ; Wed, 21 Nov 2012 18:00:08 +0000 (UTC) (envelope-from vandj@securenet.net) Received: from ms.securenet.net (ms.securenet.net [205.236.147.20]) by mx1.freebsd.org (Postfix) with ESMTP id DDA298FC12 for ; Wed, 21 Nov 2012 18:00:07 +0000 (UTC) Received: from ms.securenet.net (localhost [127.0.0.1]) by ms.securenet.net (Postfix) with ESMTP id 66FAFBA9F for ; Wed, 21 Nov 2012 12:52:48 -0500 (EST) X-Virus-Scanned: by f-prot anti-virus, and clamav anti-virus at SecureNet Inc. Received: from ms.securenet.net ([127.0.0.1]) by ms.securenet.net (ms.securenet.net [127.0.0.1]) (amavisd-new, port 10025) with ESMTP id oh4MX13AjB2j for ; Wed, 21 Nov 2012 12:52:48 -0500 (EST) Received: from HomePC (mtl-pppoe-adsl245.securenet.net [66.38.236.245]) by ms.securenet.net (Postfix) with ESMTP id 04028B9E1 for ; Wed, 21 Nov 2012 12:52:48 -0500 (EST) From: "Jean M Vandette" To: References: <20121120174554.GC50873@equilibrium.bsdes.net> <20121121145758.GD50873@equilibrium.bsdes.net> In-Reply-To: Subject: RE: FreeBSD boxes as a 'router'... Date: Wed, 21 Nov 2012 12:52:47 -0500 Message-ID: <04f401cdc811$04207e50$0c617af0$@net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Ac3IBvwwsX6k/2THT3+fa+eaIQN8qAAAOKZA Content-Language: en-us X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 18:00:08 -0000 Greetings all. We did allot of testing a few years ago with and without polling at that time what we had found it was not the nic's it was the bus that locked and caused a problem. The best we were able to get was 80~90MB throughput before the box would lock and reboot. We also found a single CPU ran circles around a dual CPU for this purpose. At that time we were running 3 BGP sessions so lots of memory was needed to hold the 240K routing table entries (at the time). CPU load was never an issue. We were unsuccessful and moved to a Juniper routing platform, we just could not get fast enough I/O to the bus. I know allot of the hardware out there is much faster now a days and cheaper. The best advice I can give you is build a current state-of-the-art machine bus speed is the biggest issue, faster is better, I/O is critical, you can have the fastest bus out there but if all you have is a PCI slot for example your dead in the water, it's just not fast enough. You cannot have a bottle neck. You need to check with the mfg for the bus speed of the nic. Just because the nic is built in many times it is just to save a expansion slot, the speed to the bus is not as fast as you might expect. Once you get the fastest of I/O to a fast bus put a current release and then hammer it with tests to tweak it. Expect high interrupt rates if not using polling. Once your results are what you expect and stable put it into production. You have had allot of good tweaks and info given to you over the last few days so I won't repeat them. Good luck Jean M. Vandette SecureNet Information Services -----Original Message----- From: owner-freebsd-isp@freebsd.org [mailto:owner-freebsd-isp@freebsd.org] On Behalf Of John Fretby Sent: Wednesday, November 21, 2012 11:41 AM To: Victor Balada Diaz Cc: freebsd-isp@freebsd.org Subject: Re: FreeBSD boxes as a 'router'... On 21 November 2012 14:57, Victor Balada Diaz wrote: > I think you forgot to CC the list. I'll add it so you can get more > answers. > I did forget, thanks for that! :) > em(4) and igb(4) are both drivers for Intel NICs. They just have > different capabilities. The sysctl you're asking for controls behavior > of adaptive interrupt moderation. It's a recommended tuning for end > hosts more than routers. You can read more about interrupt moderation > on this > document: > > http://www.intel.com/design/network/applnots/ap450.htm > > em(4) NICs don't have all the capabilities of igb(4) ones. Some em(4) > NICs have interrupt moderation (eg: 82574L) but not all of them do. If > your em(4) card does have interrupt moderation you can tune it with: > > hw.em.rx_int_delay > hw.em.rx_abs_int_delay > hw.em.tx_int_delay > hw.em.tx_abs_int_delay > > Exchanging latency to get more throughput. > > You can take a look at this document explaining capabilities of > different > NICs: > > > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phy > s-brochure.pdf > > You should ask supermicro what's the exact model they'll put on your > server and then decide if it's OK for you. They are apparently: em0: port 0xf020-0xf03f mem 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt ... em0: flags=8c02 metric 0 mtu 1500 options=4219b > About the interrupt storm: We've had various interrupt storms that > were caused by different problems. The most common was a software bug > with interrupts. > After > reporting on the lists it was fixed and we didn't have problems again. > > If you have a problem with high interrupts because too many small > packets (eg a DoS), getting a card with interrupt moderation should > help a lot. Most probably your problem with interrupt storms was > caused by something else like a shared interrupt with other device or > software bug. Without more analysis it's impossible to really say. > I have some details from when it happened - it doesn't look like it was a shared interrupt issue - it just literally looks like the host came up, with a stampeding hurd of "other" hosts hitting it for services that weren't yet running, and it folded :( That's why I was wondering if there was a similar sysctl for the em driver - in order to raise the number of interrupts the system allows, before declaring it "a storm". > > Keep in mind that i'm not an expert on this area, so you might get > better answers on frebsd-net@ :) > > Hope it helps. > It has - half the problem is there are *so* many options, combinations - and no matter what you pick, if you look them up enough you'll find someone finding fault with them, or casting doubts on their performance. Doesn't really help when all you want is something that has a good chance of "working" :) -Jon _______________________________________________ freebsd-isp@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-isp To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" From owner-freebsd-isp@FreeBSD.ORG Fri Nov 23 16:05:50 2012 Return-Path: Delivered-To: freebsd-isp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C08948F8 for ; Fri, 23 Nov 2012 16:05:50 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm32-vm3.bullet.mail.ne1.yahoo.com (nm32-vm3.bullet.mail.ne1.yahoo.com [98.138.229.51]) by mx1.freebsd.org (Postfix) with ESMTP id 76E458FC12 for ; Fri, 23 Nov 2012 16:05:50 +0000 (UTC) Received: from [98.138.226.176] by nm32.bullet.mail.ne1.yahoo.com with NNFMP; 23 Nov 2012 16:03:44 -0000 Received: from [98.138.89.199] by tm11.bullet.mail.ne1.yahoo.com with NNFMP; 23 Nov 2012 16:03:44 -0000 Received: from [127.0.0.1] by omp1057.mail.ne1.yahoo.com with NNFMP; 23 Nov 2012 16:03:44 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 510357.18753.bm@omp1057.mail.ne1.yahoo.com Received: (qmail 38771 invoked by uid 60001); 23 Nov 2012 16:03:44 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1353686624; bh=MWsJF3G91PlgDzhtAiN7AONlk+HXKD9hqsKfd2Zh2gw=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=muvkaSMtEOingGTc7zC+1V3YZQmLapZ8CvNQx+CY7/YNm6OJIuctwhkuQm7NhCic3vuuMfMtjuQveEJH+41g+rXaBKvR0ulHvlc+W2clcgVU5XfGGQzQKZlOhYSr1qlgaIgT2VU6n/acrrRrwGU31/wRwouit/yzG5JGn7lHyQg= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=Ysfku7NSkmwtH23H7pxdfdVU831QyguD297HeSF0jtDIiBtqYSs/Wiogpmsa4nd0PMHLXHwL3vgKgRBZrmSVlxQva23SL0+A3EF3RMgZBVVXVUoJ3pXWQzXy5rf/VOWCjqp9f1la6oKquS70iltMMJZjI2OFLW0ZmGAgE7Tz/xU=; X-YMail-OSG: TSGKNsEVM1nRA4ZCIYYTzE2Yi37d_Bwo.SZbsg2jxFW4HkA dW_6laRrUZGVvaWNaEj01CwrXSaO0meEjJAF.VAAjpPJHTGCBMDr1FvUB9rR 0HMZFsklLWQ3xiQq5fSXGCMUsl41Oskb3mJ4SkxS15sibkDEWOVRjrCbkx0O aCq9gL6PypXU9792q.dqLR07ziP741qkzw75qo0Xm7uaQmdbUMzDnw68tqlq FHtDnakZ2s3wCmeDVG8mubh6Jsh0iBHNQmRVfDvzxLMQuVG16IjsSVyaUEi1 KLa1.YdZf8v4xATLjgS2Tn8NnVNKyiWsGG495hc_bysqDuiCUZfW1blBtVwI Fj5pqu8T0BgD.KqGi9Jn4LDTkBSFdjbj7oAOWqqSNwPgGQP3For5g3PprMcy WRVSNjKGZsyZaZUMmtI4aBcwnugoidliAVK1mtFdG4V2lBeB0uL3OD6ZHfxy 77twgjzrbXeLQxAjFD7Vl5zeF3bcji.uRTfEsDA7_X8IC8dr1QWrH7MqBLUy 4Kh6iRsrkjy9A.aRdyvloUaVNoAwndakKDY00.ou5RLxpudRcY7heuKj6bv1 4uzJop23barMi1MmlC8Hfl5Rg6AzWm7KrcpLAwOqgulWjuZaF.p659DCPkw- - Received: from [174.48.128.27] by web121606.mail.ne1.yahoo.com via HTTP; Fri, 23 Nov 2012 08:03:44 PST X-Rocket-MIMEInfo: 001.001, R2l2ZW4gdGhhdCBidXNzZXMgYXJlIGZhc3RlciBhbmQgU01QIGlzIGJldHRlciBub3cgSSdkIHNheSB0aGF0IHlvdXIKdGVzdHMgbmVlZCBzb21lIHJlZnJlc2hpbmcuCgotLS0gT24gV2VkLCAxMS8yMS8xMiwgSmVhbiBNIFZhbmRldHRlIDx2YW5kakBzZWN1cmVuZXQubmV0PiB3cm90ZToKCj4gRnJvbTogSmVhbiBNIFZhbmRldHRlIDx2YW5kakBzZWN1cmVuZXQubmV0Pgo.IFN1YmplY3Q6IFJFOiBGcmVlQlNEIGJveGVzIGFzIGEgJ3JvdXRlcicuLi4KPiBUbzogZnJlZWJzZC1pc3BAZnJlZWJzZC5vcmcKPiABMAEBAQE- X-Mailer: YahooMailClassic/15.0.8 YahooMailWebService/0.8.123.460 Message-ID: <1353686624.27263.YahooMailClassic@web121606.mail.ne1.yahoo.com> Date: Fri, 23 Nov 2012 08:03:44 -0800 (PST) From: Barney Cordoba Subject: RE: FreeBSD boxes as a 'router'... To: freebsd-isp@freebsd.org, Jean M Vandette In-Reply-To: <04f401cdc811$04207e50$0c617af0$@net> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-isp@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Internet Services Providers List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Nov 2012 16:05:50 -0000 Given that busses are faster and SMP is better now I'd say that your=0Atest= s need some refreshing.=0A=0A--- On Wed, 11/21/12, Jean M Vandette wrote:=0A=0A> From: Jean M Vandette =0A>= Subject: RE: FreeBSD boxes as a 'router'...=0A> To: freebsd-isp@freebsd.or= g=0A> Date: Wednesday, November 21, 2012, 12:52 PM=0A> Greetings all.=0A> = =0A> We did allot of testing a few years ago with and without=0A> polling= =0A> at that time what we had found it was not the nic's it was=0A> the bus= =0A> that locked and caused a problem.=A0 The best we were=0A> able to get = was 80~90MB=0A> throughput =0A> before the box would lock and reboot.=A0 We= also found a=0A> single CPU ran=0A> circles around=0A> a dual CPU for this= purpose.=A0 At that time we were=0A> running 3 BGP sessions so=0A> lots of= memory=0A> was needed to hold the 240K routing table entries (at the=0A> t= ime).=A0 CPU load=0A> was never an issue.=0A> We were unsuccessful and move= d to a Juniper routing=0A> platform, we just could=0A> not =0A> get fast en= ough I/O to the bus.=0A> =0A> I know allot of the hardware out there is muc= h faster now a=0A> days and=0A> cheaper.=A0 =0A> =0A> The best advice I can= give you is build a current=0A> state-of-the-art machine=0A> bus speed is = the biggest issue, faster is better, I/O is=0A> critical, you can=0A> have = the fastest bus=0A> out there but if all you have is a PCI slot for example= your=0A> dead in the=0A> water, it's just not fast enough.=0A> You cannot = have a bottle neck.=A0 You need to check with=0A> the mfg for the bus=0A> s= peed of the nic.=A0 =0A> Just because the nic is built in many times it is = just to=0A> save a expansion=0A> slot, the speed=0A> to the bus is not as f= ast as you might expect. =0A> =0A> Once you get the fastest of I/O to a fas= t bus put a current=0A> release and=A0 =0A> then hammer it with tests to tw= eak it.=A0 Expect high=0A> interrupt rates if not=0A> using polling.=0A> On= ce your results are what you expect and stable put it into=0A> production.= =0A> =0A> You have had allot of good tweaks and info given to you over=0A> = the last few=0A> days=0A> so I won't repeat them.=A0 =0A> =0A> Good luck=0A= > =0A> Jean M. Vandette=0A> SecureNet Information Services=0A> =0A> -----Or= iginal Message-----=0A> From: owner-freebsd-isp@freebsd.org=0A> [mailto:own= er-freebsd-isp@freebsd.org]=0A> On Behalf Of John Fretby=0A> Sent: Wednesda= y, November 21, 2012 11:41 AM=0A> To: Victor Balada Diaz=0A> Cc: freebsd-is= p@freebsd.org=0A> Subject: Re: FreeBSD boxes as a 'router'...=0A> =0A> On 2= 1 November 2012 14:57, Victor Balada Diaz =0A> wrote:=0A>= =0A> =0A> > I think you forgot to CC the list. I'll add it so you=0A> can = get more =0A> > answers.=0A> >=0A> =0A> I did forget, thanks for that! :)= =0A> =0A> =0A> > em(4) and igb(4) are both drivers for Intel NICs. They=0A>= just have =0A> > different capabilities. The sysctl you're asking for=0A> = controls behavior =0A> > of adaptive interrupt moderation. It's a recommend= ed=0A> tuning for end =0A> > hosts more than routers. You can read more abo= ut=0A> interrupt moderation =0A> > on this=0A> > document:=0A> >=0A> > http= ://www.intel.com/design/network/applnots/ap450.htm=0A> >=0A> > em(4) NICs d= on't have all the capabilities of igb(4)=0A> ones. Some em(4) =0A> > NICs h= ave interrupt moderation (eg: 82574L) but not all=0A> of them do. If =0A> >= your em(4) card does have interrupt moderation you can=0A> tune it with:= =0A> >=0A> > hw.em.rx_int_delay=0A> > hw.em.rx_abs_int_delay=0A> > hw.em.tx= _int_delay=0A> > hw.em.tx_abs_int_delay=0A> >=0A> > Exchanging latency to g= et more throughput.=0A> >=0A> > You can take a look at this document explai= ning=0A> capabilities of =0A> > different=0A> > NICs:=0A> >=0A> >=0A> > htt= p://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phy=0A> > s= -brochure.pdf=0A> >=0A> > You should ask supermicro what's the exact model= =0A> they'll put on your =0A> > server and then decide if it's OK for you.= =0A> =0A> =0A> They are apparently:=0A> =0A> em0: port=0A> 0xf020-0xf03f mem=0A> 0xdfa00000-0xdfa1ffff,0x= dfa25000-0xdfa25fff irq 20 at device=0A> 25.0 on pci0=0A> em0: Using an MSI= interrupt=0A> ...=0A> em0: flags=3D8c02=0A> metric 0 mtu 1500=0A> options=3D4219b IC,VLAN_HWTSO>=0A> =0A> =0A> > About t= he interrupt storm: We've had various interrupt=0A> storms that =0A> > were= caused by different problems. The most common was=0A> a software bug =0A> = > with interrupts.=0A> > After=0A> > reporting on the lists it was fixed an= d we didn't have=0A> problems again.=0A> >=0A> > If you have a problem with= high interrupts because too=0A> many small =0A> > packets (eg a DoS), gett= ing a card with interrupt=0A> moderation should =0A> > help a lot. Most pro= bably your problem with interrupt=0A> storms was =0A> > caused by something= else like a shared interrupt with=0A> other device or =0A> > software bug.= Without more analysis it's impossible to=0A> really say.=0A> >=0A> =0A> I = have some details from when it happened - it doesn't look=0A> like it was a= =0A> shared interrupt issue - it just literally looks like the=0A> host cam= e up, with=0A> a stampeding hurd of "other" hosts hitting it for services= =0A> that weren't yet=0A> running, and it folded :(=0A> =0A> That's why I w= as wondering if there was a similar sysctl for=0A> the em driver=0A> - in o= rder to raise the number of interrupts the system=0A> allows, before=0A> de= claring it "a storm".=0A> =0A> =0A> >=0A> > Keep in mind that i'm not an ex= pert on this area, so=0A> you might get =0A> > better answers on frebsd-net= @ :)=0A> >=0A> > Hope it helps.=0A> >=0A> =0A> It has - half the problem is= there are *so* many options,=0A> combinations - and=0A> no matter what you= pick, if you look them up enough you'll=0A> find someone=0A> finding fault= with them, or casting doubts on their=0A> performance.=0A> =0A> Doesn't re= ally help when all you want is something that has=0A> a good chance of=0A> = "working" :)=0A> =0A> -Jon=0A> ____________________________________________= ___=0A> freebsd-isp@freebsd.org=0A> mailing list=0A> http://lists.freebsd.o= rg/mailman/listinfo/freebsd-isp=0A> To unsubscribe, send any mail to "freeb= sd-isp-unsubscribe@freebsd.org"=0A> =0A> __________________________________= _____________=0A> freebsd-isp@freebsd.org=0A> mailing list=0A> http://lists= .freebsd.org/mailman/listinfo/freebsd-isp=0A> To unsubscribe, send any mail= to "freebsd-isp-unsubscribe@freebsd.org"=0A>