Date: Fri, 23 Nov 2012 08:03:44 -0800 (PST) From: Barney Cordoba <barney_cordoba@yahoo.com> To: freebsd-isp@freebsd.org, Jean M Vandette <vandj@securenet.net> Subject: RE: FreeBSD boxes as a 'router'... Message-ID: <1353686624.27263.YahooMailClassic@web121606.mail.ne1.yahoo.com> In-Reply-To: <04f401cdc811$04207e50$0c617af0$@net>
next in thread | previous in thread | raw e-mail | index | archive | help
Given that busses are faster and SMP is better now I'd say that your=0Atest= s need some refreshing.=0A=0A--- On Wed, 11/21/12, Jean M Vandette <vandj@s= ecurenet.net> wrote:=0A=0A> From: Jean M Vandette <vandj@securenet.net>=0A>= Subject: RE: FreeBSD boxes as a 'router'...=0A> To: freebsd-isp@freebsd.or= g=0A> Date: Wednesday, November 21, 2012, 12:52 PM=0A> Greetings all.=0A> = =0A> We did allot of testing a few years ago with and without=0A> polling= =0A> at that time what we had found it was not the nic's it was=0A> the bus= =0A> that locked and caused a problem.=A0 The best we were=0A> able to get = was 80~90MB=0A> throughput =0A> before the box would lock and reboot.=A0 We= also found a=0A> single CPU ran=0A> circles around=0A> a dual CPU for this= purpose.=A0 At that time we were=0A> running 3 BGP sessions so=0A> lots of= memory=0A> was needed to hold the 240K routing table entries (at the=0A> t= ime).=A0 CPU load=0A> was never an issue.=0A> We were unsuccessful and move= d to a Juniper routing=0A> platform, we just could=0A> not =0A> get fast en= ough I/O to the bus.=0A> =0A> I know allot of the hardware out there is muc= h faster now a=0A> days and=0A> cheaper.=A0 =0A> =0A> The best advice I can= give you is build a current=0A> state-of-the-art machine=0A> bus speed is = the biggest issue, faster is better, I/O is=0A> critical, you can=0A> have = the fastest bus=0A> out there but if all you have is a PCI slot for example= your=0A> dead in the=0A> water, it's just not fast enough.=0A> You cannot = have a bottle neck.=A0 You need to check with=0A> the mfg for the bus=0A> s= peed of the nic.=A0 =0A> Just because the nic is built in many times it is = just to=0A> save a expansion=0A> slot, the speed=0A> to the bus is not as f= ast as you might expect. =0A> =0A> Once you get the fastest of I/O to a fas= t bus put a current=0A> release and=A0 =0A> then hammer it with tests to tw= eak it.=A0 Expect high=0A> interrupt rates if not=0A> using polling.=0A> On= ce your results are what you expect and stable put it into=0A> production.= =0A> =0A> You have had allot of good tweaks and info given to you over=0A> = the last few=0A> days=0A> so I won't repeat them.=A0 =0A> =0A> Good luck=0A= > =0A> Jean M. Vandette=0A> SecureNet Information Services=0A> =0A> -----Or= iginal Message-----=0A> From: owner-freebsd-isp@freebsd.org=0A> [mailto:own= er-freebsd-isp@freebsd.org]=0A> On Behalf Of John Fretby=0A> Sent: Wednesda= y, November 21, 2012 11:41 AM=0A> To: Victor Balada Diaz=0A> Cc: freebsd-is= p@freebsd.org=0A> Subject: Re: FreeBSD boxes as a 'router'...=0A> =0A> On 2= 1 November 2012 14:57, Victor Balada Diaz <victor@bsdes.net>=0A> wrote:=0A>= =0A> =0A> > I think you forgot to CC the list. I'll add it so you=0A> can = get more =0A> > answers.=0A> >=0A> =0A> I did forget, thanks for that! :)= =0A> =0A> =0A> > em(4) and igb(4) are both drivers for Intel NICs. They=0A>= just have =0A> > different capabilities. The sysctl you're asking for=0A> = controls behavior =0A> > of adaptive interrupt moderation. It's a recommend= ed=0A> tuning for end =0A> > hosts more than routers. You can read more abo= ut=0A> interrupt moderation =0A> > on this=0A> > document:=0A> >=0A> > http= ://www.intel.com/design/network/applnots/ap450.htm=0A> >=0A> > em(4) NICs d= on't have all the capabilities of igb(4)=0A> ones. Some em(4) =0A> > NICs h= ave interrupt moderation (eg: 82574L) but not all=0A> of them do. If =0A> >= your em(4) card does have interrupt moderation you can=0A> tune it with:= =0A> >=0A> > hw.em.rx_int_delay=0A> > hw.em.rx_abs_int_delay=0A> > hw.em.tx= _int_delay=0A> > hw.em.tx_abs_int_delay=0A> >=0A> > Exchanging latency to g= et more throughput.=0A> >=0A> > You can take a look at this document explai= ning=0A> capabilities of =0A> > different=0A> > NICs:=0A> >=0A> >=0A> > htt= p://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phy=0A> > s= -brochure.pdf=0A> >=0A> > You should ask supermicro what's the exact model= =0A> they'll put on your =0A> > server and then decide if it's OK for you.= =0A> =0A> =0A> They are apparently:=0A> =0A> em0: <Intel(R) PRO/1000 Networ= k Connection 7.3.2> port=0A> 0xf020-0xf03f mem=0A> 0xdfa00000-0xdfa1ffff,0x= dfa25000-0xdfa25fff irq 20 at device=0A> 25.0 on pci0=0A> em0: Using an MSI= interrupt=0A> ...=0A> em0: flags=3D8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAS= T>=0A> metric 0 mtu 1500=0A> options=3D4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HW= TAGGING,VLAN_HWCSUM,TSO4,WOL_MAG=0A> IC,VLAN_HWTSO>=0A> =0A> =0A> > About t= he interrupt storm: We've had various interrupt=0A> storms that =0A> > were= caused by different problems. The most common was=0A> a software bug =0A> = > with interrupts.=0A> > After=0A> > reporting on the lists it was fixed an= d we didn't have=0A> problems again.=0A> >=0A> > If you have a problem with= high interrupts because too=0A> many small =0A> > packets (eg a DoS), gett= ing a card with interrupt=0A> moderation should =0A> > help a lot. Most pro= bably your problem with interrupt=0A> storms was =0A> > caused by something= else like a shared interrupt with=0A> other device or =0A> > software bug.= Without more analysis it's impossible to=0A> really say.=0A> >=0A> =0A> I = have some details from when it happened - it doesn't look=0A> like it was a= =0A> shared interrupt issue - it just literally looks like the=0A> host cam= e up, with=0A> a stampeding hurd of "other" hosts hitting it for services= =0A> that weren't yet=0A> running, and it folded :(=0A> =0A> That's why I w= as wondering if there was a similar sysctl for=0A> the em driver=0A> - in o= rder to raise the number of interrupts the system=0A> allows, before=0A> de= claring it "a storm".=0A> =0A> =0A> >=0A> > Keep in mind that i'm not an ex= pert on this area, so=0A> you might get =0A> > better answers on frebsd-net= @ :)=0A> >=0A> > Hope it helps.=0A> >=0A> =0A> It has - half the problem is= there are *so* many options,=0A> combinations - and=0A> no matter what you= pick, if you look them up enough you'll=0A> find someone=0A> finding fault= with them, or casting doubts on their=0A> performance.=0A> =0A> Doesn't re= ally help when all you want is something that has=0A> a good chance of=0A> = "working" :)=0A> =0A> -Jon=0A> ____________________________________________= ___=0A> freebsd-isp@freebsd.org=0A> mailing list=0A> http://lists.freebsd.o= rg/mailman/listinfo/freebsd-isp=0A> To unsubscribe, send any mail to "freeb= sd-isp-unsubscribe@freebsd.org"=0A> =0A> __________________________________= _____________=0A> freebsd-isp@freebsd.org=0A> mailing list=0A> http://lists= .freebsd.org/mailman/listinfo/freebsd-isp=0A> To unsubscribe, send any mail= to "freebsd-isp-unsubscribe@freebsd.org"=0A>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1353686624.27263.YahooMailClassic>