Date: Fri, 23 Nov 2012 08:03:44 -0800 (PST) From: Barney Cordoba <barney_cordoba@yahoo.com> To: freebsd-isp@freebsd.org, Jean M Vandette <vandj@securenet.net> Subject: RE: FreeBSD boxes as a 'router'... Message-ID: <1353686624.27263.YahooMailClassic@web121606.mail.ne1.yahoo.com> In-Reply-To: <04f401cdc811$04207e50$0c617af0$@net>
index | next in thread | previous in thread | raw e-mail
Given that busses are faster and SMP is better now I'd say that your tests need some refreshing. --- On Wed, 11/21/12, Jean M Vandette <vandj@securenet.net> wrote: > From: Jean M Vandette <vandj@securenet.net> > Subject: RE: FreeBSD boxes as a 'router'... > To: freebsd-isp@freebsd.org > Date: Wednesday, November 21, 2012, 12:52 PM > Greetings all. > > We did allot of testing a few years ago with and without > polling > at that time what we had found it was not the nic's it was > the bus > that locked and caused a problem. The best we were > able to get was 80~90MB > throughput > before the box would lock and reboot. We also found a > single CPU ran > circles around > a dual CPU for this purpose. At that time we were > running 3 BGP sessions so > lots of memory > was needed to hold the 240K routing table entries (at the > time). CPU load > was never an issue. > We were unsuccessful and moved to a Juniper routing > platform, we just could > not > get fast enough I/O to the bus. > > I know allot of the hardware out there is much faster now a > days and > cheaper. > > The best advice I can give you is build a current > state-of-the-art machine > bus speed is the biggest issue, faster is better, I/O is > critical, you can > have the fastest bus > out there but if all you have is a PCI slot for example your > dead in the > water, it's just not fast enough. > You cannot have a bottle neck. You need to check with > the mfg for the bus > speed of the nic. > Just because the nic is built in many times it is just to > save a expansion > slot, the speed > to the bus is not as fast as you might expect. > > Once you get the fastest of I/O to a fast bus put a current > release and > then hammer it with tests to tweak it. Expect high > interrupt rates if not > using polling. > Once your results are what you expect and stable put it into > production. > > You have had allot of good tweaks and info given to you over > the last few > days > so I won't repeat them. > > Good luck > > Jean M. Vandette > SecureNet Information Services > > -----Original Message----- > From: owner-freebsd-isp@freebsd.org > [mailto:owner-freebsd-isp@freebsd.org] > On Behalf Of John Fretby > Sent: Wednesday, November 21, 2012 11:41 AM > To: Victor Balada Diaz > Cc: freebsd-isp@freebsd.org > Subject: Re: FreeBSD boxes as a 'router'... > > On 21 November 2012 14:57, Victor Balada Diaz <victor@bsdes.net> > wrote: > > > > I think you forgot to CC the list. I'll add it so you > can get more > > answers. > > > > I did forget, thanks for that! :) > > > > em(4) and igb(4) are both drivers for Intel NICs. They > just have > > different capabilities. The sysctl you're asking for > controls behavior > > of adaptive interrupt moderation. It's a recommended > tuning for end > > hosts more than routers. You can read more about > interrupt moderation > > on this > > document: > > > > http://www.intel.com/design/network/applnots/ap450.htm > > > > em(4) NICs don't have all the capabilities of igb(4) > ones. Some em(4) > > NICs have interrupt moderation (eg: 82574L) but not all > of them do. If > > your em(4) card does have interrupt moderation you can > tune it with: > > > > hw.em.rx_int_delay > > hw.em.rx_abs_int_delay > > hw.em.tx_int_delay > > hw.em.tx_abs_int_delay > > > > Exchanging latency to get more throughput. > > > > You can take a look at this document explaining > capabilities of > > different > > NICs: > > > > > > http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phy > > s-brochure.pdf > > > > You should ask supermicro what's the exact model > they'll put on your > > server and then decide if it's OK for you. > > > They are apparently: > > em0: <Intel(R) PRO/1000 Network Connection 7.3.2> port > 0xf020-0xf03f mem > 0xdfa00000-0xdfa1ffff,0xdfa25000-0xdfa25fff irq 20 at device > 25.0 on pci0 > em0: Using an MSI interrupt > ... > em0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> > metric 0 mtu 1500 > options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAG > IC,VLAN_HWTSO> > > > > About the interrupt storm: We've had various interrupt > storms that > > were caused by different problems. The most common was > a software bug > > with interrupts. > > After > > reporting on the lists it was fixed and we didn't have > problems again. > > > > If you have a problem with high interrupts because too > many small > > packets (eg a DoS), getting a card with interrupt > moderation should > > help a lot. Most probably your problem with interrupt > storms was > > caused by something else like a shared interrupt with > other device or > > software bug. Without more analysis it's impossible to > really say. > > > > I have some details from when it happened - it doesn't look > like it was a > shared interrupt issue - it just literally looks like the > host came up, with > a stampeding hurd of "other" hosts hitting it for services > that weren't yet > running, and it folded :( > > That's why I was wondering if there was a similar sysctl for > the em driver > - in order to raise the number of interrupts the system > allows, before > declaring it "a storm". > > > > > > Keep in mind that i'm not an expert on this area, so > you might get > > better answers on frebsd-net@ :) > > > > Hope it helps. > > > > It has - half the problem is there are *so* many options, > combinations - and > no matter what you pick, if you look them up enough you'll > find someone > finding fault with them, or casting doubts on their > performance. > > Doesn't really help when all you want is something that has > a good chance of > "working" :) > > -Jon > _______________________________________________ > freebsd-isp@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-isp > To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-isp@freebsd.org > mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-isp > To unsubscribe, send any mail to "freebsd-isp-unsubscribe@freebsd.org" >help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1353686624.27263.YahooMailClassic>