Date: Wed, 21 Nov 2012 15:57:58 +0100 From: Victor Balada Diaz <victor@bsdes.net> To: John Fretby <jfretby@googlemail.com> Cc: freebsd-isp@freebsd.org Subject: Re: FreeBSD boxes as a 'router'... Message-ID: <20121121145758.GD50873@equilibrium.bsdes.net> In-Reply-To: <CAN9kdQmZhsOKmgB0hns_f8ENcpUauLocBX6-VXDqE_2D8EdKzA@mail.gmail.com> References: <CAN9kdQnkaF3NdEsoBh2q%2Bxf73rur%2B1JSVGcUo8xfhugJMQ_oMw@mail.gmail.com> <20121120174554.GC50873@equilibrium.bsdes.net> <CAN9kdQmZhsOKmgB0hns_f8ENcpUauLocBX6-VXDqE_2D8EdKzA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Nov 21, 2012 at 09:40:37AM +0000, John Fretby wrote: > On 20 November 2012 17:45, Victor Balada Diaz <victor@bsdes.net> wrote: > > > > Hello John, > > > > You might find interesting to read this thread: > > > > > > http://lists.freebsd.org/pipermail/freebsd-current/2012-November/037590.html > > > > In short: device polling can decrease performance on modern hardware. > > > > You might want to try upgrading to a new FreeBSD version and tuning it > > somehow > > before buying a new server. More info on tuning the network stack: > > > > http://wiki.freebsd.org/NetworkPerformanceTuning > > > > Regards. > > Victor. > > -- > > La prueba más fehaciente de que existe vida inteligente en otros > > planetas, es que no han intentado contactar con nosotros. > > > > Ok, general consensus appears to be: > > > - Update the O/S (we will change the hardware anyway, it's pretty old) - > try to use -STABLE not -RELEASE. > - *Don't* use polling. > - Do consider some things like 'net.inet.fastforwarding=1' > - Keep IPFW to a minimum, use tables where possible. > - Don't use network to feed /dev/random > > The tuning guide mentions: > > hw.igb.max_interrupt_rate=32000 (or more) > > Is there anything for em / Intel NIC's similar to this? - A while ago we > had a server that kept killing itself while being restarted - logging > something similar to: > > interrupt storm detected on "irq256:"; throttling interrupt source > (Which on checking, irq256 = em0) That was on an 8.1 box. I think as the > network came up it was hit with so many repeat requests for services that > were not yet running - it generated a storm of SYN/RST packets etc. > > So you can see why I ask if there's a similar sysctl for em0/Intel NIC :) > > What's likely to happen to a system like this when it gets hit by DoS / > masses of small packets? - Same as existing? (which appears to spiral to > 100% interrupt time, and become completely unresponsive)? > > Thanks all for your time, > > -Jon I think you forgot to CC the list. I'll add it so you can get more answers. em(4) and igb(4) are both drivers for Intel NICs. They just have different capabilities. The sysctl you're asking for controls behavior of adaptive interrupt moderation. It's a recommended tuning for end hosts more than routers. You can read more about interrupt moderation on this document: http://www.intel.com/design/network/applnots/ap450.htm em(4) NICs don't have all the capabilities of igb(4) ones. Some em(4) NICs have interrupt moderation (eg: 82574L) but not all of them do. If your em(4) card does have interrupt moderation you can tune it with: hw.em.rx_int_delay hw.em.rx_abs_int_delay hw.em.tx_int_delay hw.em.tx_abs_int_delay Exchanging latency to get more throughput. You can take a look at this document explaining capabilities of different NICs: http://www.intel.com/content/dam/doc/brochure/ethernet-controllers-phys-brochure.pdf You should ask supermicro what's the exact model they'll put on your server and then decide if it's OK for you. I would just try to buy an igb(4) card. The networking tunning wiki recomends intel 82576. About the interrupt storm: We've had various interrupt storms that were caused by different problems. The most common was a software bug with interrupts. After reporting on the lists it was fixed and we didn't have problems again. If you have a problem with high interrupts because too many small packets (eg a DoS), getting a card with interrupt moderation should help a lot. Most probably your problem with interrupt storms was caused by something else like a shared interrupt with other device or software bug. Without more analysis it's impossible to really say. Keep in mind that i'm not an expert on this area, so you might get better answers on frebsd-net@ :) Hope it helps. Regards. Victor. -- La prueba más fehaciente de que existe vida inteligente en otros planetas, es que no han intentado contactar con nosotros.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20121121145758.GD50873>