Date: Tue, 7 Apr 2009 14:48:58 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Robert Watson <rwatson@freebsd.org>, Sepherosa Ziehau <sepherosa@gmail.com> Cc: freebsd-net@freebsd.org, Ivan Voras <ivoras@freebsd.org> Subject: Re: Advice on a multithreaded netisr patch? Message-ID: <409843.2186.qm@web63904.mail.re1.yahoo.com> In-Reply-To: <ea7b9c170904070657y1670fc80qd67dc1fa9cde2ff6@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--- On Tue, 4/7/09, Sepherosa Ziehau <sepherosa@gmail.com> wrote:
> From: Sepherosa Ziehau <sepherosa@gmail.com>
> Subject: Re: Advice on a multithreaded netisr patch?
> To: "Robert Watson" <rwatson@freebsd.org>
> Cc: freebsd-net@freebsd.org, "Ivan Voras" <ivoras@freebsd.org>
> Date: Tuesday, April 7, 2009, 9:57 AM
> On Tue, Apr 7, 2009 at 8:54 PM, Robert Watson
> <rwatson@freebsd.org> wrote:
> >
> > On Tue, 7 Apr 2009, Sepherosa Ziehau wrote:
> >
> >> On Sun, Apr 5, 2009 at 9:34 PM, Ivan Voras
> <ivoras@freebsd.org> wrote:
> >>>
> >>> Robert Watson wrote:
> >>>>
> >>>> On Sun, 5 Apr 2009, Ivan Voras wrote:
> >>>>
> >>>>> I thought this has something to deal
> with NIC moderation (em) but
> >>>>> can't really explain it. The bad
> performance part (not the jump) is
> >>>>> also visible over the loopback
> interface.
> >>>>
> >>>> FYI, if you want high performance, you
> really want a card supporting
> >>>> multiple input queues -- igb, cxgb, mxge,
> etc. if_em-only cards are
> >>
> >> PCI-E em(4) supports 2 RX queues. 82571/82572
> support 2 TX queues. I have
> >> not tested multi-TX queues, but em(4) multi-RX
> queues work well in dfly
> >> (tested with 82573 and 82571)
> >
> > You may not have seen, but in FreeBSD 7.x and higher,
> we have a new if_igb
> > driver to support more recent Intel gigabit devices,
> which now probes a few
> > of the devices historically associated with if_em.
> For example, on one of
> > the boxes I use:
>
> If I understand the code correctly, it only takes 82575 and
> 82576; I
> don't have the hardware, else I would have already
> added dfly support
> (with multi rx queues at least, it seems 82576 supports 16
> RX queues
> :)
Regarding if_igb:
1) Multiple TX queues are not supported. There's some hokey code to
test, but it doesn't properly separate flows to the queues.
2) 2 Rx queues don't work, so only 1 and 4 work
3) With 4 queues, it just sucks up CPU under heavy load on 4 cpus. It will
blow 4 cpus at a lower load than em will with 1
4) You'll need to fix DMA setup, as it sets the alignment requirement
to PAGE_SIZE. I haven't been able to convince Jack that its wrong, not
that I've tried very hard since its easy to just fix myself.
Barney
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?409843.2186.qm>