Date: Thu, 13 Aug 2009 04:56:32 -0700 (PDT) From: Barney Cordoba <barney_cordoba@yahoo.com> To: Peter Steele <psteele@webmail.maxiscale.com>, pyunyh@gmail.com Cc: freebsd-net@freebsd.org Subject: Re: nfe taskq performance issues Message-ID: <430428.63263.qm@web63902.mail.re1.yahoo.com> In-Reply-To: <20090812213521.GG55129@michelle.cdnetworks.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--- On Wed, 8/12/09, Pyun YongHyeon <pyunyh@gmail.com> wrote: > From: Pyun YongHyeon <pyunyh@gmail.com> > Subject: Re: nfe taskq performance issues > To: "Peter Steele" <psteele@webmail.maxiscale.com> > Cc: freebsd-net@freebsd.org > Date: Wednesday, August 12, 2009, 5:35 PM > On Thu, Jul 23, 2009 at 08:58:07AM > -0700, Peter Steele wrote: > > We've been hitting serious nfe taskq performance > issues during stress > > tests and in doing some research on the problem we > came across this old > > email: > > > > > > > > From: Ivan Voras <ivoras@freebsd.org> > > Date: April 28, 2009 3:53:14 AM PDT > > To: freebsd-threads@freebsd.org > > Cc: freebsd-net@freebsd.org, > freebsd-performance@freebsd.org > > Subject: Re: FreeBSD 7.1 taskq em performance > > > > > > I have been hitting some barrier with FreeBSD 7.1 > network performance. > > I > > > have written an application which contains two > kernel threads that > > takes > > > mbufs directly from a network interface and > forwards to another > > network > > > interface. This idea is to simulate different > network environment. > > > > > > I have been using FreeBSD 6.4 amd64 and tested > with an Ixia box > > > (specialised hardware firing very high packet > rate). The PC was a > > Core2 2.6 > > > GHz with dual ports Intel PCIE Gigabit network > card. It can manage up > > to 1.2 > > > million pps. > > > > > > I have a higher spec PC with FreeBSD 7.1 amd64 > and Quadcore 2.3 GHz > > and > > > PCIE Gigabit network card. The performance can > only achieve up to 600k > > pps. > > > I notice the 'taskq em0' and 'taskq em1' is solid > 100% CPU but it is > > not in > > > FreeBSD 6.4. > > > > > > > > In our case we are running FreeBSD 7.0, but we are > seeing our boxes > > experience serious thread starvation issues as the > nfe0 cpu percentage > > climbs steadily while cpu idle time drops at times to > 0 percent. This > > email thread mentioned a patch for the em driver > here: > > > > > > > > http://people.yandex-team.ru/~wawa/ > > <http://people.yandex-team.ru/%7Ewawa/>; > > > > > > > > > Does anyone know if this patch will work with the nfe > driver? > > > > That's for em(4). > > AFAIK all nfe(4) controllers lacks intelligent interrupts > moderation so driver should be prepared to handle > excessive > interrupt loads. I'm not sure whether NVIDIA ethernet > controllers > really lacks efficient interrupt mitigation mechanism but > it > seems Linux also faces the same hardware problem. > As you might know there is no publicly available data sheet > for > NVIDIA controllers so setting it right looks very hard to > me. > Try removing the INTR_MPSAFE flag from the bus_setup_intr() call. The entire point of using filters is to reduce lock contention. It might not solve the problem but its clearly an unnecessary potential bottleneck. Barney
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?430428.63263.qm>