From owner-freebsd-net@FreeBSD.ORG Thu Aug 13 12:37:26 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 093BD1065678 for ; Thu, 13 Aug 2009 12:37:26 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from web63904.mail.re1.yahoo.com (web63904.mail.re1.yahoo.com [69.147.97.119]) by mx1.freebsd.org (Postfix) with SMTP id A6B928FC72 for ; Thu, 13 Aug 2009 12:37:25 +0000 (UTC) Received: (qmail 18759 invoked by uid 60001); 13 Aug 2009 12:37:25 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1250167044; bh=2fByGnqq2nnQmlt9ur5GcEKLpL7Kn1WMCuk9faQ0YgM=; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=wCrOhpZ31ocKVZl182ZHIAZu8Z+fuOUsjIilFrWa4I/3X4KcGd5D69VJ5h2D6yZZ42we07lUthPDBGd2X4O+annNqqOURXNQ20cfiPa8AsqHww01/995RcrL4DHyXG/5h27/No2br/YZ84tS+QIFNvdf1hkJhbZ7pBPWp2Lz9+o= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=M5D819koxHV3hkfgd2+RvY3O1P4/dwsMx1AZ667VRpkE2/YDjhp4OqZ8A9t4ozDhpW8z0GkgmypTjN0K/OTyHFKVuC59ToKw18WLwuxBTKOkDecl94A57jKY6OeYYhVLe4SChdWn+If2HtLVK3Y134XcLN3hICqlp6VLE8OQZSo=; Message-ID: <978222.18685.qm@web63904.mail.re1.yahoo.com> X-YMail-OSG: sIq0h_UVM1m3KRFbVcKqm17BHgOvRhOyqVwJx8RilKGGXLuHyGQn8eVcoIa5wg2xxgQevN84.MGiheFhNc6cscXVMSeE6gs6o73IU4ZLU9VCTg7RH9GcMhvcLXhMVB0IH2K.5hEpR77P7TmeyMVpek6.Toh7jumkuKVMyoSP3GMP_qD1V1Mu4EQEshC.z5rCpYj6DJsRsbYUUGlzIexkC8Ti5agAPoj5vjFTJkD.OQgGuf9vOtRhDHX_APrS4c6LnE2RyiV1NUCyUN2PuGBxuKNgqEF.NSvz8FSk10cokTLHMoqYgcr4ApvjZg-- Received: from [66.176.162.245] by web63904.mail.re1.yahoo.com via HTTP; Thu, 13 Aug 2009 05:37:24 PDT X-Mailer: YahooMailClassic/6.1.2 YahooMailWebService/0.7.338.2 Date: Thu, 13 Aug 2009 05:37:24 -0700 (PDT) From: Barney Cordoba To: Peter Steele , pyunyh@gmail.com In-Reply-To: <430428.63263.qm@web63902.mail.re1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org Subject: Re: nfe taskq performance issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Aug 2009 12:37:26 -0000 =0A=0A--- On Thu, 8/13/09, Barney Cordoba wrote:= =0A=0A> From: Barney Cordoba =0A> Subject: Re: nf= e taskq performance issues=0A> To: "Peter Steele" , pyunyh@gmail.com=0A> Cc: freebsd-net@freebsd.org=0A> Date: Thursday= , August 13, 2009, 7:56 AM=0A> =0A> =0A> --- On Wed, 8/12/09, Pyun YongHyeo= n =0A> wrote:=0A> =0A> > From: Pyun YongHyeon =0A> > Subject: Re: nfe taskq performance issues=0A> > To: "Peter St= eele" =0A> > Cc: freebsd-net@freebsd.org=0A>= > Date: Wednesday, August 12, 2009, 5:35 PM=0A> > On Thu, Jul 23, 2009 at = 08:58:07AM=0A> > -0700, Peter Steele wrote:=0A> > > We've been hitting seri= ous nfe taskq performance=0A> > issues during stress=0A> > > tests and in d= oing some research on the problem=0A> we=0A> > came across this old=0A> > >= email:=0A> > > =0A> > >=A0 =0A> > > =0A> > > From: Ivan Voras =0A> > > Date: April 28, 2009 3:53:14 AM PDT=0A> > > To: freebsd-th= reads@freebsd.org=0A> > > Cc: freebsd-net@freebsd.org,=0A> > freebsd-perfor= mance@freebsd.org=0A> > > Subject: Re: FreeBSD 7.1 taskq em performance=0A>= > > >=0A> > > > I have been hitting some barrier with=0A> FreeBSD 7.1=0A> = > network performance.=0A> > > I=0A> > > > have written an application whic= h contains=0A> two=0A> > kernel threads that=0A> > > takes=0A> > > > mbufs = directly from a network interface and=0A> > forwards to another=0A> > > net= work=0A> > > > interface. This idea is to simulate=0A> different=0A> > netw= ork environment.=0A> > > >=0A> > > > I have been using FreeBSD 6.4 amd64 an= d=0A> tested=0A> > with an Ixia box=0A> > > > (specialised hardware firing = very high=0A> packet=0A> > rate). The PC was a=0A> > > Core2 2.6=0A> > > > = GHz with dual ports Intel PCIE Gigabit=0A> network=0A> > card. It can manag= e up=0A> > > to 1.2=0A> > > > million pps.=0A> > > >=0A> > > > I have a hig= her spec PC with FreeBSD 7.1=0A> amd64=0A> > and Quadcore 2.3 GHz=0A> > > a= nd=0A> > > > PCIE Gigabit network card. The performance=0A> can=0A> > only = achieve up to 600k=0A> > > pps.=0A> > > > I notice the 'taskq em0' and 'tas= kq em1' is=0A> solid=0A> > 100% CPU but it is=0A> > > not in=0A> > > > Free= BSD 6.4. =0A> > > =0A> > >=A0 =0A> > > =0A> > > In our case we are running = FreeBSD 7.0, but we=0A> are=0A> > seeing our boxes=0A> > > experience serio= us thread starvation issues as=0A> the=0A> > nfe0 cpu percentage=0A> > > cl= imbs steadily while cpu idle time drops at=0A> times to=0A> > 0 percent. Th= is=0A> > > email thread mentioned a patch for the em driver=0A> > here:=0A>= > > =0A> > >=A0 =0A> > > =0A> > > http://people.yandex-team.ru/~wawa/ =0A>= > > =0A> > =0A> > > =0A> > >=A0 =0A= > > > =0A> > > Does anyone know if this patch will work with the=0A> nfe=0A= > > driver?=0A> > > =0A> > =0A> > That's for em(4).=0A> > =0A> > AFAIK all = nfe(4) controllers lacks intelligent=0A> interrupts=0A> > moderation so dri= ver should be prepared to handle=0A> > excessive=0A> > interrupt loads. I'm= not sure whether NVIDIA ethernet=0A> > controllers=0A> > really lacks effi= cient interrupt mitigation mechanism=0A> but=0A> > it=0A> > seems Linux als= o faces the same hardware problem.=0A> > As you might know there is no publ= icly available data=0A> sheet=0A> > for=0A> > NVIDIA controllers so setting= it right looks very hard=0A> to=0A> > me.=0A> > =0A> =0A> Try removing the= INTR_MPSAFE flag from the bus_setup_intr()=0A> call.=0A> The entire point = of using filters is to reduce lock=0A> contention.=0A> It might not solve t= he problem but its clearly an=0A> unnecessary=0A> potential bottleneck.=0A>= =0A> Barney=0A=0AI'm curious as to the statistics on your system. Your qua= d core adapter=0Amay actually be hurting the performance. What is the cpu u= sage=0Adistribution shown by top -SH when you are loaded, and how many =0Ai= nterrupts/second are you getting per nic? It looks like the=0Adefault moder= ation is set to 8000 ints/sec, which is probably ok=0Afor 1 interrupt per N= IC. Its not clear whether multiple msix =0Ainterrupts are allocated. Spread= ing interrupts isn't always a=0Agood thing as it increases lock contention = so much as to be=0Acounterproductive, unless you have properly written mute= x =0Amanagement code; which nfe doesn't. =0A=0ABarney=0A=0A=0A