From owner-freebsd-net@FreeBSD.ORG Thu May 9 16:25:28 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4B637BDB for ; Thu, 9 May 2013 16:25:28 +0000 (UTC) (envelope-from barney_cordoba@yahoo.com) Received: from nm15-vm2.bullet.mail.ne1.yahoo.com (nm15-vm2.bullet.mail.ne1.yahoo.com [98.138.91.91]) by mx1.freebsd.org (Postfix) with ESMTP id DEBF07C for ; Thu, 9 May 2013 16:25:27 +0000 (UTC) Received: from [98.138.226.176] by nm15.bullet.mail.ne1.yahoo.com with NNFMP; 09 May 2013 16:25:21 -0000 Received: from [98.138.89.164] by tm11.bullet.mail.ne1.yahoo.com with NNFMP; 09 May 2013 16:25:20 -0000 Received: from [127.0.0.1] by omp1020.mail.ne1.yahoo.com with NNFMP; 09 May 2013 16:25:20 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 971680.98652.bm@omp1020.mail.ne1.yahoo.com Received: (qmail 51753 invoked by uid 60001); 9 May 2013 16:25:20 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1368116720; bh=sbQswaQvDIJgE7Iy8BZkzuBd2514ZPzDPqpb/Qln1Ro=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=E4V+b0CllZJg6b2b9SlqBok6g3npz+5NmddOf+XEa8wD8JZaTM+Tw/W7Ljn1r/4pJjxwbeExYP7Q2MnW+g+urr7VGwS/WAMmoS7xYTMFbMKjW8+Nlww6IElD6s1oXlkn14ms2GDlTGUBrfiLl1LGBO09ZoFPnpy+c8c8QOu+T0U= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:Message-ID:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=yPnbkaOC8UmBm8qV0L2ye6ZeylXEqFMS81DtvAGhXxf+JZuQxMCT+gm9NTMyY8fw4uA/nV7IDbUJSyRwC91qVEHDvvQowv5dXj4G6hFZayvOOv79LMnqQjx0WigeCVqJfEUVqjp2tmowd/l6DMnyVH+bfQ5euIMt71b5iQb/SX4=; X-YMail-OSG: H.XcFvgVM1n4KDeuMPlkcHYekqQ1pBWhHdj2XviZCG92q5M 8nQQFQtI7W0K7jEnMF7hAEjdIrwhbzCVJ49JlhreHeGRNyAhuLGYapH9Axxr .HfvXezpS4nezYxdqa1eCBYkE3vpzO3tK_XjsuYALk9ux33gXeOXWReGArKM arLjyxkz1A5yCNMnRCsYx8J4Y7RlXmRyqgDWZAvBmgM7TMc5eL_ejpVmDeZ5 1AYP2pOX4AAM8j4xnfeKHGiRxHVDer.XgbbzEiY7lHAHV8vU2H_eN1K_U9B2 pc51nwvp3zTEMT2EAXjZXgV2mL.Wn3ZbemApwiUaUSAWYCYybPk0p6C0gZg4 Ry8Vc.EkQE1Z4CtXHLW8puYw.hRfdh6hg2bB_cLiDFj9YW2lu4VMuqpQno1V 8uaLP2tEdaAt7V_cKqep32a4A5ru3GRVgB48HEe0zLClg4QhUZC5dqAhjD8F V0iJjkp_8q8xDJVbTbpZFeZS3C9ikS_cDnn7aLrFeuLLKdzokgxBQ8dEgTTw Ws2mp99OYkxVMlQSdeM6yeSzbNXLyiw-- Received: from [98.203.118.124] by web121606.mail.ne1.yahoo.com via HTTP; Thu, 09 May 2013 09:25:20 PDT X-Rocket-MIMEInfo: 002.001, CgotLS0gT24gVGh1LCA1LzkvMTMsIEV1Z2VuZSBHcm9zYmVpbiA8ZWdyb3NiZWluQHJkdGMucnU.IHdyb3RlOgoKPiBGcm9tOiBFdWdlbmUgR3Jvc2JlaW4gPGVncm9zYmVpbkByZHRjLnJ1Pgo.IFN1YmplY3Q6IFJlOiBIaWdoIENQVSBpbnRlcnJ1cHQgbG9hZCBvbiBpbnRlbCBJMzUwVDQgd2l0aCBpZ2Igb24gOC4zCj4gVG86ICIiQ2zDqW1lbnQgSGVybWFubiAobm9kZW5zKSIiIDxub2RlbnMyMDk5QGdtYWlsLmNvbT4KPiBDYzogZnJlZWJzZC1uZXRAZnJlZWJzZC5vcmcKPiBEYXRlOiBUaHVyc2RheSwgTWEBMAEBAQE- X-Mailer: YahooMailClassic/15.1.8 YahooMailWebService/0.8.141.536 Message-ID: <1368116720.51479.YahooMailClassic@web121606.mail.ne1.yahoo.com> Date: Thu, 9 May 2013 09:25:20 -0700 (PDT) From: Barney Cordoba Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 To: =?iso-8859-1?Q?Cl=E9ment_Hermann_=28nodens=29?= , Eugene Grosbein In-Reply-To: <518BB8EE.2080705@rdtc.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 May 2013 16:25:28 -0000 =0A=0A--- On Thu, 5/9/13, Eugene Grosbein wrote:=0A=0A>= From: Eugene Grosbein =0A> Subject: Re: High CPU interr= upt load on intel I350T4 with igb on 8.3=0A> To: ""Cl=E9ment Hermann (noden= s)"" =0A> Cc: freebsd-net@freebsd.org=0A> Date: Thurs= day, May 9, 2013, 10:55 AM=0A> On 26.04.2013 18:31, "Cl=E9ment=0A> Hermann = (nodens)" wrote:=0A> > Hi list,=0A> > =0A> > We use pf+ALTQ for trafic shap= ing on some routers.=0A> > =0A> > We are switching to new servers : Dell Po= werEdge R620=0A> with 2 8-cores =0A> > Intel Processor (E5-2650L), 8GB RAM = and Intel I350T4=0A> (quad port) using =0A> > igb driver. The old hardware = is using em driver, the=0A> CPU load is high =0A> > but mostly due to kerne= l and a large pf ruleset.=0A> > =0A> > On the new hardware, we see high CPU= Interrupt load (up=0A> to 95%), even =0A> > though there is not much trafi= c currently (peaks about=0A> 150Mbps and =0A> > 40Kpps). All queues are use= d and binded to a cpu=0A> according to top, but a =0A> > lot of CPU time is= spent on igb queues (interrupt or=0A> wait). The load is =0A> > fine when = we stay below 20Kpps.=0A> > =0A> > We see no mbuf shortage, no dropped pack= et, but there=0A> is little margin =0A> > left on CPU time (about 25% idle = at best, most of CPU=0A> time is spent on =0A> > interrupts), which is dist= urbing.=0A> =0A> It seems you suffer from pf lock contention. You should st= op=0A> using pf=0A> with multi-core systems with 8.3. Move to ipfw+dummynet= or=0A> ng_car for 8.3=0A> or move to 10.0-CURRENT having new, rewritten pf= that does=0A> not have this problem.=0A> =0A> Network device driver is not= guilty here, that's just pf's=0A> contention=0A> running in igb's context.= =0A> =0A> Eugene Grosbein=0A=0AThey're both at play. Single threadedness ag= gravates subsystems that =0Ahave too many lock points.=0A=0AIt can also be = "solved" with using 1 queue, because then you don't=0Ahave 4 queues going i= nto a single thread.=0A=0ABC