From owner-freebsd-net@FreeBSD.ORG Mon May 20 20:42:24 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0DAB12F2 for ; Mon, 20 May 2013 20:42:24 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id D4A0C1892 for ; Mon, 20 May 2013 20:42:23 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id BFD61B917; Mon, 20 May 2013 16:42:22 -0400 (EDT) From: John Baldwin To: freebsd-net@freebsd.org Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3 Date: Mon, 20 May 2013 16:02:08 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <517A657B.7060003@gmail.com> In-Reply-To: <517A657B.7060003@gmail.com> MIME-Version: 1.0 Message-Id: <201305201602.08281.jhb@freebsd.org> Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 20 May 2013 16:42:22 -0400 (EDT) Cc: "=?iso-8859-1?q?Cl=E9ment_Hermann?= \(nodens\)" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 May 2013 20:42:24 -0000 On Friday, April 26, 2013 7:31:07 am Cl=E9ment Hermann (nodens) wrote: > Hi list, >=20 > We use pf+ALTQ for trafic shaping on some routers. >=20 > We are switching to new servers : Dell PowerEdge R620 with 2 8-cores=20 > Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 (quad port) using=20 > igb driver. The old hardware is using em driver, the CPU load is high=20 > but mostly due to kernel and a large pf ruleset. >=20 > On the new hardware, we see high CPU Interrupt load (up to 95%), even=20 > though there is not much trafic currently (peaks about 150Mbps and=20 > 40Kpps). All queues are used and binded to a cpu according to top, but a= =20 > lot of CPU time is spent on igb queues (interrupt or wait). The load is=20 > fine when we stay below 20Kpps. >=20 > We see no mbuf shortage, no dropped packet, but there is little margin=20 > left on CPU time (about 25% idle at best, most of CPU time is spent on=20 > interrupts), which is disturbing. >=20 > We have done some tuning, but to no avail : If you have the processing_limit set to -1, you should never see CPU time=20 spent in the igb task threads (any such time means there is a bug). One su= ch=20 bug was fixed in 8.x here (that is after 8.3): http://svnweb.freebsd.org/base?view=3Drevision&revision=3D235553 This may not help with any issues in pf(4), but we had workloads at work (n= ot=20 involving pf) where this bug could cause boxes to spend 100% CPU in igb=20 threads. =2D-=20 John Baldwin