From owner-freebsd-net@FreeBSD.ORG  Thu May  9 14:55:52 2013
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by hub.freebsd.org (Postfix) with ESMTP id D5EC9472
 for <freebsd-net@freebsd.org>; Thu,  9 May 2013 14:55:52 +0000 (UTC)
 (envelope-from egrosbein@rdtc.ru)
Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13::5])
 by mx1.freebsd.org (Postfix) with ESMTP id 74249BE9
 for <freebsd-net@freebsd.org>; Thu,  9 May 2013 14:55:52 +0000 (UTC)
Received: from eg.sd.rdtc.ru (localhost [127.0.0.1])
 by eg.sd.rdtc.ru (8.14.6/8.14.6) with ESMTP id r49EtlfF007170;
 Thu, 9 May 2013 21:55:48 +0700 (NOVT)
 (envelope-from egrosbein@rdtc.ru)
Message-ID: <518BB8EE.2080705@rdtc.ru>
Date: Thu, 09 May 2013 21:55:42 +0700
From: Eugene Grosbein <egrosbein@rdtc.ru>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:17.0) Gecko/20130415 Thunderbird/17.0.5
MIME-Version: 1.0
To: =?UTF-8?B?IkNsw6ltZW50IEhlcm1hbm4gKG5vZGVucyki?=
 <nodens2099@gmail.com>
Subject: Re: High CPU interrupt load on intel I350T4 with igb on 8.3
References: <517A657B.7060003@gmail.com>
In-Reply-To: <517A657B.7060003@gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Cc: freebsd-net@freebsd.org
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 09 May 2013 14:55:52 -0000

On 26.04.2013 18:31, "Clément Hermann (nodens)" wrote:
> Hi list,
> 
> We use pf+ALTQ for trafic shaping on some routers.
> 
> We are switching to new servers : Dell PowerEdge R620 with 2 8-cores 
> Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 (quad port) using 
> igb driver. The old hardware is using em driver, the CPU load is high 
> but mostly due to kernel and a large pf ruleset.
> 
> On the new hardware, we see high CPU Interrupt load (up to 95%), even 
> though there is not much trafic currently (peaks about 150Mbps and 
> 40Kpps). All queues are used and binded to a cpu according to top, but a 
> lot of CPU time is spent on igb queues (interrupt or wait). The load is 
> fine when we stay below 20Kpps.
> 
> We see no mbuf shortage, no dropped packet, but there is little margin 
> left on CPU time (about 25% idle at best, most of CPU time is spent on 
> interrupts), which is disturbing.

It seems you suffer from pf lock contention. You should stop using pf
with multi-core systems with 8.3. Move to ipfw+dummynet or ng_car for 8.3
or move to 10.0-CURRENT having new, rewritten pf that does not have this problem.

Network device driver is not guilty here, that's just pf's contention
running in igb's context.

Eugene Grosbein