From owner-freebsd-net@FreeBSD.ORG Thu Nov 11 18:05:45 2010 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6299A106566C for ; Thu, 11 Nov 2010 18:05:45 +0000 (UTC) (envelope-from john@traktor.dnepro.net) Received: from smtp-out.dnepro.net (smtp-out.dnepro.net [195.24.131.41]) by mx1.freebsd.org (Postfix) with ESMTP id D41618FC14 for ; Thu, 11 Nov 2010 18:05:43 +0000 (UTC) Received: from traktor.dnepro.net (localhost [127.0.0.1]) by traktor.dnepro.net (8.14.3/8.14.3) with ESMTP id oABI5ei6057981 for ; Thu, 11 Nov 2010 20:05:40 +0200 (EET) (envelope-from john@traktor.dnepro.net) Received: (from john@localhost) by traktor.dnepro.net (8.14.3/8.14.3/Submit) id oABI5eF4057974 for freebsd-net@freebsd.org; Thu, 11 Nov 2010 20:05:40 +0200 (EET) (envelope-from john) Date: Thu, 11 Nov 2010 20:05:40 +0200 From: Eugene Perevyazko To: freebsd-net@freebsd.org Message-ID: <20101111180539.GC11275@traktor.dnepro.net> Mail-Followup-To: freebsd-net@freebsd.org References: <20101110110428.GA3505@traktor.dnepro.net> <20101111104952.GA11275@traktor.dnepro.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20101111104952.GA11275@traktor.dnepro.net> User-Agent: Mutt/1.4.2.3i Subject: Re: igb dual-port adapter 1200Mbps limit - what to tune? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Nov 2010 18:05:45 -0000 On Thu, Nov 11, 2010 at 12:49:52PM +0200, Eugene Perevyazko wrote: > On Thu, Nov 11, 2010 at 01:47:02AM +0100, Ivan Voras wrote: > > On 11/10/10 12:04, Eugene Perevyazko wrote: > > > > >Tried 2 queues and 1 queue per iface, neither hitting cpu limit. > > > > Are you sure you are not hitting the CPU limit on individual cores? Have > > you tried running "top -H -S"? > > > Sure, even with 1queue per iface load is 40-60% on busy core, with 2 queues it was much lower. > Now I've got the module for mb with 2 more ports, going to see if it helps. The IO module has em interfaces on it and somehow I've already got 2 panics after moving one of vlans to it. In the mean time, can someone explain me what is processed by threads marked like "irq256: igb0" and "igb0 que". May be understanding this will let me pin those threads to cores more optimally. There are (hw.igb.num_queues+1) "irq" threads and (hw.igb.num_queues) "que" threads. Now I just pin them sequentially to even cores (odd ones are HT). Now I use hw.igb.num_queues=2, and with traffic limited to 1200Mbits the busiest core is still 60% idle... -- Eugene Perevyazko