From owner-freebsd-net@FreeBSD.ORG  Wed Apr 25 13:35:30 2012
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id ABBA6106564A;
	Wed, 25 Apr 2012 13:35:30 +0000 (UTC) (envelope-from jhb@freebsd.org)
Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net
	[IPv6:2001:470:1f10:75::2])
	by mx1.freebsd.org (Postfix) with ESMTP id 81F128FC15;
	Wed, 25 Apr 2012 13:35:30 +0000 (UTC)
Received: from jhbbsd.localnet (unknown [209.249.190.124])
	by bigwig.baldwin.cx (Postfix) with ESMTPSA id E204CB915;
	Wed, 25 Apr 2012 09:35:29 -0400 (EDT)
From: John Baldwin <jhb@freebsd.org>
To: sbruno@freebsd.org
Date: Wed, 25 Apr 2012 09:32:21 -0400
User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; )
References: <1335312667.11564.13.camel@powernoodle-l7.corp.yahoo.com>
In-Reply-To: <1335312667.11564.13.camel@powernoodle-l7.corp.yahoo.com>
MIME-Version: 1.0
Content-Type: Text/Plain;
  charset="utf-8"
Content-Transfer-Encoding: 7bit
Message-Id: <201204250932.21378.jhb@freebsd.org>
X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7
	(bigwig.baldwin.cx); Wed, 25 Apr 2012 09:35:30 -0400 (EDT)
Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>,
	Jack Vogel <jfvogel@gmail.com>
Subject: Re: igb(4) Pondering a bind to cpu patch
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2012 13:35:30 -0000

On Tuesday, April 24, 2012 8:11:07 pm Sean Bruno wrote:
> http://people.freebsd.org/~sbruno/if_igb.c.txt
> 
> Scenario I've just seen:
> 
> 8 core machine
> 2 igb(4) interfaces
> set num_queues=4
> 
> igb0:0 --> cpu0
> igb0:1 --> cpu1
> igb0:2 --> cpu2
> igb0:3 --> cpu3
> 
> igb1:0 --> cpu0
> igb1:1 --> cpu1
> igb1:2 --> cpu2
> igb1:3 --> cpu3
> 
> I suspect, that we need a static global to keep track of what cpu last
> was last bound to a queue.  My patch does do this, but I don't know if
> its the right thing.
> 
> Since I'm doing multiple interfaces, I need to make sure I don't
> schedule a queue to a non existent cpu, so take a modulo of the counter
> and the number of cpus in the box.  
> 
> Perhaps not the most elegant solution, but its a thing?

CPU IDs are not guaranteed to be dense.  However, you can use CPU_FIRST() and 
CPU_NEXT() with your static global instead.

OTOH, if igb were to just leave the interrupts alone instead of binding them 
by hand, they would get round-robin assigned among available cores already.  I 
think in this case the best approach might be to add a tunable to disable 
igb's manual binding and instead let the default system round-robin be 
preserved.

-- 
John Baldwin