From owner-freebsd-net@FreeBSD.ORG Wed Apr 25 01:02:25 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E585B1065675 for ; Wed, 25 Apr 2012 01:02:25 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-wg0-f50.google.com (mail-wg0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 35E438FC1A for ; Wed, 25 Apr 2012 01:02:25 +0000 (UTC) Received: by wgbds12 with SMTP id ds12so1227245wgb.31 for ; Tue, 24 Apr 2012 18:02:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding:x-gm-message-state; bh=Ka2gCNteb+T/hw41ZsjC7eECi0JHNkQjrP0d3hVkVq4=; b=Ft7YfQOqv1zCJ0D5C0HdLe2TdB14ZAyn2QjOqSCcL/6zMiYckQ8gVflJeiq9ePVsAA Awcb/vDDPxJF5/BQVCXQyF7NSYQ7XkAMYl1/zNerdf8FBj3LrCQYC3m1vlpkQRVl7GIA CRXU57fwk5f1/szZ98jhQ28sCWiLn4sSneYFriiXs++QXnJT5IaISwiwrUip+0gArsjA sXzlTv0l4d4yiBFFF8dOTpL9CNfk3AFxjIeVXDzXpXunEtXWV5cDXEBdpmSaBM+f9uKY xmF09VduOWj8Sp0zQ9fDmlqRLHyg4ZE1Zj/zg2lXLgQEIEKLr+hNY0GLSvf3pjPvllbt 7Gww== Received: by 10.216.135.223 with SMTP id u73mr317687wei.117.1335315744089; Tue, 24 Apr 2012 18:02:24 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.216.110.67 with HTTP; Tue, 24 Apr 2012 18:02:03 -0700 (PDT) In-Reply-To: <1335312667.11564.13.camel@powernoodle-l7.corp.yahoo.com> References: <1335312667.11564.13.camel@powernoodle-l7.corp.yahoo.com> From: Juli Mallett Date: Tue, 24 Apr 2012 18:02:03 -0700 X-Google-Sender-Auth: 6H6E7qrbB6iYJbZSN3OzziLnZSQ Message-ID: To: sbruno@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Gm-Message-State: ALoCoQlL46sd0VKAvsK29HccjFHSir6BsqPxAvmnpDB8Ne8+HZ9I+fnciFBm9o7rqXXki9NH7Pv5 Cc: "freebsd-net@freebsd.org" , Jack Vogel , John Baldwin Subject: Re: igb(4) Pondering a bind to cpu patch X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Apr 2012 01:02:26 -0000 On Tue, Apr 24, 2012 at 17:11, Sean Bruno wrote: > http://people.freebsd.org/~sbruno/if_igb.c.txt > > 8 core machine > 2 igb(4) interfaces > set num_queues=3D4 > > igb0:0 --> cpu0 > [...] > igb1:0 --> cpu0 > [...] > > I suspect, that we need a static global to keep track of what cpu last > was last bound to a queue. =C2=A0My patch does do this, but I don't know = if > its the right thing. > > Since I'm doing multiple interfaces, I need to make sure I don't > schedule a queue to a non existent cpu, so take a modulo of the counter > and the number of cpus in the box. This seems like a plausible approach, and certainly much more DWIM than what I've done in the past, which is to use cpuset with -x to bind each IRQ to a core by hand. If there were a way for interfaces to export queue information including any relevant IRQs, it would be easy to make a frontend that would use cpuset in a usable way, but right now that's the solution I've found most tenable and uninvasive. The question here is what behavior to assume the user wants when they restrict the number of queues, which is why putting the policy bits into userland would be preferable to this sort of scheme, I suppose.