From owner-freebsd-hackers@FreeBSD.ORG  Tue Mar 31 22:59:51 2015
Return-Path: <owner-freebsd-hackers@FreeBSD.ORG>
Delivered-To: freebsd-hackers@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 61BE554A;
 Tue, 31 Mar 2015 22:59:51 +0000 (UTC)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 14F5C6C6;
 Tue, 31 Mar 2015 22:59:51 +0000 (UTC)
Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD))
 (envelope-from <slw@zxy.spb.ru>)
 id 1Yd58N-000HbM-M4; Wed, 01 Apr 2015 01:59:47 +0300
Date: Wed, 1 Apr 2015 01:59:47 +0300
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: Adrian Chadd <adrian@freebsd.org>
Subject: Re: irq cpu binding
Message-ID: <20150331225947.GC23643@zxy.spb.ru>
References: <20150328112035.GZ23643@zxy.spb.ru>
 <CAJ-Vmo=sxtLgn1HABbWchu1Hm8kZ4Qy+scPeD7ezkkBzr+QijA@mail.gmail.com>
 <20150328154031.GA23643@zxy.spb.ru>
 <CAJ-VmokpWNy-arZ_=4hNbyAvzBYgDWu4F=vYSDwqPgpjaGHk=w@mail.gmail.com>
 <20150328181026.GB23643@zxy.spb.ru>
 <CAJ-Vmok=0=C+HLvLXEM5pbYp_zP6C498wudAfEA40s4AVpGa_w@mail.gmail.com>
 <20150328183147.GC23643@zxy.spb.ru>
 <CAJ-VmokPdnOSASLcrzHtqs_r6GfUW6Q5ap7r9Tnp1OvKhDh20A@mail.gmail.com>
 <20150328192505.GD23643@zxy.spb.ru>
 <CAJ-Vmo=R5J=LT9ntgZJ4xHZwn8_WEAUC=3wSqPKn3=z8ki_N6A@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <CAJ-Vmo=R5J=LT9ntgZJ4xHZwn8_WEAUC=3wSqPKn3=z8ki_N6A@mail.gmail.com>
User-Agent: Mutt/1.5.23 (2014-03-12)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
Cc: "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org>
X-BeenThere: freebsd-hackers@freebsd.org
X-Mailman-Version: 2.1.18-1
Precedence: list
List-Id: Technical Discussions relating to FreeBSD
 <freebsd-hackers.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-hackers>, 
 <mailto:freebsd-hackers-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-hackers/>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Help: <mailto:freebsd-hackers-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-hackers>,
 <mailto:freebsd-hackers-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 31 Mar 2015 22:59:51 -0000

On Sat, Mar 28, 2015 at 12:33:52PM -0700, Adrian Chadd wrote:

> That's done deferred by the bus interrupt wiring. That's something
> John's been looking into as part of the general NUMA work (and I'm
> trying to debug right now, on dual-socket boxes with ixgbe. :-)
> 
> Look at bus_bind_intr() and the twisty path to intr_event_bind(), then
> x86/x86/intr_machdep.c:intr_assign_cpu(), then intr_shuffle_cpus() at
> boot, versus what happens via calls to pic_assign_cpu to setup the
> wiring.

I am do simple, ugle hack ixgbe driver for let start cpu binding.
I am still see ixgbe in pmc output.

What may be wrong?
What may be miss?

=====
static int ixgbe_start_cpu = 0;
TUNABLE_INT("hw.ix.start_cpu", &ixgbe_start_cpu);
SYSCTL_INT(_hw_ix, OID_AUTO, start_cpu, CTLFLAG_RDTUN, &ixgbe_start_cpu, 0,
    "Start CPU for next IRQ binding");

[...]
                if (adapter->num_queues > 1)
                        bus_bind_intr(dev, que->res, i+ixgbe_start_cpu);

#ifndef IXGBE_LEGACY_TX
                TASK_INIT(&txr->txq_task, 0, ixgbe_deferred_mq_start, txr);
#endif
                TASK_INIT(&que->que_task, 0, ixgbe_handle_que, que);
                que->tq = taskqueue_create_fast("ixgbe_que", M_NOWAIT,
                    taskqueue_thread_enqueue, &que->tq);
                taskqueue_start_threads(&que->tq, 1, PI_NET, "%s que",
                    device_get_nameunit(adapter->dev));
        }
        ixgbe_start_cpu += adapter->num_queues;