From owner-freebsd-hackers@FreeBSD.ORG Wed Apr 1 06:23:09 2015 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A80ED0C; Wed, 1 Apr 2015 06:23:09 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C013B7D3; Wed, 1 Apr 2015 06:23:08 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YdC3I-000PIx-4Q; Wed, 01 Apr 2015 09:23:00 +0300 Date: Wed, 1 Apr 2015 09:23:00 +0300 From: Slawa Olhovchenkov To: Adrian Chadd Subject: Re: irq cpu binding Message-ID: <20150401062259.GD23643@zxy.spb.ru> References: <20150328154031.GA23643@zxy.spb.ru> <20150328181026.GB23643@zxy.spb.ru> <20150328183147.GC23643@zxy.spb.ru> <20150328192505.GD23643@zxy.spb.ru> <20150331225947.GC23643@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 Apr 2015 06:23:09 -0000 On Tue, Mar 31, 2015 at 04:14:04PM -0700, Adrian Chadd wrote: > You also have to do the taskqueue threads. # procstat -ta | grep ix 0 101305 kernel ix0 que 6 8 sleep - 0 101306 kernel ix0 que 7 8 sleep - 0 101307 kernel ix0 que 8 8 sleep - 0 101308 kernel ix0 linkq 7 8 sleep - 0 101309 kernel ix1 que 9 8 sleep - 0 101310 kernel ix1 que 10 8 sleep - 0 101311 kernel ix1 que 11 8 sleep - 0 101312 kernel ix1 linkq 1 8 sleep - 12 100064 intr irq270: ix0:que 6 8 run - 12 100066 intr irq271: ix0:que 7 8 run - 12 100068 intr irq272: ix0:que 8 8 run - 12 100070 intr irq273: ix0:link 5 8 wait - 12 100072 intr irq274: ix1:que 9 8 wait - 12 100074 intr irq275: ix1:que 10 8 wait - 12 100076 intr irq276: ix1:que 11 8 run - 12 100078 intr irq277: ix1:link 9 8 wait - # cpuset -g -t 101305 tid 101305 mask: 6 # cpuset -g -t 101306 tid 101306 mask: 7 # cpuset -g -t 101307 tid 101307 mask: 8 # cpuset -g -t 101309 tid 101309 mask: 9 # cpuset -g -t 101310 tid 101310 mask: 10 # cpuset -g -t 101311 tid 101311 mask: 11 # cpuset -g -t 100064 tid 100064 mask: 6 # cpuset -g -t 100066 tid 100066 mask: 7 # cpuset -g -t 100068 tid 100068 mask: 8 # cpuset -g -t 100072 tid 100072 mask: 9 # cpuset -g -t 100074 tid 100074 mask: 10 # cpuset -g -t 100076 tid 100076 mask: 11 Or you talk abot somewere different? > (I mean, pmc could also be broken...) Why? > -a > > > On 31 March 2015 at 15:59, Slawa Olhovchenkov wrote: > > On Sat, Mar 28, 2015 at 12:33:52PM -0700, Adrian Chadd wrote: > > > >> That's done deferred by the bus interrupt wiring. That's something > >> John's been looking into as part of the general NUMA work (and I'm > >> trying to debug right now, on dual-socket boxes with ixgbe. :-) > >> > >> Look at bus_bind_intr() and the twisty path to intr_event_bind(), then > >> x86/x86/intr_machdep.c:intr_assign_cpu(), then intr_shuffle_cpus() at > >> boot, versus what happens via calls to pic_assign_cpu to setup the > >> wiring. > > > > I am do simple, ugle hack ixgbe driver for let start cpu binding. > > I am still see ixgbe in pmc output. > > > > What may be wrong? > > What may be miss? > > > > ===== > > static int ixgbe_start_cpu = 0; > > TUNABLE_INT("hw.ix.start_cpu", &ixgbe_start_cpu); > > SYSCTL_INT(_hw_ix, OID_AUTO, start_cpu, CTLFLAG_RDTUN, &ixgbe_start_cpu, 0, > > "Start CPU for next IRQ binding"); > > > > [...] > > if (adapter->num_queues > 1) > > bus_bind_intr(dev, que->res, i+ixgbe_start_cpu); > > > > #ifndef IXGBE_LEGACY_TX > > TASK_INIT(&txr->txq_task, 0, ixgbe_deferred_mq_start, txr); > > #endif > > TASK_INIT(&que->que_task, 0, ixgbe_handle_que, que); > > que->tq = taskqueue_create_fast("ixgbe_que", M_NOWAIT, > > taskqueue_thread_enqueue, &que->tq); > > taskqueue_start_threads(&que->tq, 1, PI_NET, "%s que", > > device_get_nameunit(adapter->dev)); > > } > > ixgbe_start_cpu += adapter->num_queues; > >