From owner-freebsd-hackers@FreeBSD.ORG Fri May 4 15:06:02 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 99C6910657C0; Fri, 4 May 2012 15:06:02 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 703908FC0C; Fri, 4 May 2012 15:06:02 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C5F6AB96F; Fri, 4 May 2012 11:06:01 -0400 (EDT) From: John Baldwin To: freebsd-hackers@freebsd.org Date: Fri, 4 May 2012 08:28:17 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p13; KDE/4.5.5; amd64; ; ) References: <4FA1A421.6020601@FreeBSD.org> In-Reply-To: <4FA1A421.6020601@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201205040828.18027.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 04 May 2012 11:06:01 -0400 (EDT) Cc: Navdeep Parhar Subject: Re: CPU selection for ithreads on 8.3 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 May 2012 15:06:02 -0000 On Wednesday, May 02, 2012 5:16:17 pm Navdeep Parhar wrote: > There seems to be a regression in 8.3 in the way the kernel selects CPUs > for interrupts. For example, cxgb(4) on 8.3 ends up with all > its ithreads on the same CPU (CPU7 in this case). > > 12 root -68 - 0K 816K WAIT 7 0:55 0.00% intr{irq279: > cxgbc0} > 12 root -68 - 0K 816K WAIT 7 0:52 0.00% intr{irq275: > cxgbc0} > 12 root -68 - 0K 816K WAIT 7 0:47 0.00% intr{irq278: > cxgbc0} > 12 root -68 - 0K 816K WAIT 7 0:43 0.00% intr{irq277: > cxgbc0} > 12 root -68 - 0K 816K WAIT 7 0:43 0.00% intr{irq282: > cxgbc0} > 12 root -68 - 0K 816K WAIT 7 0:41 0.00% intr{irq281: > cxgbc0} > 12 root -68 - 0K 816K WAIT 7 0:32 0.00% intr{irq276: > cxgbc0} > 12 root -68 - 0K 816K WAIT 7 0:31 0.00% intr{irq280: > cxgbc0} > > Back in the day there used to be code in cxgb to bind different > interrupts to different CPUs but it was removed because the kernel > distributed them across CPUs anyway. So what changed? This appears 8.3 > specific. I don't see it on head and I don't have a 9 system readily > available right now. Hmmm, that seems odd. It is true that the round-robin that the OS does only pins the low-level message from the APIC/MSI vector to the CPU. It does not affect the thread. However, ULE prefers to run ithreads on the CPU that processes the interrupt: static int sched_pickcpu(struct thread *td, int flags) { ... /* * Prefer to run interrupt threads on the processors that generate * the interrupt. */ if (td->td_priority <= PRI_MAX_ITHD && THREAD_CAN_SCHED(td, self) && curthread->td_intr_nesting_level && ts->ts_cpu != self) { SCHED_STAT_INC(pickcpu_intrbind); ts->ts_cpu = self; } } -- John Baldwin