Date: Fri, 4 May 2012 08:28:17 -0400 From: John Baldwin <jhb@freebsd.org> To: freebsd-hackers@freebsd.org Cc: Navdeep Parhar <np@freebsd.org> Subject: Re: CPU selection for ithreads on 8.3 Message-ID: <201205040828.18027.jhb@freebsd.org> In-Reply-To: <4FA1A421.6020601@FreeBSD.org> References: <4FA1A421.6020601@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wednesday, May 02, 2012 5:16:17 pm Navdeep Parhar wrote:
> There seems to be a regression in 8.3 in the way the kernel selects CPUs
> for interrupts. For example, cxgb(4) on 8.3 ends up with all
> its ithreads on the same CPU (CPU7 in this case).
>
> 12 root -68 - 0K 816K WAIT 7 0:55 0.00% intr{irq279:
> cxgbc0}
> 12 root -68 - 0K 816K WAIT 7 0:52 0.00% intr{irq275:
> cxgbc0}
> 12 root -68 - 0K 816K WAIT 7 0:47 0.00% intr{irq278:
> cxgbc0}
> 12 root -68 - 0K 816K WAIT 7 0:43 0.00% intr{irq277:
> cxgbc0}
> 12 root -68 - 0K 816K WAIT 7 0:43 0.00% intr{irq282:
> cxgbc0}
> 12 root -68 - 0K 816K WAIT 7 0:41 0.00% intr{irq281:
> cxgbc0}
> 12 root -68 - 0K 816K WAIT 7 0:32 0.00% intr{irq276:
> cxgbc0}
> 12 root -68 - 0K 816K WAIT 7 0:31 0.00% intr{irq280:
> cxgbc0}
>
> Back in the day there used to be code in cxgb to bind different
> interrupts to different CPUs but it was removed because the kernel
> distributed them across CPUs anyway. So what changed? This appears 8.3
> specific. I don't see it on head and I don't have a 9 system readily
> available right now.
Hmmm, that seems odd. It is true that the round-robin that the OS does only
pins the low-level message from the APIC/MSI vector to the CPU. It does not
affect the thread. However, ULE prefers to run ithreads on the CPU that
processes the interrupt:
static int
sched_pickcpu(struct thread *td, int flags)
{
...
/*
* Prefer to run interrupt threads on the processors that generate
* the interrupt.
*/
if (td->td_priority <= PRI_MAX_ITHD && THREAD_CAN_SCHED(td, self) &&
curthread->td_intr_nesting_level && ts->ts_cpu != self) {
SCHED_STAT_INC(pickcpu_intrbind);
ts->ts_cpu = self;
}
}
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201205040828.18027.jhb>