Date: Sat, 28 Jan 2012 04:41:45 +0100 From: Attilio Rao <attilio@freebsd.org> To: Ryan Stone <rysto32@gmail.com> Cc: freebsd-hackers@freebsd.org Subject: Re: Kernel threads inherit CPU affinity from random sibling Message-ID: <CAJ-FndDHsQRRNmrS7fsELEVohozWvvfaZ6eW_GipwHdjU9ZwxA@mail.gmail.com> In-Reply-To: <CAFMmRNxF1uMOr39BbZkpPN=uM7G09dtcckAYw8ag6n6bi=FeOw@mail.gmail.com> References: <CAFMmRNxF1uMOr39BbZkpPN=uM7G09dtcckAYw8ag6n6bi=FeOw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
2012/1/28 Ryan Stone <rysto32@gmail.com>:
> Right now, whenever a thread is spawned, it inherits CPU affinity from
> its "parent" thread. =C2=A0I put parent in scare quotes because as far as=
I
> can tell, for a kernel thread the parent is essentially chosen
> arbitrarily (it looks like it is the most recent thread spawned in
> that kernel process). =C2=A0Inheriting affinity is arguably correct for
> userland threads (at the very least, an explicit design decision to
> implement inheritance was clearly implemented). =C2=A0However for kernel
> threads, this behaviour leads to bizarre results that clearly weren't
> intended. =C2=A0For example, I added a printf to sched_fork_thread() that
> prints a message every time a thread inherits affinity from its
> "parent". =C2=A0Here's the results from booting a dual-core VM with
> net.isr.bindthreads=3D1:
>
> Thread 100006 inheriting affinity from parent swi1: netisr 0(100005)
> Thread 100007 inheriting affinity from parent swi3: vm(100006)
> Thread 100008 inheriting affinity from parent swi4: clock(100007)
> Thread 100014 inheriting affinity from parent swi4: clock(100008)
> Thread 100017 inheriting affinity from parent swi5: +(100014)
> Thread 100018 inheriting affinity from parent swi6: Giant taskq(100017)
> Thread 100022 inheriting affinity from parent swi6: task queue(100018)
> Thread 100025 inheriting affinity from parent swi2: cambio(100022)
> Thread 100026 inheriting affinity from parent irq14: ata0(100025)
> Thread 100027 inheriting affinity from parent irq15: ata1(100026)
> Thread 100028 inheriting affinity from parent irq19: le0(100027)
> Thread 100029 inheriting affinity from parent irq21: pcm0(100028)
> Thread 100034 inheriting affinity from parent irq22: ohci0(100029)
> Thread 100035 inheriting affinity from parent irq1: atkbd0(100034)
>
> The result is that every thread in intr kernel process ends up
> inheriting the affinity(my favourite part is that both softclock
> threads are now forced to fight over the same CPU). =C2=A0I am working on
> the following patch that adds a flag to
> sched_fork()/sched_fork_thread() that can be passed to force the child
> thread to *not* inherit affinity. =C2=A0However, this doesn't address
> affinity inheritance from calls to kproc_create(). =C2=A0I suppose that I
> could push the flag up into fork1(), but I'd prefer not to. =C2=A0I am
> wondering if it even makes sense for affinity to be inherited from a
> parent process even for userland processes.
>
> What are peoples thoughts? =C2=A0Is this the right approach?
I think what you found out is very sensitive.
However, the patch is not correct as you cannot call
cpuset_setthread() with thread_lock held. Also, I don't think we need
the flag bloat, to be honest.
IMHO, we should choose a default policy for sched_fork{_thread}() and
have the opposite case to fix the thread inherited affinity manually.
For example, for the shortest patch, we can go with
sched_fork{_thread}() functions to directly inherited parent affinity
which is ok in the userland case but needs to be fixed in the
kthread/kproc case. Hence this is my fix:
http://www.freebsd.org/~attilio/cpuset_root.patch
(just test-compiled, so if you agree and want to test further, be my guest)=
.
Maybe we can also add a comment to sched_fork() guys explaining the
default policy, but the code seems self-explanatory enough to me.
What do you think about it?
Thanks,
Attilio
--=20
Peace can only be achieved by understanding - A. Einstein
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-FndDHsQRRNmrS7fsELEVohozWvvfaZ6eW_GipwHdjU9ZwxA>