From owner-freebsd-hackers@FreeBSD.ORG Sat Jan 28 03:41:46 2012 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D7DA3106566B for ; Sat, 28 Jan 2012 03:41:46 +0000 (UTC) (envelope-from asmrookie@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 6AF6E8FC0A for ; Sat, 28 Jan 2012 03:41:46 +0000 (UTC) Received: by werm13 with SMTP id m13so390059wer.13 for ; Fri, 27 Jan 2012 19:41:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=HUYna00jqQPK7ZLZyfSIXoKZjIW0pD7c8MhL7lqhvR4=; b=PhV1a6B5XVTeqYECEExVrv19I95sNKaYvgBPKsPjRA9VkDl/OSiMAhxpAZhca+jCR7 HDCWRwhvK92naOK1PwRMsiRx7h2mvQdWPAx4G088KpVzll2VFcv7bYedCP5EtAT3x8I8 dGvZYoeUX+68m9dNIZOUPHdCPO5cxdH3i/UNE= MIME-Version: 1.0 Received: by 10.216.133.82 with SMTP id p60mr3984361wei.59.1327722105411; Fri, 27 Jan 2012 19:41:45 -0800 (PST) Sender: asmrookie@gmail.com Received: by 10.216.177.73 with HTTP; Fri, 27 Jan 2012 19:41:45 -0800 (PST) In-Reply-To: References: Date: Sat, 28 Jan 2012 04:41:45 +0100 X-Google-Sender-Auth: PpD7FXlRm4TtMm9sTYks6mkB3ik Message-ID: From: Attilio Rao To: Ryan Stone Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org Subject: Re: Kernel threads inherit CPU affinity from random sibling X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jan 2012 03:41:47 -0000 2012/1/28 Ryan Stone : > Right now, whenever a thread is spawned, it inherits CPU affinity from > its "parent" thread. =C2=A0I put parent in scare quotes because as far as= I > can tell, for a kernel thread the parent is essentially chosen > arbitrarily (it looks like it is the most recent thread spawned in > that kernel process). =C2=A0Inheriting affinity is arguably correct for > userland threads (at the very least, an explicit design decision to > implement inheritance was clearly implemented). =C2=A0However for kernel > threads, this behaviour leads to bizarre results that clearly weren't > intended. =C2=A0For example, I added a printf to sched_fork_thread() that > prints a message every time a thread inherits affinity from its > "parent". =C2=A0Here's the results from booting a dual-core VM with > net.isr.bindthreads=3D1: > > Thread 100006 inheriting affinity from parent swi1: netisr 0(100005) > Thread 100007 inheriting affinity from parent swi3: vm(100006) > Thread 100008 inheriting affinity from parent swi4: clock(100007) > Thread 100014 inheriting affinity from parent swi4: clock(100008) > Thread 100017 inheriting affinity from parent swi5: +(100014) > Thread 100018 inheriting affinity from parent swi6: Giant taskq(100017) > Thread 100022 inheriting affinity from parent swi6: task queue(100018) > Thread 100025 inheriting affinity from parent swi2: cambio(100022) > Thread 100026 inheriting affinity from parent irq14: ata0(100025) > Thread 100027 inheriting affinity from parent irq15: ata1(100026) > Thread 100028 inheriting affinity from parent irq19: le0(100027) > Thread 100029 inheriting affinity from parent irq21: pcm0(100028) > Thread 100034 inheriting affinity from parent irq22: ohci0(100029) > Thread 100035 inheriting affinity from parent irq1: atkbd0(100034) > > The result is that every thread in intr kernel process ends up > inheriting the affinity(my favourite part is that both softclock > threads are now forced to fight over the same CPU). =C2=A0I am working on > the following patch that adds a flag to > sched_fork()/sched_fork_thread() that can be passed to force the child > thread to *not* inherit affinity. =C2=A0However, this doesn't address > affinity inheritance from calls to kproc_create(). =C2=A0I suppose that I > could push the flag up into fork1(), but I'd prefer not to. =C2=A0I am > wondering if it even makes sense for affinity to be inherited from a > parent process even for userland processes. > > What are peoples thoughts? =C2=A0Is this the right approach? I think what you found out is very sensitive. However, the patch is not correct as you cannot call cpuset_setthread() with thread_lock held. Also, I don't think we need the flag bloat, to be honest. IMHO, we should choose a default policy for sched_fork{_thread}() and have the opposite case to fix the thread inherited affinity manually. For example, for the shortest patch, we can go with sched_fork{_thread}() functions to directly inherited parent affinity which is ok in the userland case but needs to be fixed in the kthread/kproc case. Hence this is my fix: http://www.freebsd.org/~attilio/cpuset_root.patch (just test-compiled, so if you agree and want to test further, be my guest)= . Maybe we can also add a comment to sched_fork() guys explaining the default policy, but the code seems self-explanatory enough to me. What do you think about it? Thanks, Attilio --=20 Peace can only be achieved by understanding - A. Einstein