From owner-freebsd-arch@FreeBSD.ORG Fri Feb 14 19:10:44 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 700EE2D6; Fri, 14 Feb 2014 19:10:44 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 434551B9F; Fri, 14 Feb 2014 19:10:44 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 990C7B9A0; Fri, 14 Feb 2014 14:10:40 -0500 (EST) From: John Baldwin To: Andrey Chernov Subject: Re: can the scheduler decide to schedule an interrupted but runnable thread on another CPU core? What are the implications for code? Date: Fri, 14 Feb 2014 14:10:29 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <201402141318.44743.jhb@freebsd.org> <52FE5FBF.3090104@freebsd.org> In-Reply-To: <52FE5FBF.3090104@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="koi8-r" Content-Transfer-Encoding: 7bit Message-Id: <201402141410.29325.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Fri, 14 Feb 2014 14:10:40 -0500 (EST) Cc: "freebsd-hackers@freebsd.org" , Adrian Chadd , Ryan Stone , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 14 Feb 2014 19:10:44 -0000 On Friday, February 14, 2014 1:26:07 pm Andrey Chernov wrote: > On 14.02.2014 22:18, John Baldwin wrote: > > On Friday, February 14, 2014 12:55:03 pm Adrian Chadd wrote: > >> On 14 February 2014 08:39, John Baldwin wrote: > >>> On Friday, February 14, 2014 4:22:34 am Adrian Chadd wrote: > >>>> Ok, so now I remember the other odd thing. > >>>> > >>>> I was seeing the sending context(s) jumping from one CPU to another > >>>> during flowtable_insert_common(), around the locking bits. > >>>> > >>>> But I thread pinned all the sender user threads! > >>>> > >>>> So, why would the senders still be scheduled on other CPUs if I've > >>>> pinned the userland threads? > >>>> > >>>> (and yes, I verified that the userland threads weren't moving around.) > >>> > >>> Can you clarify a bit? It's not clear how sender thraeds differ from > >>> userland threads differ from sender user threads. (I.e. one reading > >>> is that these are all the same thing and should thus all be pinned > >>> (I assume you mean using cpuset to bind them to specific cores rather > >>> than sched_pin)) > >> > >> Yup, I'm doing a manual, poor-mans RSS in lieu of merging in roberts stuff: > >> > >> * the userland threads are using the cpuset call to map a thread into > >> a cpuset, yes > >> * the NIC TX/RX ring routines in cxgbe are pinned to the same CPU as > >> the userland threads > > > > If they are all cpuset to a single CPU, they should not migrate, though > > I think sched_bind() can override that. However, that requires code to > > explicitly call sched_bind() which should be rare. > > > > Due to this bug, not fixed yet, the real picture is more complex: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/163585 Eh, that bug report has no useful details, as in, it doesn't list the actual commands run. If you do 'cpuset -l 6 -s 1' to force all processes to only use CPU6, then yes, of course the other CPUs are idle because that's what you _asked_ for. AFAICT, that is all the original reporter did. At work we regularly add and remove CPUs from the default set (set 1) on hundreds of machines every day with ULE without any issues. -- John Baldwin