From owner-freebsd-arch@FreeBSD.ORG Thu Feb 20 19:17:38 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0424DC1A; Thu, 20 Feb 2014 19:17:38 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id CBCD714C7; Thu, 20 Feb 2014 19:17:37 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id DBF12B9D0; Thu, 20 Feb 2014 14:17:36 -0500 (EST) From: John Baldwin To: freebsd-current@freebsd.org, "freebsd-arch@freebsd.org" Subject: Re: [rfc] bind per-cpu timeout threads to each CPU Date: Thu, 20 Feb 2014 14:17:34 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <530508B7.7060102@FreeBSD.org> <201402191602.54465.jhb@freebsd.org> In-Reply-To: <201402191602.54465.jhb@freebsd.org> MIME-Version: 1.0 Message-Id: <201402201417.34148.jhb@freebsd.org> Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 20 Feb 2014 14:17:37 -0500 (EST) Cc: Adrian Chadd , Alexander Motin X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 20 Feb 2014 19:17:38 -0000 On Wednesday, February 19, 2014 4:02:54 pm John Baldwin wrote: > On Wednesday, February 19, 2014 3:04:51 pm Adrian Chadd wrote: > > On 19 February 2014 11:59, Alexander Motin wrote: > > > > >> So if we're moving towards supporting (among others) a pcbgroup / RSS > > >> hash style work load distribution across CPUs to minimise > > >> per-connection lock contention, we really don't want the scheduler to > > >> decide it can schedule things on other CPUs under enough pressure. > > >> That'll just make things worse. > > > > > True, though it is also not obvious that putting second thread on CPU run > > > queue is better then executing it right now on another core. > > > > Well, it depends if you're trying to optimise for "run all runnable > > tasks as quickly as possible" or "run all runnable tasks in contexts > > that minimise lock contention." > > > > The former sounds great as long as there's no real lock contention > > going on. But as you add more chances for contention (something like > > "100,000 concurrent TCP flows") then you may end up having your TCP > > timer firing stuff interfere with more TXing or RXing on the same > > connection. > > > > Chasing this stuff down is a pain, because it only really shows up > > when you're doing lots of concurrency. > > > > I'm happy to make this a boot-time option and leave it off for the > > time being. How's that? > > I think having it be a tunable would be good. OTOH, I could also > see another option which would be to pin all clock threads except > for the "default" one by default and only have the option control > whether or not the default thread is pinned to CPU 0 as callers > who use callout_on() are explicitly asking to run the callout on a > specific CPU. (A further variant of this would be to divorce cpu0's swi from the catch-all softclock and let the catch-all softclock float, but bind all the per-cpu swis) -- John Baldwin