From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 5 20:26:15 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4AB28567; Thu, 5 Jun 2014 20:26:15 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1EED125E8; Thu, 5 Jun 2014 20:26:15 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 5319DB94B; Thu, 5 Jun 2014 16:26:13 -0400 (EDT) From: John Baldwin To: "Alexander V. Chernikov" Subject: Re: Permit init(8) use its own cpuset group. Date: Thu, 5 Jun 2014 15:59:11 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <538C8F9A.4020301@FreeBSD.org> <201406051009.59432.jhb@freebsd.org> <5390C907.1070405@FreeBSD.org> In-Reply-To: <5390C907.1070405@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201406051559.11274.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 05 Jun 2014 16:26:13 -0400 (EDT) Cc: Konstantin Belousov , freebsd-hackers@freebsd.org, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jun 2014 20:26:15 -0000 On Thursday, June 05, 2014 3:46:15 pm Alexander V. Chernikov wrote: > On 05.06.2014 18:09, John Baldwin wrote: > > On Wednesday, June 04, 2014 3:16:59 pm Alexander V. Chernikov wrote: > >> On 04.06.2014 19:06, John Baldwin wrote: > >>> On Monday, June 02, 2014 12:48:50 pm Konstantin Belousov wrote: > >>>> On Mon, Jun 02, 2014 at 06:52:10PM +0400, Alexander V. Chernikov wrote: > >>>>> Hello list! > >>>>> > >>>>> Currently init(8) uses group 1 which is root group. > >>>>> Modifications of this group affects both kernel and userland threads. > >>>>> Additionally, such modifications are impossible, for example, in > > presence > >>>>> of multi-queue NIC drivers (like igb or ixgbe) which binds their threads > >>> to > >>>>> particular cpus. > >>>>> > >>>>> Proposed change ("init_cpuset" loader tunable) permits changing cpu > >>>>> masks for > >>>>> userland more easily. Restricting user processes to migrate to/from CPU > >>>>> cores > >>>>> used for network traffic processing is one of the cases. > >>>>> > >>>>> Phabricator: https://phabric.freebsd.org/D141 (the same version attached > >>>>> inline) > >>>>> > >>>>> If there are no objections, I'll commit this next week. > >>>> Why is the tunable needed ? > >>> Because some people already depend on doing 'cpuset -l 0 -s 1'. It is > > also > >>> documented in our manpages that processes start in cpuset 1 by default so > >>> that you can use 'cpuset -l 0 -s 1' to move all processes, etc. > >>> > >>> For the stated problem (bound ithreads in drivers), I would actually like > > to > >>> fix ithreads that are bound to a specific CPU to create a different cpuset > >>> instead so they don't conflict with set 1. > >> Yes, this seem to be much better approach. > >> Please take a look on the new patch (here or in the phabricator). > >> Comments: > >> > >> Use different approach for modifyable root set: > >> > >> * Make sets in 0..15 internal > >> * Define CPUSET_SET1 & CPUSET_ITHREAD in that range > >> * Add cpuset_lookup_builtin() to retrieve such cpu sets by id > >> * Create additional root set for ithreads > >> * Use this set in ithread_create() > >> * Change intr_setaffinity() to use cpuset_iroot (do we really need this)? > >> > >> We can probably do the same for kprocs, but I'm unsure if we really need it. > > > > I imagined something a bit simpler. Just create a new set in intr_event_bind > > and bind the ithread to the new set. No need to have more magic set ids, etc. > Well, we also have userland which can modify given changes via `cpuset > -x', so we need to be able to add some more logic on set > allocation/keeping. Additionally, we can try to do the same via `cpuset > -t', so introducing something like cpuset_setIthread() and hooking into > intr_event_bind() won't probably be enough. At least I can't think out a > quick and easy way to do this. cpuset -x calls intr_event_bind(). If you just do it there you fix both places. > > That also means that an ithread that isn't bound to a specific CPU via either > > 'cpuset -x' or BUS_BIND_INTR() will honor 'cpuset -s 1' like other > > kernel processes. I think that's probably fine and sensible. The issue is > Well, it is questionable. Kernel threads are a bit different in terms of > TLB changes, memory working set and so on. (Personally I'd prefer to > separate user / kthreads / ithreads to different sets in HEAD but that's > another story). > > Anyway, we probably can (and should) MFC a bit different version which > tries to change several sets at once if user supplied set 1 as argument. No, I don't think we need umpteen special sets. I think we just need to fix this one specific case of bound ithreads and everything else will work fine. If someone wants to move kprocs out of set 1, they can already do that with the existing tools via cpuset -C, etc. -- John Baldwin