From owner-freebsd-hackers@FreeBSD.ORG Mon Jun 9 18:09:23 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AF2C8A51; Mon, 9 Jun 2014 18:09:23 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6FAF4216B; Mon, 9 Jun 2014 18:09:23 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 72C65B94F; Mon, 9 Jun 2014 14:09:22 -0400 (EDT) From: John Baldwin To: "Alexander V. Chernikov" Subject: Re: Permit init(8) use its own cpuset group. Date: Mon, 9 Jun 2014 13:43:39 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <538C8F9A.4020301@FreeBSD.org> <5390DB64.6010704@FreeBSD.org> <53923C09.6020302@FreeBSD.org> In-Reply-To: <53923C09.6020302@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201406091343.39584.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 09 Jun 2014 14:09:22 -0400 (EDT) Cc: Konstantin Belousov , freebsd-hackers@freebsd.org, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jun 2014 18:09:23 -0000 On Friday, June 06, 2014 6:09:13 pm Alexander V. Chernikov wrote: > On 06.06.2014 01:04, Alexander V. Chernikov wrote: > > On 05.06.2014 23:59, John Baldwin wrote: > >> On Thursday, June 05, 2014 3:46:15 pm Alexander V. Chernikov wrote: > >>> On 05.06.2014 18:09, John Baldwin wrote: > >>>> On Wednesday, June 04, 2014 3:16:59 pm Alexander V. Chernikov wrote: > >>>>> On 04.06.2014 19:06, John Baldwin wrote: > >>>>>> On Monday, June 02, 2014 12:48:50 pm Konstantin Belousov wrote: > >>>>>>> On Mon, Jun 02, 2014 at 06:52:10PM +0400, Alexander V. Chernikov wrote: > >>>>>>>> Hello list! > >>>>>>>> > >>>>>>>> Currently init(8) uses group 1 which is root group. > >>>>>>>> Modifications of this group affects both kernel and userland threads. > >>>>>>>> Additionally, such modifications are impossible, for example, in > >>>> presence > >>>>>>>> of multi-queue NIC drivers (like igb or ixgbe) which binds their > >> threads > >>>>>> to > >>>>>>>> particular cpus. > >>>>>>>> > >>>>>>>> Proposed change ("init_cpuset" loader tunable) permits changing cpu > >>>>>>>> masks for > >>>>>>>> userland more easily. Restricting user processes to migrate to/from > >> CPU > >>>>>>>> cores > >>>>>>>> used for network traffic processing is one of the cases. > >>>>>>>> > >>>>>>>> Phabricator: https://phabric.freebsd.org/D141 (the same version > >> attached > >>>>>>>> inline) > >>>>>>>> > >>>>>>>> If there are no objections, I'll commit this next week. > >>>>>>> Why is the tunable needed ? > >>>>>> Because some people already depend on doing 'cpuset -l 0 -s 1'. It is > >>>> also > >>>>>> documented in our manpages that processes start in cpuset 1 by default > >> so > >>>>>> that you can use 'cpuset -l 0 -s 1' to move all processes, etc. > >>>>>> > >>>>>> For the stated problem (bound ithreads in drivers), I would actually > >> like > >>>> to > >>>>>> fix ithreads that are bound to a specific CPU to create a different > >> cpuset > >>>>>> instead so they don't conflict with set 1. > >>>>> Yes, this seem to be much better approach. > >>>>> Please take a look on the new patch (here or in the phabricator). > >>>>> Comments: > >>>>> > >>>>> Use different approach for modifyable root set: > >>>>> > >>>>> * Make sets in 0..15 internal > >>>>> * Define CPUSET_SET1 & CPUSET_ITHREAD in that range > >>>>> * Add cpuset_lookup_builtin() to retrieve such cpu sets by id > >>>>> * Create additional root set for ithreads > >>>>> * Use this set in ithread_create() > >>>>> * Change intr_setaffinity() to use cpuset_iroot (do we really need > >> this)? > >>>>> > >>>>> We can probably do the same for kprocs, but I'm unsure if we really need > >> it. > >>>> > >>>> I imagined something a bit simpler. Just create a new set in > >> intr_event_bind > >>>> and bind the ithread to the new set. No need to have more magic set ids, > >> etc. > >>> Well, we also have userland which can modify given changes via `cpuset > >>> -x', so we need to be able to add some more logic on set > >>> allocation/keeping. Additionally, we can try to do the same via `cpuset > >>> -t', so introducing something like cpuset_setIthread() and hooking into > >>> intr_event_bind() won't probably be enough. At least I can't think out a > >>> quick and easy way to do this. > >> > >> cpuset -x calls intr_event_bind(). If you just do it there you fix both > >> places. > Yup. I was wrong. This version seems to be simpler while doing the right > thing. Hmm, I do think this patch is doing the right thing, but I'm worried you might have weird effects, e.g. I worry that because the 'intr' proc is in set 1 still, the thread masks will be (incorrectly) checked when changing set 1 and you still won't be able to 'cpuset -l 0 -s 1'. Also, strictly speaking, it would probably be best to return threads to set 1 when NOCPU is passed to intr_event_bind() to unbind an interrupt. -- John Baldwin