Date: Thu, 5 Jun 2014 10:09:59 -0400 From: John Baldwin <jhb@freebsd.org> To: "Alexander V. Chernikov" <melifaro@freebsd.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-hackers@freebsd.org, hackers@freebsd.org Subject: Re: Permit init(8) use its own cpuset group. Message-ID: <201406051009.59432.jhb@freebsd.org> In-Reply-To: <538F70AB.5030701@FreeBSD.org> References: <538C8F9A.4020301@FreeBSD.org> <201406041106.11659.jhb@freebsd.org> <538F70AB.5030701@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wednesday, June 04, 2014 3:16:59 pm Alexander V. Chernikov wrote:
> On 04.06.2014 19:06, John Baldwin wrote:
> > On Monday, June 02, 2014 12:48:50 pm Konstantin Belousov wrote:
> >> On Mon, Jun 02, 2014 at 06:52:10PM +0400, Alexander V. Chernikov wrote:
> >>> Hello list!
> >>>
> >>> Currently init(8) uses group 1 which is root group.
> >>> Modifications of this group affects both kernel and userland threads.
> >>> Additionally, such modifications are impossible, for example, in
presence
> >>> of multi-queue NIC drivers (like igb or ixgbe) which binds their threads
> > to
> >>> particular cpus.
> >>>
> >>> Proposed change ("init_cpuset" loader tunable) permits changing cpu
> >>> masks for
> >>> userland more easily. Restricting user processes to migrate to/from CPU
> >>> cores
> >>> used for network traffic processing is one of the cases.
> >>>
> >>> Phabricator: https://phabric.freebsd.org/D141 (the same version attached
> >>> inline)
> >>>
> >>> If there are no objections, I'll commit this next week.
> >> Why is the tunable needed ?
> > Because some people already depend on doing 'cpuset -l 0 -s 1'. It is
also
> > documented in our manpages that processes start in cpuset 1 by default so
> > that you can use 'cpuset -l 0 -s 1' to move all processes, etc.
> >
> > For the stated problem (bound ithreads in drivers), I would actually like
to
> > fix ithreads that are bound to a specific CPU to create a different cpuset
> > instead so they don't conflict with set 1.
> Yes, this seem to be much better approach.
> Please take a look on the new patch (here or in the phabricator).
> Comments:
>
> Use different approach for modifyable root set:
>
> * Make sets in 0..15 internal
> * Define CPUSET_SET1 & CPUSET_ITHREAD in that range
> * Add cpuset_lookup_builtin() to retrieve such cpu sets by id
> * Create additional root set for ithreads
> * Use this set in ithread_create()
> * Change intr_setaffinity() to use cpuset_iroot (do we really need this)?
>
> We can probably do the same for kprocs, but I'm unsure if we really need it.
I imagined something a bit simpler. Just create a new set in intr_event_bind
and bind the ithread to the new set. No need to have more magic set ids, etc.
That also means that an ithread that isn't bound to a specific CPU via either
'cpuset -x' or BUS_BIND_INTR() will honor 'cpuset -s 1' like other
kernel processes. I think that's probably fine and sensible. The issue is
not the default set of ithreads, the issue is only with ithreads that are
bound to specific CPUs.
Unfortunately, this really needs ithreads to be separate kprocs again to work
correctly as the cpuset code doesn't really permit threads to use independent
sets.
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201406051009.59432.jhb>