Date: Thu, 5 Jun 2014 15:59:11 -0400 From: John Baldwin <jhb@freebsd.org> To: "Alexander V. Chernikov" <melifaro@freebsd.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-hackers@freebsd.org, hackers@freebsd.org Subject: Re: Permit init(8) use its own cpuset group. Message-ID: <201406051559.11274.jhb@freebsd.org> In-Reply-To: <5390C907.1070405@FreeBSD.org> References: <538C8F9A.4020301@FreeBSD.org> <201406051009.59432.jhb@freebsd.org> <5390C907.1070405@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday, June 05, 2014 3:46:15 pm Alexander V. Chernikov wrote:
> On 05.06.2014 18:09, John Baldwin wrote:
> > On Wednesday, June 04, 2014 3:16:59 pm Alexander V. Chernikov wrote:
> >> On 04.06.2014 19:06, John Baldwin wrote:
> >>> On Monday, June 02, 2014 12:48:50 pm Konstantin Belousov wrote:
> >>>> On Mon, Jun 02, 2014 at 06:52:10PM +0400, Alexander V. Chernikov wrote:
> >>>>> Hello list!
> >>>>>
> >>>>> Currently init(8) uses group 1 which is root group.
> >>>>> Modifications of this group affects both kernel and userland threads.
> >>>>> Additionally, such modifications are impossible, for example, in
> > presence
> >>>>> of multi-queue NIC drivers (like igb or ixgbe) which binds their
threads
> >>> to
> >>>>> particular cpus.
> >>>>>
> >>>>> Proposed change ("init_cpuset" loader tunable) permits changing cpu
> >>>>> masks for
> >>>>> userland more easily. Restricting user processes to migrate to/from
CPU
> >>>>> cores
> >>>>> used for network traffic processing is one of the cases.
> >>>>>
> >>>>> Phabricator: https://phabric.freebsd.org/D141 (the same version
attached
> >>>>> inline)
> >>>>>
> >>>>> If there are no objections, I'll commit this next week.
> >>>> Why is the tunable needed ?
> >>> Because some people already depend on doing 'cpuset -l 0 -s 1'. It is
> > also
> >>> documented in our manpages that processes start in cpuset 1 by default
so
> >>> that you can use 'cpuset -l 0 -s 1' to move all processes, etc.
> >>>
> >>> For the stated problem (bound ithreads in drivers), I would actually
like
> > to
> >>> fix ithreads that are bound to a specific CPU to create a different
cpuset
> >>> instead so they don't conflict with set 1.
> >> Yes, this seem to be much better approach.
> >> Please take a look on the new patch (here or in the phabricator).
> >> Comments:
> >>
> >> Use different approach for modifyable root set:
> >>
> >> * Make sets in 0..15 internal
> >> * Define CPUSET_SET1 & CPUSET_ITHREAD in that range
> >> * Add cpuset_lookup_builtin() to retrieve such cpu sets by id
> >> * Create additional root set for ithreads
> >> * Use this set in ithread_create()
> >> * Change intr_setaffinity() to use cpuset_iroot (do we really need
this)?
> >>
> >> We can probably do the same for kprocs, but I'm unsure if we really need
it.
> >
> > I imagined something a bit simpler. Just create a new set in
intr_event_bind
> > and bind the ithread to the new set. No need to have more magic set ids,
etc.
> Well, we also have userland which can modify given changes via `cpuset
> -x', so we need to be able to add some more logic on set
> allocation/keeping. Additionally, we can try to do the same via `cpuset
> -t', so introducing something like cpuset_setIthread() and hooking into
> intr_event_bind() won't probably be enough. At least I can't think out a
> quick and easy way to do this.
cpuset -x calls intr_event_bind(). If you just do it there you fix both
places.
> > That also means that an ithread that isn't bound to a specific CPU via
either
> > 'cpuset -x' or BUS_BIND_INTR() will honor 'cpuset -s 1' like other
> > kernel processes. I think that's probably fine and sensible. The issue
is
> Well, it is questionable. Kernel threads are a bit different in terms of
> TLB changes, memory working set and so on. (Personally I'd prefer to
> separate user / kthreads / ithreads to different sets in HEAD but that's
> another story).
>
> Anyway, we probably can (and should) MFC a bit different version which
> tries to change several sets at once if user supplied set 1 as argument.
No, I don't think we need umpteen special sets. I think we just need to fix
this one specific case of bound ithreads and everything else will work fine.
If someone wants to move kprocs out of set 1, they can already do that with
the existing tools via cpuset -C, etc.
--
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201406051559.11274.jhb>