Date: Thu, 05 Jun 2014 23:46:15 +0400 From: "Alexander V. Chernikov" <melifaro@FreeBSD.org> To: John Baldwin <jhb@freebsd.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-hackers@freebsd.org, hackers@freebsd.org Subject: Re: Permit init(8) use its own cpuset group. Message-ID: <5390C907.1070405@FreeBSD.org> In-Reply-To: <201406051009.59432.jhb@freebsd.org> References: <538C8F9A.4020301@FreeBSD.org> <201406041106.11659.jhb@freebsd.org> <538F70AB.5030701@FreeBSD.org> <201406051009.59432.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 05.06.2014 18:09, John Baldwin wrote:
> On Wednesday, June 04, 2014 3:16:59 pm Alexander V. Chernikov wrote:
>> On 04.06.2014 19:06, John Baldwin wrote:
>>> On Monday, June 02, 2014 12:48:50 pm Konstantin Belousov wrote:
>>>> On Mon, Jun 02, 2014 at 06:52:10PM +0400, Alexander V. Chernikov wrote:
>>>>> Hello list!
>>>>>
>>>>> Currently init(8) uses group 1 which is root group.
>>>>> Modifications of this group affects both kernel and userland threads.
>>>>> Additionally, such modifications are impossible, for example, in
> presence
>>>>> of multi-queue NIC drivers (like igb or ixgbe) which binds their threads
>>> to
>>>>> particular cpus.
>>>>>
>>>>> Proposed change ("init_cpuset" loader tunable) permits changing cpu
>>>>> masks for
>>>>> userland more easily. Restricting user processes to migrate to/from CPU
>>>>> cores
>>>>> used for network traffic processing is one of the cases.
>>>>>
>>>>> Phabricator: https://phabric.freebsd.org/D141 (the same version attached
>>>>> inline)
>>>>>
>>>>> If there are no objections, I'll commit this next week.
>>>> Why is the tunable needed ?
>>> Because some people already depend on doing 'cpuset -l 0 -s 1'. It is
> also
>>> documented in our manpages that processes start in cpuset 1 by default so
>>> that you can use 'cpuset -l 0 -s 1' to move all processes, etc.
>>>
>>> For the stated problem (bound ithreads in drivers), I would actually like
> to
>>> fix ithreads that are bound to a specific CPU to create a different cpuset
>>> instead so they don't conflict with set 1.
>> Yes, this seem to be much better approach.
>> Please take a look on the new patch (here or in the phabricator).
>> Comments:
>>
>> Use different approach for modifyable root set:
>>
>> * Make sets in 0..15 internal
>> * Define CPUSET_SET1 & CPUSET_ITHREAD in that range
>> * Add cpuset_lookup_builtin() to retrieve such cpu sets by id
>> * Create additional root set for ithreads
>> * Use this set in ithread_create()
>> * Change intr_setaffinity() to use cpuset_iroot (do we really need this)?
>>
>> We can probably do the same for kprocs, but I'm unsure if we really need it.
>
> I imagined something a bit simpler. Just create a new set in intr_event_bind
> and bind the ithread to the new set. No need to have more magic set ids, etc.
Well, we also have userland which can modify given changes via `cpuset
-x', so we need to be able to add some more logic on set
allocation/keeping. Additionally, we can try to do the same via `cpuset
-t', so introducing something like cpuset_setIthread() and hooking into
intr_event_bind() won't probably be enough. At least I can't think out a
quick and easy way to do this.
> That also means that an ithread that isn't bound to a specific CPU via either
> 'cpuset -x' or BUS_BIND_INTR() will honor 'cpuset -s 1' like other
> kernel processes. I think that's probably fine and sensible. The issue is
Well, it is questionable. Kernel threads are a bit different in terms of
TLB changes, memory working set and so on. (Personally I'd prefer to
separate user / kthreads / ithreads to different sets in HEAD but that's
another story).
Anyway, we probably can (and should) MFC a bit different version which
tries to change several sets at once if user supplied set 1 as argument.
> not the default set of ithreads, the issue is only with ithreads that are
> bound to specific CPUs.
>
> Unfortunately, this really needs ithreads to be separate kprocs again to work
> correctly as the cpuset code doesn't really permit threads to use independent
> sets.
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5390C907.1070405>