From owner-freebsd-hackers@FreeBSD.ORG Thu Jun 5 14:58:48 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB700721; Thu, 5 Jun 2014 14:58:48 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B2E2E26B5; Thu, 5 Jun 2014 14:58:48 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id AEA23B94C; Thu, 5 Jun 2014 10:58:47 -0400 (EDT) From: John Baldwin To: "Alexander V. Chernikov" Subject: Re: Permit init(8) use its own cpuset group. Date: Thu, 5 Jun 2014 10:09:59 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <538C8F9A.4020301@FreeBSD.org> <201406041106.11659.jhb@freebsd.org> <538F70AB.5030701@FreeBSD.org> In-Reply-To: <538F70AB.5030701@FreeBSD.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201406051009.59432.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 05 Jun 2014 10:58:47 -0400 (EDT) Cc: Konstantin Belousov , freebsd-hackers@freebsd.org, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Jun 2014 14:58:48 -0000 On Wednesday, June 04, 2014 3:16:59 pm Alexander V. Chernikov wrote: > On 04.06.2014 19:06, John Baldwin wrote: > > On Monday, June 02, 2014 12:48:50 pm Konstantin Belousov wrote: > >> On Mon, Jun 02, 2014 at 06:52:10PM +0400, Alexander V. Chernikov wrote: > >>> Hello list! > >>> > >>> Currently init(8) uses group 1 which is root group. > >>> Modifications of this group affects both kernel and userland threads. > >>> Additionally, such modifications are impossible, for example, in presence > >>> of multi-queue NIC drivers (like igb or ixgbe) which binds their threads > > to > >>> particular cpus. > >>> > >>> Proposed change ("init_cpuset" loader tunable) permits changing cpu > >>> masks for > >>> userland more easily. Restricting user processes to migrate to/from CPU > >>> cores > >>> used for network traffic processing is one of the cases. > >>> > >>> Phabricator: https://phabric.freebsd.org/D141 (the same version attached > >>> inline) > >>> > >>> If there are no objections, I'll commit this next week. > >> Why is the tunable needed ? > > Because some people already depend on doing 'cpuset -l 0 -s 1'. It is also > > documented in our manpages that processes start in cpuset 1 by default so > > that you can use 'cpuset -l 0 -s 1' to move all processes, etc. > > > > For the stated problem (bound ithreads in drivers), I would actually like to > > fix ithreads that are bound to a specific CPU to create a different cpuset > > instead so they don't conflict with set 1. > Yes, this seem to be much better approach. > Please take a look on the new patch (here or in the phabricator). > Comments: > > Use different approach for modifyable root set: > > * Make sets in 0..15 internal > * Define CPUSET_SET1 & CPUSET_ITHREAD in that range > * Add cpuset_lookup_builtin() to retrieve such cpu sets by id > * Create additional root set for ithreads > * Use this set in ithread_create() > * Change intr_setaffinity() to use cpuset_iroot (do we really need this)? > > We can probably do the same for kprocs, but I'm unsure if we really need it. I imagined something a bit simpler. Just create a new set in intr_event_bind and bind the ithread to the new set. No need to have more magic set ids, etc. That also means that an ithread that isn't bound to a specific CPU via either 'cpuset -x' or BUS_BIND_INTR() will honor 'cpuset -s 1' like other kernel processes. I think that's probably fine and sensible. The issue is not the default set of ithreads, the issue is only with ithreads that are bound to specific CPUs. Unfortunately, this really needs ithreads to be separate kprocs again to work correctly as the cpuset code doesn't really permit threads to use independent sets. -- John Baldwin