From owner-freebsd-arch@FreeBSD.ORG Tue Feb 26 22:11:02 2008 Return-Path: Delivered-To: arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37A421065671 for ; Tue, 26 Feb 2008 22:11:02 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from webaccess-cl.virtdom.com (webaccess-cl.virtdom.com [216.240.101.25]) by mx1.freebsd.org (Postfix) with ESMTP id 07A6D13C4E7 for ; Tue, 26 Feb 2008 22:11:01 +0000 (UTC) (envelope-from jroberson@chesapeake.net) Received: from [192.168.1.107] (cpe-24-94-75-93.hawaii.res.rr.com [24.94.75.93]) (authenticated bits=0) by webaccess-cl.virtdom.com (8.13.6/8.13.6) with ESMTP id m1QMApKV049274; Tue, 26 Feb 2008 17:10:56 -0500 (EST) (envelope-from jroberson@chesapeake.net) Date: Tue, 26 Feb 2008 12:12:34 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Nick Evans In-Reply-To: <20080226135439.27db7400@pleiades.nextvenue.com> Message-ID: <20080226120956.V920@desktop> References: <20080220105333.G44565@fledge.watson.org> <47BCEFDB.5040207@freebsd.org> <20080220175532.Q920@desktop> <20080220213253.A920@desktop> <20080221092011.J52922@fledge.watson.org> <20080222121253.N920@desktop> <20080222231245.GA28788@lor.one-eyed-alien.net> <20080222134923.M920@desktop> <20080223194047.GB38485@lor.one-eyed-alien.net> <20080223111659.K920@desktop> <20080223213507.GD39699@lor.one-eyed-alien.net> <20080224001902.J920@desktop> <20080226135439.27db7400@pleiades.nextvenue.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@freebsd.org Subject: Re: cpuset and affinity implementation X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Feb 2008 22:11:02 -0000 On Tue, 26 Feb 2008, Nick Evans wrote: > On Sun, 24 Feb 2008 00:40:31 -1000 (HST) > Jeff Roberson wrote: > >> Please see: >> http://people.freebsd.org/~jeff/cpuset.diff >> >> This is unfortunately intertwined with ULE's new CPU selection algorithm >> so that code is in the patch as well. Otherwise, this includes a simple, >> ugly userland tool called cpuset and all of the kernel support required. >> I have tested this by creating sets and subsets and modifying their cpu >> masks under load. I'm able to dynamically reprovision without issue. >> >> This doesn't have support for jails but the infrastructure is there. It >> also fails to modify sets if it would leave threads without a valid cpu >> to run on. I have not implemented a force option but it will be trivial >> to do so. The initial cpu set is also created before we know all_cpus so >> it's faked up with all cpus set for now. >> >> I mostly want people to look at the interface in cpuset.h and make sure >> they agree with it before I start polishing to commit. I'm fairly happy >> with the way the syscall api looks now. The code itself ended up being >> much more complicated than I'd hoped due to locking considerations. Try >> not to look at cpuset_setproc() ;). >> >> If you want to actually try the patch, here's a couple of neat things to >> do with cpuset: >> >> cpuset -l 0-4 /bin/sh >> >> This creates a new group with a list (-l) of cpus 0-4 inclusive and runs >> sh in it. >> >> cpuset -g -p >> >> This will get (-g) the mask of cpus pid (-p) is allowed to run on. >> >> cpuset -l 0,2 -p >> >> This will restrict sh to running on cpus 0, 2 while its group is still >> allowed 0-4. >> >> cpuset -l 0,2 -c -p >> >> This will modify the cpuset (-c) that the sh belongs to. >> >> cpuset -l 0-3 -s 1 >> >> This will modify the set (-s) that all threads are in by default to >> contain the first 4 cpus leaving the rest idled. >> >> Feedback is appreciated. >> > > Jeff, > > Is it currently, or will it eventually be possible to assign network threads > to different cores? Everything appears to be driven by pid, but at least > according to top all interrupt "processes" show as pid 12. Also, if > kern.sched.topology returns 0 is it safe to assume I'm not getting the > benefit of the topology distinction between packages vs cores? I forgot to remove the topology sysctl. It is meaningless now. If your machine on boot say something like: Cores per package: 2 Then the scheduler is aware of the layout. As for binding interrupt/kernel threads, you need to get the tid. Then you can set a mask. er, except I guess I didn't make a -t tid argument for cpuset yet. I'll add that to the next patch. Thanks, Jeff > > Nick >