From owner-svn-src-head@FreeBSD.ORG Fri Feb 4 17:38:05 2011 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45186106564A; Fri, 4 Feb 2011 17:38:05 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 1C7098FC14; Fri, 4 Feb 2011 17:38:05 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id B897146B0C; Fri, 4 Feb 2011 12:38:04 -0500 (EST) Date: Fri, 4 Feb 2011 17:38:04 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: John Baldwin In-Reply-To: <201102031529.25072.jhb@freebsd.org> Message-ID: References: <201102031922.p13JML8i055697@svn.freebsd.org> <201102031529.25072.jhb@freebsd.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: svn-src-head@freebsd.org, Randall Stewart , svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r218232 - head/sys/netinet X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Feb 2011 17:38:05 -0000 On Thu, 3 Feb 2011, John Baldwin wrote: >> 1) Move per John Baldwin to mp_maxid >> 2) Some signed/unsigned errors found by Mac OS compiler (from Michael) >> 3) a couple of copyright updates on the effected files. > > Note that mp_maxid is the maxium valid ID, so you typically have to do > things like: > > for (i = 0; i <= mp_maxid; i++) { > if (CPU_ABSENT(i)) > continue; > ... > } > > There is a CPU_FOREACH() macro that does the above (but assumes you want to > skip over non-existent CPUs). I'm finding the network stack requires quite a bit more along these lines, btw. I'd love also to have: PACKAGE_FOREACH() CORE_FOREACH() HWTHREAD_FOREACH() CURPACKAGE() CURCORE() CURTHREAD() Available when putting together thread worker pools, distributing work, identifying where to channel work, making dispatch decisions and so on. It seems likely that in some scenarios, it will be desirable to have worker thread topology linked to hardware topology -- for example, a network stack worker per core, with distribution of work targeting the closest worker (subject to ordering constraints)... > Hmmm, this is more complicated. Can sctp_queue_to_mcore() handle the fact > that a cpu_to_use value might not be valid? If not you might want to > maintain a separate "dense" virtual CPU ID table numbered 0 .. mp_ncpus - 1 > that maps to "present" FreeBSD CPU IDs. I think Robert has done something > similar to support RSS in TCP. Does that make sense? This proves somewhat complicated. I basically have two models, depending on whether RSS is involved (which adds an external factor). Without RSS, I build a contiguous workstream number space, which is then mapped via a table to the CPU ID space, allowing mappings and hashing to be done easily -- however, these refer to ordered flow processing streams (i.e., "threads") rather than CPUs, in the strict sense. In the future with dynamic configuration, this becomes important because what I do is rebalance ordered processing streams rather than work to CPUs. With RSS there has to be a link between work distribution and the CPU identifiers shared by device drivers, hardware, etc, in which case RSS identifies viable CPUs as it starts (probably not quite correctly, I'll be looking for a review of that code shortly, cleaning it up currently). This issue came up some at the BSDCan devsummit last year: as more and more kernel subsystems need to exploit parallelism explicitly, the thread programming model isn't bad, but lacks a strong tie to hardware topology in order to help manage work distribution. One idea idly bandied around was to do something along the lines of KSE/GCD for the kernel: provide a layered "work" model with ordering constraints, rather than exploit threads directly, for work-oriented subsystems. This is effectively what netisr does, but in a network stack-specific way. But with crypto code, IPSEC, storage stuff, etc, all looking to exploit parallelism, perhaps a more general model is called for. Robert