From owner-svn-src-all@FreeBSD.ORG Fri Feb 4 18:56:15 2011 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB502106566C; Fri, 4 Feb 2011 18:56:14 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id B4B788FC15; Fri, 4 Feb 2011 18:56:14 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p14IUUUZ025431 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Fri, 4 Feb 2011 10:30:32 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <4D4C45C9.4080105@freebsd.org> Date: Fri, 04 Feb 2011 10:30:33 -0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: Robert Watson References: <201102031922.p13JML8i055697@svn.freebsd.org> <201102031529.25072.jhb@freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: svn-src-head@freebsd.org, Randall Stewart , svn-src-all@freebsd.org, src-committers@freebsd.org, John Baldwin Subject: Re: svn commit: r218232 - head/sys/netinet X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Feb 2011 18:56:15 -0000 On 2/4/11 9:38 AM, Robert Watson wrote: > > On Thu, 3 Feb 2011, John Baldwin wrote: > >>> 1) Move per John Baldwin to mp_maxid >>> 2) Some signed/unsigned errors found by Mac OS compiler (from >>> Michael) >>> 3) a couple of copyright updates on the effected files. >> >> Note that mp_maxid is the maxium valid ID, so you typically have to >> do things like: >> >> for (i = 0; i <= mp_maxid; i++) { >> if (CPU_ABSENT(i)) >> continue; >> ... >> } >> >> There is a CPU_FOREACH() macro that does the above (but assumes you >> want to skip over non-existent CPUs). > > I'm finding the network stack requires quite a bit more along these > lines, btw. I'd love also to have: > > PACKAGE_FOREACH() > CORE_FOREACH() > HWTHREAD_FOREACH() > I agree, which is why I usually support adding such iterators though some people scream about them. (e.g. FOREACH_THREAD_IN_PROC and there is one for iterating through vnets too.) > CURPACKAGE() > CURCORE() > CURTHREAD() also current jail, vnet, etc. (these (kinda) exist) > > Available when putting together thread worker pools, distributing > work, identifying where to channel work, making dispatch decisions > and so on. It seems likely that in some scenarios, it will be > desirable to have worker thread topology linked to hardware topology > -- for example, a network stack worker per core, with distribution > of work targeting the closest worker (subject to ordering > constraints)... > >> Hmmm, this is more complicated. Can sctp_queue_to_mcore() handle >> the fact that a cpu_to_use value might not be valid? If not you >> might want to maintain a separate "dense" virtual CPU ID table >> numbered 0 .. mp_ncpus - 1 that maps to "present" FreeBSD CPU IDs. >> I think Robert has done something similar to support RSS in TCP. >> Does that make sense? > > This proves somewhat complicated. I basically have two models, > depending on whether RSS is involved (which adds an external > factor). Without RSS, I build a contiguous workstream number space, > which is then mapped via a table to the CPU ID space, allowing > mappings and hashing to be done easily -- however, these refer to > ordered flow processing streams (i.e., "threads") rather than CPUs, > in the strict sense. In the future with dynamic configuration, this > becomes important because what I do is rebalance ordered processing > streams rather than work to CPUs. With RSS there has to be a link > between work distribution and the CPU identifiers shared by device > drivers, hardware, etc, in which case RSS identifies viable CPUs as > it starts (probably not quite correctly, I'll be looking for a > review of that code shortly, cleaning it up currently). > > This issue came up some at the BSDCan devsummit last year: as more > and more kernel subsystems need to exploit parallelism explicitly, > the thread programming model isn't bad, but lacks a strong tie to > hardware topology in order to help manage work distribution. One > idea idly bandied around was to do something along the lines of > KSE/GCD for the kernel: provide a layered "work" model with ordering > constraints, rather than exploit threads directly, for work-oriented > subsystems. This is effectively what netisr does, but in a network > stack-specific way. But with crypto code, IPSEC, storage stuff, > etc, all looking to exploit parallelism, perhaps a more general > model is called for. > > Robert >