From owner-freebsd-smp@FreeBSD.ORG Thu Nov 13 01:32:20 2008 Return-Path: Delivered-To: freebsd-smp@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 367A1106564A for ; Thu, 13 Nov 2008 01:32:20 +0000 (UTC) (envelope-from marc@freshaire.wiz.com) Received: from freshaire.wiz.com (freshaire.wiz.com [66.143.183.129]) by mx1.freebsd.org (Postfix) with ESMTP id EE6E68FC14 for ; Thu, 13 Nov 2008 01:32:19 +0000 (UTC) (envelope-from marc@freshaire.wiz.com) Received: from freshaire.wiz.com (localhost.wiz.com [127.0.0.1]) by freshaire.wiz.com (8.14.2/8.14.2) with ESMTP id mAD12LDt020333 for ; Wed, 12 Nov 2008 19:02:21 -0600 (CST) (envelope-from marc@freshaire.wiz.com) Received: (from marc@localhost) by freshaire.wiz.com (8.14.2/8.14.2/Submit) id mAD12LwF020332 for freebsd-smp@freebsd.org; Wed, 12 Nov 2008 19:02:21 -0600 (CST) (envelope-from marc) Date: Wed, 12 Nov 2008 19:02:21 -0600 From: Marc Wiz To: freebsd-smp@freebsd.org Message-ID: <20081113010221.GB20056@wiz.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.2i Subject: Re: NUMA? X-BeenThere: freebsd-smp@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: FreeBSD SMP implementation group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2008 01:32:20 -0000 On Thu, Nov 13, 2008 at 01:35:28AM +0100, Ivan Voras wrote: > Hi, > > As even Intel's new CPUs have integrated memory controllers and thus > become NUMA, I'm interested in what is, in theory (I'm not proposing to > do it, I'm just curious), necessary to change in an OS to support NUMA. > My guess is: > > 1) node topology detection - something similar to what ULE does but also > recording which memory ranges are "close" to which CPU and the > "distance" between nodes/CPUs > 2) on new image load (exec), pick a node for it, among "least used" > nodes and record the choice per-proc; on fork, keep the new process on > the same node > 3) schedule threads on a CPU from the proc's node if at all possible > (e.g, when a 6-core CPU is still 1 node), then on a "near" node from a > list of distances sorted in order of cost > 4) allocate new pages for a proc from its node's memory range(s) if at > all possible. One good source of information on this topic is IBM's AIX on the Power 4 - 6 processors. There is the concept of distant vs. close memory and processors as well as what is referred to as memory affinity. Marc -- Marc Wiz marc@wiz.com Yes, that really is my last name.