From owner-freebsd-hackers@FreeBSD.ORG Wed Jan 8 08:23:48 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6B9CB560; Wed, 8 Jan 2014 08:23:48 +0000 (UTC) Received: from mail-oa0-x22c.google.com (mail-oa0-x22c.google.com [IPv6:2607:f8b0:4003:c02::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 23F241DFA; Wed, 8 Jan 2014 08:23:48 +0000 (UTC) Received: by mail-oa0-f44.google.com with SMTP id h16so1448336oag.3 for ; Wed, 08 Jan 2014 00:23:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7KbaJBub0PEUq+IuqydLIC1GIc3yqzFffAi+nbHvnwM=; b=ypKLq0adFDcP9ZSSzwpFNNM1+pFlxFl0obkNH7GE4gWqJUwLe2W12b4vlcvt8j5Ch9 Ug7a1e6iVg679NKeKqLkHc0FhVD8X/pYJUgieyA+bFwDERGxq0VIHW9+JbRGFtdbTS6U A8TgJJjllzc4iUO2arkEK4VSKcs4YhzqkkR+c0XKwHX0T7zrdu197jVT0oju274uHeee Q+Nk89gjkfE2NJOAhl9U54YfTslL+0nZAgFGYocTBMutMuxsOwIxSi5IpCDJyuBsNSJq DGx7ux6ko1IM8ktPW5RvtRLee1u63w50t0WpneGbl+0cidE51xiD93Uqwiv8HPdqk23f vksg== MIME-Version: 1.0 X-Received: by 10.182.232.4 with SMTP id tk4mr81799267obc.9.1389169427317; Wed, 08 Jan 2014 00:23:47 -0800 (PST) Received: by 10.182.166.39 with HTTP; Wed, 8 Jan 2014 00:23:47 -0800 (PST) In-Reply-To: References: Date: Wed, 8 Jan 2014 00:23:47 -0800 Message-ID: Subject: Re: Working on NUMA support From: Andrew Bates To: Adrian Chadd Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jan 2014 08:23:48 -0000 Hey Adrian, We spent the last few months on research/design and plan on spending the next few months preparing tests, building the physical server(s) to test on, and finishing the prototype'd functions. There is some white-board and pen/paper design that hasn't yet made it to the github repo, but the prototypes online are hopefully a solid base for what we intend to complete. In the meantime, we are excited to hear who else is interested in NUMA and if anyone has suggestions or concerns about our approach. On Wed, Jan 8, 2014 at 12:03 AM, Adrian Chadd wrote: > Cool! Do you have any working code to implement the API, or is this > just in the design phase right now? > > > -a > > > On 6 January 2014 12:11, Andrew Bates wrote: > > Hey all, > > > > My name is Andrew Bates, and I would like to take a bit of your time to > > talk about NUMA support. > > > > Supporting Non-Uniform Memory Access in FreeBSD is something that has > been > > brought up in the past < > > > http://freebsd.1045724.n5.nabble.com/NUMA-Support-is-there-in-FreeBSD-td4= 865200.html > >. > > This is becoming increasingly important now that multiprocessor > > systems > > are an expanding technology, thus performance is scaling in terms of cp= u > > count, rather than just clock rate. > > > > There is a great opportunity here to optimize performance. After being > > asked to look into this by the EMC Isilon Storage Division, myself and = a > > few colleagues advised by Andrew Pilloud and Jeff Roberson would like t= o > > propose APIs to handle basic memory allocation/management to specific > NUMA > > domains. > > > > What we have devised so far consists of two levels. First there are th= e > > KPIs, to expose NUMA functionality at a thread level of domain affinity= . > > Secondly, there would be a userspace/interface to take advantage of th= e > > proposed APIs, thus giving users the capability to make their > applications > > NUMA-aware. > > > > We took the time to look into how many other systems (Linux, Macintosh, > > Solaris, Windows) already approach this problem, so there are some > aspects > > of our solution that are similar to how Linux and Solaris handle NUMA. > > Unlike Linux libnuma, we are only proposing a few additions and a minim= al > > library that can easily be expanded later to suit users=92 needs. > > > > > > KISS in mind, we came up with the following KPI prototypes > (freebsdnuma.h) > > to uncover NUMA in a usable fashion: > > > > > > - > > > > cpuset_get_memory_affinity() > > - > > > > cpuset_set_memory_affinity() > > - > > > > move_pages() > > - > > > > migrate_pages() > > - > > > > get_numa_cpus() > > - > > > > get_numa_weights() > > > > > > Then to the second part, we have the following userspace API prototypes > > (numanor.h) for our interface and testing purposes: > > > > > > - > > > > is_numa_available() > > - > > > > set_thread_on_domain() > > - > > > > set_memory_policy() > > - > > > > move_thread() > > > > > > In much much more detail, you can learn more about these prototypes, th= is > > project, view our progress, track along, and give input on our github > repo > > < https://github.com/andrewbates09/freebsd-numa > or simply via email. > This > > repo currently includes fully commented prototypes (like a mini man pag= e) > > and will later include additions to the project. > > > > If anyone has any comments, suggestions, concerns, quandaries, or just > > general thoughts please feel free to contact us, as we would love to he= ar > > your input! > > > > The Leaders: Sakire Arslan Ay, Andrew Pilloud, Jeff Roberson > > The Team: Andrew Bates, Joshua Clark, Alex Schuldberg, Dustin Walker > > > > -- > > V/Respectfully, > > Andrew M Bates > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > --=20 V/Respectfully, Andrew M Bates