From owner-freebsd-hackers@FreeBSD.ORG Mon Jan 6 20:11:52 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 220B413D for ; Mon, 6 Jan 2014 20:11:52 +0000 (UTC) Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [IPv6:2607:f8b0:4003:c01::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id DD9DB1637 for ; Mon, 6 Jan 2014 20:11:51 +0000 (UTC) Received: by mail-ob0-f172.google.com with SMTP id gq1so19051091obb.31 for ; Mon, 06 Jan 2014 12:11:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=Hf9CyilTOTfor22GLTKOmAQvmw4VaN1iljQl2p24e/c=; b=cPt75b16i44vzO4FFXnM/weGFB01XzTkkYGW6Y5cb3VTZJSY1pm7LSCDr0CkrTZ5x6 2Q5WbZPU/jGm1iSojHzju61gTT8bOT4f4VXhnayhpzVEK/XstJ4Eno70XIh6K3ilFPbx b6T04OL+nydQDD7X4EwtonbX/ebiOu8FS8+yOGen7Moe7jN9A3MbGYVDlrQltg0rfWGK nw0eQpobc2lRiAs8YsQNChkstF6At0SX/Vf6weAv7KcaZF6tE5tx+vBffez4X4JtOf71 ki8oLQXxUtxA3wvfakQbl5/+X2TraUijytDY2vXtiJGXy+18BOdQLOXkZ2uRYky8rsqY PZig== MIME-Version: 1.0 X-Received: by 10.182.148.106 with SMTP id tr10mr1842912obb.65.1389039111085; Mon, 06 Jan 2014 12:11:51 -0800 (PST) Received: by 10.182.166.39 with HTTP; Mon, 6 Jan 2014 12:11:50 -0800 (PST) Date: Mon, 6 Jan 2014 12:11:50 -0800 Message-ID: Subject: Working on NUMA support From: Andrew Bates To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Jan 2014 20:11:52 -0000 Hey all, My name is Andrew Bates, and I would like to take a bit of your time to talk about NUMA support. Supporting Non-Uniform Memory Access in FreeBSD is something that has been brought up in the past < http://freebsd.1045724.n5.nabble.com/NUMA-Support-is-there-in-FreeBSD-td486= 5200.html>. This is becoming increasingly important now that multiprocessor systems are an expanding technology, thus performance is scaling in terms of cpu count, rather than just clock rate. There is a great opportunity here to optimize performance. After being asked to look into this by the EMC Isilon Storage Division, myself and a few colleagues advised by Andrew Pilloud and Jeff Roberson would like to propose APIs to handle basic memory allocation/management to specific NUMA domains. What we have devised so far consists of two levels. First there are the KPIs, to expose NUMA functionality at a thread level of domain affinity. Secondly, there would be a userspace/interface to take advantage of the proposed APIs, thus giving users the capability to make their applications NUMA-aware. We took the time to look into how many other systems (Linux, Macintosh, Solaris, Windows) already approach this problem, so there are some aspects of our solution that are similar to how Linux and Solaris handle NUMA. Unlike Linux libnuma, we are only proposing a few additions and a minimal library that can easily be expanded later to suit users=92 needs. KISS in mind, we came up with the following KPI prototypes (freebsdnuma.h) to uncover NUMA in a usable fashion: - cpuset_get_memory_affinity() - cpuset_set_memory_affinity() - move_pages() - migrate_pages() - get_numa_cpus() - get_numa_weights() Then to the second part, we have the following userspace API prototypes (numanor.h) for our interface and testing purposes: - is_numa_available() - set_thread_on_domain() - set_memory_policy() - move_thread() In much much more detail, you can learn more about these prototypes, this project, view our progress, track along, and give input on our github repo < https://github.com/andrewbates09/freebsd-numa > or simply via email. This repo currently includes fully commented prototypes (like a mini man page) and will later include additions to the project. If anyone has any comments, suggestions, concerns, quandaries, or just general thoughts please feel free to contact us, as we would love to hear your input! The Leaders: Sakire Arslan Ay, Andrew Pilloud, Jeff Roberson The Team: Andrew Bates, Joshua Clark, Alex Schuldberg, Dustin Walker --=20 V/Respectfully, Andrew M Bates