From owner-freebsd-arch@FreeBSD.ORG Sun Apr 26 20:30:53 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5DB2889C for ; Sun, 26 Apr 2015 20:30:53 +0000 (UTC) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB5E31EAA for ; Sun, 26 Apr 2015 20:30:52 +0000 (UTC) Received: by wizk4 with SMTP id k4so77337706wiz.1 for ; Sun, 26 Apr 2015 13:30:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=004It6X1U8ovvM8sfNhtpNeQYamSHDfyMXVS68kIXDA=; b=rvRJX/IGGGZIGrA1SVZDXVovUmS8RMtsPe5cYJPQfHAA/GWbVnOthqfDDhXUClHSGW d2+Sdd92vrULL+YmM7XRAIV1muozDaycrsjUJuYY+4v0XWui5yTWGePOoR0UX9gZBxJh odXEv8fheMcm8H9znJlKBZyMf/gtHt7HTPyGALqT7CdgKZuiFq5huwh9ksiF1s1OUQC9 8d+FNKzo3wH13oSbWcnR2W4h9yUGqZO4V/pga+0+Z0YeIwJ8Kxk2tG+3mjJiiJVfBg91 uUBKbEfh/uatsH5vSV2JKxULf2T7GQnKdyMHhpK2i3/+0MwZkbmV/8IuaCXDYgWxf2WU AQuQ== MIME-Version: 1.0 X-Received: by 10.180.208.42 with SMTP id mb10mr14445543wic.80.1430080251349; Sun, 26 Apr 2015 13:30:51 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.180.42.68 with HTTP; Sun, 26 Apr 2015 13:30:51 -0700 (PDT) In-Reply-To: References: Date: Sun, 26 Apr 2015 13:30:51 -0700 X-Google-Sender-Auth: JR-IUyNetz_wBzufHa-RsInf0I4 Message-ID: Subject: Re: RFT: numa policy branch From: Adrian Chadd To: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Apr 2015 20:30:53 -0000 Hi! Another update: * updated to recent -HEAD; * numactl now can set memory policy and cpuset domain information - so it's easy to say "this runs in memory domain X and cpu domain Y" in one pass with it; * the locality matrix is now available. Here's an example from scott's 2x haswell v3, with cluster-on-die enabled: vm.phys_locality: 0: 10 21 31 31 1: 21 10 31 31 2: 31 31 10 21 3: 31 31 21 10 And on the westmere-ex box, with no SLIT table: vm.phys_locality: 0: -1 -1 -1 -1 1: -1 -1 -1 -1 2: -1 -1 -1 -1 3: -1 -1 -1 -1 * I've tested in on westmere-ex (4x socket), sandybridge, ivybridge, haswell v3 and haswell v3 cluster on die. * I've discovered that our implementation of libgomp (from gcc-4.2) is very old and doesn't include some of the thread control environment variables, grr. * .. and that the gcc libgomp code doesn't at all have freebsd thread affinity routines, so I added them to gcc-4.8. Testing with a local copy of stream - using gcc-4.9 and the updated libgomp to support thread pinning - shows that yes, it all works as expected, and yes for NUMA workloads its quite a big difference. I'd appreciate any reviews / testing people are able to provide. I'm about at the functionality point where I'd like to submit it for formal review and try to land it in -HEAD. -adrian