From owner-freebsd-arch@FreeBSD.ORG Wed Mar 11 17:24:02 2015 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A8FFE436 for ; Wed, 11 Mar 2015 17:24:02 +0000 (UTC) Received: from mail-ie0-x236.google.com (mail-ie0-x236.google.com [IPv6:2607:f8b0:4001:c03::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 716B6E8B for ; Wed, 11 Mar 2015 17:24:02 +0000 (UTC) Received: by iecsl2 with SMTP id sl2so26980iec.1 for ; Wed, 11 Mar 2015 10:24:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=f7ESP680Q4gelcGh/EK+yXDyLuARK+3rzStqU3InMFY=; b=TpPDHWW255ypYQqhBTr803XT1NN6shSkN7LwV1l+m38DRMs4rrp9WiY2BXEOOhM/Em NNllqmF2vQGoYgpKWKA8fsLSNgSALoTfJs7BNsj+avRqCgCVYdQASThWrB/HGssu3tgu SsFSvx+Mvgji8NWi0XUbmiMlCuSdwWi67nw5ds/Z6v58UaU/HvSGb7nefkcLEpoLn1Rm fh9llwojyenRPdsdFdJLMF3LosKSYZtXXfTj78ddZi5S48nyKJrfZfsGFx8TicyOQQrz +7agFIfzge4xtOBFhmOYDjNy/bwGOP7b1gx/5G6qJ5bBiipFOXNpZN7MGWPgGqq4rzob DPXw== MIME-Version: 1.0 X-Received: by 10.50.66.170 with SMTP id g10mr92972897igt.49.1426094641882; Wed, 11 Mar 2015 10:24:01 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.36.17.194 with HTTP; Wed, 11 Mar 2015 10:24:01 -0700 (PDT) Date: Wed, 11 Mar 2015 10:24:01 -0700 X-Google-Sender-Auth: 2icC6gujeusRD4nNCNaq8mmYxaQ Message-ID: Subject: re-visiting the NUMA physical allocator work in -HEAD From: Adrian Chadd To: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Mar 2015 17:24:02 -0000 hi, I've started fleshing out the vm_phys allocator work in -HEAD. This is based on what's in -9 with some discussions from others - namely, we'd like policies and iterators, rather than just hard-coded stuff. So, here's where i'm doing it: https://github.com/freebsd/freebsd/compare/master...erikarn:local/adrian_numa_policy My aim is to get the following done: * add vm_phys default allocator policies - round robin, first touch, first touch + round robin (done); * convert the vm_phys domain iteration to use an iterator instead of hard-coded stuff (done); * add some basic policy description objects and API - likely just something that's copied around to begin with; * add counters for allocations - by domain; looking at allocation types and whether they were met (ie, things like "domain-local allocation to domain 0 success, domain-local allocation to domain 0 failure; domain 0 round-robin allocation, etc." * add some very basic domain and domain policy awareness to busdma. At this point I'll stop and get buy-in for pushing it into -HEAD. Trying to plumb things through contigmalloc / kmem_alloc / malloc all the way down to vm_phys for domain aware kernel allocations won't happen this pass - the vm_reserv code needs to know about domains and some basic domain policy. I want to get the above done and pushed into -HEAD so we can then build upon it, and that includes teaching vm_reserv about domains and domain policies. I'm treating all of this NUMA stuff as "if it's in -HEAD then great, but the API will be subject to change." With the basic iterator stuff and policy stuff done, I can already measure the effectiveness of first-touch on my dual and quad socket test boxes. It's indeed doing the right thing for some userland use cases. It seems completely silly to not have this bare minimum functionality in -HEAD. :( -adrian