From owner-freebsd-current@freebsd.org Tue Jan 9 19:51:00 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0656CE6DE76 for ; Tue, 9 Jan 2018 19:51:00 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D6BF76D48C for ; Tue, 9 Jan 2018 19:50:59 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mailman.ysv.freebsd.org (Postfix) id D6004E6DE74; Tue, 9 Jan 2018 19:50:59 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5A33E6DE73 for ; Tue, 9 Jan 2018 19:50:59 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pf0-x235.google.com (mail-pf0-x235.google.com [IPv6:2607:f8b0:400e:c00::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AEFEA6D48B for ; Tue, 9 Jan 2018 19:50:59 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mail-pf0-x235.google.com with SMTP id d23so9309676pfe.9 for ; Tue, 09 Jan 2018 11:50:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20150623.gappssmtp.com; s=20150623; h=date:from:to:subject:message-id:user-agent:mime-version; bh=HFkDc53lcoFqrkAlwZW4Jb/CfGjE+U8FWPcYyrrlTUo=; b=ROPJA0u/rjNKwHdobhM5krVRpXIyryElT7LSDouVRURg9JBAGpz6m8YyYXCoWMniYD Ia0dxNFjmUatRF76UnvrX67Mgy4R1hFaPFY8Wif3CLo2cIRbDiDwfPtgyugJxAjDCv8G Lb6hM1HmTYZuM9fREfmvAWfZ6bmO5O0Bym/EmzyWfa+9kzzWs5p+BeEvQK3uZzhRKYde zpo2crTrWQwvPotTEXevGpRcLZQB7SrQRWhUyO3DJPQ5RgThqh/eZGXSAw3yQxMEC5Bo DRt0030fL8CIOLx4GW8ExXpgKsv5KmWaUihzdzJcjWxPPTp9CWPg29MYiO27lHEDig0B e8MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:user-agent :mime-version; bh=HFkDc53lcoFqrkAlwZW4Jb/CfGjE+U8FWPcYyrrlTUo=; b=TXEbQCs3vrxzdOqSkN053ynf4IKz6fqoOrrSvAOaBL4e8VM8fdReCKQE9BgO6JyafH UPcc9FIBTrIoxnzI15y4d8URWiCUnoWQ0XXZOsu2WlnQuyDqFYNqd7ra1bsqVsUhb+AB n/EARnPP5CK4diYdmDB2k6fQYn6fDd1JXTcaUfqRs5bBuKah92tA/ujzsZPg1Tktq/hq CQVgu8ReuAydW1wKRUXE7/kUNNH6cwXlNvRJeCn6DqRJDbxdku3t8OySSKrgacUHvOdU WKKorIzxdla4XrbZHRza6ZbEX2ExMQav24qD8lwgFvpze/hDJGQN86bn6PsJlzpQzUfd s+lA== X-Gm-Message-State: AKGB3mJ+FK48a8gkcMwEgMElgb0+km/TLVk20KsUL5G3dwtTr67SVsQz A3ckL99u85en2W8ps+byGhSXVygn X-Google-Smtp-Source: ACJfBos6hiCzZy+1b3RYR+rwrgDwBnC7JohfkJdVIskLiBppr/5YADy/fZI4j8XgJacxeL+AHk0mSQ== X-Received: by 10.84.140.133 with SMTP id 5mr16798724plt.207.1515527458801; Tue, 09 Jan 2018 11:50:58 -0800 (PST) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by smtp.gmail.com with ESMTPSA id k2sm28057599pff.150.2018.01.09.11.50.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Jan 2018 11:50:58 -0800 (PST) Date: Tue, 9 Jan 2018 09:46:54 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: current@freebsd.org Subject: New NUMA support coming to CURRENT Message-ID: User-Agent: Alpine 2.21 (BSF 202 2017-01-01) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset=US-ASCII X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jan 2018 19:51:00 -0000 Hello folks, I am working on merging improved NUMA support with policy implemented by cpuset(2) over the next week. This work has been supported by Dell/EMC's Isilon product division and Netflix. You can see some discussion of these changes here: https://reviews.freebsd.org/D13403 https://reviews.freebsd.org/D13289 https://reviews.freebsd.org/D13545 The work has been done in user/jeff/numa if you want to look at svn history or experiment with the branch. It has been tested by Peter Holm on i386 and amd64 and it has been verified to work on arm at various points. We are working towards compatibility with libnuma and linux mbind. These commits will bring in improved support for NUMA in the kernel. There are new domain specific allocation functions available to kernel for UMA, malloc, kmem_, and vm_page*. busdmamem consumers will automatically be placed in the correct domain, bringing automatic improvements to some device performance. cpuset will be able to constrains processes, groups of processes, jails, etc. to subsets of the system memory domains, just as it can with sets of cpus. It can set default policy for any of the above. Threads can use cpusets to set policy that specifies a subset of their visible domains. Available policies are first-touch (local in linux terms), round-robin (similar to linux interleave), and preferred. For now, the default is round-robin. You can achieve a fixed domain policy by using round-robin with a bitmask of a single domain. As the scheduler and VM become more sophisticated we may switch the default to first-touch as linux does. Currently these features are enabled with VM_NUMA_ALLOC and MAXMEMDOM. It will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA syscalls and VM_NUMA_ALLOC code was 'experimental' and will be deprecated. numactl will continue to be supported although cpuset should be preferred going forward as it supports the full feature set of the new API. Thank you for your patience as I deal with the inevitable fallout of such sweeping changes. If you do have bugs, please file them in bugzilla, or reach out to me directly. I don't always have time to catch up on all of my mailing list mail and regretfully things slip through the cracks when they are not addressed directly to me. Thanks, Jeff