From owner-freebsd-current@freebsd.org Sun Jan 14 03:44:02 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 98D0EE76232 for ; Sun, 14 Jan 2018 03:44:02 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 73B51682DE for ; Sun, 14 Jan 2018 03:44:02 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mailman.ysv.freebsd.org (Postfix) id 73081E76231; Sun, 14 Jan 2018 03:44:02 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 729C0E76230 for ; Sun, 14 Jan 2018 03:44:02 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pg0-x229.google.com (mail-pg0-x229.google.com [IPv6:2607:f8b0:400e:c05::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 420B4682DD for ; Sun, 14 Jan 2018 03:44:02 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: by mail-pg0-x229.google.com with SMTP id m20so6553351pgc.11 for ; Sat, 13 Jan 2018 19:44:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=jroberson-net.20150623.gappssmtp.com; s=20150623; h=date:from:to:subject:in-reply-to:message-id:references:user-agent :mime-version; bh=aPOz1ueFWfYdLjV9xLywibQgFsj9yPYZ1V5xrtYWnoQ=; b=tY2E2YJfeS950AcM8aG+ZKA8jUpQ2FLPdAljnz5YYFx9cDxEUt2fJUX5ZxH6r/YWlJ zHjq01VGo0rAqLTQrvqNhV9Yn2WO3g/ZVkoyr+fuj0dGtNQE3Kxi6HtnRuSQEWY+JZYZ A7bXHrn9yQkwQ9QeNVbiPa3C1gHqwd5/FcpjToPBIM922gmDXOAxkJRb7FMPsPIP1J97 IQ05ryisOCJ/twfqX43jouzPH06H3a1YbobFup1b53ssAtf1qRqwGf0DpPl6DKSDplk3 oWFuUNJdAInFn+4q1i3ufyB1M9JHwP/BJbSorLn0WpP4WS5nwchyVs+Brl90r28z0kZx 51bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:in-reply-to:message-id :references:user-agent:mime-version; bh=aPOz1ueFWfYdLjV9xLywibQgFsj9yPYZ1V5xrtYWnoQ=; b=ooL0qaSRxQnyrRHTGah4Gt3QpJKblk/6kKxsgitydl3TwqN4JbWKNFCqcPz8pcwsl3 Mwr2uQjMuvOXyJTEJUvOUD7VZUlxEJT22S3Aap6QzS88gaYfwDWkaBRMgw8JWcDeMuYA JTfUxlGWIeDLe4kSl/BcOa0uxoLlbugAjuR7hrGygrfa9S6RjRfmbJrzFDnzn2SfVlIF N0UJNBcgZIHvgklNvsr8vD8+Ygowg6Rd/GOcZ+tKDitCqgOA+3rgpsQ9NvISgLSXUZAk l9OKgtLZPqx9wShKHRUCPstydJavFOfN6XrfJA/0eLQeZLEEpXJjpYDAN8FWyfRAyUAu kt/A== X-Gm-Message-State: AKwxytdO5BOpddEm2lrfBFCYVSFWwjbhfOSrZvoyUbhCUmGvpc66VVDY +hacRAfYdSw7PJyvmCZC8679q/nr X-Google-Smtp-Source: ACJfBosPYi+UfjdPrFf/UEm3O2QHd4XcBkgRK6cQM0WE5G1lIaDllL9GXjC0B0vdKPUM64kyOWsGrw== X-Received: by 10.101.73.71 with SMTP id q7mr2035228pgs.66.1515901441486; Sat, 13 Jan 2018 19:44:01 -0800 (PST) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by smtp.gmail.com with ESMTPSA id s65sm55905474pfk.7.2018.01.13.19.44.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 13 Jan 2018 19:44:00 -0800 (PST) Date: Sat, 13 Jan 2018 17:39:49 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: current@freebsd.org Subject: Re: New NUMA support coming to CURRENT In-Reply-To: Message-ID: References: User-Agent: Alpine 2.21 (BSF 202 2017-01-01) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Jan 2018 03:44:02 -0000 Hello, This work has been committed. It is governed by a new 'NUMA' config option and 'DEVICE_NUMA' and 'VM_NUMA_ALLOC' have both been retired. This option is fairly light weight and I will likely enable it in GENERIC before 12.0 release. I have heard reports that switching from a default policy of first-touch to round-robin has caused some performance regression. You can change the default policy at runtime by doing the following: cpuset -s 1 -n first-touch:all This is the default set that all others inherit from. You can query the current default with: cpuset -g -s 1 I will be investigating the regression and tweaking the default policy based on performance feedback from multiple workloads. This may take some time. numactl is still functional but deprecated. Man pages will be updated soonish. Thank you for your patience as I work on refining this somewhat involved feature. Thanks, Jeff On Tue, 9 Jan 2018, Jeff Roberson wrote: > Hello folks, > > I am working on merging improved NUMA support with policy implemented by > cpuset(2) over the next week. This work has been supported by Dell/EMC's > Isilon product division and Netflix. You can see some discussion of these > changes here: > > https://reviews.freebsd.org/D13403 > https://reviews.freebsd.org/D13289 > https://reviews.freebsd.org/D13545 > > The work has been done in user/jeff/numa if you want to look at svn history > or experiment with the branch. It has been tested by Peter Holm on i386 and > amd64 and it has been verified to work on arm at various points. > > We are working towards compatibility with libnuma and linux mbind. These > commits will bring in improved support for NUMA in the kernel. There are new > domain specific allocation functions available to kernel for UMA, malloc, > kmem_, and vm_page*. busdmamem consumers will automatically be placed in the > correct domain, bringing automatic improvements to some device performance. > > cpuset will be able to constrains processes, groups of processes, jails, etc. > to subsets of the system memory domains, just as it can with sets of cpus. > It can set default policy for any of the above. Threads can use cpusets to > set policy that specifies a subset of their visible domains. > > Available policies are first-touch (local in linux terms), round-robin > (similar to linux interleave), and preferred. For now, the default is > round-robin. You can achieve a fixed domain policy by using round-robin with > a bitmask of a single domain. As the scheduler and VM become more > sophisticated we may switch the default to first-touch as linux does. > > Currently these features are enabled with VM_NUMA_ALLOC and MAXMEMDOM. It > will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA > syscalls and VM_NUMA_ALLOC code was 'experimental' and will be deprecated. > numactl will continue to be supported although cpuset should be preferred > going forward as it supports the full feature set of the new API. > > Thank you for your patience as I deal with the inevitable fallout of such > sweeping changes. If you do have bugs, please file them in bugzilla, or > reach out to me directly. I don't always have time to catch up on all of my > mailing list mail and regretfully things slip through the cracks when they > are not addressed directly to me. > > Thanks, > Jeff >