From owner-freebsd-current@freebsd.org Thu Jan 18 02:48:24 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EF8C3E75D66 for ; Thu, 18 Jan 2018 02:48:24 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CA50571170 for ; Thu, 18 Jan 2018 02:48:24 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id C9A03E75D64; Thu, 18 Jan 2018 02:48:24 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9421E75D63 for ; Thu, 18 Jan 2018 02:48:24 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: from mail-yw0-x22f.google.com (mail-yw0-x22f.google.com [IPv6:2607:f8b0:4002:c05::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 84E737116F for ; Thu, 18 Jan 2018 02:48:24 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: by mail-yw0-x22f.google.com with SMTP id m19so10053894ywh.12 for ; Wed, 17 Jan 2018 18:48:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=TYRxIRKBeCqTcdJK1Q8DIPEdX+lfrDHk4ky2nA16ycQ=; b=gzHx+q1LFLezPTO+UtzWhGoB9q5wl+LSJZUB0+Ydqw9DU8uE1gz0sSpE3oJZIGnUkz bAdgOWgF2GaoyQjEcplXainVu9zQC/nxa4ICGE3/oBDfv5WHSifXZkUoXWFDaNO6mE88 iHY80zFzDoMrh1yAGkE8ualNgcsEks2Rcnqq/AUIlHiS7xTdikAcBgXclz2cWZONq4M0 s7zejbl16RNkXs1QB1DHbFDz8rw2xA3H1vQ8WJNriOP+DWX/15QsxrVk72DO9J9hEBjn 2KbaNGS3A77oHEf93t+Na7CqN/pzpEHpi4mwhCLdd1Ik7Jarm6TbiZIKP7D/4bCB9GLP vwDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=TYRxIRKBeCqTcdJK1Q8DIPEdX+lfrDHk4ky2nA16ycQ=; b=YAgUKLaQDyG0JnV5LZmLDAL5BuqmrbAstQzZEAONkOXkaPpiU7k6N1in6t2RrKZFW0 oiLJ+ysE30n2qSxrDO6VI7rCgnINNwAT8lSFbgWUPBXqKP3exJUTDm9wPvQO4WtY+/Ds SKQBF0O813EtBygodG9YXiekaL7wMVdznfTgKHOZv9fx2LqgTG65DwPyPx0h2LWiYMGO dE2V1J79SxXdi3qwWzVBFmEIUoUNEvtHF7ZGVkAUSivzCVJkoCA+RHXHzRfhpZ/S+Euz qo5uVovOeuPbbuzkXq/jdiQsqVsPc0peElNmhqBQBy3T9MKUpS3Vyb4WI/6zUtqckh5x Hewg== X-Gm-Message-State: AKwxytd2IPLQaM+ycpXr5ay/6it/yVKyDyjPfrXaVUrgtq9NDGSEcGbs 8GmeZOF1VNZbKuV+blQ0uPdZLbpNasgy17ltcNarhzrG X-Google-Smtp-Source: ACJfBou1u7kGtvTIw7RCdMLzTbc7uO6sW7HeGeWs9GmFpFiMws43TRCzg04w8Y2j0AN2wAqnr2mGkpJs1brSoDof+Vw= X-Received: by 10.129.116.8 with SMTP id p8mr4623248ywc.386.1516243703531; Wed, 17 Jan 2018 18:48:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.129.74.214 with HTTP; Wed, 17 Jan 2018 18:48:23 -0800 (PST) In-Reply-To: References: From: Ultima Date: Wed, 17 Jan 2018 18:48:23 -0800 Message-ID: Subject: Re: New NUMA support coming to CURRENT To: Jeff Roberson Cc: current@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 18 Jan 2018 02:48:25 -0000 Hello Jeff, Few days ago I upgraded my system firmware, upgraded base to r327991 and altered snooping to cluster-on-die. Its hard to say what is making the server feels like a completely system due to all these changes (also running llvm 6.0), but I am betting it is the NUMA optimizations. It is so responsive! Thanks for the amazing work! Best regards, Richard Gallamore On Sat, Jan 13, 2018 at 7:39 PM, Jeff Roberson wrote: > Hello, > > This work has been committed. It is governed by a new 'NUMA' config > option and 'DEVICE_NUMA' and 'VM_NUMA_ALLOC' have both been retired. This > option is fairly light weight and I will likely enable it in GENERIC before > 12.0 release. > > I have heard reports that switching from a default policy of first-touch > to round-robin has caused some performance regression. You can change the > default policy at runtime by doing the following: > > cpuset -s 1 -n first-touch:all > > This is the default set that all others inherit from. You can query the > current default with: > cpuset -g -s 1 > > I will be investigating the regression and tweaking the default policy > based on performance feedback from multiple workloads. This may take some > time. > > numactl is still functional but deprecated. Man pages will be updated > soonish. > > Thank you for your patience as I work on refining this somewhat involved > feature. > > Thanks, > Jeff > > > On Tue, 9 Jan 2018, Jeff Roberson wrote: > > Hello folks, >> >> I am working on merging improved NUMA support with policy implemented by >> cpuset(2) over the next week. This work has been supported by Dell/EMC's >> Isilon product division and Netflix. You can see some discussion of these >> changes here: >> >> https://reviews.freebsd.org/D13403 >> https://reviews.freebsd.org/D13289 >> https://reviews.freebsd.org/D13545 >> >> The work has been done in user/jeff/numa if you want to look at svn >> history or experiment with the branch. It has been tested by Peter Holm on >> i386 and amd64 and it has been verified to work on arm at various points. >> >> We are working towards compatibility with libnuma and linux mbind. These >> commits will bring in improved support for NUMA in the kernel. There are >> new domain specific allocation functions available to kernel for UMA, >> malloc, kmem_, and vm_page*. busdmamem consumers will automatically be >> placed in the correct domain, bringing automatic improvements to some >> device performance. >> >> cpuset will be able to constrains processes, groups of processes, jails, >> etc. to subsets of the system memory domains, just as it can with sets of >> cpus. It can set default policy for any of the above. Threads can use >> cpusets to set policy that specifies a subset of their visible domains. >> >> Available policies are first-touch (local in linux terms), round-robin >> (similar to linux interleave), and preferred. For now, the default is >> round-robin. You can achieve a fixed domain policy by using round-robin >> with a bitmask of a single domain. As the scheduler and VM become more >> sophisticated we may switch the default to first-touch as linux does. >> >> Currently these features are enabled with VM_NUMA_ALLOC and MAXMEMDOM. >> It will eventually be NUMA/MAXMEMDOM to match SMP/MAXCPU. The current NUMA >> syscalls and VM_NUMA_ALLOC code was 'experimental' and will be deprecated. >> numactl will continue to be supported although cpuset should be preferred >> going forward as it supports the full feature set of the new API. >> >> Thank you for your patience as I deal with the inevitable fallout of such >> sweeping changes. If you do have bugs, please file them in bugzilla, or >> reach out to me directly. I don't always have time to catch up on all of >> my mailing list mail and regretfully things slip through the cracks when >> they are not addressed directly to me. >> >> Thanks, >> Jeff >> >> _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" >