From owner-freebsd-hackers@FreeBSD.ORG Mon Apr 22 19:17:45 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8B5A751B for ; Mon, 22 Apr 2013 19:17:45 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 682BD1F80 for ; Mon, 22 Apr 2013 19:17:45 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 54793B962; Mon, 22 Apr 2013 15:17:42 -0400 (EDT) From: John Baldwin To: Freddie Cash Subject: Re: NUMA, cpuset and malloc Date: Mon, 22 Apr 2013 13:17:10 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <201304221132.08194.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201304221317.10722.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Mon, 22 Apr 2013 15:17:42 -0400 (EDT) Cc: FreeBSD Hackers , Robert Waksmundzki X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Apr 2013 19:17:45 -0000 On Monday, April 22, 2013 12:29:29 pm Freddie Cash wrote: > On Mon, Apr 22, 2013 at 8:32 AM, John Baldwin wrote: > > > On Saturday, April 20, 2013 6:43:26 pm Robert Waksmundzki wrote: > > > On NUMA systems allocated memory is striped across local and non-local > > banks > > in order to have consistent performance in case the task is rescheduled to > > a > > different CPU socket. > > > When a process is pinned to a single CPU socket with cpuset having the > > memory allocator prefer local banks would probably improve performance. > > Default system behavior would stay the same and the optimization would > > only be > > triggered on big multi socket systems when administrator used cpuset > > (command > > mostly used for performance optimization anyway). > > > > > > Is this something currently implemented in FreeBSD? Is this even a good > > idea? > > > > You can get something sort of like this by enabling NUMA in your kernel > > (9.0 > > and later) and always pinning your processes with cpuset. (The simple NUMA > > bits always allocate memory in the memory domain the current thread is > > running in at the time of the fault.) > > > > How does one enable NUMA? > > A "grep -i numa /usr/src/sys/conf/NOTES /usr/src/sys/amd64/conf/NOTES" > turns up 0 hits for both 9-STABLE r248547 and 10-CURRENT (April 11, used > svnup so no way to get the exact revision number, that I know of). > > Or, is it enabled automatically? You have to chagne the VM_NDOMAIN setting. In recent HEAD and 9-stable you can do it in the kernel config (options VM_NDOMAIN=4 for example). In older HEAD and 9 you have to edit sys/amd64/include/vmparam.h or sys/i386/include/vmparam.h and change VM_NDOMAIN before building your kernel. -- John Baldwin