From owner-freebsd-amd64@freebsd.org Sun Apr 8 16:39:41 2018 Return-Path: Delivered-To: freebsd-amd64@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3AC3F8EF1B for ; Sun, 8 Apr 2018 16:39:40 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic316-8.consmr.mail.gq1.yahoo.com (sonic316-8.consmr.mail.gq1.yahoo.com [98.137.69.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A09E70C1C for ; Sun, 8 Apr 2018 16:39:40 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: u_Ux55QVM1k4XIc7bIAbet2wS8WhBYeQq1NlNCkjUfQ9dWprhhpwLbH4cuFJozg 7kV9AMEjQC7LniK8U2KIPs5gaI1Dafmdl.SbI8bbjj5n8ZR82tFqvcIHG1lt2dONdwNjpFUkHFgP SF7QgiUpk2JjT4Jh8C1SU1MMz.wrenDoIMTh5ke8PBrFfrzHxx8rQ.R1EQU1btpqeKMc05vRbGax O3tRirwCdJlOgsU8ng.4WooHwNNqVjZZascZug1Tx_Ra0jlaAZk5Z95SRjWtb36SVxV4doz7PBVv zCMnDFUNfmOi34O2KOJQGvqeObKw0tGaloxviCDil9uWFBuStt92zxGUboGzvTC1EzCX1aAJCRv4 Z1hkY76p9Cln5iNNoW0RY4NZtT_n_ZopGTggcNgkGEzCuLtgBmV8pAFyjM5Qipp.o5shTAnrlIbA LPM.y8uv9hekIvw1dnUMqG0keoU.fuy0fn7XK2GOkNoi8w2bPGkldqak0dDILKaqF45CUEAhNQla sUO8BLwY7gqIW7a6tAVSdvGeQrfAKmXml5N36 Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Sun, 8 Apr 2018 16:39:38 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp431.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID d9f02034e3b178c021c97b2e13312602; Sun, 08 Apr 2018 16:29:26 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\)) Subject: Ryzen Threadripper, Hyper-V, and NUMA vs. DIMMs: 3 DIMMs on each side seems to always be in "Local" mode, not "Distributed" Message-Id: Date: Sun, 8 Apr 2018 09:29:25 -0700 To: freebsd-hackers@freebsd.org, freebsd-amd64@freebsd.org, FreeBSD Current X-Mailer: Apple Mail (2.3445.6.18) X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Apr 2018 16:39:41 -0000 Context: Ryzen Threadripper 1950X under Windows 10 Pro with Hyper-V (used to run FreeBSD). In experimenting with switching a Threadripper 1950X to have ECC RAM I discovered: A) The maximum ECC memory it would put to use was 96 GiBytes (3 DIMMs on each side, a 4th on each side was recognized but was ignored/disabled if present). B) AMD Ryzen Master classified the 96 GiByte configurations (with or without the ignored DIMMs) as "Local" without an ability switch to "Distributed". C) The downloaded Windows CoreInfo.exe utility agreed on there being 2 NUMA nodes. D) As did the result of the User Hardware Topology button in the Hyper-V Processor > NUMA settings: On a single virtual non-uniform memory architecture node: Maximum number of processors : 16 Maximum amount of memory (MB) : 48070 Maximum NUMA nodes allowed on a socket: 2 Only 1 socket. E) The CoreInfo.exe quick "Approximate Cross-NUMA Node Access Cost (relative to fastest)" tends to show the 4 numbers varying from 1.0 to 1.7 when retried repeatedly. An oddity is that sometimes the 1.0's are between 00 and 01, in fact this seems usual, and normally at most one 1.0 exists. The 00 row seems to always have the smaller numbers. An example: 00 01 00: 1.2 1.0 01: 1.3 1.5 I had no original intent of playing with NUMA but I figured that the Threadripper could be configured for such, and even has configurations that apparently require such as far as AMD Ryzen Master is concerned, could be of interest and possible use for folks testing FreeBSD NUMA support. Since I'd done nothing to build a kernel with NUMA enabled, FreeBSD 12.0 under Hyper-V did not see the NUMA structure from (D). One thing that did show during booting was getting 4 lines of: "SRAT: Ignoring memory at addr" instead of 2. === Mark Millard marklmi26-fbsd at yahoo.com ( dsl-only.net went away in early 2018-Mar)