From owner-freebsd-hackers@freebsd.org Mon Sep 23 20:28:21 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DD795FECC7 for ; Mon, 23 Sep 2019 20:28:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-20.consmr.mail.gq1.yahoo.com (sonic317-20.consmr.mail.gq1.yahoo.com [98.137.66.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46cbVr1WwCz4TLq for ; Mon, 23 Sep 2019 20:28:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: yr3jZ5sVM1mGGyjS7uFim0GLnpJRqkVKfPqh4P1Ysb2Ol0lavpF_R_JxL0LJgfV rPV446RYOVfoQLpPuSCQNQRF1pa5c5KgiVZ08XT7k57bmatkSictLO8LsBk6ps.E3JGKs8sxfgfp ICp7KM82RbGD4ljrQRKoH7QpBAvlI8cpQV63avqQpN0FqGKyGeAOvPnS..2Oia3v6lO2zL6DwgI7 GWJ2a84e3DoGr8td9Hv3pbjklFSCwBl_q..mYWCs1c7MxSRElX2U34FgYjHvxrTshWRX_kuI0M2M dvfVbi6bcGPhtUKl0TqcjjoC_S3.DXeHW9o2sDIBL2g8GfpVagG65PRMf9mi8HF16IUdGa8gD4ba d8AQp1hLMkzaF.gI08iN59Od1_HAEGGDldWxOMSibJIjU6e9kKmIBIZT4xCrXaftLadSH9tVUqql MKPZ2XCpGISmY3kfRncqCH5B7dnq_x2sSfSouB8.5z5P8M9.je0q_C9Sdl7qSYVRpBe3iR9Ons6w FwgAeHDl3ebZga0XDqF70A8KPRukKqUkLjt7RDgOCA50t8GC3r3iD4xfVTd26Dw.tF7W1J0I9lnn fgKAeCvYnOE2M.EQY1hjb5V8PrQQ8cvkmvVSMieLkr8Zbdf_iH1eLvdLJDo.sllID7EsRc1JobOi Tck9XSTw9b3OBDcCORxemCtwlxyfAPFW_vLbIHiaEaI.irAnLkE__SS.MHniJMqWIs3pNmP9kN6o Pvyoj1cIMvlhCuvQ94hX2nBlqpOjce81pcpOpvykVv4_PbANW5_cAx0BkdaydEwxxu.deWTl7YAX R4cgIlxCt2IB19mrO0mMWWyarxoPXK6Ll7MTCM0.ZnKcGrxVSCaFcyb7ew43uqHeb6Ciw14Svd7l yjyM7dZB.W_VS4fLgS2rrzSQbI8K96jXJ4R8tzA4yocBLfT7xkuxYNfn_GsBmHk_Q_PR1l4Z_Ya9 CwScmsQo3Pj4zaGMdKAkQHUT_pgtrDxAoh6eVCg0.WxvDCRuXearmr0wp6TGYO7h7WFnSFQcilUz u.yNqPDldf8BqfqAdR3CA1v4b5K7M6yCin.U8o_9GfQnd7dfrYmzuvolh6MvueaTfcKfrkNIuc8F pNMGO8iHiERfDEn1UZKLcpA8qM4OQGXewsNlQYz39sKPalftwufBGlaFo_Bgg2po856syJZHzIWK 47KaAHPh7xldVO0bGpcICsbB.mxtKykLOr2mVd272UIW.GbYBhlK2m3X2lk_WwdUV.4InIc76oPL sP.cUDMOz9qHbIx1Q..XXj45u5Cbmj7RrnxEPM6MCRmqq14dFvcp40L.hiIEvp9es8Nqz Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Mon, 23 Sep 2019 20:28:18 +0000 Received: by smtp424.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 26c35cfde03bc3c0174f09621ebbb85c; Mon, 23 Sep 2019 20:28:16 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: head -r352341 example context on ThreadRipper 1950X: cpuset -n prefer:1 with -l 0-15 vs. -l 16-31 odd performance? Message-Id: <704D4CE4-865E-4C3C-A64E-9562F4D9FC4E@yahoo.com> Date: Mon, 23 Sep 2019 13:28:15 -0700 To: freebsd-amd64@freebsd.org, freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3445.104.11) X-Rspamd-Queue-Id: 46cbVr1WwCz4TLq X-Spamd-Bar: / X-Spamd-Result: default: False [0.50 / 15.00]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_NONE(0.00)[]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.85)[-0.847,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (3.34), ipnet: 98.137.64.0/21(0.94), asn: 36647(0.75), country: US(-0.05)]; NEURAL_SPAM_MEDIUM(0.85)[0.847,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[146.66.137.98.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Sep 2019 20:28:21 -0000 Note: I have access to only one FreeBSD amd64 context, and it is also my only access to a NUMA context: 2 memory domains. A Threadripper 1950X context. Also: I have only a head FreeBSD context on any architecture, not 12.x or before. So I have limited compare/contrast material. I present the below basically to ask if the NUMA handling has been validated, or if it is going to be, at least for contexts that might apply to ThreadRipper 1950X and analogous contexts. My results suggest they are not (or libc++'s now times get messed up such that it looks like NUMA mishandling since this is based on odd benchmark results that involve mean time for laps, using a median of such across multiple trials). I ran a benchmark on both Fedora 30 and FreeBSD 13 on this 1950X got got expected results on Fedora but odd ones on FreeBSD. The benchmark is a variation on the old HINT benchmark, spanning the old multi-threading variation. I later tried Fedora because the FreeBSD results looked odd. The other architectures I tried FreeBSD benchmarking with did not look odd like this. (powerpc64 on a old PowerMac 2 socket with 2 cores per socket, aarch64 Cortex-A57 Overdrive 1000, CortextA53 Pine64+ 2GB, armv7 Cortex-A7 Orange Pi+ 2nd Ed. For these I used 4 threads, not more.) I tend to write in terms of plots made from the data instead of the raw benchmark data. FreeBSD testing based on: cpuset -l0-15 -n prefer:1 cpuset -l16-31 -n prefer:1 Fedora 30 testing based on: numactl --preferred 1 --cpunodebind 0 numactl --preferred 1 --cpunodebind 1 While I have more results, I reference primarily DSIZE and ISIZE being unsigned long long and also both being unsigned long as examples. Variations in results are not from the type differences for any LP64 architectures. (But they give an idea of benchmark variability in the test context.) The Fedora results solidly show the bandwidth limitation of using one memory controller. They also show the latency consequences for the remote memory domain case vs. the local memory domain case. There is not a lot of variability between the examples of the 2 type-pairs used for Fedora. Not true for FreeBSD on the 1950X: A) The latency-constrained part of the graph looks to normally be using the local memory domain when -l0-15 is in use for 8 threads. B) Both the -l0-15 and the -l16-31 parts of the graph for 8 threads that should be bandwidth limited show mostly examples that would have to involve both memory controllers for the bandwidth to get the results shown as far as I can tell. There is also wide variability ranging between the expected 1 controller result and, say, what a 2 controller round-robin would be expected produce. C) Even the single threaded result shows a higher result for larger total bytes for the kernel vectors. Fedora does not. I think that (B) is the most solid evidence for something being odd. For reference for FreeBSD: # cpuset -g -d 1 domain 1 mask: 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31 -r352341 allows -prefer:0 but I happen to have used -prefer:1 in these experiments. The benchmark was built via devel/g++9 but linked with system libraries, including libc++. Unfortunately, I'm not yet ready for distributing source to the benchmark, but expect to at some point. I do not expect to ever distribute binaries. The source code for normal builds involves just standard C++17 code. Such builds are what is involved here. [The powerpc64 context is a system-clang 8, ELFv1 based system context, not the usual gcc 4.2.1 based one.] More notes: In the 'kernel vectors: total Bytes' vs. 'QUality Improvement Per Second' graphs the left hand side of the curve is latency limited. On the right is bandwidth limited for LP64. (The total Bytes axis is log base 2 scaling in the graphs.) Thread creation has latency so the 8-thread curves are mostly of interest for kernel vectors total bytes being 1 MiByte or more (say) so that thread creations are not that much of the total contributions to the measured time. The thread creations are via std::async use. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)