From nobody Sun May 7 21:19:01 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QDy3W1s1Jz49WtB for ; Sun, 7 May 2023 21:19:19 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from mail.rlwinm.de (mail.rlwinm.de [138.201.35.217]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QDy3V0183z3HtR for ; Sun, 7 May 2023 21:19:17 +0000 (UTC) (envelope-from crest@rlwinm.de) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of crest@rlwinm.de designates 138.201.35.217 as permitted sender) smtp.mailfrom=crest@rlwinm.de; dmarc=none Received: from [IPV6:2001:9e8:978:1d00:3db4:6da6:3cca:fbb9] (unknown [IPv6:2001:9e8:978:1d00:3db4:6da6:3cca:fbb9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-384) server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mail.rlwinm.de (Postfix) with ESMTPSA id 4B1E533F02 for ; Sun, 7 May 2023 21:19:03 +0000 (UTC) Message-ID: Date: Sun, 7 May 2023 23:19:01 +0200 List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.10.1 Subject: Re: Support for more than 256 CPU cores Content-Language: en-US To: freebsd-arch@freebsd.org References: <84816f2f-b23c-f448-55fe-454cbb604681@selasky.org> From: Jan Bramkamp In-Reply-To: <84816f2f-b23c-f448-55fe-454cbb604681@selasky.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Result: default: False [-3.23 / 15.00]; NEURAL_HAM_SHORT(-0.99)[-0.991]; NEURAL_HAM_MEDIUM(-0.97)[-0.971]; NEURAL_HAM_LONG(-0.96)[-0.965]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:24940, ipnet:138.201.0.0/16, country:DE]; R_DKIM_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[rlwinm.de]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4QDy3V0183z3HtR X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 05.05.23 17:52, Hans Petter Selasky wrote: > On 5/5/23 17:23, Tomek CEDRO wrote: >> On Fri, May 5, 2023 at 3:38 PM Ed Maste wrote: >>> FreeBSD supports up to 256 CPU cores in the default kernel >>> configuration >>> (on Tier-1 architectures).  Systems with more than 256 cores are >>> available now, and will become increasingly common over FreeBSD 14’s >>> lifetime. (..) >> >> Congratulations! :-) >> >> I am looking after AMD Threadripper with 64 cores 2 threads each that >> will give 128 CPU to the system.. maybe this year I could afford that >> beast then I will report back after testing :-) >> >> In upcoming years variations of RISC-V will provide unheard before >> number of CPU in a single SoC (i.e. 1000 CPU) at amazing power >> efficiency and I saw reports of prototype with 3 x SoC of this kind on >> a single board :-) >> >> https://spectrum.ieee.org/risc-v-ai >> > > Hi, > > Maybe it makes sense to cluster CPU's in logical groups somehow. Some > synchronization mechanism like EPOCH() are O(N²) where N is the number > of CPUs. Not in the read-case, but in the synchronize case. It depends > a bit though. Currently EPOCH() is executed every kern.hz . Unless the implementation scales quadratic with the number of CPU logical CPUs representable by cpuset_t instead of the number of CPUs available to the kernel it should still be worth it to change the ABI in time for 14.0 even if the existing implementation of EPOCH doesn't scale well. If the ABI isn't extended in time the whole 14.x major release line will either be limited to 256 logical CPUs, require users to change the type and recompile *everything*, or have to reinterpret cpuset_t as something other than a flat bitmap of cores. In my opinion there are three important constraints:  * Don't delay the 14.0 release more than necessary.  * Don't penalize systems with 256 or less cores too much.  * Make the struct big enough to allow a better implementation later. Just increasing the bitset by a factor of four shouldn't be to expensive, but is it enough to allow the implementation of algorithms that scale better or is additional storage needed. e.g. a bit for each logical group at each level of grouping?