From nobody Fri Mar 22 18:54:06 2024
X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
	by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4V1WhY6CYzz5F5gx
	for <freebsd-hackers@mlmmj.nyi.freebsd.org>; Fri, 22 Mar 2024 18:54:21 +0000 (UTC)
	(envelope-from ticso@cicely7.cicely.de)
Received: from raven.bwct.de (raven.bwct.de [IPv6:2a02:21e0:16e0:fe::101:1])
	(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
	 key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
	 client-signature RSA-PSS (2048 bits) client-digest SHA256)
	(Client CN "raven.bwct.de", Issuer "raven.bwct.de" (not verified))
	by mx1.freebsd.org (Postfix) with ESMTPS id 4V1WhW74lfz4vVq
	for <freebsd-hackers@freebsd.org>; Fri, 22 Mar 2024 18:54:19 +0000 (UTC)
	(envelope-from ticso@cicely7.cicely.de)
Authentication-Results: mx1.freebsd.org;
	dkim=pass header.d=cicely.de header.s=default header.b=kbOAuUB3;
	dmarc=none;
	spf=pass (mx1.freebsd.org: domain of ticso@cicely7.cicely.de designates 2a02:21e0:16e0:fe::101:1 as permitted sender) smtp.mailfrom=ticso@cicely7.cicely.de
Received: from mail.cicely.de (mail.cicely.de [IPv6:2a02:21e0:16e0:20fe:0:0:101:c])
	by raven.bwct.de (8.15.2/8.15.2) with ESMTPS id 42MIs6LB069363
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL)
	for <freebsd-hackers@freebsd.org>; Fri, 22 Mar 2024 19:54:09 +0100 (CET)
	(envelope-from ticso@cicely7.cicely.de)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cicely.de; s=default;
	t=1711133649; bh=V+lyRBikAGslTqP6vYuECNTmM6vg07Lie+ua+qKUpvA=;
	h=Date:From:To:Cc:Subject;
	b=kbOAuUB3LwWGj4Uk2a4L9+lxocTPFIwB26+6nTLO8FOpCd1a9+Sg8KAXpvDQsDT2F
	 M0xgjNyA6HM0rBUOsC2AzJThQI66/TdYr4u/v4iH+nd86YUlbRslgjnn1du16h/i8Q
	 NjJnYz0L96ClTzDQG8Xzw7WzOEA175Vd19alWOQ8=
Received: from cicely7.cicely.de (c7-old.cicely.de [IPv6:2a02:21e0:16e0:20fe:0:0:101:d])
	by mail.cicely.de (8.16.1/8.16.1) with ESMTPS id 42MIs6gW062039
	(version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO);
	Fri, 22 Mar 2024 19:54:06 +0100 (CET)
	(envelope-from ticso@cicely7.cicely.de)
Received: from cicely7.cicely.de (localhost [127.0.0.1])
	by cicely7.cicely.de (8.17.1/8.17.1) with ESMTP id 42MIs6Z2039675;
	Fri, 22 Mar 2024 19:54:06 +0100 (CET)
	(envelope-from ticso@cicely7.cicely.de)
Received: (from ticso@localhost)
	by cicely7.cicely.de (8.17.1/8.17.1/Submit) id 42MIs61w039674;
	Fri, 22 Mar 2024 19:54:06 +0100 (CET)
	(envelope-from ticso)
Date: Fri, 22 Mar 2024 19:54:06 +0100
From: Bernd Walter <freebsd-hackers@cicely.de>
To: freebsd-hackers@freebsd.org
Cc: Bernd Walter <ticso@cicely7.cicely.de>
Subject: Only 1TB RAM out of 1.5TB on amd64
Message-ID: <Zf3TzpVwxirlUE2n@cicely7.cicely.de>
List-Id: Technical discussions relating to FreeBSD <freebsd-hackers.freebsd.org>
List-Archive: https://lists.freebsd.org/archives/freebsd-hackers
List-Help: <mailto:freebsd-hackers+help@freebsd.org>
List-Post: <mailto:freebsd-hackers@freebsd.org>
List-Subscribe: <mailto:freebsd-hackers+subscribe@freebsd.org>
List-Unsubscribe: <mailto:freebsd-hackers+unsubscribe@freebsd.org>
Sender: owner-freebsd-hackers@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
X-Operating-System: FreeBSD cicely7.cicely.de 13.2-RELEASE-p3 amd64
X-Rspamd-Server: rspamd.fizon.de
X-Spamd-Bar: ---
X-Spamd-Result: default: False [-3.19 / 15.00];
	NEURAL_HAM_MEDIUM(-1.00)[-1.000];
	NEURAL_HAM_LONG(-1.00)[-1.000];
	NEURAL_HAM_SHORT(-0.99)[-0.994];
	FORGED_SENDER(0.30)[freebsd-hackers@cicely.de,ticso@cicely7.cicely.de];
	R_SPF_ALLOW(-0.20)[+mx];
	R_DKIM_ALLOW(-0.20)[cicely.de:s=default];
	MIME_GOOD(-0.10)[text/plain];
	RCVD_TLS_LAST(0.00)[];
	FROM_HAS_DN(0.00)[];
	RCPT_COUNT_TWO(0.00)[2];
	DMARC_NA(0.00)[cicely.de];
	RCVD_COUNT_THREE(0.00)[4];
	ARC_NA(0.00)[];
	DKIM_TRACE(0.00)[cicely.de:+];
	FREEFALL_USER(0.00)[ticso];
	MIME_TRACE(0.00)[0:+];
	TO_MATCH_ENVRCPT_SOME(0.00)[];
	FROM_NEQ_ENVFROM(0.00)[freebsd-hackers@cicely.de,ticso@cicely7.cicely.de];
	MISSING_XM_UA(0.00)[];
	PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org];
	MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org];
	MID_RHS_MATCH_FROMTLD(0.00)[];
	ASN(0.00)[asn:44700, ipnet:2a02:21e0::/32, country:DE];
	TO_DN_SOME(0.00)[]
X-Rspamd-Queue-Id: 4V1WhW74lfz4vVq

...
real memory  = 1649265344512 (1572862 MB)
avail memory = 1057138888704 (1008166 MB)
...

I suspect address space issues, but don't know for sure.
Changing NUMA nodes per socket in the BIOS had an influence to make it
worse, but not better.


Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.3-RELEASE releng/13.3-n257428-80d2b634ddf0 GENERIC amd64
FreeBSD clang version 17.0.6 (https://github.com/llvm/llvm-project.git llvmorg-17.0.6-0-g6009708b4367)
VT(efifb): resolution 1024x768
CPU: AMD EPYC 7352 24-Core Processor                 (2300.07-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x830f10  Family=0x17  Model=0x31  Stepping=0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x7ed8320b<SSE3,PCLMULQDQ,MON,SSSE3,FMA,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
  AMD Features2=0x75c237ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT,TCE,Topology,PCXC,PNXC,DBE,PL2I,MWAITX,ADMSKX>
  Structured Extended Features=0x219c91a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,PQE,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA>
  Structured Extended Features2=0x400004<UMIP,RDPID>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  AMD Extended Feature Extensions ID EBX=0x18cf757<CLZERO,IRPerf,XSaveErPtr,RDPRU,MCOMMIT,WBNOINVD,IBPB,IBRS,STIBP,PREFER_IBRS,PPIN,SSBD>
  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics
real memory  = 1649265344512 (1572862 MB)
avail memory = 1057138888704 (1008166 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 96 CPUs
FreeBSD/SMP: 2 package(s) x 8 cache groups x 3 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.1> irqs 0-23
ioapic1 <Version 2.1> irqs 24-55
ioapic2 <Version 2.1> irqs 56-87
ioapic3 <Version 2.1> irqs 88-119
ioapic4 <Version 2.1> irqs 120-151
ioapic5 <Version 2.1> irqs 152-183
ioapic6 <Version 2.1> irqs 184-215
ioapic7 <Version 2.1> irqs 216-247
ioapic8 <Version 2.1> irqs 248-279
Launching APs: 32 30 34 13 14 16 23 21 19 24 28 9 86 10 87 7 26 37 39 41 45 46 43 66 71 69 54 57 59 80 82 78 49 53 51 60 65 62 84 88 91 95 74 73 76 92 33 40 11 2 27 22 20 5 44 18 15 36 6 8 25 89 79 85 93 56 48 70 31 81 12 50 38 77 55 63 75 58 1 90 29 94 52 72 67 83 35 42 61 64 68 47 4 17 3
random: entropy device external interface
kbd1 at kbdmux0
...

kern.maxphys: 1048576
hw.physmem: 1099322220544
hw.usermem: 1091323355136
hw.realmem: 1649265344512
vm.phys_locality: 
0: 10 32 
1: 32 10 

vm.phys_segs: 
SEGMENT 0:

start:     0x10000
end:       0xa0000
domain:    0
free list: 0xffffffff81ec7ea0

SEGMENT 1:

start:     0x100000
end:       0x1000000
domain:    0
free list: 0xffffffff81ec7ea0

SEGMENT 2:

start:     0x1000000
end:       0x74000000
domain:    0
free list: 0xffffffff81ec7c30

SEGMENT 3:

start:     0x74043000
end:       0x75db0000
domain:    0
free list: 0xffffffff81ec7c30

SEGMENT 4:

start:     0x76000000
end:       0x963b8000
domain:    0
free list: 0xffffffff81ec7c30

SEGMENT 5:

start:     0x963c2000
end:       0x963f9000
domain:    0
free list: 0xffffffff81ec7c30

SEGMENT 6:

start:     0x96500000
end:       0xa57f7000
domain:    0
free list: 0xffffffff81ec7c30

SEGMENT 7:

start:     0xa8e96000
end:       0xac000000
domain:    0
free list: 0xffffffff81ec7c30

SEGMENT 8:

start:     0x10000e000
end:       0x7d07800000
domain:    0
free list: 0xffffffff81ec79c0

SEGMENT 9:

start:     0x10000019000
end:       0x1797de00000
domain:    1
free list: 0xffffffff81ec8110

SEGMENT 10:

start:     0x17ffdc00000
end:       0x17ffdde7000
domain:    1
free list: 0xffffffff81ec8110



-- 
B.Walter <bernd@bwct.de> https://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.