Date: Sun, 27 Dec 2020 22:56:13 -0800 From: Mark Millard <marklmi@yahoo.com> To: bob prohaska <fbsd@www.zefox.net> Cc: freebsd-arm <freebsd-arm@freebsd.org> Subject: Re: Migrating from -current to stable/12 on RPI2B (ARMv7) Message-ID: <D8278682-17B3-4969-905D-D1F084BBB2B8@yahoo.com> In-Reply-To: <F9CB3890-5E07-46C9-AC40-D968F8B51B1F@yahoo.com> References: <20201228044840.GA28380@www.zefox.net> <F9CB3890-5E07-46C9-AC40-D968F8B51B1F@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2020-Dec-27, at 22:10, Mark Millard <marklmi at yahoo.com> wrote:
> On 2020-Dec-27, at 20:48, bob prohaska <fbsd@www.zefox.net> wrote:
>=20
>> Having a bit of trouble migrating to stable/12 from 13.0-CURRENT=20
>> FreeBSD 13.0-CURRENT #3 r368820: Sat Dec 26 19:01:50 PST 2020 =20
>> bob@www.zefox.com:/usr/obj/usr/src/arm.armv7/sys/GENERIC-MMCCAM arm
>>=20
>> This is on a Pi2B v1.1 The MMCCAM kernel option was added for sake=20
>> of the experiment and hasn't, up to now, been accompanied by any
>> visible problems. World and kernel have been built many times, no
>> problems.=20
>>=20
>> Buildworld keeps stopping at
>> --- clang.full ---
>> c++ -O -pipe -fno-common -mlong-calls =
-I/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libclang =
-I/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libllvm =
-I/usr/freebsd-src/contrib/llvm-project/clang/include =
-I/usr/freebsd-src/lib/clang/include =
-I/usr/freebsd-src/contrib/llvm-project/llvm/include =
-D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS =
-DHAVE_VCS_VERSION_INC -DNDEBUG =
-DLLVM_DEFAULT_TARGET_TRIPLE=3D\"armv7-unknown-freebsd12.2-gnueabihf\" =
-DLLVM_HOST_TRIPLE=3D\"armv7-unknown-freebsd12.2\" =
-DDEFAULT_SYSROOT=3D\"/usr/obj/usr/freebsd-src/arm.armv7/tmp\" =
-DLLVM_TARGET_ENABLE_ARM =
-DLLVM_NATIVE_ASMPARSER=3DLLVMInitializeARMAsmParser =
-DLLVM_NATIVE_ASMPRINTER=3DLLVMInitializeARMAsmPrinter =
-DLLVM_NATIVE_DISASSEMBLER=3DLLVMInitializeARMDisassembler =
-DLLVM_NATIVE_TARGET=3DLLVMInitializeARMTarget =
-DLLVM_NATIVE_TARGETINFO=3DLLVMInitializeARMTargetInfo =
-DLLVM_NATIVE_TARGETMC=3DLLVMInitializeARMTargetMC -ffunction-sections =
-fdata-sections -gline-tables-only -Wno-for
>> mat-zero-length -Qunused-arguments =
-I/usr/obj/usr/freebsd-src/arm.armv7/tmp/legacy/usr/include =
-fno-exceptions -fno-rtti -std=3Dc++14 -stdlib=3Dlibc++ =
-Wno-c++11-extensions -Wl,--gc-sections -static =
-L/usr/obj/usr/freebsd-src/arm.armv7/tmp/legacy/usr/lib -o clang.full =
cc1_main.o cc1as_main.o cc1gen_reproducer_main.o driver.o =
/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libclang/libcla=
ng.a =
/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/clang/libllvm/libllvm=
.a -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libz -lz =
-L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libexecinfo =
-lexecinfo -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libelf =
-lelf =
-L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/ncurses/ncursesw =
-lncursesw -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools/lib/libthr =
-lpthread -legacy
>> ld: error: failed to open clang.full: Cannot allocate memory
>> c++: error: linker command failed with exit code 1 (use -v to see =
invocation)
>> *** [clang.full] Error code 1
>>=20
>> make[4]: stopped in /usr/freebsd-src/usr.bin/clang/clang
>> 1 error
>>=20
>> The failure occurs many minutes after the log entry appears.=20
>> No errors on the console, swap tops near half a gig. The=20
>> stable/12 sources being compiled were obtained via git. The=20
>> -current sources used to compile the running system were=20
>> obtained using svnlite.=20
>>=20
>> I don't recall seeing this much swap used before by armv7,=20
>> but that's the only notable oddity. =20
>>=20
>> The stable/12 sources have been updated every day for the=20
>> past few in hopes the trouble might go away, the -current=20
>> sources seem to be updated as far as svnlite goes.=20
>>=20
>> Any suggestions appreciated, thanks for reading
>>=20
>=20
> I presume the historical system tuning for small memory systems
> from our past investigations and help that we got.
>=20
> You do not mention -j4 vs. -j1 or the like. I'm assuming -j1 has
> been tried.
>=20
> You have not said if you have adjusted the likes of:
> kern.maxswapzone or VM_SWZONE_SIZE_MAX or if you get the boot
> notices about the likes of:
>=20
> warning: total configured swap (??? pages) exceeds maximum recommended =
amount (??? pages).
>=20
> Mistuning these could lead to RAM fragmentation issues as I
> understand.
>=20
> Since the allocation failure is reported by ld while it tries to
> produce clang.full , I wonder if you have used:
>=20
> LDFLAGS.lld+=3D -Wl,--no-threads
>=20
> in, say, /etc/make.conf ? If not, the llvm ld will try to use
> all 4 RPi2 v1.1 cores (via threads, not via processes) and, from
> what I've seen, will end up trying to use more RAM than when it
> runs single-threaded (main thread only). Of course, links then
> take longer to complete. (LLVM ld actually ends up up with 4+2
> threads, counting the processes's main thread as well.)
>=20
> Except on the machines with plenty of RAM (by a large margin for
> the hardware-thread count), I use --no-threads for contexts where
> I can readily control such.
>=20
> (Ports do not use LDFLAGS.lld and gnu's linker does not tolerate
> --no-threads as a command line option. So "readily" is limited to
> buildkernel and buildworld .)
Another type of thing that you have not mentioned: non-debug vs.
debug status of various things (debug usually meaning more RAM
ends up in use during builds). But I've not experimented to know
if some of these could make a nontrivial difference in the RAM
use.
Is the host 13 built with debug information and debug code generally?
kernel? world? Is its llvm toolchain build with debug asserts and the
like?
Is the clang.full being built one that was built with debug information?
Debug asserts? Similar types of questions for the stable/12 "cross"
toolchain (older llvm vintage?) if such is built and used.
In stable/12 there are (for /etc/src.conf use):
WITHOUT_ASSERT_DEBUG
Set to compile programs and libraries without the assert(3)
checks.
and:
WITH_LLVM_ASSERTIONS
Set to enable debugging assertions in LLVM.
and:
WITHOUT_MALLOC_PRODUCTION
Set to enable assertions and statistics gathering in =
malloc(3).
It also defaults the A and J runtime options to on.
So using WITHOUT_ASSERT_DEBUG=3D but not using the other two would be
appropriate for having less RAM use.
But, be warned, 13 is still at a stage where it has
reversed two of the orientations (default vs. what it
can be changed to):
WITHOUT_LLVM_ASSERTIONS
Set to disable debugging assertions in LLVM.
WITH_MALLOC_PRODUCTION
Set to disable assertions and statistics gathering in =
malloc(3).
It also defaults the A and J runtime options to off.
So care in handling src.conf is required if you have been explicit
about those last two in past build activity. The combination for
minimizing RAM use from these is:
WITHOUT_ASSERT_DEBUG=3D
WITHOUT_LLVM_ASSERTIONS=3D
WITH_MALLOC_PRODUCTION=3D
If I understand right, both stable/12 and 13 will tolerate listing
those 3 in /etc/src.conf . (Despite the src.conf man pages not saying
so.)
There are, of course, other tradeoffs involved in minimizing RAM
use (host live use generally and target output-files related) for
what these options control.
For 13's kernel options: what of the status of (shown disabled) . . .
nooptions DEADLKRES # the deadlock resolver
nooptions INVARIANTS # calls of extra sanity checking
nooptions INVARIANT_SUPPORT # Extra sanity checks of =
internal structures, required by INVARIANTS
nooptions WITNESS # Enable checks to detect =
deadlocks and cycles
nooptions WITNESS_SKIPSPIN # Don't run witness on spinlocks =
for speed
nooptions DIAGNOSTIC
nooptions MALLOC_DEBUG_MAXZONES # Separate malloc(9) zones
and possibly the status of (shown not enabled):
#makeoptions DEBUG=3D-g # Build kernel with gdb(1) =
debug symbols
and ( in, say, /etc/src.conf ):
WITH_DEBUG_FILES=3D
(if the DEBUG=3D-g was used in makeoptions).
(Avoiding debug information being in the loaded kernel files
helps avoid as much RAM use, for example.)
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?D8278682-17B3-4969-905D-D1F084BBB2B8>