Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 28 Dec 2020 10:56:22 -0800
From:      bob prohaska <fbsd@www.zefox.net>
To:        Mark Millard <marklmi@yahoo.com>
Cc:        freebsd-arm <freebsd-arm@freebsd.org>, bob prohaska <fbsd@www.zefox.net>
Subject:   Re: Migrating from -current to stable/12 on RPI2B (ARMv7)
Message-ID:  <20201228185622.GB28380@www.zefox.net>
In-Reply-To: <F9CB3890-5E07-46C9-AC40-D968F8B51B1F@yahoo.com>
References:  <20201228044840.GA28380@www.zefox.net> <F9CB3890-5E07-46C9-AC40-D968F8B51B1F@yahoo.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Dec 27, 2020 at 10:10:18PM -0800, Mark Millard wrote:
>=20
>=20
> On 2020-Dec-27, at 20:48, bob prohaska <fbsd@www.zefox.net> wrote:
>=20
> > Having a bit of trouble migrating to stable/12 from 13.0-CURRENT=20
> > FreeBSD 13.0-CURRENT #3 r368820: Sat Dec 26 19:01:50 PST 2020    =20
> > bob@www.zefox.com:/usr/obj/usr/src/arm.armv7/sys/GENERIC-MMCCAM  arm
> >=20
> > This is on a Pi2B v1.1 The MMCCAM kernel option was added for sake=20
> > of the experiment and hasn't, up to now, been accompanied by any
> > visible problems. World and kernel have been built many times, no
> > problems.=20
> >=20
> > Buildworld keeps stopping at
> > --- clang.full ---
> > c++ -O -pipe -fno-common -mlong-calls -I/usr/obj/usr/freebsd-src/arm.ar=
mv7/tmp/obj-tools/lib/clang/libclang -I/usr/obj/usr/freebsd-src/arm.armv7/t=
mp/obj-tools/lib/clang/libllvm -I/usr/freebsd-src/contrib/llvm-project/clan=
g/include -I/usr/freebsd-src/lib/clang/include -I/usr/freebsd-src/contrib/l=
lvm-project/llvm/include -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D=
__STDC_LIMIT_MACROS -DHAVE_VCS_VERSION_INC -DNDEBUG -DLLVM_DEFAULT_TARGET_T=
RIPLE=3D\"armv7-unknown-freebsd12.2-gnueabihf\" -DLLVM_HOST_TRIPLE=3D\"armv=
7-unknown-freebsd12.2\" -DDEFAULT_SYSROOT=3D\"/usr/obj/usr/freebsd-src/arm.=
armv7/tmp\" -DLLVM_TARGET_ENABLE_ARM -DLLVM_NATIVE_ASMPARSER=3DLLVMInitiali=
zeARMAsmParser -DLLVM_NATIVE_ASMPRINTER=3DLLVMInitializeARMAsmPrinter -DLLV=
M_NATIVE_DISASSEMBLER=3DLLVMInitializeARMDisassembler -DLLVM_NATIVE_TARGET=
=3DLLVMInitializeARMTarget -DLLVM_NATIVE_TARGETINFO=3DLLVMInitializeARMTarg=
etInfo -DLLVM_NATIVE_TARGETMC=3DLLVMInitializeARMTargetMC -ffunction-sectio=
ns -fdata-sections -gline-tables-only -Wno-for
> > mat-zero-length -Qunused-arguments -I/usr/obj/usr/freebsd-src/arm.armv7=
/tmp/legacy/usr/include -fno-exceptions -fno-rtti -std=3Dc++14 -stdlib=3Dli=
bc++ -Wno-c++11-extensions  -Wl,--gc-sections -static  -L/usr/obj/usr/freeb=
sd-src/arm.armv7/tmp/legacy/usr/lib -o clang.full  cc1_main.o cc1as_main.o =
cc1gen_reproducer_main.o driver.o /usr/obj/usr/freebsd-src/arm.armv7/tmp/ob=
j-tools/lib/clang/libclang/libclang.a /usr/obj/usr/freebsd-src/arm.armv7/tm=
p/obj-tools/lib/clang/libllvm/libllvm.a -L/usr/obj/usr/freebsd-src/arm.armv=
7/tmp/obj-tools/lib/libz -lz -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-t=
ools/lib/libexecinfo -lexecinfo -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/ob=
j-tools/lib/libelf -lelf -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/obj-tools=
/lib/ncurses/ncursesw -lncursesw -L/usr/obj/usr/freebsd-src/arm.armv7/tmp/o=
bj-tools/lib/libthr -lpthread -legacy
> > ld: error: failed to open clang.full: Cannot allocate memory
> > c++: error: linker command failed with exit code 1 (use -v to see invoc=
ation)
> > *** [clang.full] Error code 1
> >=20
> > make[4]: stopped in /usr/freebsd-src/usr.bin/clang/clang
> > 1 error
> >=20
> > The failure occurs many minutes after the log entry appears.=20
> > No errors on the console, swap tops near half a gig. The=20
> > stable/12 sources being compiled were obtained via git. The=20
> > -current sources used to compile the running system were=20
> > obtained using svnlite.=20
> >=20
> > I don't recall seeing this much swap used before by armv7,=20
> > but that's the only notable oddity. =20
> >=20
> > The stable/12 sources have been updated every day for the=20
> > past few in hopes the trouble might go away, the -current=20
> > sources seem to be updated as far as svnlite goes.=20
> >=20
> > Any suggestions appreciated, thanks for reading
> >=20
>=20
> I presume the historical system tuning for small memory systems
> from our past investigations and help that we got.
>=20
Correct. /boot/loader.conf contains=20
vm.pageout_oom_seq=3D"4096"
vm.pfault_oom_attempts=3D"-1"


> You do not mention -j4 vs. -j1 or the like. I'm assuming -j1 has
> been tried.
>=20
j values of 4, 2 and 1 have been tried, to no effect

> You have not said if you have adjusted the likes of:
> kern.maxswapzone or VM_SWZONE_SIZE_MAX or if you get the boot
> notices about the likes of:
>=20
> warning: total configured swap (??? pages) exceeds maximum recommended am=
ount (??? pages).
>=20
> Mistuning these could lead to RAM fragmentation issues as I
> understand.
>
Dmesg reports=20
warning: total configured swap (579552 pages) exceeds maximum recomm
ended amount (467936 pages).
warning: increase kern.maxswzone or reduce amount of swap.
but that hasn't caused recognizable trouble up to now. Adding
even more swap doesn't make things any worse.=20
=20
> Since the allocation failure is reported by ld while it tries to
> produce clang.full , I wonder if you have used:
>=20
> LDFLAGS.lld+=3D -Wl,--no-threads
>=20
> in, say, /etc/make.conf ? If not, the llvm ld will try to use
> all 4 RPi2 v1.1 cores (via threads, not via processes) and, from
> what I've seen, will end up trying to use more RAM than when it
> runs single-threaded (main thread only). Of course, links then
> take longer to complete. (LLVM ld actually ends up up with 4+2
> threads, counting the processes's main thread as well.)
>=20
> Except on the machines with plenty of RAM (by a large margin for
> the hardware-thread count), I use --no-threads for contexts where
> I can readily control such.
>=20
> (Ports do not use LDFLAGS.lld and gnu's linker does not tolerate
> --no-threads as a command line option. So "readily" is limited to
> buildkernel and buildworld .)
>
I didn't know about LDFLAGS, but a re-try with=20
-j1 and LDFLAGS.lld+=3D -Wl,--threads=3D1
(apparently the syntax changed) in /etc/make.conf=20
promptly reproduced the error.=20

At this point the host has two source directories:
/usr/src, obtained using svnlite, holding -current
and owned by root, along with=20
/usr/freebsd-src, obtained using git, holding -stable/12=20
owned by bob (member of wheel) where the error is occuring.

Could this be either a permissions problem or a conflict
among intermediate files left over between make attempts?
Each source directory is matched by its own object directory,
so that seems unlikely. There have been no warnings on the=20
console related to memory or swap, only the cryptic little=20
"cannot allocate memory" in the buildworld log.

It's especially puzzling that the problem is evident only
when compiling stable/12 as a regular user, but not when
compiling -current as root.

I'm trying a sanity check now by deleting clang.full from
/usr/obj/usr/src (two copies) and restarting buildword with
-j4 and no LDFLAGS to verify the error isn't present in
-current. That might take a day or two.=20

Many thanks for readying and replying!

bob prohaska




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20201228185622.GB28380>