Date: Sat, 4 Jun 2022 17:28:07 -0700 From: Mark Millard <marklmi@yahoo.com> To: Marcin Cieslak <saper@saper.info> Cc: Free BSD <freebsd-arm@freebsd.org>, bob prohaska <fbsd@www.zefox.net>, Mark Johnston <markj@freebsd.org> Subject: Re: devel/llvm13 failed to reclaim memory on 8 GB Pi4 running -current Message-ID: <324C9134-542F-4954-A0E6-E75A08491A8D@yahoo.com> In-Reply-To: <99rror60-3pnn-s3q6-2q70-5ss2p968r658@fncre.vasb> References: <20220127164512.GA51200@www.zefox.net> <C8BDF77F-5144-4234-A453-8DEC9EA9E227@yahoo.com> <B5F34703-F217-4719-B563-09B76D32401E@yahoo.com> <99rror60-3pnn-s3q6-2q70-5ss2p968r658@fncre.vasb>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2022-Jun-4, at 14:49, Marcin Cieslak <saper@saper.info> wrote:
> On Fri, 28 Jan 2022, Mark Millard wrote:
>=20
> [ Reviving old thread ]
>=20
>> After that I intend runs with 30 GiBytes of swap (so RAM+SWAP 38 =
GiBytes).
>> Hopefully that will complete and I'll be able to report how much swap =
was
>> observed to have been used.
>=20
> I thought I will get LLVM14 + OpenJDK built quickly so I fired up
> a c6g.4xlarge AWS instance (16 vCPUs, 32 GB RAM),
> or even c6g.8xlarge (32 vCPUs, 64 GB RAM) and even these
> are unable to build llvm14 under FreeBSD 13.1-RELEASE
> with poudri=C3=A8re enabling MAKE_JOBS for llvm build.
It depends on the build options. Do you need fortran,
a.k.a. flang ?
Building flang requires building MLIR because building
flang uses MLIR. Without needing flang, both can normally
be turned off.
flang does not build for armv7 and other 32-bit environments,
so recently the /usr/ports/devel/llvm14/Makefile was modified
to not have flang as a default for armv7 (or armv6). My
understanding, the intent is to later also turn off MLIR
for these.
I have built devel/llvm14 with FLANG and MLIR disabled on
a 8 GiByte RPi4B in an armv7 poudriere jail. This was
as part of an ongoing "bulk -a -c" on the RPi4B. (I will
not be able to let it run to completion but am testing
how things go until I stop it.) The below notes are
somewhat biased by my also having used BE_NATIVE for the
devel/llvm14 build:
---Begin OPTIONS List---
=3D=3D=3D> The following configuration options are available for =
llvm14-14.0.2:
BE_AMDGPU=3Don: AMD GPU backend (required by mesa)
BE_WASM=3Don: WebAssembly backend (required by firefox via wasi)
CLANG=3Don: Build clang
DOCS=3Don: Build and/or install documentation
EXTRAS=3Don: Extra clang tools
LIT=3Don: Install lit and FileCheck test tools
LLD=3Don: Install lld, the LLVM linker
LLDB=3Don: Install lldb, the LLVM debugger
MLIR=3Doff: Multi-Level Intermediate Representation
PYCLANG=3Don: Install python bindings to libclang
=3D=3D=3D=3D> Options available for the single BACKENDS: you have to =
select exactly one of them
BE_FREEBSD=3Doff: Backends for FreeBSD architectures
BE_NATIVE=3Don: Backend(s) for this architecture (ARM)
BE_STANDARD=3Doff: All non-experimental backends
=3D=3D=3D> Use 'make config' to modify these settings
---End OPTIONS List---
The RPi4B has 4 cores and I had poudriere using 4 builders
and ALLOW_MAKE_JOBS=3D with no explicit constraint on the make
jobs per builder, so implicitly 4. Thus it can have periods
with load averages around 16 when when things do as I said
to do. (Some ports do not limit themselves to the HWthread
count.) Result:
[54:41:45] [01] [13:40:48] Finished devel/llvm14 | llvm14-14.0.2: =
Success
Note that that is with 2 GiBytes of RAM per core, the
same ratio that your report indicates.
I will also report that I used a UFS context, not ZFS,
and that my patched top indicated little swap usage
during any of my "bulk -a -c" atttempt so far. (The
top tracks and reports various MaxObs???? figures,
i.e., "Maximum Observed" figures.)
My environments do have /boot/loader.conf ,
/etc/sysctl.conf , and /usr/local/etc/poudriere.conf
settings appropriate to avoiding OOM kills for long
running [but bounded duration] build activity for the
hardware that I have access to --and that is also
part of why I prefer UFS for 2 GiBytes/HWthread for
doing builds.
(I do have access to 3 systems with 4 GiBytes
per HWthread and on 2 of them I normally use
ZFS, including for build activity.)
> The build proceeds on 1 CPU only now
Such was not true of my build reported above.
Using one core at a time, the RPi4B would have
taken much longer to do the build.
> - and casual observation
> with top confirms that compilation of certain C++ files
> requires 7..8 GB of RAM.
My understanding is that such has a known status for
flang/MLIR being in use.
> If 16 of them are built concurrently,
> no wonder that there is not enough memory.
>=20
> Files tha crashed my compiliation:
>=20
> =
/wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Eva=
luate/tools.cpp
> =
/wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/clang/lib/Sem=
a/SemaOpenMP.cpp
> =
/wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Sem=
antics/check-omp-structure.cpp
> =
/wrkdirs/usr/ports/devel/llvm14/work/llvm-project-14.0.4.src/flang/lib/Eva=
luate/fold-integer.cpp
You do not report the related console messages related (or
dmsg -a output or /var/log/messages content). I can think
of multiple distinct possibilities for the reports that
would have different implications. (But such need not be
of any importance for you.)
I see 3 /flang/ paths in the list of 4 paths, not surprising.
(Your /usr/ports/ context is more recent than mine.)
> Sure, poudri=C3=A8re could get Kubernetes-like capabilities to control
> resources during the build one day, but aren't amounts of memory
> needed to build the compiler excessive a bit?
"the compiler" ignores a fair mount of what is built.
Part of what I'm suggesting above is to configure
the devel/llvm14 build to build fewer compilers/toolchains
(no flang, no MLIR). You might not want, say, EXTRAS, or
some other things I have "on". BE_NATIVE may not fit
your usage context well but BE_FREEBSD could be considered.
And so on.
Hopefully my notes above are of some use unless
you are required to use the default OPTIONS.
Side note:
The RPi4B bulk actually built llvm13 and rust with
a vast part of the time frames overlapping and other
things going through the other 2 builders.
It later worked on llvm12, ghc, and suitesparse-graphblas
with the vast part of the time being overlapped. Again:
all 4 builders doing something. (suitesparse-graphblas
does not last as many hours as the other 2.)
No kernel OOM process kills. Under 300 MiByte swap used
at any point.
(ghc's haddock runs out of 32-bit process memory in
a normal core-dump way, towards the end the ghc build,
so ghc failed. But the build had been going for most
of 24 hours before that point, as had llvm12.)
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?324C9134-542F-4954-A0E6-E75A08491A8D>