Date: Tue, 16 Aug 2022 11:11:41 +0100 From: Nuno Teixeira <eduardo@freebsd.org> To: Mark Millard <marklmi@yahoo.com> Cc: FreeBSD Mailing List <freebsd-ports@freebsd.org> Subject: Re: Resolved: devel/llvm13 build: "ninja: build stopped: subcommand failed" Message-ID: <CAFDf7UJztLqTjdD_=kt5cZawUvr8y5cjNkKshr27YprQV0Jyeg@mail.gmail.com> In-Reply-To: <4106D776-402C-45E2-A903-4734DC3FF749@yahoo.com> References: <1D4C14BD-8955-4B86-9C99-3E58D7603122.ref@yahoo.com> <1D4C14BD-8955-4B86-9C99-3E58D7603122@yahoo.com> <CAFDf7UK-pAFXCrZZA9veASaa-wf9HKMdX52fxmcmDgRFiNOF7A@mail.gmail.com> <7CDC63F3-8B68-420E-8012-B1692667E293@yahoo.com> <B8C17283-0C5E-4D84-B10F-0712B26BDCB9@yahoo.com> <4106D776-402C-45E2-A903-4734DC3FF749@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--0000000000003e99b805e658fa4c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mark, *** Sorry about TMPFS(=3Dno) when I want to mention USE_TMPFS(=3Dno). Because I was in a hurry and I needed llvm13 package in one of my jails, I did what you told me and turned off FLANG, MLIR and DOCS (to avoid textproc/py-sphinx-markdown-tables problem). I've used 6c+6t (hw.ncpu: 12) and compiled with success that consumed ~1GB swap in ~40min. Because I only use llvm* as dependencies in poudriere jails for ports testing I could use a general llvm* options to reduce time and mem+swap or if I want to stick with default options then I need to use /usr/local/etc/poudriere.d/make.conf MAKE_JOBS_NUMBER=3Dn to be able to compile it with my hardware. Thanks, Mark Millard <marklmi@yahoo.com> escreveu no dia ter=C3=A7a, 16/08/2022 =C3= =A0(s) 01:08: > On 2022-Aug-14, at 18:40, Mark Millard <marklmi@yahoo.com> wrote: > > > On 2022-Aug-14, at 09:35, Mark Millard <marklmi@yahoo.com> wrote: > > > >> On 2022-Aug-14, at 07:50, Nuno Teixeira <eduardo@freebsd.org> wrote: > >> > >> . . . > >>> have some time now and it's caused by a build peak of memory that > affects people with less than 32/64GB mem and to solve building it must b= e > build using one builder with one core thats takes about 7 hours on my > machine or with 6c+6t on 12.3 i386 that takes about 45min (123i386 is the > only jail that I can use all cores). > >> > >> Last I tried I built all the various devel/llvm* on a 8 GiByte > >> RPi4B, 4 builders active and ALLOW_MAKE_JOBS=3Dyes in use. > >> 4 FreeBSD cpus. So the load average would have been around 16+ > >> much of the time during devel/llvm13 's builder activity. > >> USE_TMPFS=3Ddata in use. > >> > >> Similarly for a 16 GiByte machine --but it is also an aarch64 > >> context, also 4 FreebSD cpus. > >> > >> But I use in /boot/loader.conf: > >> > >> # > >> # Delay when persistent low free RAM leads to > >> # Out Of Memory killing of processes: > >> vm.pageout_oom_seq=3D120 > >> > >> This has been historically important to avoiding the likes of > >> "was killed: failed to reclaim memory" and related notices on > >> various armv7 and aarch64 small board computers used to > >> buildworld buildkernel and build ports, using all the cores. > >> > >> The only amd64 system that I've access to has 32 FreeBSD cpus > >> and 128 GiBytes of RAM. Not a good basis for a comparison test > >> with your context. I've no i386 access at all. > >> > >>> llvm 12 build without problems > >> > >> Hmm. I'll try building devel/llvm13 on aarch64 with periodic > >> sampling of the memory use to see maximum observed figures > >> for SWAP and for various categories of RAM, as well as the > >> largest observed load averages. > >> > >> ZFS context use. I could try UFS as well. > >> > >> Swap: 30720Mi Total on the 8GiByte RPi4B. > >> So about 38 GiBytes RAM+SWAP available. > >> We should see how much SWAP is used. > >> > >> Before starting poudriere, shortly after a reboot: > >> > >> 19296Ki MaxObs(Act+Lndry+SwapUsed) > >> (No SWAP in use at the time.) > >> > >> # poudriere bulk -jmain-CA72-bulk_a -w devel/llvm13 > >> > >> for the from scratch build: reports: > >> > >> [00:00:34] Building 91 packages using up to 4 builders > >> > >> The ports tree is about a month back: > >> > >> # ~/fbsd-based-on-what-commit.sh -C /usr/ports/ > >> branch: main > >> merge-base: 872199326a916efbb4bf13c97bc1af910ba1482e > >> merge-base: CommitDate: 2022-07-14 01:26:04 +0000 > >> 872199326a91 (HEAD -> main, freebsd/main, freebsd/HEAD) > devel/ruby-build: Update to 20220713 > >> n589512 (--first-parent --count for merge-base) > >> > >> But, if I gather right, the problem you see goes back > >> before that. > >> > >> I can not tell how 4 FreeBSD cpus compares to the > >> count that the Lenovo Legion 5 gets. > >> > >> I'll report on its maximum observed figures once the > >> build stops. It will be a while before the RPi4B > >> gets that far. > >> > >> The ports built prior to devel/llvm13's builder starting > >> will lead to load averages over 4 from up to 4 > >> builders, each potentially using up to around 4 > >> processes. I'll see about starting a separate tracking > >> once devel/llvm13 's builder has started if I happen > >> to observe it at the right time frame for doing such. > >> > >> . . . > > > > I actually have tried a few builds on different > > machines. The 8GiByte RPi4B takes a long time and > > is currently omitted from this report. > > > > > > 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): > > but using MAKE_JOBS_NUMBER=3D4 (with both FLANG and MLIR) > > > > On amd64 I started a build with FLANG and MLIR enabled, > > using MAKE_JOBS_NUMBER=3D4 in devel/llvm13/Makefile to > > limit the build to 4 FreeBSD cpus. It is a ZFS context. > > Given the 128 GiBytes of RAM, there will not be much > > for effects of memory-pressure. But will record the > > MaxObs(Act+Lndry+SwapUsed) and the like. > > > > ---Begin OPTIONS List--- > > =3D=3D=3D> The following configuration options are available for > llvm13-13.0.1_3: > > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > > CLANG=3Don: Build clang > > COMPILER_RT=3Don: Sanitizer libraries > > DOCS=3Don: Build and/or install documentation > > EXTRAS=3Don: Extra clang tools > > FLANG=3Don: Flang FORTRAN compiler > > GOLD=3Don: Build the LLVM Gold plugin for LTO > > LIT=3Don: Install lit and FileCheck test tools > > LLD=3Don: Install lld, the LLVM linker > > LLDB=3Don: Install lldb, the LLVM debugger > > MLIR=3Don: Multi-Level Intermediate Representation > > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > > PYCLANG=3Don: Install python bindings to libclang > > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to se= lect > exactly one of them > > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > > BE_NATIVE=3Doff: Backend(s) for this architecture (X86) > > BE_STANDARD=3Don: All non-experimental backends > > =3D=3D=3D> Use 'make config' to modify these settings > > ---End OPTIONS List--- > > > > [02:23:55] [01] [02:04:29] Finished devel/llvm13 | llvm13-13.0.1_3: > Success > > > > For just the devel/llvm13 builder activity, no parallel > > builds and excluding the prerequisites being built: > > > > load averages: . . . MaxObs: 6.76, 4.75, 4.38 > > > > 6812Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed. > > > > Note: MAKE_JOBS_NUMBER does not constrain any lld > > procoess from using all available FreeBSD cpus > > (via threading) --and multiple lld's can be > > active at the same time. > > > > So this looks to fit in a 16 GiByte RAM context just fine, > > no SWAP needed. > > > > I'll try MAKE_JOBS_NUMBER=3D12 instead and rerun on the same > > machine. > > > > > > > > 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): > > but using MAKE_JOBS_NUMBER=3D12 (with both FLANG and MLIR) > > > > ---Begin OPTIONS List--- > > =3D=3D=3D> The following configuration options are available for > llvm13-13.0.1_3: > > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > > CLANG=3Don: Build clang > > COMPILER_RT=3Don: Sanitizer libraries > > DOCS=3Don: Build and/or install documentation > > EXTRAS=3Don: Extra clang tools > > FLANG=3Don: Flang FORTRAN compiler > > GOLD=3Don: Build the LLVM Gold plugin for LTO > > LIT=3Don: Install lit and FileCheck test tools > > LLD=3Don: Install lld, the LLVM linker > > LLDB=3Don: Install lldb, the LLVM debugger > > MLIR=3Don: Multi-Level Intermediate Representation > > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > > PYCLANG=3Don: Install python bindings to libclang > > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to se= lect > exactly one of them > > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > > BE_NATIVE=3Doff: Backend(s) for this architecture (X86) > > BE_STANDARD=3Don: All non-experimental backends > > =3D=3D=3D> Use 'make config' to modify these settings > > ---End OPTIONS List--- > > > > [00:55:37] [01] [00:55:30] Finished devel/llvm13 | llvm13-13.0.1_3: > Success > > > > load averages: . . . MaxObs: 12.45, 12.20, 11.52 > > > > 13074Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed. > > > > Note: MAKE_JOBS_NUMBER does not constrain any lld > > procoess from using all available FreeBSD cpus > > (via threading) --and multiple lld's can be > > active at the same time. > > > > (16+4)*1024 Mi - 13074 Mi =3D=3D 7406 Mi for other RAM+SWAP use. > > (Crude estimates relative to your context.) That would seem > > to be plenty. > > > > > > Conclusion: > > > > It is far from clear what all was contributing to your > > (16+4)*1024 MiBytes proving to be insufficient. > > Unintentional tmpfs use, such as a typo in USE_TMPFS > > in /usr/local/etc/poudriere.conf ? I really have no > > clue: the example is arbitrary. > > > > > > > > Other notes: > > > > # uname -apKU > > FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #50 > main-n256584-5bc926af9fd1-dirty: Wed Jul 6 17:44:43 PDT 2022 > root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64= .amd64/sys/GENERIC-NODBG > amd64 amd64 1400063 1400063 > > > > Note that the above is without WITNESS and without INVARIANTS and the > like. > > > > The only thing commited to main's contrib/llvm-project after that > > was 9ef1127008 : > > > > QUOTE > > Apply tentative llvm fix for avoiding fma on PowerPC SPE > > Merge llvm review D77558, by Justin Hibbits: > > > > PowerPC: Don't hoist float multiply + add to fused operation on SPE > > > > SPE doesn't have a fmadd instruction, so don't bother hoisting a > > multiply and add sequence to this, as it'd become just a library call. > > Hoisting happens too late for the CTR usability test to veto using the > CTR > > in a loop, and results in an assert "Invalid PPC CTR loop!". > > END QUOTE > > Here I add the 8 GiByte RPi4B results, in this case > both with FLANG/MLIR results and without results. The > context has and uses 4 FreeBSD cpus. No other builder > is active when devel/llvm13's builder is. > > With both FLANG and MLIR: > > ---Begin OPTIONS List--- > =3D=3D=3D> The following configuration options are available for llvm13-1= 3.0.1_3: > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > CLANG=3Don: Build clang > DOCS=3Don: Build and/or install documentation > EXTRAS=3Don: Extra clang tools > FLANG=3Don: Flang FORTRAN compiler > LIT=3Don: Install lit and FileCheck test tools > LLD=3Don: Install lld, the LLVM linker > LLDB=3Don: Install lldb, the LLVM debugger > MLIR=3Don: Multi-Level Intermediate Representation > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > PYCLANG=3Don: Install python bindings to libclang > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to sele= ct > exactly one of them > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > BE_NATIVE=3Doff: Backend(s) for this architecture (AArch64) > BE_STANDARD=3Don: All non-experimental backends > =3D=3D=3D> Use 'make config' to modify these settings > ---End OPTIONS List--- > > [13:49:55] [01] [13:49:17] Finished devel/llvm13 | llvm13-13.0.1_3: Succe= ss > > load averages: . . . MaxObs: 6.43, 5.91, 5.77 > (Note: spanned overnight so the nightly cron job was > spanned.) > > Note: Given that SWAP was used, I report more > Max(imum)Obs(erved) figures for this case than > I've been reporting for other tests: > > 5696Mi MaxObsActive > 1775Mi MaxObsSwapUsed > 7374Mi MaxObs(Act+Lndry+SwapUsed) > 9333Mi MaxObs(Act+Wir+Lndry+SwapUsed) > > Reminder: MaximumOfASum <=3D TheSumOfTheMaximums > Note: The various Maximums need not be from the same time. > > > By contrast . . . > > No FLANG, no MLIR: > > ---Begin OPTIONS List--- > =3D=3D=3D> The following configuration options are available for llvm13-1= 3.0.1_3: > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > CLANG=3Don: Build clang > DOCS=3Don: Build and/or install documentation > EXTRAS=3Don: Extra clang tools > FLANG=3Doff: Flang FORTRAN compiler > LIT=3Don: Install lit and FileCheck test tools > LLD=3Don: Install lld, the LLVM linker > LLDB=3Don: Install lldb, the LLVM debugger > MLIR=3Doff: Multi-Level Intermediate Representation > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > PYCLANG=3Don: Install python bindings to libclang > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to sele= ct > exactly one of them > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > BE_NATIVE=3Don: Backend(s) for this architecture (AArch64) > BE_STANDARD=3Doff: All non-experimental backends > =3D=3D=3D> Use 'make config' to modify these settings > ---End OPTIONS List--- > > [11:07:48] [01] [08:58:53] Finished devel/llvm13 | llvm13-13.0.1_3: Succe= ss > > load averages: . . . MaxObs: 5.31, 4.94, 4.79 > > 1479Mi MaxObs(Act+Lndry+SwapUsed) > > So, vastly less RAM+SWAP space use. Somewhat under > 5 hours less build time (about 9hr vs. somewhat under 14hr). > > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --=20 Nuno Teixeira FreeBSD Committer (ports) --0000000000003e99b805e658fa4c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>Hi Mark,</div><div><br></div><div>**= * Sorry about TMPFS(=3Dno) when I want to mention USE_TMPFS(=3Dno).</div><d= iv><br></div><div>Because I was in a hurry and I needed llvm13 package in o= ne of my jails, I did what you told me and turned off FLANG, MLIR and DOCS = (to avoid textproc/py-sphinx-markdown-tables problem).</div><div><br></div>= <div>I've used 6c+6t (hw.ncpu: 12) and compiled with success that consu= med ~1GB swap in ~40min.</div><div><br></div><div>Because I only use llvm* = as dependencies in poudriere jails for ports testing I could use a general = llvm* options to reduce time and mem+swap or if I want to stick with defaul= t options then I need to use /usr/local/etc/poudriere.d/make.conf MAKE_JOBS= _NUMBER=3Dn to be able to compile it with my hardware.</div><div><br></div>= <div>Thanks,<br></div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr"= class=3D"gmail_attr">Mark Millard <<a href=3D"mailto:marklmi@yahoo.com"= target=3D"_blank">marklmi@yahoo.com</a>> escreveu no dia ter=C3=A7a, 16= /08/2022 =C3=A0(s) 01:08:<br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex">On 2022-Aug-14, at 18:40, Mark Millard <<a href=3D"mailto:mar= klmi@yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>> wrote:<br> <br> > On 2022-Aug-14, at 09:35, Mark Millard <<a href=3D"mailto:marklmi@y= ahoo.com" target=3D"_blank">marklmi@yahoo.com</a>> wrote:<br> > <br> >> On 2022-Aug-14, at 07:50, Nuno Teixeira <<a href=3D"mailto:edua= rdo@freebsd.org" target=3D"_blank">eduardo@freebsd.org</a>> wrote:<br> >> <br> >> . . .<br> >>> have some time now and it's caused by a build peak of memo= ry that affects people with less than 32/64GB mem and to solve building it = must be build using one builder with one core thats takes about 7 hours on = my machine or with 6c+6t on 12.3 i386 that takes about 45min (123i386 is th= e only jail that I can use all cores).<br> >> <br> >> Last I tried I built all the various devel/llvm* on a 8 GiByte<br> >> RPi4B, 4 builders active and ALLOW_MAKE_JOBS=3Dyes in use.<br> >> 4 FreeBSD cpus. So the load average would have been around 16+<br> >> much of the time during devel/llvm13 's builder activity.<br> >> USE_TMPFS=3Ddata in use.<br> >> <br> >> Similarly for a 16 GiByte machine --but it is also an aarch64<br> >> context, also 4 FreebSD cpus.<br> >> <br> >> But I use in /boot/loader.conf:<br> >> <br> >> #<br> >> # Delay when persistent low free RAM leads to<br> >> # Out Of Memory killing of processes:<br> >> vm.pageout_oom_seq=3D120<br> >> <br> >> This has been historically important to avoiding the likes of<br> >> "was killed: failed to reclaim memory" and related notic= es on<br> >> various armv7 and aarch64 small board computers used to<br> >> buildworld buildkernel and build ports, using all the cores.<br> >> <br> >> The only amd64 system that I've access to has 32 FreeBSD cpus<= br> >> and 128 GiBytes of RAM. Not a good basis for a comparison test<br> >> with your context. I've no i386 access at all.<br> >> <br> >>> llvm 12 build without problems<br> >> <br> >> Hmm. I'll try building devel/llvm13 on aarch64 with periodic<b= r> >> sampling of the memory use to see maximum observed figures<br> >> for SWAP and for various categories of RAM, as well as the<br> >> largest observed load averages.<br> >> <br> >> ZFS context use. I could try UFS as well.<br> >> <br> >> Swap: 30720Mi Total on the 8GiByte RPi4B.<br> >> So about 38 GiBytes RAM+SWAP available.<br> >> We should see how much SWAP is used.<br> >> <br> >> Before starting poudriere, shortly after a reboot:<br> >> <br> >> 19296Ki MaxObs(Act+Lndry+SwapUsed)<br> >> (No SWAP in use at the time.)<br> >> <br> >> # poudriere bulk -jmain-CA72-bulk_a -w devel/llvm13<br> >> <br> >> for the from scratch build: reports:<br> >> <br> >> [00:00:34] Building 91 packages using up to 4 builders<br> >> <br> >> The ports tree is about a month back:<br> >> <br> >> # ~/fbsd-based-on-what-commit.sh -C /usr/ports/<br> >> branch: main<br> >> merge-base: 872199326a916efbb4bf13c97bc1af910ba1482e<br> >> merge-base: CommitDate: 2022-07-14 01:26:04 +0000<br> >> 872199326a91 (HEAD -> main, freebsd/main, freebsd/HEAD) devel/r= uby-build: Update to 20220713<br> >> n589512 (--first-parent --count for merge-base)<br> >> <br> >> But, if I gather right, the problem you see goes back<br> >> before that.<br> >> <br> >> I can not tell how 4 FreeBSD cpus compares to the<br> >> count that the Lenovo Legion 5 gets.<br> >> <br> >> I'll report on its maximum observed figures once the<br> >> build stops. It will be a while before the RPi4B<br> >> gets that far.<br> >> <br> >> The ports built prior to devel/llvm13's builder starting<br> >> will lead to load averages over 4 from up to 4<br> >> builders, each potentially using up to around 4<br> >> processes. I'll see about starting a separate tracking<br> >> once devel/llvm13 's builder has started if I happen<br> >> to observe it at the right time frame for doing such.<br> >> <br> >> . . .<br> > <br> > I actually have tried a few builds on different<br> > machines. The 8GiByte RPi4B takes a long time and<br> > is currently omitted from this report.<br> > <br> > <br> > 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus):<br= > > but using MAKE_JOBS_NUMBER=3D4 (with both FLANG and MLIR)<br> > <br> > On amd64 I started a build with FLANG and MLIR enabled,<br> > using MAKE_JOBS_NUMBER=3D4 in devel/llvm13/Makefile to<br> > limit the build to 4 FreeBSD cpus. It is a ZFS context.<br> > Given the 128 GiBytes of RAM, there will not be much<br> > for effects of memory-pressure. But will record the <br> > MaxObs(Act+Lndry+SwapUsed) and the like.<br> > <br> > ---Begin OPTIONS List---<br> > =3D=3D=3D> The following configuration options are available for ll= vm13-13.0.1_3:<br> >=C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)<= br> >=C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by fire= fox via wasi)<br> >=C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang<br> >=C2=A0 =C2=A0 =C2=A0COMPILER_RT=3Don: Sanitizer libraries<br> >=C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation<br> >=C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools<br> >=C2=A0 =C2=A0 =C2=A0FLANG=3Don: Flang FORTRAN compiler<br> >=C2=A0 =C2=A0 =C2=A0GOLD=3Don: Build the LLVM Gold plugin for LTO<br> >=C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools<br> >=C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker<br> >=C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger<br> >=C2=A0 =C2=A0 =C2=A0MLIR=3Don: Multi-Level Intermediate Representation<= br> >=C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtim= e library<br> >=C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang<b= r> > =3D=3D=3D=3D> Options available for the single BACKENDS: you have t= o select exactly one of them<br> >=C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architecture= s<br> >=C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Doff: Backend(s) for this architecture (= X86)<br> >=C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Don: All non-experimental backends<br> > =3D=3D=3D> Use 'make config' to modify these settings<br> > ---End OPTIONS List---<br> > <br> > [02:23:55] [01] [02:04:29] Finished devel/llvm13 | llvm13-13.0.1_3: Su= ccess<br> > <br> > For just the devel/llvm13 builder activity, no parallel<br> > builds and excluding the prerequisites being built:<br> > <br> > load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A06.76,=C2=A0 =C2= =A04.75,=C2=A0 =C2=A04.38<br> > <br> > 6812Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed.<br> > <br> > Note: MAKE_JOBS_NUMBER does not constrain any lld <br> >=C2=A0 =C2=A0 =C2=A0 procoess from using all available FreeBSD cpus<br> >=C2=A0 =C2=A0 =C2=A0 (via threading) --and multiple lld's can be<br= > >=C2=A0 =C2=A0 =C2=A0 active at the same time.<br> > <br> > So this looks to fit in a 16 GiByte RAM context just fine,<br> > no SWAP needed.<br> > <br> > I'll try MAKE_JOBS_NUMBER=3D12 instead and rerun on the same<br> > machine.<br> > <br> > <br> > <br> > 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus):<br= > > but using MAKE_JOBS_NUMBER=3D12 (with both FLANG and MLIR)<br> > <br> > ---Begin OPTIONS List---<br> > =3D=3D=3D> The following configuration options are available for ll= vm13-13.0.1_3:<br> >=C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)<= br> >=C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by fire= fox via wasi)<br> >=C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang<br> >=C2=A0 =C2=A0 =C2=A0COMPILER_RT=3Don: Sanitizer libraries<br> >=C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation<br> >=C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools<br> >=C2=A0 =C2=A0 =C2=A0FLANG=3Don: Flang FORTRAN compiler<br> >=C2=A0 =C2=A0 =C2=A0GOLD=3Don: Build the LLVM Gold plugin for LTO<br> >=C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools<br> >=C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker<br> >=C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger<br> >=C2=A0 =C2=A0 =C2=A0MLIR=3Don: Multi-Level Intermediate Representation<= br> >=C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtim= e library<br> >=C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang<b= r> > =3D=3D=3D=3D> Options available for the single BACKENDS: you have t= o select exactly one of them<br> >=C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architecture= s<br> >=C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Doff: Backend(s) for this architecture (= X86)<br> >=C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Don: All non-experimental backends<br> > =3D=3D=3D> Use 'make config' to modify these settings<br> > ---End OPTIONS List---<br> > <br> > [00:55:37] [01] [00:55:30] Finished devel/llvm13 | llvm13-13.0.1_3: Su= ccess<br> > <br> > load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A012.45,=C2=A0 12.2= 0,=C2=A0 11.52<br> > <br> > 13074Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed.<br> > <br> > Note: MAKE_JOBS_NUMBER does not constrain any lld <br> >=C2=A0 =C2=A0 =C2=A0 procoess from using all available FreeBSD cpus<br> >=C2=A0 =C2=A0 =C2=A0 (via threading) --and multiple lld's can be<br= > >=C2=A0 =C2=A0 =C2=A0 active at the same time.<br> > <br> > (16+4)*1024 Mi - 13074 Mi =3D=3D 7406 Mi for other RAM+SWAP use.<br> > (Crude estimates relative to your context.) That would seem<br> > to be plenty.<br> > <br> > <br> > Conclusion:<br> > <br> > It is far from clear what all was contributing to your<br> > (16+4)*1024 MiBytes proving to be insufficient.<br> > Unintentional tmpfs use, such as a typo in USE_TMPFS<br> > in /usr/local/etc/poudriere.conf ? I really have no<br> > clue: the example is arbitrary.<br> > <br> > <br> > <br> > Other notes:<br> > <br> > # uname -apKU<br> > FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #50 main-n256584-5= bc926af9fd1-dirty: Wed Jul=C2=A0 6 17:44:43 PDT 2022=C2=A0 =C2=A0 =C2=A0roo= t@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64= /sys/GENERIC-NODBG amd64 amd64 1400063 1400063<br> > <br> > Note that the above is without WITNESS and without INVARIANTS and the = like.<br> > <br> > The only thing commited to main's contrib/llvm-project after that<= br> > was 9ef1127008 :<br> > <br> > QUOTE<br> > Apply tentative llvm fix for avoiding fma on PowerPC SPE<br> > Merge llvm review D77558, by Justin Hibbits:<br> > <br> >=C2=A0 PowerPC: Don't hoist float multiply + add to fused operation= on SPE<br> > <br> >=C2=A0 SPE doesn't have a fmadd instruction, so don't bother ho= isting a<br> >=C2=A0 multiply and add sequence to this, as it'd become just a lib= rary call.<br> >=C2=A0 Hoisting happens too late for the CTR usability test to veto usi= ng the CTR<br> >=C2=A0 in a loop, and results in an assert "Invalid PPC CTR loop!&= quot;.<br> > END QUOTE<br> <br> Here I add the 8 GiByte RPi4B results, in this case<br> both with FLANG/MLIR results and without results. The<br> context has and uses 4 FreeBSD cpus. No other builder<br> is active when devel/llvm13's builder is.<br> <br> With both FLANG and MLIR:<br> <br> ---Begin OPTIONS List---<br> =3D=3D=3D> The following configuration options are available for llvm13-= 13.0.1_3:<br> =C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)<br> =C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by firefox = via wasi)<br> =C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang<br> =C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation<br> =C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools<br> =C2=A0 =C2=A0 =C2=A0FLANG=3Don: Flang FORTRAN compiler<br> =C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools<br> =C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker<br> =C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger<br> =C2=A0 =C2=A0 =C2=A0MLIR=3Don: Multi-Level Intermediate Representation<br> =C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtime li= brary<br> =C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang<br> =3D=3D=3D=3D> Options available for the single BACKENDS: you have to sel= ect exactly one of them<br> =C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architectures<br= > =C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Doff: Backend(s) for this architecture (AArc= h64)<br> =C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Don: All non-experimental backends<br> =3D=3D=3D> Use 'make config' to modify these settings<br> ---End OPTIONS List---<br> <br> [13:49:55] [01] [13:49:17] Finished devel/llvm13 | llvm13-13.0.1_3: Success= <br> <br> load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A06.43,=C2=A0 =C2=A05.91= ,=C2=A0 =C2=A05.77<br> (Note: spanned overnight so the nightly cron job was<br> spanned.)<br> <br> Note: Given that SWAP was used, I report more<br> Max(imum)Obs(erved) figures for this case than<br> I've been reporting for other tests:<br> <br> 5696Mi MaxObsActive<br> 1775Mi MaxObsSwapUsed<br> 7374Mi MaxObs(Act+Lndry+SwapUsed)<br> 9333Mi MaxObs(Act+Wir+Lndry+SwapUsed)<br> <br> Reminder: MaximumOfASum <=3D TheSumOfTheMaximums<br> Note: The various Maximums need not be from the same time.<br> <br> <br> By contrast . . .<br> <br> No FLANG, no MLIR:<br> <br> ---Begin OPTIONS List---<br> =3D=3D=3D> The following configuration options are available for llvm13-= 13.0.1_3:<br> =C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)<br> =C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by firefox = via wasi)<br> =C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang<br> =C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation<br> =C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools<br> =C2=A0 =C2=A0 =C2=A0FLANG=3Doff: Flang FORTRAN compiler<br> =C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools<br> =C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker<br> =C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger<br> =C2=A0 =C2=A0 =C2=A0MLIR=3Doff: Multi-Level Intermediate Representation<br> =C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtime li= brary<br> =C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang<br> =3D=3D=3D=3D> Options available for the single BACKENDS: you have to sel= ect exactly one of them<br> =C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architectures<br= > =C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Don: Backend(s) for this architecture (AArch= 64)<br> =C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Doff: All non-experimental backends<br> =3D=3D=3D> Use 'make config' to modify these settings<br> ---End OPTIONS List---<br> <br> [11:07:48] [01] [08:58:53] Finished devel/llvm13 | llvm13-13.0.1_3: Success= <br> <br> load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A05.31,=C2=A0 =C2=A04.94= ,=C2=A0 =C2=A04.79<br> <br> 1479Mi MaxObs(Act+Lndry+SwapUsed)<br> <br> So, vastly less RAM+SWAP space use. Somewhat under<br> 5 hours less build time (about 9hr vs. somewhat under 14hr).<br> <br> <br> =3D=3D=3D<br> Mark Millard<br> marklmi at <a href=3D"http://yahoo.com" rel=3D"noreferrer" target=3D"_blank= ">yahoo.com</a><br> <br> </blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr"><div dir= =3D"ltr"><span style=3D"color:rgb(102,102,102)">Nuno Teixeira<br>FreeBSD Co= mmitter (ports)</span></div></div> </div> --0000000000003e99b805e658fa4c--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFDf7UJztLqTjdD_=kt5cZawUvr8y5cjNkKshr27YprQV0Jyeg>