From nobody Tue Aug 16 10:11:41 2022 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M6RlG22MSz4Yfcr for ; Tue, 16 Aug 2022 10:11:54 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M6RlG1C7jz3ykx for ; Tue, 16 Aug 2022 10:11:54 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1660644714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dDcOqNrukwy7BVlDMPJM5WyBiSx0wMTswGPd0Q6nBvI=; b=XTT5289NPTY3aIjlKOUbYfdDORn8R5jy2Ra1kOHVmpKPa12v3O1nzgE99ICoGQeeVc/WBK k8vB9kkAOZ2lA3qTdFcD5XJYz+llQC5e/v7/74FUYhE2BiyqPgPh3+SbgvQLE7gMGLoxi8 CdT4exqOD7nVYfzcS2LE06NcDhUkZHnlY7auEheUgubmsFABJAVjJsgo1BHa1zqoy0BQ18 NZsV+OBjoXNcUoRjD3Q2FzA6hGqcmIr3CX8lzaKtzZftXHtnVEjtGqndGWWzA+nu841aMZ R2GWL6xJ6b5UqgIEeAqv2Hwctgd//5eoNTq7KgRCsStTxIOTgmEB/MhVgAOVDQ== Received: from mail-vk1-f180.google.com (mail-vk1-f180.google.com [209.85.221.180]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4M6RlG0FRbzvqD for ; Tue, 16 Aug 2022 10:11:53 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-vk1-f180.google.com with SMTP id c22so4910217vko.7 for ; Tue, 16 Aug 2022 03:11:53 -0700 (PDT) X-Gm-Message-State: ACgBeo3HqM6JMOLFPi4XtripMTvKoaFrI+0X73E7GxGxNvyZiIM55I2C v6LrIo78cHK0lZ9b//gB49yx0Wbr2mZantFlNEY= X-Google-Smtp-Source: AA6agR55mvFEoVg4Xl52y3upopwUh8X4Vu0tMeW5EWX/dU1jHuvDD0AuGbJQm0VQyRJWoHWdflYp0CDtY1ONR2eNVFY= X-Received: by 2002:a1f:41c8:0:b0:377:1352:8f9d with SMTP id o191-20020a1f41c8000000b0037713528f9dmr8265519vka.25.1660644713468; Tue, 16 Aug 2022 03:11:53 -0700 (PDT) List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 References: <1D4C14BD-8955-4B86-9C99-3E58D7603122.ref@yahoo.com> <1D4C14BD-8955-4B86-9C99-3E58D7603122@yahoo.com> <7CDC63F3-8B68-420E-8012-B1692667E293@yahoo.com> <4106D776-402C-45E2-A903-4734DC3FF749@yahoo.com> In-Reply-To: <4106D776-402C-45E2-A903-4734DC3FF749@yahoo.com> From: Nuno Teixeira Date: Tue, 16 Aug 2022 11:11:41 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Resolved: devel/llvm13 build: "ninja: build stopped: subcommand failed" To: Mark Millard Cc: FreeBSD Mailing List Content-Type: multipart/alternative; boundary="0000000000003e99b805e658fa4c" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1660644714; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=dDcOqNrukwy7BVlDMPJM5WyBiSx0wMTswGPd0Q6nBvI=; b=JdqGwMJsrPGTMa4GH5sQ1EBp2Ot5apzEbw+3vsPs8vb81OEwoEOaC3hssD11ubTGNiJi2u gNHYlkbHlby0XZdyMuYfU2My2hPo0oLLmQp96BCYqIGYmhL3BTu/z0idJ/CdiA0zBDDBEY D/WDzmDXT8ZDly+CzTe432E/eUnKkUzyc7BRnEGxTEteW4y3ZOtLAxAwvwJgvI5nnK3ntE kl9t037QcA6bGH7OJZKCV3hcQWx7B7DBp+yJ5Zg0bpzCnWG+neMgD+XzntGqojEEJRFjsq yNYc7xP3fW4ipuSHF7mcguNsc2OQYsT7S7tpLBK0+hVEavD0sRwdyKM5qCxFnA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1660644714; a=rsa-sha256; cv=none; b=RWqGCSXvhsvguc2zI3mtpTr+ejFlIsaLFtvyE46NmUjtm4hqJfGscow6yAnEauaww5Cz9F KZnE9nS5QCQHuvvoVdz4cHQYjOiVLPA1eyYZNci7BsbB+N+NJe8sYpcljIZyrCMz6bZj9R 8Xv8zs8S74KvAFE9wI0pX/E5lzjEC7/lIR7sekVO1kQoQ1zg75K2z5ZHKIc673Ph5STAvq 4MMeyYOE8fTKsgxZZYVMsz8FUTLQDO8Gc35FY1p/Oi8TMqyrhAuxjuDBTQiBdUYViIfcTt PNw1/8OeZL/tIPjobNQNIdWHzNYAfndAlqnpgTWtQwqLuprM8PiIkTcjqsxrOA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N --0000000000003e99b805e658fa4c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Mark, *** Sorry about TMPFS(=3Dno) when I want to mention USE_TMPFS(=3Dno). Because I was in a hurry and I needed llvm13 package in one of my jails, I did what you told me and turned off FLANG, MLIR and DOCS (to avoid textproc/py-sphinx-markdown-tables problem). I've used 6c+6t (hw.ncpu: 12) and compiled with success that consumed ~1GB swap in ~40min. Because I only use llvm* as dependencies in poudriere jails for ports testing I could use a general llvm* options to reduce time and mem+swap or if I want to stick with default options then I need to use /usr/local/etc/poudriere.d/make.conf MAKE_JOBS_NUMBER=3Dn to be able to compile it with my hardware. Thanks, Mark Millard escreveu no dia ter=C3=A7a, 16/08/2022 =C3= =A0(s) 01:08: > On 2022-Aug-14, at 18:40, Mark Millard wrote: > > > On 2022-Aug-14, at 09:35, Mark Millard wrote: > > > >> On 2022-Aug-14, at 07:50, Nuno Teixeira wrote: > >> > >> . . . > >>> have some time now and it's caused by a build peak of memory that > affects people with less than 32/64GB mem and to solve building it must b= e > build using one builder with one core thats takes about 7 hours on my > machine or with 6c+6t on 12.3 i386 that takes about 45min (123i386 is the > only jail that I can use all cores). > >> > >> Last I tried I built all the various devel/llvm* on a 8 GiByte > >> RPi4B, 4 builders active and ALLOW_MAKE_JOBS=3Dyes in use. > >> 4 FreeBSD cpus. So the load average would have been around 16+ > >> much of the time during devel/llvm13 's builder activity. > >> USE_TMPFS=3Ddata in use. > >> > >> Similarly for a 16 GiByte machine --but it is also an aarch64 > >> context, also 4 FreebSD cpus. > >> > >> But I use in /boot/loader.conf: > >> > >> # > >> # Delay when persistent low free RAM leads to > >> # Out Of Memory killing of processes: > >> vm.pageout_oom_seq=3D120 > >> > >> This has been historically important to avoiding the likes of > >> "was killed: failed to reclaim memory" and related notices on > >> various armv7 and aarch64 small board computers used to > >> buildworld buildkernel and build ports, using all the cores. > >> > >> The only amd64 system that I've access to has 32 FreeBSD cpus > >> and 128 GiBytes of RAM. Not a good basis for a comparison test > >> with your context. I've no i386 access at all. > >> > >>> llvm 12 build without problems > >> > >> Hmm. I'll try building devel/llvm13 on aarch64 with periodic > >> sampling of the memory use to see maximum observed figures > >> for SWAP and for various categories of RAM, as well as the > >> largest observed load averages. > >> > >> ZFS context use. I could try UFS as well. > >> > >> Swap: 30720Mi Total on the 8GiByte RPi4B. > >> So about 38 GiBytes RAM+SWAP available. > >> We should see how much SWAP is used. > >> > >> Before starting poudriere, shortly after a reboot: > >> > >> 19296Ki MaxObs(Act+Lndry+SwapUsed) > >> (No SWAP in use at the time.) > >> > >> # poudriere bulk -jmain-CA72-bulk_a -w devel/llvm13 > >> > >> for the from scratch build: reports: > >> > >> [00:00:34] Building 91 packages using up to 4 builders > >> > >> The ports tree is about a month back: > >> > >> # ~/fbsd-based-on-what-commit.sh -C /usr/ports/ > >> branch: main > >> merge-base: 872199326a916efbb4bf13c97bc1af910ba1482e > >> merge-base: CommitDate: 2022-07-14 01:26:04 +0000 > >> 872199326a91 (HEAD -> main, freebsd/main, freebsd/HEAD) > devel/ruby-build: Update to 20220713 > >> n589512 (--first-parent --count for merge-base) > >> > >> But, if I gather right, the problem you see goes back > >> before that. > >> > >> I can not tell how 4 FreeBSD cpus compares to the > >> count that the Lenovo Legion 5 gets. > >> > >> I'll report on its maximum observed figures once the > >> build stops. It will be a while before the RPi4B > >> gets that far. > >> > >> The ports built prior to devel/llvm13's builder starting > >> will lead to load averages over 4 from up to 4 > >> builders, each potentially using up to around 4 > >> processes. I'll see about starting a separate tracking > >> once devel/llvm13 's builder has started if I happen > >> to observe it at the right time frame for doing such. > >> > >> . . . > > > > I actually have tried a few builds on different > > machines. The 8GiByte RPi4B takes a long time and > > is currently omitted from this report. > > > > > > 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): > > but using MAKE_JOBS_NUMBER=3D4 (with both FLANG and MLIR) > > > > On amd64 I started a build with FLANG and MLIR enabled, > > using MAKE_JOBS_NUMBER=3D4 in devel/llvm13/Makefile to > > limit the build to 4 FreeBSD cpus. It is a ZFS context. > > Given the 128 GiBytes of RAM, there will not be much > > for effects of memory-pressure. But will record the > > MaxObs(Act+Lndry+SwapUsed) and the like. > > > > ---Begin OPTIONS List--- > > =3D=3D=3D> The following configuration options are available for > llvm13-13.0.1_3: > > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > > CLANG=3Don: Build clang > > COMPILER_RT=3Don: Sanitizer libraries > > DOCS=3Don: Build and/or install documentation > > EXTRAS=3Don: Extra clang tools > > FLANG=3Don: Flang FORTRAN compiler > > GOLD=3Don: Build the LLVM Gold plugin for LTO > > LIT=3Don: Install lit and FileCheck test tools > > LLD=3Don: Install lld, the LLVM linker > > LLDB=3Don: Install lldb, the LLVM debugger > > MLIR=3Don: Multi-Level Intermediate Representation > > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > > PYCLANG=3Don: Install python bindings to libclang > > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to se= lect > exactly one of them > > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > > BE_NATIVE=3Doff: Backend(s) for this architecture (X86) > > BE_STANDARD=3Don: All non-experimental backends > > =3D=3D=3D> Use 'make config' to modify these settings > > ---End OPTIONS List--- > > > > [02:23:55] [01] [02:04:29] Finished devel/llvm13 | llvm13-13.0.1_3: > Success > > > > For just the devel/llvm13 builder activity, no parallel > > builds and excluding the prerequisites being built: > > > > load averages: . . . MaxObs: 6.76, 4.75, 4.38 > > > > 6812Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed. > > > > Note: MAKE_JOBS_NUMBER does not constrain any lld > > procoess from using all available FreeBSD cpus > > (via threading) --and multiple lld's can be > > active at the same time. > > > > So this looks to fit in a 16 GiByte RAM context just fine, > > no SWAP needed. > > > > I'll try MAKE_JOBS_NUMBER=3D12 instead and rerun on the same > > machine. > > > > > > > > 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): > > but using MAKE_JOBS_NUMBER=3D12 (with both FLANG and MLIR) > > > > ---Begin OPTIONS List--- > > =3D=3D=3D> The following configuration options are available for > llvm13-13.0.1_3: > > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > > CLANG=3Don: Build clang > > COMPILER_RT=3Don: Sanitizer libraries > > DOCS=3Don: Build and/or install documentation > > EXTRAS=3Don: Extra clang tools > > FLANG=3Don: Flang FORTRAN compiler > > GOLD=3Don: Build the LLVM Gold plugin for LTO > > LIT=3Don: Install lit and FileCheck test tools > > LLD=3Don: Install lld, the LLVM linker > > LLDB=3Don: Install lldb, the LLVM debugger > > MLIR=3Don: Multi-Level Intermediate Representation > > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > > PYCLANG=3Don: Install python bindings to libclang > > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to se= lect > exactly one of them > > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > > BE_NATIVE=3Doff: Backend(s) for this architecture (X86) > > BE_STANDARD=3Don: All non-experimental backends > > =3D=3D=3D> Use 'make config' to modify these settings > > ---End OPTIONS List--- > > > > [00:55:37] [01] [00:55:30] Finished devel/llvm13 | llvm13-13.0.1_3: > Success > > > > load averages: . . . MaxObs: 12.45, 12.20, 11.52 > > > > 13074Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed. > > > > Note: MAKE_JOBS_NUMBER does not constrain any lld > > procoess from using all available FreeBSD cpus > > (via threading) --and multiple lld's can be > > active at the same time. > > > > (16+4)*1024 Mi - 13074 Mi =3D=3D 7406 Mi for other RAM+SWAP use. > > (Crude estimates relative to your context.) That would seem > > to be plenty. > > > > > > Conclusion: > > > > It is far from clear what all was contributing to your > > (16+4)*1024 MiBytes proving to be insufficient. > > Unintentional tmpfs use, such as a typo in USE_TMPFS > > in /usr/local/etc/poudriere.conf ? I really have no > > clue: the example is arbitrary. > > > > > > > > Other notes: > > > > # uname -apKU > > FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #50 > main-n256584-5bc926af9fd1-dirty: Wed Jul 6 17:44:43 PDT 2022 > root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64= .amd64/sys/GENERIC-NODBG > amd64 amd64 1400063 1400063 > > > > Note that the above is without WITNESS and without INVARIANTS and the > like. > > > > The only thing commited to main's contrib/llvm-project after that > > was 9ef1127008 : > > > > QUOTE > > Apply tentative llvm fix for avoiding fma on PowerPC SPE > > Merge llvm review D77558, by Justin Hibbits: > > > > PowerPC: Don't hoist float multiply + add to fused operation on SPE > > > > SPE doesn't have a fmadd instruction, so don't bother hoisting a > > multiply and add sequence to this, as it'd become just a library call. > > Hoisting happens too late for the CTR usability test to veto using the > CTR > > in a loop, and results in an assert "Invalid PPC CTR loop!". > > END QUOTE > > Here I add the 8 GiByte RPi4B results, in this case > both with FLANG/MLIR results and without results. The > context has and uses 4 FreeBSD cpus. No other builder > is active when devel/llvm13's builder is. > > With both FLANG and MLIR: > > ---Begin OPTIONS List--- > =3D=3D=3D> The following configuration options are available for llvm13-1= 3.0.1_3: > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > CLANG=3Don: Build clang > DOCS=3Don: Build and/or install documentation > EXTRAS=3Don: Extra clang tools > FLANG=3Don: Flang FORTRAN compiler > LIT=3Don: Install lit and FileCheck test tools > LLD=3Don: Install lld, the LLVM linker > LLDB=3Don: Install lldb, the LLVM debugger > MLIR=3Don: Multi-Level Intermediate Representation > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > PYCLANG=3Don: Install python bindings to libclang > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to sele= ct > exactly one of them > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > BE_NATIVE=3Doff: Backend(s) for this architecture (AArch64) > BE_STANDARD=3Don: All non-experimental backends > =3D=3D=3D> Use 'make config' to modify these settings > ---End OPTIONS List--- > > [13:49:55] [01] [13:49:17] Finished devel/llvm13 | llvm13-13.0.1_3: Succe= ss > > load averages: . . . MaxObs: 6.43, 5.91, 5.77 > (Note: spanned overnight so the nightly cron job was > spanned.) > > Note: Given that SWAP was used, I report more > Max(imum)Obs(erved) figures for this case than > I've been reporting for other tests: > > 5696Mi MaxObsActive > 1775Mi MaxObsSwapUsed > 7374Mi MaxObs(Act+Lndry+SwapUsed) > 9333Mi MaxObs(Act+Wir+Lndry+SwapUsed) > > Reminder: MaximumOfASum <=3D TheSumOfTheMaximums > Note: The various Maximums need not be from the same time. > > > By contrast . . . > > No FLANG, no MLIR: > > ---Begin OPTIONS List--- > =3D=3D=3D> The following configuration options are available for llvm13-1= 3.0.1_3: > BE_AMDGPU=3Don: AMD GPU backend (required by mesa) > BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) > CLANG=3Don: Build clang > DOCS=3Don: Build and/or install documentation > EXTRAS=3Don: Extra clang tools > FLANG=3Doff: Flang FORTRAN compiler > LIT=3Don: Install lit and FileCheck test tools > LLD=3Don: Install lld, the LLVM linker > LLDB=3Don: Install lldb, the LLVM debugger > MLIR=3Doff: Multi-Level Intermediate Representation > OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library > PYCLANG=3Don: Install python bindings to libclang > =3D=3D=3D=3D> Options available for the single BACKENDS: you have to sele= ct > exactly one of them > BE_FREEBSD=3Doff: Backends for FreeBSD architectures > BE_NATIVE=3Don: Backend(s) for this architecture (AArch64) > BE_STANDARD=3Doff: All non-experimental backends > =3D=3D=3D> Use 'make config' to modify these settings > ---End OPTIONS List--- > > [11:07:48] [01] [08:58:53] Finished devel/llvm13 | llvm13-13.0.1_3: Succe= ss > > load averages: . . . MaxObs: 5.31, 4.94, 4.79 > > 1479Mi MaxObs(Act+Lndry+SwapUsed) > > So, vastly less RAM+SWAP space use. Somewhat under > 5 hours less build time (about 9hr vs. somewhat under 14hr). > > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --=20 Nuno Teixeira FreeBSD Committer (ports) --0000000000003e99b805e658fa4c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Mark,

**= * Sorry about TMPFS(=3Dno) when I want to mention USE_TMPFS(=3Dno).

Because I was in a hurry and I needed llvm13 package in o= ne of my jails, I did what you told me and turned off FLANG, MLIR and DOCS = (to avoid textproc/py-sphinx-markdown-tables problem).

=
I've used 6c+6t (hw.ncpu: 12) and compiled with success that consu= med ~1GB swap in ~40min.

Because I only use llvm* = as dependencies in poudriere jails for ports testing I could use a general = llvm* options to reduce time and mem+swap or if I want to stick with defaul= t options then I need to use /usr/local/etc/poudriere.d/make.conf MAKE_JOBS= _NUMBER=3Dn to be able to compile it with my hardware.

=
Thanks,

Mark Millard <marklmi@yahoo.com> escreveu no dia ter=C3=A7a, 16= /08/2022 =C3=A0(s) 01:08:
On 2022-Aug-14, at 18:40, Mark Millard <marklmi@yahoo.com> wrote:

> On 2022-Aug-14, at 09:35, Mark Millard <marklmi@yahoo.com> wrote:
>
>> On 2022-Aug-14, at 07:50, Nuno Teixeira <eduardo@freebsd.org> wrote:
>>
>> . . .
>>> have some time now and it's caused by a build peak of memo= ry that affects people with less than 32/64GB mem and to solve building it = must be build using one builder with one core thats takes about 7 hours on = my machine or with 6c+6t on 12.3 i386 that takes about 45min (123i386 is th= e only jail that I can use all cores).
>>
>> Last I tried I built all the various devel/llvm* on a 8 GiByte
>> RPi4B, 4 builders active and ALLOW_MAKE_JOBS=3Dyes in use.
>> 4 FreeBSD cpus. So the load average would have been around 16+
>> much of the time during devel/llvm13 's builder activity.
>> USE_TMPFS=3Ddata in use.
>>
>> Similarly for a 16 GiByte machine --but it is also an aarch64
>> context, also 4 FreebSD cpus.
>>
>> But I use in /boot/loader.conf:
>>
>> #
>> # Delay when persistent low free RAM leads to
>> # Out Of Memory killing of processes:
>> vm.pageout_oom_seq=3D120
>>
>> This has been historically important to avoiding the likes of
>> "was killed: failed to reclaim memory" and related notic= es on
>> various armv7 and aarch64 small board computers used to
>> buildworld buildkernel and build ports, using all the cores.
>>
>> The only amd64 system that I've access to has 32 FreeBSD cpus<= br> >> and 128 GiBytes of RAM. Not a good basis for a comparison test
>> with your context. I've no i386 access at all.
>>
>>> llvm 12 build without problems
>>
>> Hmm. I'll try building devel/llvm13 on aarch64 with periodic >> sampling of the memory use to see maximum observed figures
>> for SWAP and for various categories of RAM, as well as the
>> largest observed load averages.
>>
>> ZFS context use. I could try UFS as well.
>>
>> Swap: 30720Mi Total on the 8GiByte RPi4B.
>> So about 38 GiBytes RAM+SWAP available.
>> We should see how much SWAP is used.
>>
>> Before starting poudriere, shortly after a reboot:
>>
>> 19296Ki MaxObs(Act+Lndry+SwapUsed)
>> (No SWAP in use at the time.)
>>
>> # poudriere bulk -jmain-CA72-bulk_a -w devel/llvm13
>>
>> for the from scratch build: reports:
>>
>> [00:00:34] Building 91 packages using up to 4 builders
>>
>> The ports tree is about a month back:
>>
>> # ~/fbsd-based-on-what-commit.sh -C /usr/ports/
>> branch: main
>> merge-base: 872199326a916efbb4bf13c97bc1af910ba1482e
>> merge-base: CommitDate: 2022-07-14 01:26:04 +0000
>> 872199326a91 (HEAD -> main, freebsd/main, freebsd/HEAD) devel/r= uby-build: Update to 20220713
>> n589512 (--first-parent --count for merge-base)
>>
>> But, if I gather right, the problem you see goes back
>> before that.
>>
>> I can not tell how 4 FreeBSD cpus compares to the
>> count that the Lenovo Legion 5 gets.
>>
>> I'll report on its maximum observed figures once the
>> build stops. It will be a while before the RPi4B
>> gets that far.
>>
>> The ports built prior to devel/llvm13's builder starting
>> will lead to load averages over 4 from up to 4
>> builders, each potentially using up to around 4
>> processes. I'll see about starting a separate tracking
>> once devel/llvm13 's builder has started if I happen
>> to observe it at the right time frame for doing such.
>>
>> . . .
>
> I actually have tried a few builds on different
> machines. The 8GiByte RPi4B takes a long time and
> is currently omitted from this report.
>
>
> 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): > but using MAKE_JOBS_NUMBER=3D4 (with both FLANG and MLIR)
>
> On amd64 I started a build with FLANG and MLIR enabled,
> using MAKE_JOBS_NUMBER=3D4 in devel/llvm13/Makefile to
> limit the build to 4 FreeBSD cpus. It is a ZFS context.
> Given the 128 GiBytes of RAM, there will not be much
> for effects of memory-pressure. But will record the
> MaxObs(Act+Lndry+SwapUsed) and the like.
>
> ---Begin OPTIONS List---
> =3D=3D=3D> The following configuration options are available for ll= vm13-13.0.1_3:
>=C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)<= br> >=C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by fire= fox via wasi)
>=C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang
>=C2=A0 =C2=A0 =C2=A0COMPILER_RT=3Don: Sanitizer libraries
>=C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation
>=C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools
>=C2=A0 =C2=A0 =C2=A0FLANG=3Don: Flang FORTRAN compiler
>=C2=A0 =C2=A0 =C2=A0GOLD=3Don: Build the LLVM Gold plugin for LTO
>=C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools
>=C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker
>=C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger
>=C2=A0 =C2=A0 =C2=A0MLIR=3Don: Multi-Level Intermediate Representation<= br> >=C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtim= e library
>=C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang > =3D=3D=3D=3D> Options available for the single BACKENDS: you have t= o select exactly one of them
>=C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architecture= s
>=C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Doff: Backend(s) for this architecture (= X86)
>=C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Don: All non-experimental backends
> =3D=3D=3D> Use 'make config' to modify these settings
> ---End OPTIONS List---
>
> [02:23:55] [01] [02:04:29] Finished devel/llvm13 | llvm13-13.0.1_3: Su= ccess
>
> For just the devel/llvm13 builder activity, no parallel
> builds and excluding the prerequisites being built:
>
> load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A06.76,=C2=A0 =C2= =A04.75,=C2=A0 =C2=A04.38
>
> 6812Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed.
>
> Note: MAKE_JOBS_NUMBER does not constrain any lld
>=C2=A0 =C2=A0 =C2=A0 procoess from using all available FreeBSD cpus
>=C2=A0 =C2=A0 =C2=A0 (via threading) --and multiple lld's can be >=C2=A0 =C2=A0 =C2=A0 active at the same time.
>
> So this looks to fit in a 16 GiByte RAM context just fine,
> no SWAP needed.
>
> I'll try MAKE_JOBS_NUMBER=3D12 instead and rerun on the same
> machine.
>
>
>
> 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): > but using MAKE_JOBS_NUMBER=3D12 (with both FLANG and MLIR)
>
> ---Begin OPTIONS List---
> =3D=3D=3D> The following configuration options are available for ll= vm13-13.0.1_3:
>=C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)<= br> >=C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by fire= fox via wasi)
>=C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang
>=C2=A0 =C2=A0 =C2=A0COMPILER_RT=3Don: Sanitizer libraries
>=C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation
>=C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools
>=C2=A0 =C2=A0 =C2=A0FLANG=3Don: Flang FORTRAN compiler
>=C2=A0 =C2=A0 =C2=A0GOLD=3Don: Build the LLVM Gold plugin for LTO
>=C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools
>=C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker
>=C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger
>=C2=A0 =C2=A0 =C2=A0MLIR=3Don: Multi-Level Intermediate Representation<= br> >=C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtim= e library
>=C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang > =3D=3D=3D=3D> Options available for the single BACKENDS: you have t= o select exactly one of them
>=C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architecture= s
>=C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Doff: Backend(s) for this architecture (= X86)
>=C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Don: All non-experimental backends
> =3D=3D=3D> Use 'make config' to modify these settings
> ---End OPTIONS List---
>
> [00:55:37] [01] [00:55:30] Finished devel/llvm13 | llvm13-13.0.1_3: Su= ccess
>
> load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A012.45,=C2=A0 12.2= 0,=C2=A0 11.52
>
> 13074Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed.
>
> Note: MAKE_JOBS_NUMBER does not constrain any lld
>=C2=A0 =C2=A0 =C2=A0 procoess from using all available FreeBSD cpus
>=C2=A0 =C2=A0 =C2=A0 (via threading) --and multiple lld's can be >=C2=A0 =C2=A0 =C2=A0 active at the same time.
>
> (16+4)*1024 Mi - 13074 Mi =3D=3D 7406 Mi for other RAM+SWAP use.
> (Crude estimates relative to your context.) That would seem
> to be plenty.
>
>
> Conclusion:
>
> It is far from clear what all was contributing to your
> (16+4)*1024 MiBytes proving to be insufficient.
> Unintentional tmpfs use, such as a typo in USE_TMPFS
> in /usr/local/etc/poudriere.conf ? I really have no
> clue: the example is arbitrary.
>
>
>
> Other notes:
>
> # uname -apKU
> FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #50 main-n256584-5= bc926af9fd1-dirty: Wed Jul=C2=A0 6 17:44:43 PDT 2022=C2=A0 =C2=A0 =C2=A0roo= t@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64= /sys/GENERIC-NODBG amd64 amd64 1400063 1400063
>
> Note that the above is without WITNESS and without INVARIANTS and the = like.
>
> The only thing commited to main's contrib/llvm-project after that<= br> > was 9ef1127008 :
>
> QUOTE
> Apply tentative llvm fix for avoiding fma on PowerPC SPE
> Merge llvm review D77558, by Justin Hibbits:
>
>=C2=A0 PowerPC: Don't hoist float multiply + add to fused operation= on SPE
>
>=C2=A0 SPE doesn't have a fmadd instruction, so don't bother ho= isting a
>=C2=A0 multiply and add sequence to this, as it'd become just a lib= rary call.
>=C2=A0 Hoisting happens too late for the CTR usability test to veto usi= ng the CTR
>=C2=A0 in a loop, and results in an assert "Invalid PPC CTR loop!&= quot;.
> END QUOTE

Here I add the 8 GiByte RPi4B results, in this case
both with FLANG/MLIR results and without results. The
context has and uses 4 FreeBSD cpus. No other builder
is active when devel/llvm13's builder is.

With both FLANG and MLIR:

---Begin OPTIONS List---
=3D=3D=3D> The following configuration options are available for llvm13-= 13.0.1_3:
=C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)
=C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by firefox = via wasi)
=C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang
=C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation
=C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools
=C2=A0 =C2=A0 =C2=A0FLANG=3Don: Flang FORTRAN compiler
=C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools
=C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker
=C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger
=C2=A0 =C2=A0 =C2=A0MLIR=3Don: Multi-Level Intermediate Representation
=C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtime li= brary
=C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang
=3D=3D=3D=3D> Options available for the single BACKENDS: you have to sel= ect exactly one of them
=C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architectures =C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Doff: Backend(s) for this architecture (AArc= h64)
=C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Don: All non-experimental backends
=3D=3D=3D> Use 'make config' to modify these settings
---End OPTIONS List---

[13:49:55] [01] [13:49:17] Finished devel/llvm13 | llvm13-13.0.1_3: Success=

load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A06.43,=C2=A0 =C2=A05.91= ,=C2=A0 =C2=A05.77
(Note: spanned overnight so the nightly cron job was
spanned.)

Note: Given that SWAP was used, I report more
Max(imum)Obs(erved) figures for this case than
I've been reporting for other tests:

5696Mi MaxObsActive
1775Mi MaxObsSwapUsed
7374Mi MaxObs(Act+Lndry+SwapUsed)
9333Mi MaxObs(Act+Wir+Lndry+SwapUsed)

Reminder: MaximumOfASum <=3D TheSumOfTheMaximums
Note: The various Maximums need not be from the same time.


By contrast . . .

No FLANG, no MLIR:

---Begin OPTIONS List---
=3D=3D=3D> The following configuration options are available for llvm13-= 13.0.1_3:
=C2=A0 =C2=A0 =C2=A0BE_AMDGPU=3Don: AMD GPU backend (required by mesa)
=C2=A0 =C2=A0 =C2=A0BE_WASM=3Don: WebAssembly backend (required by firefox = via wasi)
=C2=A0 =C2=A0 =C2=A0CLANG=3Don: Build clang
=C2=A0 =C2=A0 =C2=A0DOCS=3Don: Build and/or install documentation
=C2=A0 =C2=A0 =C2=A0EXTRAS=3Don: Extra clang tools
=C2=A0 =C2=A0 =C2=A0FLANG=3Doff: Flang FORTRAN compiler
=C2=A0 =C2=A0 =C2=A0LIT=3Don: Install lit and FileCheck test tools
=C2=A0 =C2=A0 =C2=A0LLD=3Don: Install lld, the LLVM linker
=C2=A0 =C2=A0 =C2=A0LLDB=3Don: Install lldb, the LLVM debugger
=C2=A0 =C2=A0 =C2=A0MLIR=3Doff: Multi-Level Intermediate Representation
=C2=A0 =C2=A0 =C2=A0OPENMP=3Don: Install libomp, the LLVM OpenMP runtime li= brary
=C2=A0 =C2=A0 =C2=A0PYCLANG=3Don: Install python bindings to libclang
=3D=3D=3D=3D> Options available for the single BACKENDS: you have to sel= ect exactly one of them
=C2=A0 =C2=A0 =C2=A0BE_FREEBSD=3Doff: Backends for FreeBSD architectures =C2=A0 =C2=A0 =C2=A0BE_NATIVE=3Don: Backend(s) for this architecture (AArch= 64)
=C2=A0 =C2=A0 =C2=A0BE_STANDARD=3Doff: All non-experimental backends
=3D=3D=3D> Use 'make config' to modify these settings
---End OPTIONS List---

[11:07:48] [01] [08:58:53] Finished devel/llvm13 | llvm13-13.0.1_3: Success=

load averages:=C2=A0 =C2=A0. . . MaxObs:=C2=A0 =C2=A05.31,=C2=A0 =C2=A04.94= ,=C2=A0 =C2=A04.79

1479Mi MaxObs(Act+Lndry+SwapUsed)

So, vastly less RAM+SWAP space use. Somewhat under
5 hours less build time (about 9hr vs. somewhat under 14hr).


=3D=3D=3D
Mark Millard
marklmi at yahoo.com



--
Nuno Teixeira
FreeBSD Co= mmitter (ports)
--0000000000003e99b805e658fa4c--