From nobody Sun Aug 14 17:15:28 2022 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M5PF864mmz4ZpRh for ; Sun, 14 Aug 2022 17:15:40 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M5PF85whBz3yKV for ; Sun, 14 Aug 2022 17:15:40 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1660497340; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kmEgciZYWAA0Hrpxj7Ng9cRtC7KaekU1If2wClzDoRo=; b=uedpb/4vTtoWmnVLhMwyGwoOPW7Z1mMzyVaYmUyUxYWcIeTGRzbMB5WfoqNa5FLVss/MFo uWTvAX1YUQEeS4vY5ZBC6RBD9uCpJH6Y2gw2WMi/4vux07yG/phhXvS1uQuu/RXZqkTuge tg1bdW2EBX2Cx4YwcsoMjje6DrVPI88AQ8YYwiZ1Sr9qJrCWN+ZM80o2BVFoXSQVlVFAq9 JKTzxt7vGb/2/+3q42cS1+9tNYyjQ5d9AUHc9a53B2ZlWWJC2UiY5GHozQMsbZoedJpolv f21iT3iZtAhodL4wm2CaPF3Tvj5mK9aiqagceoT3B5BVIExXiyubjIKtSWpjOg== Received: from mail-vs1-f42.google.com (mail-vs1-f42.google.com [209.85.217.42]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4M5PF84tBVz1CVQ for ; Sun, 14 Aug 2022 17:15:40 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-vs1-f42.google.com with SMTP id 125so5355932vsd.5 for ; Sun, 14 Aug 2022 10:15:40 -0700 (PDT) X-Gm-Message-State: ACgBeo1YFSwSJWtTw6Nx/3t8YR7YqoLvhVNjuETA4czMZ8yT0eZAzCw1 m5SmfI/GS/vHi8cS46UBGoqrMWwNBmNT2ClTVck= X-Google-Smtp-Source: AA6agR6is7fEU/3K52Nkh8oVpXAOqDMVfefU5nvBLLctyBpzg1OLosY7LesNPWvFkqdz4/5tyrh4qpuY8rAoXsSUuvk= X-Received: by 2002:a05:6102:3e02:b0:388:984b:e082 with SMTP id j2-20020a0561023e0200b00388984be082mr4814330vsv.58.1660497340276; Sun, 14 Aug 2022 10:15:40 -0700 (PDT) List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 References: <1D4C14BD-8955-4B86-9C99-3E58D7603122.ref@yahoo.com> <1D4C14BD-8955-4B86-9C99-3E58D7603122@yahoo.com> <7CDC63F3-8B68-420E-8012-B1692667E293@yahoo.com> In-Reply-To: <7CDC63F3-8B68-420E-8012-B1692667E293@yahoo.com> From: Nuno Teixeira Date: Sun, 14 Aug 2022 18:15:28 +0100 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Resolved: devel/llvm13 build: "ninja: build stopped: subcommand failed" To: Mark Millard Cc: FreeBSD Mailing List Content-Type: multipart/alternative; boundary="0000000000001e25c205e636aa0f" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1660497340; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=kmEgciZYWAA0Hrpxj7Ng9cRtC7KaekU1If2wClzDoRo=; b=ysReSWo2b0xRarNV1dJ/A/73jmuqvu4JNyu+x3LVtL89vhFBDvFAFketDlnqt172GsvoFf hPvSi9sdY8q+j2Cvu7Yp08Stt51YvQwWLtX5DdKoGxy94X1tjLjdcneCCM9RhiPFphF4UB Hmuchz0CVP4JSTW9G+M5k2WK2OkBZqwQman3f0sSs7qlpSSsqPvQa4eaiW3vnKCbbPGoQ1 saBPO1OWjEN+OWDVfeA9XWTGD5ztbjPT+jGLC9+VPdJj2QMs0PBZ9RD8rFrRW63xPA59wC jWRIriaaiXludTDe3PgQM5ieu7teVk9YR0akjjvD/pw5uXldsnApPkFuicnpRA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1660497340; a=rsa-sha256; cv=none; b=t19J0cMin8QAVVeBRDT/edHogZptJfwK9yKoOtlDZ/qED9LuA2s+VQOEBTGoMLcHEw7rsg 8k84a+0AWKek3Uo6AfZBajtxpe7WBpO9vC9jL07c7GuZE1ZB4owzTu+QnB3XwHJi4mwPJp NyHrIqqRtfOP5CoYmW84JDb8/nvRpRD7FmzILl6kt3fE++OLloQuy9z1/aBbUma3KWeGYp cM0l0BjfQ45Qw5cEDfQ65LzKOXl4Cmaij7Jsw9IZlvZwSuA6VAV0sTScY+kd5EZextamp+ hTL7QPY/WUZUNLZxJjAlEv2aZhi8uKSXNJ1CuhACSM69W7/yuElfjRehxm/cpg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N --0000000000001e25c205e636aa0f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable I use ZFS. I will follow your recomendations and use a swap of 64GB and then test it again. In the meanwhile I will take a look at freebsd docs to see how do I increase swap, by adding a new swap file or resize actual one if possible. Mark Millard escreveu no dia domingo, 14/08/2022 =C3=A0= (s) 17:35: > On 2022-Aug-14, at 07:50, Nuno Teixeira wrote: > > Hello Mark, > > > I use poudriere with USE_TMPFS=3Dno, ofc because of low mem) > > The problem "ninja: build stopped: subcommand failed" > > That is never the original error, just ninja reporting after > it observed an error that occurred, generally in another > process that is involved. A wide variety of errors will > end up with a "ninja: build stopped: subcommand failed" > notice as well. > > The original error should be earlier in the log or on the > console ( or in /var/log/messages ). The "was killed: failed > to reclaim memory" is an example. > > With 16 GiBytes of RAM you could have up to something like > 60 GiByte of swap without FreeBSD complaining about being > potentially mistuned. (It would complain before 64 GiBytes > of SWAP.) 16+60 would be 76 GiBytes for RAM+SWAP. > > I forgot to ask about UFS vs. ZFS being in use: which is in > use? (ZFS uses more RAM.) > > > have some time now and it's caused by a build peak of memory that > affects people with less than 32/64GB mem and to solve building it must b= e > build using one builder with one core thats takes about 7 hours on my > machine or with 6c+6t on 12.3 i386 that takes about 45min (123i386 is the > only jail that I can use all cores). > > Last I tried I built all the various devel/llvm* on a 8 GiByte > RPi4B, 4 builders active and ALLOW_MAKE_JOBS=3Dyes in use. > 4 FreeBSD cpus. So the load average would have been around 16+ > much of the time during devel/llvm13 's builder activity. > USE_TMPFS=3Ddata in use. > > Similarly for a 16 GiByte machine --but it is also an aarch64 > context, also 4 FreebSD cpus. > > But I use in /boot/loader.conf: > > # > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 > > This has been historically important to avoiding the likes of > "was killed: failed to reclaim memory" and related notices on > various armv7 and aarch64 small board computers used to > buildworld buildkernel and build ports, using all the cores. > > The only amd64 system that I've access to has 32 FreeBSD cpus > and 128 GiBytes of RAM. Not a good basis for a comparison test > with your context. I've no i386 access at all. > > > llvm 12 build without problems > > Hmm. I'll try building devel/llvm13 on aarch64 with periodic > sampling of the memory use to see maximum observed figures > for SWAP and for various categories of RAM, as well as the > largest observed load averages. > > ZFS context use. I could try UFS as well. > > Swap: 30720Mi Total on the 8GiByte RPi4B. > So about 38 GiBytes RAM+SWAP available. > We should see how much SWAP is used. > > Before starting poudriere, shortly after a reboot: > > 19296Ki MaxObs(Act+Lndry+SwapUsed) > (No SWAP in use at the time.) > > # poudriere bulk -jmain-CA72-bulk_a -w devel/llvm13 > > for the from scratch build: reports: > > [00:00:34] Building 91 packages using up to 4 builders > > The ports tree is about a month back: > > # ~/fbsd-based-on-what-commit.sh -C /usr/ports/ > branch: main > merge-base: 872199326a916efbb4bf13c97bc1af910ba1482e > merge-base: CommitDate: 2022-07-14 01:26:04 +0000 > 872199326a91 (HEAD -> main, freebsd/main, freebsd/HEAD) devel/ruby-build: > Update to 20220713 > n589512 (--first-parent --count for merge-base) > > But, if I gather right, the problem you see goes back > before that. > > I can not tell how 4 FreeBSD cpus compares to the > count that the Lenovo Legion 5 gets. > > I'll report on its maximum observed figures once the > build stops. It will be a while before the RPi4B > gets that far. > > The ports built prior to devel/llvm13's builder starting > will lead to load averages over 4 from up to 4 > builders, each potentially using up to around 4 > processes. I'll see about starting a separate tracking > once devel/llvm13 's builder has started if I happen > to observe it at the right time frame for doing such. > > > Cheers > > > > Mark Millard escreveu no dia domingo, 14/08/2022 > =C3=A0(s) 03:54: > > Nuno Teixeira wrote on > > Date: Sat, 13 Aug 2022 16:52:09 UTC : > > > > > . . . > > > I've tested it but it still fails: > > > --- > > > pid 64502 (c++), jid 7, uid 65534, was killed: failed to reclaim memo= ry > > > swap_pager: out of swap space > > > --- > > > on a Lenovo Legion 5, 16GB RAM and 4GB swap. > > > . . . > > > > This leaves various points unclear: > > > > poudriere style build? Some other style? > > > > (I'll state questions in a form generally for a poudriere style > > context. Some could be converted to analogous points for other > > build-styles.) > > > > How many poudriere builders allowed (-JN) ? > > > > /usr/local/etc/poudreire.conf : > > ALLOW_MAKE_JOBS=3Dyes in use? > > ALLOW_MAKE_JOBS_PACKAGES=3D??? in use? > > USE_TMPFS=3D??? With what value? Anything other that "data" or "no"? > > > > /usr/local/etc/poudriere.d/make.conf (or the like): > > MAKE_JOBS_NUMBER=3D??? in use? With what value? > > > > Is tmpfs in use such that it will use RAM+SWAP when the > > used tmpfs space is large? > > > > How much free space is available for /tmp ? > > > > Are you using something like ( in, say, /boot/loader/conf ): > > That should have been: /boot/loader.conf > > Sorry. > > > # > > # Delay when persistent low free RAM leads to > > # Out Of Memory killing of processes: > > vm.pageout_oom_seq=3D120 > > > > > > How many FreeBSD cpus does a Lenovo Legion 5 present > > in the configuration used? > > > > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --=20 Nuno Teixeira FreeBSD Committer (ports) --0000000000001e25c205e636aa0f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I use ZFS.

I will follow your recom= endations and use a swap of 64GB and then test it again.

In th= e meanwhile I will take a look at freebsd docs to see how do I increase swa= p, by adding a new swap file or resize actual one if possible.
Mark Mill= ard <marklmi@yahoo.com> escr= eveu no dia domingo, 14/08/2022 =C3=A0(s) 17:35:
On 2022-Aug-14, at 07:50, Nuno Teixeira &l= t;eduardo@freebsd.= org> wrote:

Hello Mark,

> I use poudriere with USE_TMPFS=3Dno, ofc because of low mem)
> The problem "ninja: build stopped: subcommand failed"

That is never the original error, just ninja reporting after
it observed an error that occurred, generally in another
process that is involved. A wide variety of errors will
end up with a "ninja: build stopped: subcommand failed"
notice as well.

The original error should be earlier in the log or on the
console ( or in /var/log/messages ). The "was killed: failed
to reclaim memory" is an example.

With 16 GiBytes of RAM you could have up to something like
60 GiByte of swap without FreeBSD complaining about being
potentially mistuned. (It would complain before 64 GiBytes
of SWAP.) 16+60 would be 76 GiBytes for RAM+SWAP.

I forgot to ask about UFS vs. ZFS being in use: which is in
use? (ZFS uses more RAM.)

> have some time now and it's caused by a build peak of memory that = affects people with less than 32/64GB mem and to solve building it must be = build using one builder with one core thats takes about 7 hours on my machi= ne or with 6c+6t on 12.3 i386 that takes about 45min (123i386 is the only j= ail that I can use all cores).

Last I tried I built all the various devel/llvm* on a 8 GiByte
RPi4B, 4 builders active and ALLOW_MAKE_JOBS=3Dyes in use.
4 FreeBSD cpus. So the load average would have been around 16+
much of the time during devel/llvm13 's builder activity.
USE_TMPFS=3Ddata in use.

Similarly for a 16 GiByte machine --but it is also an aarch64
context, also 4 FreebSD cpus.

But I use in /boot/loader.conf:

#
# Delay when persistent low free RAM leads to
# Out Of Memory killing of processes:
vm.pageout_oom_seq=3D120

This has been historically important to avoiding the likes of
"was killed: failed to reclaim memory" and related notices on
various armv7 and aarch64 small board computers used to
buildworld buildkernel and build ports, using all the cores.

The only amd64 system that I've access to has 32 FreeBSD cpus
and 128 GiBytes of RAM. Not a good basis for a comparison test
with your context. I've no i386 access at all.

> llvm 12 build without problems

Hmm. I'll try building devel/llvm13 on aarch64 with periodic
sampling of the memory use to see maximum observed figures
for SWAP and for various categories of RAM, as well as the
largest observed load averages.

ZFS context use. I could try UFS as well.

Swap: 30720Mi Total on the 8GiByte RPi4B.
So about 38 GiBytes RAM+SWAP available.
We should see how much SWAP is used.

Before starting poudriere, shortly after a reboot:

19296Ki MaxObs(Act+Lndry+SwapUsed)
(No SWAP in use at the time.)

# poudriere bulk -jmain-CA72-bulk_a -w devel/llvm13

for the from scratch build: reports:

[00:00:34] Building 91 packages using up to 4 builders

The ports tree is about a month back:

# ~/fbsd-based-on-what-commit.sh -C /usr/ports/
branch: main
merge-base: 872199326a916efbb4bf13c97bc1af910ba1482e
merge-base: CommitDate: 2022-07-14 01:26:04 +0000
872199326a91 (HEAD -> main, freebsd/main, freebsd/HEAD) devel/ruby-build= : Update to 20220713
n589512 (--first-parent --count for merge-base)

But, if I gather right, the problem you see goes back
before that.

I can not tell how 4 FreeBSD cpus compares to the
count that the Lenovo Legion 5 gets.

I'll report on its maximum observed figures once the
build stops. It will be a while before the RPi4B
gets that far.

The ports built prior to devel/llvm13's builder starting
will lead to load averages over 4 from up to 4
builders, each potentially using up to around 4
processes. I'll see about starting a separate tracking
once devel/llvm13 's builder has started if I happen
to observe it at the right time frame for doing such.

> Cheers
>
> Mark Millard <marklmi@yahoo.com> escreveu no dia domingo, 14/08/2022 =C3=A0(s) 0= 3:54:
> Nuno Teixeira <eduardo_at_freebsd.org> wrote on
> Date: Sat, 13 Aug 2022 16:52:09 UTC :
>
> > . . .
> > I've tested it but it still fails:
> > ---
> > pid 64502 (c++), jid 7, uid 65534, was killed: failed to reclaim = memory
> > swap_pager: out of swap space
> > ---
> > on a Lenovo Legion 5, 16GB RAM and 4GB swap.
> > . . .
>
> This leaves various points unclear:
>
> poudriere style build? Some other style?
>
> (I'll state questions in a form generally for a poudriere style > context. Some could be converted to analogous points for other
> build-styles.)
>
> How many poudriere builders allowed (-JN) ?
>
> /usr/local/etc/poudreire.conf :
> ALLOW_MAKE_JOBS=3Dyes in use?
> ALLOW_MAKE_JOBS_PACKAGES=3D??? in use?
> USE_TMPFS=3D??? With what value? Anything other that "data" = or "no"?
>
> /usr/local/etc/poudriere.d/make.conf (or the like):
> MAKE_JOBS_NUMBER=3D??? in use? With what value?
>
> Is tmpfs in use such that it will use RAM+SWAP when the
> used tmpfs space is large?
>
> How much free space is available for /tmp ?
>
> Are you using something like ( in, say, /boot/loader/conf ):

That should have been: /boot/loader.conf

Sorry.

> #
> # Delay when persistent low free RAM leads to
> # Out Of Memory killing of processes:
> vm.pageout_oom_seq=3D120
>
>
> How many FreeBSD cpus does a Lenovo Legion 5 present
> in the configuration used?
>


=3D=3D=3D
Mark Millard
marklmi at yahoo.com



--
Nun= o Teixeira
FreeBSD Committer (ports)
--0000000000001e25c205e636aa0f--