From nobody Mon May 5 22:36:25 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZrxGF6gW2z5txqs for ; Mon, 05 May 2025 22:36:37 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZrxGF3y7Mz3JNX for ; Mon, 05 May 2025 22:36:37 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1746484597; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=k/Xo+XiSR3eBsqXxrPWrmDo+FFICEDS8EN5gpX7e9Sw=; b=JxtuqoLfc9Lkx2hu73gCAcR7en9ImGNOE1Pc7aFxivExRF8HANy34ZzlnRnR+GOuaH4hIw zGHPTOT/HRFqMDPc8xObMDv+L+TBqGT+GrGJWkJCYlhce7xzvs5pfKpuVeqXdtfvwfyvv0 Z2KlfsGdQmO1olpkNOvkVwSgUqBqbOFbTVD2Rr95MWPjOeFZmAq+Gd9FV3ja2+aB0VXBgg qpmhfr473G5v0sCz79aZMAfLltis5gAsdD9dBBMK2+vJejE6q8/85AfaoKyX3hdZMvctlF JxQTNCCXjp/J7NZCXCUGFPkgLp2OxatVcajv87oKDd057TxTRCLrZ3LwQ+cdZA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1746484597; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=k/Xo+XiSR3eBsqXxrPWrmDo+FFICEDS8EN5gpX7e9Sw=; b=CSGLC1lcgqRQi5WaeGlBJ7frkdUdY/vq/1d/wCY7VJz/AhCaUnCHahSmM3Pb0GFbXhGQhA rHU5fwMKdCojPPwkr22wh2tZfEQwIxDru568GBdgQEQjUWjQlLn4+yOlBTbgROOx0rKAvp 0QQ0P9Ua9ZQxRGhPP/7k7mMFhryQ/tDReyyaQNCQXmBRpIGvWBqseLpDKbvrNjV9CzBNgY EcEH3zucedlePcUBcbeYFacW5k7sHNd07flQ/6wU6EJwsDyufJ6fSk9rhqVT+EGq3IkpqL VL2b595su9xj9APJTinMvAKD1VhqvJmxakl/4lTz3I39X3c1v3CyaRB7MYfi9A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1746484597; a=rsa-sha256; cv=none; b=gfFw57j0yhmSoAZJ15F8hy1pvc/eKOIP+ENytXWHekWOh1D5e0JhySSzUEiEJZTidhGWmp 8Z6I71ypupsZChIIzzMbkapkogg7D17uW5OzvN5L9pZD90Ea23bZTF0hAwJg3NXb0goDgW zubUF8uPhDYoKPV/Lxio+f0e+DlfRQzyXS7RTg+g8r4pleMIgR16vk39tJQMxzNXn+2iaF Gwg0mc0q+KzQRCxy2AEzMWsGxgMfNbrPbjlDRmzuegFnrQlYEsS4dou9iA/A+xMa85atjL EfQDRiGRZ5j6gTKXo/XGQ29gdJ3LO+oGVHfy1eit/3jibFgeAbCl1ktDKVJz2g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4ZrxGF339SzFYF for ; Mon, 05 May 2025 22:36:37 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7c5f20d512fso69490385a.2 for ; Mon, 05 May 2025 15:36:37 -0700 (PDT) X-Gm-Message-State: AOJu0Yxk7QUit9boYswFYU+djGmobdnLGuYxdhbhLNbkC1W8USOlBh6Q vYaPtD8ErjaYmcNpmfZ6d9IHMXIv8HpLhv4D7HdGVyaExFl9LpaYYYQ1GvjI3WVh477fnWm+4gD fcNcBk+8qIMJVJzwpXAE+tRpI8Uw= X-Google-Smtp-Source: AGHT+IGFPb3gGKwPqXUWaC8XTF4bEKLENJ3+OeoSOsfd5lIHA8NRFTBOqXkLtObC2Xm0I+9ckkpbNOwSlO1K45Y2OgM= X-Received: by 2002:a05:622a:14d4:b0:475:820:9f6f with SMTP id d75a77b69052e-48c3192cea7mr84163121cf.9.1746484596871; Mon, 05 May 2025 15:36:36 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <28F2BDE7-5903-4C04-A570-6A407F19D5F2.ref@yahoo.com> <28F2BDE7-5903-4C04-A570-6A407F19D5F2@yahoo.com> In-Reply-To: <28F2BDE7-5903-4C04-A570-6A407F19D5F2@yahoo.com> From: Nuno Teixeira Date: Mon, 5 May 2025 23:36:25 +0100 X-Gmail-Original-Message-ID: X-Gm-Features: ATxdqUFIR1OQ2IX0GZJCZljEratMjChQjHQi3i6v8Z-9K956HDUEQ_G7TAVOldo Message-ID: Subject: Re: incremental bulds from scratch with beinstall.sh To: Mark Millard Cc: FreeBSD Current Content-Type: multipart/alternative; boundary="00000000000000972606346b224b" --00000000000000972606346b224b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Mark, Minutes ago I used a more tradicional way: # make buildworld-jobs buildkernel-jobs ``` RELEASE=3DWhatever > bectl create ${RELEASE} > bectl mount ${RELEASE} BASEDIR=3D/tmp/be_mount.XXXX # Use mount point returned by bectl mount > make DESTDIR=3D${BASEDIR} installkernel > etcupdate -p -D ${BASEDIR} > make DESTDIR=3D${BASEDIR} installworld > etcupdate -D ${BASEDIR} > bectl activate ${RELEASE} > reboot ``` # make buildworld-jobs[1] buildkernel-jobs[2] [1] did not compile [2] compiled # make buildworld-jobs[1] buildkernel-jobs[2] [1] did not compile [2] did not compile Cheers, Mark Millard escreveu (segunda, 5/05/2025 =C3=A0(s) 23:= 18): > Nuno Teixeira wrote on > Date: Mon, 05 May 2025 20:37:09 UTC : > > > (...) > > > > Don't forget to `env NO_PKG_UPGRADE=3Dyes beinstall.sh` to not mess wit= h > > ports and stuff. > > > > Nuno Teixeira escreveu (segunda, 5/05/2025 =C3=A0= (s) > > 21:34): > > > > > Hello, > > > > > > Using incremental WITH_META_MODE builds, after installation with > > > beinstall.sh, building same src, a complete compilation happens. > > > > > > If someone uses this script, could you please do the following test: > > > > > > WITH_META_MODE=3Dyes (/etc/src-env.conf) > > > filemon module loaded > > > > > > # cd /usr/src > > > # make buildworld-jobs buildkernel-jobs > > The above used older commands and files from before > the following install. META_MODE recorded the use of > those commands. > > Example .meta mode file content: > > # Meta data file > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/l= ibc++.a.meta > CMD @echo building static c++ library > CMD @rm -f libc++.a > CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o > charconv.o chrono.o condition_variable.o condition_variable_destructor.o > debug.o exception.o filesystem/directory_iterator.o filesyste > m/int128_builtins.o filesystem/operations.o functional.o future.o hash.o > ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o > optional.o random.o random_shuffle.o regex.o shared_mutex.o > stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o > utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o > cxxrt_dynamic_cast.o cxxrt_exception.o cxxrt_guard.o cxxrt_libelftc_dem_g > nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o cxxrt_typeinfo.o > CWD /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc= ++ > TARGET libc++.a > -- command output -- > building static c++ library > > -- filemon acquired metadata -- > # filemon version 5 > # Target pid 22471 > # Start 1611359217.214996 > V 5 > E 22961 /bin/sh > R 22961 /etc/libmap.conf > R 22961 /var/run/ld-elf.so.hints > R 22961 /lib/libedit.so.7 > R 22961 /lib/libc.so.7 > R 22961 /lib/libncursesw.so.9 > R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE > F 22961 22962 > E 22962 > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/u= sr/sbin/rm > R 22962 /etc/libmap.conf > R 22962 /var/run/ld-elf.so.hints > R 22962 /lib/libc.so.7 > R 22962 /usr/share/locale/C.UTF-8/LC_CTYPE > D 22962 libc++.a > X 22962 0 0 > . . . > > So META_MODE has lots of files that were used > that it can later detect being newer than the > prior build results, leading to rebuilds based > on those newer files. > > > > # ./tools/build/beinstall.sh > > The new be will have various updated files > that could be different by content and, for > commands, could behave differently than those > used to do the prior build. Detecting newer > time stamps on such used files leads to > rebuild activity. > > More than /usr/src/ and /usr/obj/ content > are involved as well. > > Note that the new be is based on somewhat > different files than the original > buildworld-jobs buildkernel-jobs was based > on. > > > > # reboot > > > > > (I presume booting into the new be here.) > > > > # cd /usr/src > > > # make buildworld-jobs buildkernel-jobs > > META_MODE will notice when commands are used > that are newer than when the prior build was > done. Similarly for other files that may be > read. It will make sure that the newer commands > and files are allowed to produce new results > that potentially could be distinct in content > from what the old context produced for results. > > > > > > > Since src and obj are the same from one BE to newer BE, minimal > > > compilation should happen, not a full one. > > META_MODE is more careful than that. > > Note: I'm not claiming that new behavior that is > needed is likely for lots of the files with new > dates. But META_MODE is biased to avoiding leaving > in place something that should have been updated. > > > > > > > Am I missing something here? > > > > > Note that make has a -dM option: > > M Print debugging information about =E2=80=9Cmeta=E2= =80=9D mode > decisions > about targets. > > So, for example, > > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/awk' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/cap_mkdb' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/cat' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/cp' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/crunchgen' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/crunchide' > is newer than the target... > file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/dd' > is newer than the target... > . . . > > Various . . ./tmp/legacy/. . ./*bin/ actually were > links to files. > > Ultimately buildworld then installworld lead to new > dates for a bunch of files used. Later use of > META_MODE notices such and rebuilds based on the > newer files. (It is a lot of detail to go through > it all.) > > Back in 2021 and 2023 I got help with exploring > avoiding lots of these. But, in the end, it > involved use of experimental code in > share/mk/src.sys.obj.mk to provide a new > definition to use to build some paths with. > > The experiments were an unsupported activity that > produced an unsupported change to allow > configurable enabling of taking risks with not > updating files that possibly should be updated. > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --=20 Nuno Teixeira FreeBSD UNIX: Web: https://FreeBSD.org --00000000000000972606346b224b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Mark,

Minutes ago I us= ed a more tradicional way:

# make buildworld-j= obs buildkernel-jobs

```
RELEASE=3DWhate= ver
> bectl create ${RELEASE}
> bectl mount ${RELEASE}
BASED= IR=3D/tmp/be_mount.XXXX # Use mount point returned by bectl mount

&g= t; make DESTDIR=3D${BASEDIR} installkernel
> etcupdate -p -D ${BASEDI= R}
> make DESTDIR=3D${BASEDIR} installworld
> etcupdate -D ${BA= SEDIR}
> bectl activate ${RELEASE}
> reboot
```
=

# make buildworld-jobs[1] buildkernel-jobs[2]
[1] did not compile
[2] compiled

# make buildw= orld-jobs[1] buildkernel-jobs[2]
[1] did not compile
[2] did not compile

Cheers,
Mark Millard <marklmi@yahoo.com<= /a>> escreveu (segunda, 5/05/2025 =C3=A0(s) 23:18):
Nuno Teixeira <eduardo_at_fre= ebsd.org> wrote on
Date: Mon, 05 May 2025 20:37:09 UTC :

> (...)
>
> Don't forget to `env NO_PKG_UPGRADE=3Dyes beinstall.sh` to not mes= s with
> ports and stuff.
>
> Nuno Teixeira <eduardo@freebsd.org> escreveu (segunda, 5/05/2025 =C3=A0(s)
> 21:34):
>
> > Hello,
> >
> > Using incremental WITH_META_MODE builds, after installation with<= br> > > beinstall.sh, building same src, a complete compilation happens.<= br> > >
> > If someone uses this script, could you please do the following te= st:
> >
> > WITH_META_MODE=3Dyes (/etc/src-env.conf)
> > filemon module loaded
> >
> > # cd /usr/src
> > # make buildworld-jobs buildkernel-jobs

The above used older commands and files from before
the following install. META_MODE recorded the use of
those commands.

Example .meta mode file content:

# Meta data file /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd= 64/lib/libc++/libc++.a.meta
CMD @echo building static c++ library
CMD @rm -f libc++.a
CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o charconv.= o chrono.o condition_variable.o condition_variable_destructor.o debug.o exc= eption.o filesystem/directory_iterator.o filesyste
m/int128_builtins.o filesystem/operations.o functional.o future.o hash.o io= s.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o optional.= o random.o random_shuffle.o regex.o shared_mutex.o
stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o utility= .o valarray.o variant.o vector.o cxxrt_auxhelper.o cxxrt_dynamic_cast.o cxx= rt_exception.o cxxrt_guard.o cxxrt_libelftc_dem_g
nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o cxxrt_typeinfo.o CWD /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++=
TARGET libc++.a
-- command output --
building static c++ library

-- filemon acquired metadata --
# filemon version 5
# Target pid 22471
# Start 1611359217.214996
V 5
E 22961 /bin/sh
R 22961 /etc/libmap.conf
R 22961 /var/run/ld-elf.so.hints
R 22961 /lib/libedit.so.7
R 22961 /lib/libc.so.7
R 22961 /lib/libncursesw.so.9
R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE
F 22961 22962
E 22962 /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/le= gacy/usr/sbin/rm
R 22962 /etc/libmap.conf
R 22962 /var/run/ld-elf.so.hints
R 22962 /lib/libc.so.7
R 22962 /usr/share/locale/C.UTF-8/LC_CTYPE
D 22962 libc++.a
X 22962 0 0
. . .

So META_MODE has lots of files that were used
that it can later detect being newer than the
prior build results, leading to rebuilds based
on those newer files.

> > # ./tools/build/beinstall.sh

The new be will have various updated files
that could be different by content and, for
commands, could behave differently than those
used to do the prior build. Detecting newer
time stamps on such used files leads to
rebuild activity.

More than /usr/src/ and /usr/obj/ content
are involved as well.

Note that the new be is based on somewhat
different files than the original
buildworld-jobs buildkernel-jobs was based
on.

> > # reboot
> >

(I presume booting into the new be here.)

> > # cd /usr/src
> > # make buildworld-jobs buildkernel-jobs

META_MODE will notice when commands are used
that are newer than when the prior build was
done. Similarly=C2=A0 for other files that may be
read. It will make sure that the newer commands
and files are allowed to produce new results
that potentially could be distinct in content
from what the old context produced for results.

> >
> > Since src and obj are the same from one BE to newer BE, minimal > > compilation should happen, not a full one.

META_MODE is more careful than that.

Note: I'm not claiming that new behavior that is
needed is likely for lots of the files with new
dates. But META_MODE is biased to avoiding leaving
in place something that should have been updated.

> >
> > Am I missing something here?
> >

Note that make has a -dM option:

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0M=C2=A0 =C2=A0 =C2=A0 =C2= =A0Print debugging information about =E2=80=9Cmeta=E2=80=9D mode decisions<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0about targets.

So, for example,

file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/= legacy/usr/sbin/awk' is newer than the target...
file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/= legacy/usr/sbin/cap_mkdb' is newer than the target...
file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/= legacy/usr/sbin/cat' is newer than the target...
file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/= legacy/usr/sbin/cp' is newer than the target...
file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/= legacy/usr/sbin/crunchgen' is newer than the target...
file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/= legacy/usr/sbin/crunchide' is newer than the target...
file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/= legacy/usr/sbin/dd' is newer than the target...
. . .

Various . . ./tmp/legacy/. . ./*bin/ actually were
links to files.

Ultimately buildworld then installworld lead to new
dates for a bunch of files used. Later use of
META_MODE notices such and rebuilds based on the
newer files. (It is a lot of detail to go through
it all.)

Back in 2021 and 2023 I got help with exploring
avoiding lots of these. But, in the end, it
involved use of experimental code in
share/mk/src.sys.obj.mk to provide a new
definition to use to build some paths with.

The experiments were an unsupported activity that
produced an unsupported change to allow
configurable enabling of taking risks with not
updating files that possibly should be updated.

=3D=3D=3D
Mark Millard
marklmi at yahoo.com



--
Nuno Teixeira
=
FreeBSD UNIX:=C2=A0 <eduardo@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 https://Fr= eeBSD.org
--00000000000000972606346b224b--