From nobody Tue May 6 17:47:32 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4ZsQpR5GwWz5v96X for ; Tue, 06 May 2025 17:47:43 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4ZsQpR4WCDz41L0 for ; Tue, 06 May 2025 17:47:43 +0000 (UTC) (envelope-from eduardo@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1746553663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hurQNVf+CbxuRhxDjYIX8a/Q3hwACN1I/DiKq18xkOU=; b=YrcX4wMzxcjBn4VHmnCVUwCRaMuq7DxI6oPXzidwFW4Yuu3n8ZWqvkEiMRpms23mRJmF/G obXXcgkBLJ+QgJjI8VocXaxfV6liN2Tal7yj/gGPop5kIDlp7ObyY/y7jvMivZpHrPE3M6 3rKdk7JetYeVE5e8rYym1K9gGzqOG6+4/iu9vhMsq6NyPF7Vl14XFZlO6S/zGJCicjnXsf 0gSFpFioUMOojyRoGe9xZI00rZ2rBJUM429V7Rq4B1l8yvwv5HXU8RMSH2oww/+ZF+USs6 /SBi70Zik74F9cV0oszBJZd/LQkaeV4dMeqwwkUoK479RxxMW4W6OqkgIc3Hqg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1746553663; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=hurQNVf+CbxuRhxDjYIX8a/Q3hwACN1I/DiKq18xkOU=; b=emIeCYeBAEkjwOMOlk+DW3FfGZekfALb/n/QeqM1Z7Rsnj51KM2GhGKWy3zc05mEWJVun+ HzK1DlDXim+bH5bDJCqUolk9MrRHp1zG32KEyK1b98lX/3U5fKsce2a2fuyvIlHGnJ6KWb /Imgy3kM8IBj+J/XczhAUrpzny+j3yM/SGRRcqkbnQo0SJCQKhnwK28Augy0tGFllJgvin 8xAFApkkeX52epkjSTLYXu9f7groshXdWhKMU4IPWa+33tWbuduo13WEfH2TJ8qTJtBHRr HQdlwCNaRJ+piHfTQQCWaGLJ9c5s8ETlJYS/gQIKor1fScpD1rc01oYi48SY+A== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1746553663; a=rsa-sha256; cv=none; b=gpBHdZq9ZnAOlxtvqLTUjRGez0vrtoiMAWFWOhl5Pg+I0WiEDBDlk1oS6Vc8mK1q2qjhnq k5f0kcHcrbVd3Q7SQkFCXTyvTL6AFSok5i5TA6KLX1oE17j/M0L3Crh3srjeF0JZLVfwUW IPwfB6wjNgzrxg5tzWM7RZLaGzniGWVQXuayAhIFkbxMOgJWRsT878OVCzrVZ8DoqtV3sS WyEUuTb8AfXn4lsAq/rY5wl276C2tUX4KAcqTsOvSdTommNdWPNnxl1FXdqmie7IhPp13N yrJVDax8Yn2g+Hfhh0AGc/tb5bClmMwdBaXSxZLcPbJFKaZSoGOuz80mUkTUOw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) (Authenticated sender: eduardo) by smtp.freebsd.org (Postfix) with ESMTPSA id 4ZsQpR3lQHzvHr for ; Tue, 06 May 2025 17:47:43 +0000 (UTC) (envelope-from eduardo@freebsd.org) Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7c964e8710eso45083285a.0 for ; Tue, 06 May 2025 10:47:43 -0700 (PDT) X-Gm-Message-State: AOJu0YzQ/0cjVBfzGplbf7uaIHEYbeTc7xhXiI53StPYQXJUpBgWsW5Y 66de0+wXgNIrVAq5KrrYROJX+IcLQLjyssYDIC4ubksY2hSon6pIQ8qhZm0qEi9n1F/KdiptsOf 4rjLetQdl8YIU4oYsBj28us4x11M= X-Google-Smtp-Source: AGHT+IHTz47Ql67pMiwaiug9uvL0tKp9jhJDqHIfSSoP9CnzG5FtN5IEExGtDbkcXYo42O9zh2YaWDu6onB2u2oD9xs= X-Received: by 2002:a05:622a:44c:b0:474:f601:c21e with SMTP id d75a77b69052e-49225a41fedmr306211cf.5.1746553663095; Tue, 06 May 2025 10:47:43 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <28F2BDE7-5903-4C04-A570-6A407F19D5F2.ref@yahoo.com> <28F2BDE7-5903-4C04-A570-6A407F19D5F2@yahoo.com> <5BFEF4D2-0E42-46ED-899B-D27169DFC322@yahoo.com> In-Reply-To: <5BFEF4D2-0E42-46ED-899B-D27169DFC322@yahoo.com> From: Nuno Teixeira Date: Tue, 6 May 2025 18:47:32 +0100 X-Gmail-Original-Message-ID: X-Gm-Features: ATxdqUEWnILb0icAQyIlycTvuhD6_aMf5GEhBKzj6_ZNSZj98v_z-jxxygliOL0 Message-ID: Subject: Re: incremental bulds from scratch with beinstall.sh To: Mark Millard Cc: FreeBSD Current Content-Type: multipart/alternative; boundary="000000000000ab8a0e06347b3674" --000000000000ab8a0e06347b3674 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello! I use CURRENT on my laptop and rpi4 for some time now. I like to see changes in src tree and then rebuild world and test for myself. One of the things that attracts me is building and rebuilding stuff and maybe that's why I'm so happy as a ports committer :) The only "change" that I do in build world is: /etc/src.conf: WITH_MALLOC_PRODUCTION=3Dyes WITHOUT_LLVM_ASSERTIONS=3Dyes that might be incompatible with pkgbase since those settings are for releases. Cheers, Mark Millard escreveu (ter=C3=A7a, 6/05/2025 =C3=A0(s) = 17:52): > On May 6, 2025, at 08:15, Nuno Teixeira wrote: > > > Hello Mark, > > > > Definitely I'm not getting the results I want with WITH_META_MODE using > BEs since most of the times I end up rebuilding almost everything in new = BE. > > > > Should I stop using WITH_META_MODES a go straight to clean builds (clea= n > /usr/obj) instead? > > If by "clean" you mean doing some form of "rm -fr" on all > or part of what is by default somewhere under /usr/ob/ : > You likely would be deleting files that would be > regenerated during a build instead of any of those being > reused. > > May be I've guess wrong about what you mean by "clean > /usr/obj"? > > I'll note that I've never used ccache or the like: > > https://man.freebsd.org/cgi/man.cgi?query=3Dccache&sektion=3D1&n=3D1 > > I expect that there are others that have that might > comment about such setups. > > But going in another direction: I do not know why you are > doing your own builds instead of using pre-built materials, > such as PkgBase's base_latest (updated multiple times per > day, not that you need to do updates from it that often) > or base_weekly (again, you might not update as often). > > Do you have local modifications of the system source that > you are building? (That could mean not using PkgBase > materials, for example.) Just kernel updates? Just > world updates? Both? Neither? > > I also do not know if you build your own packages/ports. So > I do not even known if you need a toolchain. Do you need > to do package/port builds to get security updates sooner > or some such? > > Can you comment on what has to be true of your system > environment(s) for such (even if it in turn means build > times that you do not like have to be involved)? > Delimiting the tradeoff structure requirements might help > in picking a path. > > > A RPi4 specific note: > > One core on the RPi4B executing code for which the L1 > (smallest) level cache is ineffective can saturate its > memory subsystem. Depending on details I've had > experiments in normal system build activity where using > -j3 instead of -j4 or more actually took somewhat less > time overall. (Not that the differences were huge or > anything. And they might not be systematic. But -j3 can > also help if there are also memory usage tradeoffs that > need to be managed. Does the RPi4 have 8 GiBytes of RAM? > 4 GiByte? 2 GiByte?) > > > A note on official PkgBase build use: > > The below illustrates that it is possible to mix > official PkgBase and personal system builds in some > ways. > > Not that it is on RPi4B's, but I have both PkgBase > kernels and my own kernel builds and use > /boot/loader.conf to pick which is the default > for booting. I do not boot kernels that are > older than the boot world on the media. > > I boot a PkgBase world and have directory trees for > chroot or other such use with my personal world > builds, some have my personal package builds > installed and others have official package builds > installed. > > I only use my personal world builds with my personal > kernel builds. (They are matched.) I never use my > personal kernel build when it is older than the > PkgBase kernel or world that I use. This means > my personal kernel build supports booting the > PkgBase based world when I use that kernel. > > Overall I can investigate if any problems I run into > are reproducible without my system or package builds > involved. > > I have various poudriere(-devel) jails, some use > official PkgBase based jails and others are using my > personal builds for the jail content: > > # poudriere jail -l > JAILNAME VERSION OSVERSION ARCH METHOD > TIMESTAMP PATH > release-aarch64 14.2-RELEASE-p1 aarch64 pkgbase > 2025-05-05 22:11:10 /usr/local/poudriere/jails/release-aarch64 > release-armv7 14.2-RELEASE-p2 armv7 pkgbase > 2025-03-13 21:50:17 /usr/local/poudriere/jails/release-armv7 > official-aarch64 14.2-STABLE aarch64 pkgbase > 2025-05-05 22:13:47 /usr/local/poudriere/jails/official-aarch64 > official-armv7 14.2-STABLE armv7 pkgbase > 2025-03-13 21:47:04 /usr/local/poudriere/jails/official-armv7 > main-aarch64 15.0-CURRENT aarch64 pkgbase > 2025-05-05 22:15:30 /usr/local/poudriere/jails/main-aarch64 > main-CA76 15.0-CURRENT arm64.aarch64 null > 2025-02-13 01:35:39 /usr/obj/DESTDIRs/main-CA76-poud > main-CA76-bulk_a 15.0-CURRENT arm64.aarch64 null > 2025-02-13 01:35:39 /usr/obj/DESTDIRs/main-CA76-poud-bulk_a > main-CA7 15.0-CURRENT arm.armv7 null > 2025-02-20 18:16:55 /usr/obj/DESTDIRs/main-CA7-poud > main-CA7-bulk_a 15.0-CURRENT arm.armv7 null > 2025-02-20 18:16:56 /usr/obj/DESTDIRs/main-CA7-poud-bulk_a > main-armv7 15.0-CURRENT armv7 pkgbase > 2025-03-14 22:48:11 /usr/local/poudriere/jails/main-armv7 > > # poudriere jail -l > JAILNAME VERSION OSVERSION ARCH METHOD TIMESTAMP > PATH > release-amd64 14.2-RELEASE-p2 amd64 pkgbase 2025-05-05 > 22:19:47 /usr/local/poudriere/jails/release-amd64 > official-amd64 14.2-STABLE amd64 pkgbase 2025-05-05 > 22:21:46 /usr/local/poudriere/jails/official-amd64 > main-ZNV4 15.0-CURRENT amd64 null 2025-02-12 > 16:03:46 /usr/obj/DESTDIRs/main-ZNV4-poud > main-ZNV4-bulk_a 15.0-CURRENT amd64 null 2025-02-12 > 16:03:46 /usr/obj/DESTDIRs/main-ZNV4-poud-bulk_a > main-ZNV4-dbg 15.0-CURRENT amd64 null 2025-04-02 > 09:20:08 /usr/obj/DESTDIRs/main-ZNV4-poud-dbg > main-amd64 15.0-CURRENT amd64 pkgbase 2025-05-05 > 22:26:12 /usr/local/poudriere/jails/main-amd64 > > (I've never bothered with i386 in amd64 contexts and > have never used FreeBSD on an actual i386 class system.) > > > My first concern is to speed up builds expecially on rpi4. > > Per some prior questions: What other constraints must > also be met? > > > Any hints are welcome. > > > > Thanks, > > > > > > Mark Millard escreveu (segunda, 5/05/2025 =C3=A0(s) > 23:18): > >> Nuno Teixeira wrote on > >> Date: Mon, 05 May 2025 20:37:09 UTC : > >> > >> > (...) > >> > > >> > Don't forget to `env NO_PKG_UPGRADE=3Dyes beinstall.sh` to not mess = with > >> > ports and stuff. > >> > > >> > Nuno Teixeira escreveu (segunda, 5/05/2025 =C3= =A0(s) > >> > 21:34): > >> > > >> > > Hello, > >> > > > >> > > Using incremental WITH_META_MODE builds, after installation with > >> > > beinstall.sh, building same src, a complete compilation happens. > >> > > > >> > > If someone uses this script, could you please do the following tes= t: > >> > > > >> > > WITH_META_MODE=3Dyes (/etc/src-env.conf) > >> > > filemon module loaded > >> > > > >> > > # cd /usr/src > >> > > # make buildworld-jobs buildkernel-jobs > >> > >> The above used older commands and files from before > >> the following install. META_MODE recorded the use of > >> those commands. > >> > >> Example .meta mode file content: > >> > >> # Meta data file > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/l= ibc++.a.meta > >> CMD @echo building static c++ library > >> CMD @rm -f libc++.a > >> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o > charconv.o chrono.o condition_variable.o condition_variable_destructor.o > debug.o exception.o filesystem/directory_iterator.o filesyste > >> m/int128_builtins.o filesystem/operations.o functional.o future.o > hash.o ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.= o > optional.o random.o random_shuffle.o regex.o shared_mutex.o > >> stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o > utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o > cxxrt_dynamic_cast.o cxxrt_exception.o cxxrt_guard.o cxxrt_libelftc_dem_g > >> nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o > cxxrt_typeinfo.o > >> CWD > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++ > >> TARGET libc++.a > >> -- command output -- > >> building static c++ library > >> > >> -- filemon acquired metadata -- > >> # filemon version 5 > >> # Target pid 22471 > >> # Start 1611359217.214996 > >> V 5 > >> E 22961 /bin/sh > >> R 22961 /etc/libmap.conf > >> R 22961 /var/run/ld-elf.so.hints > >> R 22961 /lib/libedit.so.7 > >> R 22961 /lib/libc.so.7 > >> R 22961 /lib/libncursesw.so.9 > >> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE > >> F 22961 22962 > >> E 22962 > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/u= sr/sbin/rm > >> R 22962 /etc/libmap.conf > >> R 22962 /var/run/ld-elf.so.hints > >> R 22962 /lib/libc.so.7 > >> R 22962 /usr/share/locale/C.UTF-8/LC_CTYPE > >> D 22962 libc++.a > >> X 22962 0 0 > >> . . . > >> > >> So META_MODE has lots of files that were used > >> that it can later detect being newer than the > >> prior build results, leading to rebuilds based > >> on those newer files. > >> > >> > > # ./tools/build/beinstall.sh > >> > >> The new be will have various updated files > >> that could be different by content and, for > >> commands, could behave differently than those > >> used to do the prior build. Detecting newer > >> time stamps on such used files leads to > >> rebuild activity. > >> > >> More than /usr/src/ and /usr/obj/ content > >> are involved as well. > >> > >> Note that the new be is based on somewhat > >> different files than the original > >> buildworld-jobs buildkernel-jobs was based > >> on. > >> > >> > > # reboot > >> > > > >> > >> (I presume booting into the new be here.) > >> > >> > > # cd /usr/src > >> > > # make buildworld-jobs buildkernel-jobs > >> > >> META_MODE will notice when commands are used > >> that are newer than when the prior build was > >> done. Similarly for other files that may be > >> read. It will make sure that the newer commands > >> and files are allowed to produce new results > >> that potentially could be distinct in content > >> from what the old context produced for results. > >> > >> > > > >> > > Since src and obj are the same from one BE to newer BE, minimal > >> > > compilation should happen, not a full one. > >> > >> META_MODE is more careful than that. > >> > >> Note: I'm not claiming that new behavior that is > >> needed is likely for lots of the files with new > >> dates. But META_MODE is biased to avoiding leaving > >> in place something that should have been updated. > >> > >> > > > >> > > Am I missing something here? > >> > > > >> > >> Note that make has a -dM option: > >> > >> M Print debugging information about =E2=80=9Cmeta= =E2=80=9D mode > decisions > >> about targets. > >> > >> So, for example, > >> > >> file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/awk' > is newer than the target... > >> file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/cap_mkdb' > is newer than the target... > >> file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/cat' > is newer than the target... > >> file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/cp' > is newer than the target... > >> file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/crunchgen' > is newer than the target... > >> file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/crunchide' > is newer than the target... > >> file > '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= usr/sbin/dd' > is newer than the target... > >> . . . > >> > >> Various . . ./tmp/legacy/. . ./*bin/ actually were > >> links to files. > >> > >> Ultimately buildworld then installworld lead to new > >> dates for a bunch of files used. Later use of > >> META_MODE notices such and rebuilds based on the > >> newer files. (It is a lot of detail to go through > >> it all.) > >> > >> Back in 2021 and 2023 I got help with exploring > >> avoiding lots of these. But, in the end, it > >> involved use of experimental code in > >> share/mk/src.sys.obj.mk to provide a new > >> definition to use to build some paths with. > >> > >> The experiments were an unsupported activity that > >> produced an unsupported change to allow > >> configurable enabling of taking risks with not > >> updating files that possibly should be updated. > >> > > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --=20 Nuno Teixeira FreeBSD UNIX: Web: https://FreeBSD.org --000000000000ab8a0e06347b3674 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello!

I use CURRENT on my lap= top and rpi4 for some time now.
I like to see changes in src tree = and then rebuild world and test for myself.
One of the things that att= racts me is building and rebuilding stuff and maybe that's why I'm = so happy as a ports committer :)

The only "chan= ge" that I do in build world is:
/etc/src.conf:
WITH_MALLOC_PRODUCTION=3Dyes
WITHOUT_LLVM_ASSERTIONS=3Dyes
t= hat might be incompatible with pkgbase since those settings are for release= s.

Cheers,

Mark Millard= <marklmi@yahoo.com> escreve= u (ter=C3=A7a, 6/05/2025 =C3=A0(s) 17:52):
On May 6, 2025, at 08:15, Nuno Teixeira <eduardo@freebsd.org> wrote:

> Hello Mark,
>
> Definitely I'm not getting the results I want with WITH_META_MODE = using BEs since most of the times I end up rebuilding almost everything in = new BE.
>
> Should I stop using WITH_META_MODES a go straight to clean builds (cle= an /usr/obj) instead?

If by "clean" you mean doing some form of "rm -fr" on a= ll
or part of what is by default somewhere under /usr/ob/ :
You likely would be deleting files that would be
regenerated during a build instead of any of those being
reused.

May be I've guess wrong about what you mean by "clean
/usr/obj"?

I'll note that I've never used ccache or the like:

https://man.freebsd.or= g/cgi/man.cgi?query=3Dccache&sektion=3D1&n=3D1

I expect that there are others that have that might
comment about such setups.

But going in another direction: I do not know why you are
doing your own builds instead of using pre-built materials,
such as PkgBase's base_latest (updated multiple times per
day, not that you need to do updates from it that often)
or base_weekly (again, you might not update as often).

Do you have local modifications of the system source that
you are building? (That could mean not using PkgBase
materials, for example.) Just kernel updates? Just
world updates? Both? Neither?

I also do not know if you build your own packages/ports. So
I do not even known if you need a toolchain. Do you need
to do package/port builds to get security updates sooner
or some such?

Can you comment on what has to be true of your system
environment(s) for such (even if it in turn means build
times that you do not like have to be involved)?
Delimiting the tradeoff structure requirements might help
in picking a path.


A RPi4 specific note:

One core on the RPi4B executing code for which the L1
(smallest) level cache is ineffective can saturate its
memory subsystem. Depending on details I've had
experiments in normal system build activity where using
-j3 instead of -j4 or more actually took somewhat less
time overall. (Not that the differences were huge or
anything. And they might not be systematic. But -j3 can
also help if there are also memory usage tradeoffs that
need to be managed. Does the RPi4 have 8 GiBytes of RAM?
4 GiByte? 2 GiByte?)


A note on official PkgBase build use:

The below illustrates that it is possible to mix
official PkgBase and personal system builds in some
ways.

Not that it is on RPi4B's, but I have both PkgBase
kernels and my own kernel builds and use
/boot/loader.conf to pick which is the default
for booting. I do not boot kernels that are
older than the boot world on the media.

I boot a PkgBase world and have directory trees for
chroot or other such use with my personal world
builds, some have my personal package builds
installed and others have official package builds
installed.

I only use my personal world builds with my personal
kernel builds. (They are matched.) I never use my
personal kernel build when it is older than the
PkgBase kernel or world that I use. This means
my personal kernel build supports booting the
PkgBase based world when I use that kernel.

Overall I can investigate if any problems I run into
are reproducible without my system or package builds
involved.

I have various poudriere(-devel) jails, some use
official PkgBase based jails and others are using my
personal builds for the jail content:

# poudriere jail -l
JAILNAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0VERSION=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0OSVERSION ARCH=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 METHOD=C2=A0 TIM= ESTAMP=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0PATH
release-aarch64=C2=A0 14.2-RELEASE-p1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0aarch64=C2=A0 =C2=A0 =C2=A0 =C2=A0pkgbase 2025-05-05 22:11:10 /usr/local= /poudriere/jails/release-aarch64
release-armv7=C2=A0 =C2=A0 14.2-RELEASE-p2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0armv7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pkgbase 2025-03-13 21:50:1= 7 /usr/local/poudriere/jails/release-armv7
official-aarch64 14.2-STABLE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0aarch64=C2=A0 =C2=A0 =C2=A0 =C2=A0pkgbase 2025-05-05 22:13:47 /us= r/local/poudriere/jails/official-aarch64
official-armv7=C2=A0 =C2=A014.2-STABLE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0armv7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pkgbase 2025-03-= 13 21:47:04 /usr/local/poudriere/jails/official-armv7
main-aarch64=C2=A0 =C2=A0 =C2=A015.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 aarch64=C2=A0 =C2=A0 =C2=A0 =C2=A0pkgbase 2025-05-05 22:1= 5:30 /usr/local/poudriere/jails/main-aarch64
main-CA76=C2=A0 =C2=A0 =C2=A0 =C2=A0 15.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 arm64.aarch64 null=C2=A0 =C2=A0 2025-02-13 01:35:3= 9 /usr/obj/DESTDIRs/main-CA76-poud
main-CA76-bulk_a 15.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 arm64.aarch64 null=C2=A0 =C2=A0 2025-02-13 01:35:39 /usr/obj/DESTDIRs/m= ain-CA76-poud-bulk_a
main-CA7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A015.0-CURRENT=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 arm.armv7=C2=A0 =C2=A0 =C2=A0null=C2=A0 =C2=A0 = 2025-02-20 18:16:55 /usr/obj/DESTDIRs/main-CA7-poud
main-CA7-bulk_a=C2=A0 15.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 arm.armv7=C2=A0 =C2=A0 =C2=A0null=C2=A0 =C2=A0 2025-02-20 18:16:56 = /usr/obj/DESTDIRs/main-CA7-poud-bulk_a
main-armv7=C2=A0 =C2=A0 =C2=A0 =C2=A015.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 armv7=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pkgbase 202= 5-03-14 22:48:11 /usr/local/poudriere/jails/main-armv7

# poudriere jail -l
JAILNAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0VERSION=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0OSVERSION ARCH=C2=A0 METHOD=C2=A0 TIMESTAMP=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0PATH
release-amd64=C2=A0 =C2=A0 14.2-RELEASE-p2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0amd64 pkgbase 2025-05-05 22:19:47 /usr/local/poudriere/jails/rele= ase-amd64
official-amd64=C2=A0 =C2=A014.2-STABLE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0amd64 pkgbase 2025-05-05 22:21:46 /usr/local/poudriere/= jails/official-amd64
main-ZNV4=C2=A0 =C2=A0 =C2=A0 =C2=A0 15.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 amd64 null=C2=A0 =C2=A0 2025-02-12 16:03:46 /usr/o= bj/DESTDIRs/main-ZNV4-poud
main-ZNV4-bulk_a 15.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 amd64 null=C2=A0 =C2=A0 2025-02-12 16:03:46 /usr/obj/DESTDIRs/main-ZNV4= -poud-bulk_a
main-ZNV4-dbg=C2=A0 =C2=A0 15.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 amd64 null=C2=A0 =C2=A0 2025-04-02 09:20:08 /usr/obj/DESTDIRs= /main-ZNV4-poud-dbg
main-amd64=C2=A0 =C2=A0 =C2=A0 =C2=A015.0-CURRENT=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 amd64 pkgbase 2025-05-05 22:26:12 /usr/local/poudr= iere/jails/main-amd64

(I've never bothered with i386 in amd64 contexts and
have never used FreeBSD on an actual i386 class system.)

> My first concern is to speed up builds expecially on rpi4.

Per some prior questions: What other constraints must
also be met?

> Any hints are welcome.
>
> Thanks,
>
>
> Mark Millard <marklmi@yahoo.com> escreveu (segunda, 5/05/2025 =C3=A0(s) 23:18):<= br> >> Nuno Teixeira <eduardo_at_freebsd.org> wrote on
>> Date: Mon, 05 May 2025 20:37:09 UTC :
>>
>> > (...)
>> >
>> > Don't forget to `env NO_PKG_UPGRADE=3Dyes beinstall.sh` t= o not mess with
>> > ports and stuff.
>> >
>> > Nuno Teixeira <eduardo@freebsd.org> escreveu (segunda, 5/05/2025 =C3= =A0(s)
>> > 21:34):
>> >
>> > > Hello,
>> > >
>> > > Using incremental WITH_META_MODE builds, after installat= ion with
>> > > beinstall.sh, building same src, a complete compilation = happens.
>> > >
>> > > If someone uses this script, could you please do the fol= lowing test:
>> > >
>> > > WITH_META_MODE=3Dyes (/etc/src-env.conf)
>> > > filemon module loaded
>> > >
>> > > # cd /usr/src
>> > > # make buildworld-jobs buildkernel-jobs
>>
>> The above used older commands and files from before
>> the following install. META_MODE recorded the use of
>> those commands.
>>
>> Example .meta mode file content:
>>
>> # Meta data file /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/= amd64.amd64/lib/libc++/libc++.a.meta
>> CMD @echo building static c++ library
>> CMD @rm -f libc++.a
>> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o = charconv.o chrono.o condition_variable.o condition_variable_destructor.o de= bug.o exception.o filesystem/directory_iterator.o filesyste
>> m/int128_builtins.o filesystem/operations.o functional.o future.o = hash.o ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o = optional.o random.o random_shuffle.o regex.o shared_mutex.o
>> stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.= o utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o cxxrt_dynamic_c= ast.o cxxrt_exception.o cxxrt_guard.o cxxrt_libelftc_dem_g
>> nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o cxxrt_typ= einfo.o
>> CWD /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/l= ib/libc++
>> TARGET libc++.a
>> -- command output --
>> building static c++ library
>>
>> -- filemon acquired metadata --
>> # filemon version 5
>> # Target pid 22471
>> # Start 1611359217.214996
>> V 5
>> E 22961 /bin/sh
>> R 22961 /etc/libmap.conf
>> R 22961 /var/run/ld-elf.so.hints
>> R 22961 /lib/libedit.so.7
>> R 22961 /lib/libc.so.7
>> R 22961 /lib/libncursesw.so.9
>> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE
>> F 22961 22962
>> E 22962 /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd= 64/tmp/legacy/usr/sbin/rm
>> R 22962 /etc/libmap.conf
>> R 22962 /var/run/ld-elf.so.hints
>> R 22962 /lib/libc.so.7
>> R 22962 /usr/share/locale/C.UTF-8/LC_CTYPE
>> D 22962 libc++.a
>> X 22962 0 0
>> . . .
>>
>> So META_MODE has lots of files that were used
>> that it can later detect being newer than the
>> prior build results, leading to rebuilds based
>> on those newer files.
>>
>> > > # ./tools/build/beinstall.sh
>>
>> The new be will have various updated files
>> that could be different by content and, for
>> commands, could behave differently than those
>> used to do the prior build. Detecting newer
>> time stamps on such used files leads to
>> rebuild activity.
>>
>> More than /usr/src/ and /usr/obj/ content
>> are involved as well.
>>
>> Note that the new be is based on somewhat
>> different files than the original
>> buildworld-jobs buildkernel-jobs was based
>> on.
>>
>> > > # reboot
>> > >
>>
>> (I presume booting into the new be here.)
>>
>> > > # cd /usr/src
>> > > # make buildworld-jobs buildkernel-jobs
>>
>> META_MODE will notice when commands are used
>> that are newer than when the prior build was
>> done. Similarly=C2=A0 for other files that may be
>> read. It will make sure that the newer commands
>> and files are allowed to produce new results
>> that potentially could be distinct in content
>> from what the old context produced for results.
>>
>> > >
>> > > Since src and obj are the same from one BE to newer BE, = minimal
>> > > compilation should happen, not a full one.
>>
>> META_MODE is more careful than that.
>>
>> Note: I'm not claiming that new behavior that is
>> needed is likely for lots of the files with new
>> dates. But META_MODE is biased to avoiding leaving
>> in place something that should have been updated.
>>
>> > >
>> > > Am I missing something here?
>> > >
>>
>> Note that make has a -dM option:
>>
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 M=C2=A0 =C2=A0 =C2= =A0 =C2=A0Print debugging information about =E2=80=9Cmeta=E2=80=9D mode dec= isions
>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 about targets.
>>
>> So, for example,
>>
>> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.a= md64/tmp/legacy/usr/sbin/awk' is newer than the target...
>> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.a= md64/tmp/legacy/usr/sbin/cap_mkdb' is newer than the target...
>> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.a= md64/tmp/legacy/usr/sbin/cat' is newer than the target...
>> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.a= md64/tmp/legacy/usr/sbin/cp' is newer than the target...
>> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.a= md64/tmp/legacy/usr/sbin/crunchgen' is newer than the target...
>> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.a= md64/tmp/legacy/usr/sbin/crunchide' is newer than the target...
>> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.a= md64/tmp/legacy/usr/sbin/dd' is newer than the target...
>> . . .
>>
>> Various . . ./tmp/legacy/. . ./*bin/ actually were
>> links to files.
>>
>> Ultimately buildworld then installworld lead to new
>> dates for a bunch of files used. Later use of
>> META_MODE notices such and rebuilds based on the
>> newer files. (It is a lot of detail to go through
>> it all.)
>>
>> Back in 2021 and 2023 I got help with exploring
>> avoiding lots of these. But, in the end, it
>> involved use of experimental code in
>> share/mk/src.sys.obj.mk to provide a new
>> definition to use to build some paths with.
>>
>> The experiments were an unsupported activity that
>> produced an unsupported change to allow
>> configurable enabling of taking risks with not
>> updating files that possibly should be updated.
>>
>
=3D=3D=3D
Mark Millard
marklmi at yahoo.com



--
Nuno Teixeira
=
FreeBSD UNIX:=C2=A0 <eduardo@FreeBSD.org>=C2=A0 =C2=A0Web:=C2=A0 https://Fr= eeBSD.org
--000000000000ab8a0e06347b3674--