Date: Wed, 22 Feb 2023 15:19:41 -0800 From: Mark Millard <marklmi@yahoo.com> To: "Simon J. Gerraty" <sjg@juniper.net> Cc: Bryan Drewery <bdrewery@FreeBSD.org>, Current FreeBSD <freebsd-current@FreeBSD.org>, Peter <pmc@citylink.dinoex.sub.org> Subject: Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) Message-ID: <F6BF110D-7855-4A10-A53F-52B34282234F@yahoo.com> In-Reply-To: <CB7040D0-3BF4-496F-A54F-87E5378016E0@yahoo.com> References: <B74790D9-FBC2-4818-BEAF-34E5B705C460@yahoo.com> <3345EBA5-A09C-4E3F-B94D-39F57F56BDBB@yahoo.com> <DB0C7B41-2101-4C5C-BFC8-3C95CC0B9F6F@yahoo.com> <73088.1611797582@kaos.jnpr.net> <CB7040D0-3BF4-496F-A54F-87E5378016E0@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[Added a question about a possible typo in the old original message.]
On Feb 21, 2023, at 13:31, Mark Millard <marklmi@yahoo.com> wrote:
> [This will be a question relative to the old material below.]
>=20
> On Jan 27, 2021, at 17:33, Simon J. Gerraty <sjg@juniper.net> wrote:
>=20
>> Mark Millard via freebsd-current <freebsd-current@FreeBSD.org> wrote:
>>>>> Given an already built, installed and booted system version, I've
>>>>> noted a big difference for META_MODE in 2 rebuild contexts (no
>>>>> source updates involved):
>>>>>=20
>>>>> *) Prior buildworld buildkernel, installkernel, installworld, boot
>>>>> presumed before (A) and before (B) below.
>>=20
>> Yes installworld is going to cause problems unless you tell meta mode =
to
>> ignore everything outside of the src/obj tree.
>> But even that isn't enough as you note below.
>>=20
>>>>> So I used make -dM for the commented buildworld buildkernel lines, =
logging
>>>>> the build output and later diff'ing them.
>>>>>=20
>>>>> Result that I noticed? Lots of lines uniquely from (B)'s case, =
ending with
>>>>> one of:
>>>>>=20
>>>>> file =
'/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/u=
sr/sbin/awk' is newer than the target...
>>=20
>> Yes. That would be expected.
>> I think Bryan added some knobs to mitigate some of this.
>>=20
>> grep -n META.*IGNORE share/mk/*mk
>>=20
>> might be instructive.
>>=20
>>>>> Many/most/all(?) of these seem to me to be unlikely to actually =
need to
>>>>> contribute to what needs to be rebuilt (just based on being =
newer). So
>>>>> the option to ignore (some of?) them could be useful in making =
META_MODE
>>>>> builds quicker.
>>=20
>> Yes on all counts. That's why bmake provides a number of knobs to
>> ignore such things.
>> They are listed in the man page in increasing order of expense.
>>=20
>> One of the risks of the sort of introspection meta mode does, is =
delving
>> too deep (like the dwarfs ;-)
>>=20
>> The .MAKE.META.IGNORE_* and .MAKE.META.BAILIWICK variables help to
>> contain the potential insanity.
>>=20
>>>> The following from one of the .meta files makes the point that rm =
use
>>>> in the example is unlikely to be important to needing to rebuild,
>>>> despite it actually causing a file rebuild. Nor is the specific =
echo
>>>> command listed relevant. Only the "ar" command is:
>>>>=20
>>>> # Meta data file =
/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/li=
bc++.a.meta
>>>> CMD @echo building static c++ library
>>>> CMD @rm -f libc++.a
>>>> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o
>>=20
>>>> -- filemon acquired metadata --
>>>> # filemon version 5
>>>> # Target pid 22471
>>>> # Start 1611359217.214996
>>>> V 5
>>>> E 22961 /bin/sh
>>>> R 22961 /etc/libmap.conf
>>>> R 22961 /var/run/ld-elf.so.hints
>>>> R 22961 /lib/libedit.so.7
>>>> R 22961 /lib/libc.so.7
>>>> R 22961 /lib/libncursesw.so.9
>>>> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE
>>>> F 22961 22962
>>>> E 22962 =
/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/us=
r/sbin/rm
>>=20
>>>> The timestamp on . . ./tmp/legacy/usr/sbin/rm is not
>>>> actually relevant to if libc++.a needs to be rebuilt.
>>=20
>> True.=20
>> If there is nothing under .../tmp/legacy that should be counted you =
can
>> just:
>>=20
>> .MAKE.META_IGNORE_PATHS +=3D that path
Was that supposed to be ("." vs. "_"):
.MAKE.META.IGNORE_PATHS +=3D that path
The later ones listed have ".".
>> which would be the cheapest option.
>>=20
>> If however there are things that should be considered you'd have to
>> use a much more specific list of .MAKE.META_IGNORE_PATHS or
(Same question applies to the above.)
>> perhaps something like:
>>=20
>> .MAKE.META.IGNORE_PATTERNS +=3D */rm
>>=20
>> would ignore an rm binary regardless of where found.
>>=20
>> worst case you might need something like:
>>=20
>> # realpath
>> .MAKE.META.IGNORE_FILTER +=3D tA
>> # ignore anything here
>> .MAKE.META.IGNORE_FILTER +=3D N*/tmp/legacy/usr/bin/*
>>=20
>> Unlike .MAKE.META.IGNORE_PATTERNS which is a list of patterns to =
match
>> the goal with .MAKE.META.IGNORE_FILTER is to reduced to an empty =
string
>> paths to be ignored.
>>=20
>>>> Of course, the structure also point out the judgment
>>>> is specific to understanding the sequence of CMD's
>>>> listed above. Only a hack of ignoring, not recording,
>>>> or commenting out the filemon lines ending in
>>>> /tmp/legacy/usr/sbin/rm would seem to avoid the @rm
>>>> handling issue. Such might well have its own risks.
>>=20
>> No hacking needed, there are a range of knobs to help.
>>=20
>>> Just for reference for more about the sequencing involved:
>>>=20
>>> Looks like in my example various . . ./tmp/legacy/. . ./*bin/
>>> actually are links to files in:
>>>=20
>>> =
/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bi=
n/
>>>=20
>>> and the later after-install buildworld "Rebuilding the
>>> temporary build tree" step leads to the updated dates for
>>> files in that area, updated via the code that reports:
>>>=20
>>> Linking host tools into =
/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bi=
n
>>>=20
>>> So the prior installworld leaves updated dates and the
>>> linking to those installed files makes
>>> . . ./tmp/legacy/. . ./*bin/* have the newer dates show
>>> up for the legacy paths as well.
>>>=20
>>> In turn the dependency tracking via META_MODE uses the new
>>> dates on . . ./tmp/legacy/. . ./*bin/* files to cause
>>> rebuilds of more materials than if installworld had not
>>> been done.
>>>=20
>>> It is not obvious if Bryan D. would find the effort to avoid
>>> this worthwhile for the contexts that drive FreeBSD's build
>>> environment choices.
>>=20
>> For people that use installworld and also want META_MODE
>> it would seem some additional IGNORE work may be beneficial.
>=20
> Since some of the paths reported ended up being links
> (symbolic, as I remember), what are the principles for
> which form of paths should be the basis for paths in
> the likes of:
>=20
> .MAKE.META_IGNORE_PATHS
(The above may be a typo relative to .MAKE.META.IGNORE_PATHS .)
> .MAKE.META.IGNORE_PATTERNS
> .MAKE.META.IGNORE_FILTER
>=20
> Target of link? Path to the link itself, not the
> target? Both?
>=20
> (Something had interrupted my explorations back then and
> I'd forgotten about your note and have never done the
> exploration. Trying to partially answer a question on the
> lists lead to me reviewing the old thread and running into
> your note again.)
>=20
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F6BF110D-7855-4A10-A53F-52B34282234F>