Date: Wed, 27 Jan 2021 17:33:02 -0800 From: "Simon J. Gerraty" <sjg@juniper.net> To: Mark Millard <marklmi@yahoo.com> Cc: Bryan Drewery <bdrewery@FreeBSD.org>, Current FreeBSD <freebsd-current@FreeBSD.org>, <sjg@juniper.net> Subject: Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) Message-ID: <73088.1611797582@kaos.jnpr.net> In-Reply-To: <DB0C7B41-2101-4C5C-BFC8-3C95CC0B9F6F@yahoo.com> References: <B74790D9-FBC2-4818-BEAF-34E5B705C460@yahoo.com> <3345EBA5-A09C-4E3F-B94D-39F57F56BDBB@yahoo.com> <DB0C7B41-2101-4C5C-BFC8-3C95CC0B9F6F@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Mark Millard via freebsd-current <freebsd-current@FreeBSD.org> wrote: > >> Given an already built, installed and booted system version, I've > >> noted a big difference for META_MODE in 2 rebuild contexts (no > >> source updates involved): > >> > >> *) Prior buildworld buildkernel, installkernel, installworld, boot > >> presumed before (A) and before (B) below. Yes installworld is going to cause problems unless you tell meta mode to ignore everything outside of the src/obj tree. But even that isn't enough as you note below. > >> So I used make -dM for the commented buildworld buildkernel lines, lo= gging > >> the build output and later diff'ing them. > >> > >> Result that I noticed? Lots of lines uniquely from (B)'s case, ending= with > >> one of: > >> > >> file '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tm= p/legacy/usr/sbin/awk' is newer than the target... Yes. That would be expected. I think Bryan added some knobs to mitigate some of this. grep -n META.*IGNORE share/mk/*mk might be instructive. > >> Many/most/all(?) of these seem to me to be unlikely to actually need = to > >> contribute to what needs to be rebuilt (just based on being newer). S= o > >> the option to ignore (some of?) them could be useful in making META_M= ODE > >> builds quicker. Yes on all counts. That's why bmake provides a number of knobs to ignore such things. They are listed in the man page in increasing order of expense. One of the risks of the sort of introspection meta mode does, is delving too deep (like the dwarfs ;-) The .MAKE.META.IGNORE_* and .MAKE.META.BAILIWICK variables help to contain the potential insanity. > > The following from one of the .meta files makes the point that rm use > > in the example is unlikely to be important to needing to rebuild, > > despite it actually causing a file rebuild. Nor is the specific echo > > command listed relevant. Only the "ar" command is: > > > > # Meta data file /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd6= 4.amd64/lib/libc++/libc++.a.meta > > CMD @echo building static c++ library > > CMD @rm -f libc++.a > > CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o > > -- filemon acquired metadata -- > > # filemon version 5 > > # Target pid 22471 > > # Start 1611359217.214996 > > V 5 > > E 22961 /bin/sh > > R 22961 /etc/libmap.conf > > R 22961 /var/run/ld-elf.so.hints > > R 22961 /lib/libedit.so.7 > > R 22961 /lib/libc.so.7 > > R 22961 /lib/libncursesw.so.9 > > R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE > > F 22961 22962 > > E 22962 /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/t= mp/legacy/usr/sbin/rm > > The timestamp on . . ./tmp/legacy/usr/sbin/rm is not > > actually relevant to if libc++.a needs to be rebuilt. True. = If there is nothing under .../tmp/legacy that should be counted you can just: .MAKE.META_IGNORE_PATHS +=3D that path which would be the cheapest option. If however there are things that should be considered you'd have to use a much more specific list of .MAKE.META_IGNORE_PATHS or perhaps something like: .MAKE.META.IGNORE_PATTERNS +=3D */rm would ignore an rm binary regardless of where found. worst case you might need something like: # realpath .MAKE.META.IGNORE_FILTER +=3D tA # ignore anything here .MAKE.META.IGNORE_FILTER +=3D N*/tmp/legacy/usr/bin/* Unlike .MAKE.META.IGNORE_PATTERNS which is a list of patterns to match the goal with .MAKE.META.IGNORE_FILTER is to reduced to an empty string paths to be ignored. > > Of course, the structure also point out the judgment > > is specific to understanding the sequence of CMD's > > listed above. Only a hack of ignoring, not recording, > > or commenting out the filemon lines ending in > > /tmp/legacy/usr/sbin/rm would seem to avoid the @rm > > handling issue. Such might well have its own risks. No hacking needed, there are a range of knobs to help. > Just for reference for more about the sequencing involved: > = > Looks like in my example various . . ./tmp/legacy/. . ./*bin/ > actually are links to files in: > = > /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/= bin/ > = > and the later after-install buildworld "Rebuilding the > temporary build tree" step leads to the updated dates for > files in that area, updated via the code that reports: > = > Linking host tools into /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src= /amd64.amd64/tmp/legacy/bin > > So the prior installworld leaves updated dates and the > linking to those installed files makes > . . ./tmp/legacy/. . ./*bin/* have the newer dates show > up for the legacy paths as well. > = > In turn the dependency tracking via META_MODE uses the new > dates on . . ./tmp/legacy/. . ./*bin/* files to cause > rebuilds of more materials than if installworld had not > been done. > = > It is not obvious if Bryan D. would find the effort to avoid > this worthwhile for the contexts that drive FreeBSD's build > environment choices. For people that use installworld and also want META_MODE it would seem some additional IGNORE work may be beneficial. --sjg
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?73088.1611797582>