Date: Sat, 23 Jan 2021 22:01:07 -0800 From: Mark Millard <marklmi@yahoo.com> To: bob prohaska <fbsd@www.zefox.net> Cc: freebsd-arm <freebsd-arm@freebsd.org> Subject: Re: Needless work in buildworld Message-ID: <EDF9F631-FF55-444C-BAB3-3037CEA61D65@yahoo.com> In-Reply-To: <20210123173754.GA83834@www.zefox.net> References: <A6150F56-062F-4582-853A-319C1EE4DDCB@yahoo.com> <20210121023358.GA58854@www.zefox.net> <8D0C2A4C-B616-47B9-864E-D846A6EBA3D6@yahoo.com> <20210122011535.GA66611@www.zefox.net> <655C6BAA-8B10-4130-A5C9-EDED6906207D@yahoo.com> <C502D267-5158-4658-BB02-EC67835015D3@yahoo.com> <36A2E015-78DF-40AB-BF53-FB3D26FA5AAC@yahoo.com> <20210122224656.GA76907@www.zefox.net> <2903491E-7DE7-4F17-B515-120BA447B8B3@yahoo.com> <A982C6D4-6123-40FA-8592-A3CD7B0D8E10@yahoo.com> <20210123173754.GA83834@www.zefox.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2021-Jan-23, at 09:37, bob prohaska <fbsd at www.zefox.net> wrote: > On Fri, Jan 22, 2021 at 10:20:53PM -0800, Mark Millard wrote: >>=20 >> In things such as (abbreviated): >>=20 >> file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target... >>=20 >> The "target" is the object file or whatever that might need to >> be replaced, not the . . ./tmp/legacy/usr/sbin/???. The >> examples of such lines listed the target file path before >> the word "file" above. >>=20 >> I only showed one line with the common suffix text, no hint of >> the prior text on the lines summarized. >>=20 >> So the targets were timestamped at the prior build's creation >> of them. >>=20 >> It is the installworld installkernel sorts of activity that ends >> up with . . ./tmp/legacy/usr/sbin/awk and the like having time >> stamps from the new build's time frame. But the META_MODE >> records indicate that . . ./tmp/legacy/usr/sbin/awk (or whatever) >> was involved in generating the target in the prior build so the >> updated timestamp leads to questions of needing regeneration of >> the target. But a . . ./tmp/legacy/. . ./??? generally is not >> likely to cause such a need, even with new timestamps. >>=20 >>=20 >>>> If so, would some sort of checksum be able >>>> to distinguish meaningful changes? Would it be computationally = worthwhile? >>=20 >> I leave the overall issue to Bryan Drewery to decide if there is >> something to be done or not. >>=20 > My thinking regarding checksums was absolutely wrong; I wanted to = compare > files that hadn't been created.....=20 >=20 > In deciding whether to recompile, say, /usr/bin/awk, Note that, in my example, . . ./tmp/legacy/usr/sbin/awk is not what ended up being rebuilt (at least at the time of the "newer than the target" message). I did not show a list of what was being rebuilt because of . . ./tmp/legacy/usr/sbin/awk being newer. I'll show some examples below. The .meta file for the rebuilt target is what has the "newer than the target" message, reporting what was newer that was sufficient to cause the rebuild. So a partial list is (abbreviated paths): . . ./tmp/obj-tools/kerberos5/tools/make-roken/make-roken.c.meta . . ./lib/ncurses/ncurses/names.c.meta . . ./lib/ncurses/ncurses/unctrl.c.meta . . ./lib/ncurses/ncurses/codes.c.meta . . ./lib/libthread_db/Version.map.meta . . ./obj-lib32/lib/libstdthreads/Version.map.meta . . ./stand/efi/libefi/teken_state.h.meta . . ./usr.bin/getconf/limits.c.meta . . ./rescue/rescue/usr/fbsd/mm-src/sbin/route/keywords.h.meta . . ./sys/GENERIC-NODBG/vnode_if_newproto.h.meta . . ./sys/GENERIC-NODBG/iwm8265fw.c.meta . . = ./sys/GENERIC-NODBG/modules/usr/fbsd/mm-src/sys/modules/aac/device_if.h.me= ta The full list is much larger. There are even lots of files that list . . ./tmp/legacy/usr/sbin/rm as "newer than target", including: . . ./tmp/obj-tools/tools/build/libegacy.a.meta . . ./tmp/obj-tools/lib/libopenbsd/libopenbsd.a.meta . . ./tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a.meta . . ./bin/csh/gethost.meta . . ./rescue/rescue/usr/fbsd/mm-src/bin/csh/gethost.meta . . ./include/compat.meta . . ./lib/libc_nonshared/libc_nonshared.a.meta . . ./lib/libcxxrt/libcxxrt.a.meta . . ./lib/libcxxrt/libcxxrt.so.1.full.meta . . ./cddl/lib/libspl/libspl.a.meta . . ./lib/libthr/libthr.a.meta . . ./lib/libc++/libc++.a.meta The full list is much larger. The following from one of the .meta files makes the point that . . ./tmp/legacy/usr/sbin/rm is unlikely to be important. Nor i the echo relevant. Only the "ar" command is. # Meta data file = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/li= bc++.a.meta CMD @echo building static c++ library CMD @rm -f libc++.a CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o = charconv.o chrono.o condition_variable.o condition_variable_destructor.o = debug.o exception.o filesystem/directory_iterator.o filesyste m/int128_builtins.o filesystem/operations.o functional.o future.o hash.o = ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o = optional.o random.o random_shuffle.o regex.o shared_mutex.o stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o = utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o = cxxrt_dynamic_cast.o cxxrt_exception.o cxxrt_guard.o = cxxrt_libelftc_dem_g nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o = cxxrt_typeinfo.o CWD = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++ TARGET libc++.a -- command output -- building static c++ library -- filemon acquired metadata -- # filemon version 5 # Target pid 22471 # Start 1611359217.214996 V 5 E 22961 /bin/sh R 22961 /etc/libmap.conf R 22961 /var/run/ld-elf.so.hints R 22961 /lib/libedit.so.7 R 22961 /lib/libc.so.7 R 22961 /lib/libncursesw.so.9 R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE F 22961 22962 E 22962 = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/us= r/sbin/rm . . . The vintage of . . ./tmp/legacy/usr/sbin/rm is not going to change the content of: /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++ in the above 3-"CMD" sequence. > one must compare the=20 > creation date of the installed /usr/bin/awk to every file it depends = on.=20 > If any of those files is newer, the whole chain must be recompiled. The dependency tracking builds up to the final things being built, not from the final things backwards. It frequently is the case that only some of the chain required for a final thing is rebuilt. As an example: A final link may reference a mix of changed and unchanged material, for example. But that final link is required if any of its inputs changed. Unnecessary changes to inputs might lead to a relink that otherwise would not be needed. > I > can readily see how this would avalanche, but it seems necessary. Have=20= > I got at least that part right?=20 The "avalanche" reference and implication fits my alternate wording as well as yours and so is good, whatever the other details of the avalanche structure may be. >>> The files (programs) were used during the activity that generated = the prior >>> build of the target. The worry is that the updated programs might = have >>> differing results from older ones and so the new timestamps lead to >>> rebuilding. The worry is just unlikely to be an actual problem for = many of >>> the particular programs. >>>=20 >>> It would be good if META_MODE could ignore those programs that are = in the >>> legacy area and are unlikely to cause the output to vary in some >>> significant way. >>>=20 >=20 > Is this to say that META_MODE is checking outside the dependency chain > (web?), reacting to files changed but not to be used again? I'm not sure of the reference/context-intended here. See the prior libc++.a example above for an example in which the timestamp on . . ./tmp/legacy/usr/sbin/rm is irrelevant to if the libc++.a actually needs to be rebuilt. >>>> More broadly, I've been surprised to see lots of files associated = with foreign >>>> architechtures reported in self-hosting buildworld logs on the = Pi2/3. Things >>>> with mips, ppc and i386 in the pathname. The log always reports = building "cross=20 >>>> tools" when it's compiling for itself, which puts an odd meaning on = the term "cross".=20 >>>> Are cross tools and object files for alien hardware really needed = in such a case? >>>=20 >>> Clang and the llvm tools are by default built to be cross >>> compile capable. You could compile targeting powerpc64 while >>> building on armv7, for example. (Gcc had had separate >>> compilers for such.) >>>=20 >>> In an earlier message I showed my src.conf like file that I >>> use that included turning off generating a clang that can >>> do so, limiting it to arm (including aarch64). This is >>> not the same as the "cross tools" stage issue but does get >>> rid of some of the build activity for clang. It also does >>> not change all llvm tools to completely avoid other processor >>> families. >>>=20 >=20 > Probably wisest if I leave artifice like that alone. Best to=20 > minimize the amount of tampering at the expense of wasting=20 > cpu cycles. The build systems is already immensely complicated=20 > and private variations just make it worse. =20 Up to you. "man src.conf" documents the likes of WITHOUT_LLVM_TARGET_MIPS=3D : it is part of a documented interface for controlling the builds. >>> I'll note that building main (14-CURRENT) armv7 from >>> stable/13 armv7 or 13-CURRENT armv7, is an example of >>> "cross tools" being involved. Cross-tools span more >>> issues than you might initially think of, such as some >>> differing defaults for the host targeted files (old >>> context) vs. the new context's files. Thus the: >>>=20 >>> "/usr/src/Makefile.inc1" line 335: SYSTEM_COMPILER: libclang will be = built for bootstrapping a cross-compiler. >>>=20 >>> that is involved. Example difference in context in an >>> explicit notation (amd64 context example, not armv7): >>>=20 >>> -target x86_64-unknown-freebsd13.0 >>> -target x86_64-unknown-freebsd14.0 >>>=20 >>> The processor family is not what is varying for those. >>>=20 >=20 > That puts a new light on the notion of "cross".=20 >=20 >>>>> I'll note that stable/13 now exists and git's main is now = 14-CURRENT. >>>>> Unfortunately, it seems that main (14-CURRENT) will not get the >>>>> weekly snapshot builds that are reported at any: >>>>>=20 >>>>> = https://lists.freebsd.org/pipermail/freebsd-snapshots/202*-*/thread.html >>>>>=20 >>>>> and go somewhere below (for example): >>>>>=20 >>>>> http://ftp3.freebsd.org/pub/FreeBSD/snapshots/ >>>>>=20 >>>>> until sometime after after 13.0-RELEASE : only stable/{11,12,13} >>>>> likely will, from what I'm told. Also, ci.freebsd.org is not >>>>> building main (14-CURRENT) yet. stable/13 builds are non-debug >>>>> builds. >>>>>=20 >>>>> https://artifact.ci.freebsd.org/snapshot/ being filling in has >>>>> not caught up with 13.0-STABLE , stable/13 , 14.0-CURRENT , or >>>>> main yet. >>>>=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?EDF9F631-FF55-444C-BAB3-3037CEA61D65>