From owner-freebsd-arm@freebsd.org Sun Jan 24 06:01:15 2021 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 97E174EF4DE for ; Sun, 24 Jan 2021 06:01:15 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.gq1.yahoo.com (sonic314-21.consmr.mail.gq1.yahoo.com [98.137.69.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DNj6f3s97z4Z9k for ; Sun, 24 Jan 2021 06:01:14 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1611468072; bh=JTxIpoVpoPKYXB6LR/yO+IuRVk0XXoTbIfubMRa7Gsl=; h=Subject:From:Date:To:From:Subject:Reply-To; b=fCIVI2XYRuzbIxmRdCEcD2IFp97gF4vkJekmSrKgDPVps41lvyw3A7+yv+9J/BeqOFTQo827Qgl51QigzRPmvAtw9V1kGn7ynV0UuNEyp2YU7MP941y41AVYD3mSu+IrvMKUCM2kjCq8CJtCL3TPHG40LEkV/oZ57bUkUy+Gqd41UrX+NGQ0gtQrozdWb70EMR30eSelTr1+z2GDs6pSUcZ2aazQ5ktvG3ZhGUNlbaQmJFSmxnES766MLnjuhFmITqXYH1yfgNyuc/sZIL0iTuo3Vw3PQYlUB6BcZMqAqpuxUX/2JCYrcK9dM1OgIj7VK60eycOlfDP2ki+CR9/4pw== X-YMail-OSG: 7WWwhAwVM1lY8_nm0mypxkAWCABLPA_RIqOu9fCo1tNOIEzM6JSSvQsStttZyqk kBEiDTPOkcUUOUNZ9kjSTQWg1f7q2_SVmJ9wRJhdINF4tR6WSpY8g.gs0SIATso26zfyvCYerALd 3cZju5u9YiLRx8WGoHmp.rV5nN.fmdjpBu7yN7yI0BR.kcIKAqqaPqjDuGHs2ixV.GVGs.2dtz0B nVGf2uT082mbXK5PyShOxwTY56Yn02_q.YFwSmTVaBiSpRd5LrwsdDLs4.f0oRmBMPa.8lnzF754 3yWqt83dex0BMpjaK0aEjJVe4n.LA.HQU3Rh_EOW9FJb6unrL3h8ORLG_lNz9XXJBeGCkoGRIxQV I9katPbqBcunqwXmpXU0ucAIxX.jPxh_0rLF4hcENuzGKis.ilO6kRkNONHEpf5mMlc7Hv_sLn8K _ZepRToRuTzjkLf3b7XT2Xp5NYOjNmQNLYleniX..Af24dnziI_xy4cQ4anw5Lu5gCSvygBTe2Ie 1iGZXxw_r5VSwaCbKI0yl.7SXKNoHtwSknwt8fZJpy6_N25nnW8f8GO7yCV2D0.A..0VLgaJDMt_ hb8hw3WK1qJv5jBdtjKzBy_TYeD6w9TYQw7p5nrjHsPTtFTt.En09N14c6aylNwpY1NXsKavMpIH z6oJ8hTu_QSasdmlPuqRBk84z9UmXfGJ83ArRsWGDBrWdvmhA5vrKbfA5MZDORtmgpY4_8LrcgOB SVT1wS1Fnk9Pi6KcjyaXNQukBdm5a4LFEk_qGqOlcsgfsjZsdhHHQiKXPsSEx3uWhPtONqm9.rtS N9aLnTIvhg_6ULjX74Iecv42pjv.pOEIODgkjEZPSitfxh0T18vSOj6jZ0V3dXqSSu.JOAQJM6Ot gSbnbLrujq6bfyGw293SJWFVwmR5fnVpd1KJ2BgnBzUdrBGdp9k_kC1qj9YoMkN3lOaxATwOr_fs F3X9Dzav8HMsMo2CXWmOJa6cK1sgkjWl42SzIwYBX7g5tmpZ1PWqurk.pA69o7.sNQOZKCMGKpBb JDY5wA75581WD4CTK.t3mQ3.vEyj5HIGGKwWqmnT87Vw0k881Ju9cKWdQ0nb5LxKwj0x5Fl8a7E1 pvvDQ2dINfGF2rEyD_lDv90PxsBXeHYoQQQzU8gtNWVrTkss6btV_neSXr8mEkaBxFur.o58V2KS x_p3GFJgzk5GUyyDkkK3bkoAMDMWrym2cY8xwwiM6OlJT0j5OZwxRSrBS_iBn0Bw1JrwBHdCR0QL dcs2ifdRIJpp4TgHLB0eUfQ6C6bNAw.YJqq1iXFVB5Qf2wYFvfb1Oba.W7vUunFV5Iz2_w7seXF8 50u2uLk9gnkZidoZ_FYzUaP6eTyxZRyJSxyk8UphVIlJhuD1rwNovaPEVGi.mComlYbsl4gVaaeB oUFNruepRyB6wQo1Vk6iYccUO9GffQZauyjeFrCiNYTubAhMfhN_qZ6nyDFICFGDdK14jzy0b02. OuKFUmAN.vLI34WtXh64Bhw05K3pgPRK2xxsWKdHmRrUK2Wsq4nrich30fi3vsLEIVtjVvCZWQuV boF6s3i9MmxkrdZuMB3u0nQPlPymIcrK2szmltbdSQK7poh2U7UomQBrDAzgMLkDq.TJQcN79bqR 11DYmmoNR4DwUlOX7tUVaaIMo8Ev1z3wUyycTpqGxL7rewcvgdsWTxcrIJV9v_qM4FameiUKGItg l3EVlgE0pTlvGpHTcai1N_hJ5rVDWUlzWRR0HMKCfreiHbGcmVhFiBAtvpGI.hYpmJwkhdVk03yL sqPIpFozc4NzU4gVsZ3consuSNL40y7.glBOzW4fdtH8BhhH1FJJ7bVJ3OaUAeBttf50g0WtyC3a q3WZiS.uLKywlkcdzWu._EadgKPkYCZj4ORb22wljw3IdY_gJWL.Pv_LtoFZpbH3X3OphU9_8zWN aDEzeRDDkARvI1SEc.sd0VMT3Zmsk38p8hSZGzyvH43YIzky80vPthihMdcN9PH03Hozwz7SeiIs 4J9e0.k6TXZ_BpdgnDf.gIeSzRkTgEL.chcen.bka8vmtKbdxhp3HoOzOVD_tfoldwPu6rPatRYJ BvB_kmXbj8PHWm06QD1F9Vd48Tyz1rPtRBPrFFSythE7HCV7iwdAMZ89VjO0TiuYTJovzNeFttef BdtYGHezP11b9eNoJGjpHpk_zLWgKNuu5wN8FNx17omgCnC6qP5rAjdEYzdjhT1q3l1KZinBaohU wgx3drfJR0Jtg5RA_tdbBkEzCmWpT.bnAjYiEVrMnXx6NzHeLjeIg4YCYzFlMUfyMNeAYlC6mx8h 9mbm._k2x4rfQvXqPjilvYmyJOVz4jguco3GBnRlU8uJhnEfn6aoI39y90r3bRT4n2GbVc5gD0Da sOLD1MAwfbMG4ZwOHu_fIihQdbo4bDrKt.JBdJ8sbOOdBuyy8ix9vtQrqMwUXGBq2X1Gh1VFjs6s 6QA6FFNEJao7YOxC4BO5IBoSVzRZHSFJce7w3Lxxcj.dvFds1g5yTp4KqiKMx6D6tuAVWgq.HlEL OSKM3DA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Sun, 24 Jan 2021 06:01:12 +0000 Received: by smtp418.mail.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID bf4ac8a6d08141e94cf79f5e9bfcaf32; Sun, 24 Jan 2021 06:01:09 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.40.0.2.32\)) Subject: Re: Needless work in buildworld From: Mark Millard In-Reply-To: <20210123173754.GA83834@www.zefox.net> Date: Sat, 23 Jan 2021 22:01:07 -0800 Cc: freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: References: <20210121023358.GA58854@www.zefox.net> <8D0C2A4C-B616-47B9-864E-D846A6EBA3D6@yahoo.com> <20210122011535.GA66611@www.zefox.net> <655C6BAA-8B10-4130-A5C9-EDED6906207D@yahoo.com> <36A2E015-78DF-40AB-BF53-FB3D26FA5AAC@yahoo.com> <20210122224656.GA76907@www.zefox.net> <2903491E-7DE7-4F17-B515-120BA447B8B3@yahoo.com> <20210123173754.GA83834@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3654.40.0.2.32) X-Rspamd-Queue-Id: 4DNj6f3s97z4Z9k X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.69.84:from]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.69.84:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.84:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.84:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Porting FreeBSD to ARM processors List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2021 06:01:15 -0000 On 2021-Jan-23, at 09:37, bob prohaska wrote: > On Fri, Jan 22, 2021 at 10:20:53PM -0800, Mark Millard wrote: >>=20 >> In things such as (abbreviated): >>=20 >> file '. . ./tmp/legacy/usr/sbin/awk' is newer than the target... >>=20 >> The "target" is the object file or whatever that might need to >> be replaced, not the . . ./tmp/legacy/usr/sbin/???. The >> examples of such lines listed the target file path before >> the word "file" above. >>=20 >> I only showed one line with the common suffix text, no hint of >> the prior text on the lines summarized. >>=20 >> So the targets were timestamped at the prior build's creation >> of them. >>=20 >> It is the installworld installkernel sorts of activity that ends >> up with . . ./tmp/legacy/usr/sbin/awk and the like having time >> stamps from the new build's time frame. But the META_MODE >> records indicate that . . ./tmp/legacy/usr/sbin/awk (or whatever) >> was involved in generating the target in the prior build so the >> updated timestamp leads to questions of needing regeneration of >> the target. But a . . ./tmp/legacy/. . ./??? generally is not >> likely to cause such a need, even with new timestamps. >>=20 >>=20 >>>> If so, would some sort of checksum be able >>>> to distinguish meaningful changes? Would it be computationally = worthwhile? >>=20 >> I leave the overall issue to Bryan Drewery to decide if there is >> something to be done or not. >>=20 > My thinking regarding checksums was absolutely wrong; I wanted to = compare > files that hadn't been created.....=20 >=20 > In deciding whether to recompile, say, /usr/bin/awk, Note that, in my example, . . ./tmp/legacy/usr/sbin/awk is not what ended up being rebuilt (at least at the time of the "newer than the target" message). I did not show a list of what was being rebuilt because of . . ./tmp/legacy/usr/sbin/awk being newer. I'll show some examples below. The .meta file for the rebuilt target is what has the "newer than the target" message, reporting what was newer that was sufficient to cause the rebuild. So a partial list is (abbreviated paths): . . ./tmp/obj-tools/kerberos5/tools/make-roken/make-roken.c.meta . . ./lib/ncurses/ncurses/names.c.meta . . ./lib/ncurses/ncurses/unctrl.c.meta . . ./lib/ncurses/ncurses/codes.c.meta . . ./lib/libthread_db/Version.map.meta . . ./obj-lib32/lib/libstdthreads/Version.map.meta . . ./stand/efi/libefi/teken_state.h.meta . . ./usr.bin/getconf/limits.c.meta . . ./rescue/rescue/usr/fbsd/mm-src/sbin/route/keywords.h.meta . . ./sys/GENERIC-NODBG/vnode_if_newproto.h.meta . . ./sys/GENERIC-NODBG/iwm8265fw.c.meta . . = ./sys/GENERIC-NODBG/modules/usr/fbsd/mm-src/sys/modules/aac/device_if.h.me= ta The full list is much larger. There are even lots of files that list . . ./tmp/legacy/usr/sbin/rm as "newer than target", including: . . ./tmp/obj-tools/tools/build/libegacy.a.meta . . ./tmp/obj-tools/lib/libopenbsd/libopenbsd.a.meta . . ./tmp/obj-tools/lib/clang/libllvmminimal/libllvmminimal.a.meta . . ./bin/csh/gethost.meta . . ./rescue/rescue/usr/fbsd/mm-src/bin/csh/gethost.meta . . ./include/compat.meta . . ./lib/libc_nonshared/libc_nonshared.a.meta . . ./lib/libcxxrt/libcxxrt.a.meta . . ./lib/libcxxrt/libcxxrt.so.1.full.meta . . ./cddl/lib/libspl/libspl.a.meta . . ./lib/libthr/libthr.a.meta . . ./lib/libc++/libc++.a.meta The full list is much larger. The following from one of the .meta files makes the point that . . ./tmp/legacy/usr/sbin/rm is unlikely to be important. Nor i the echo relevant. Only the "ar" command is. # Meta data file = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/li= bc++.a.meta CMD @echo building static c++ library CMD @rm -f libc++.a CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o = charconv.o chrono.o condition_variable.o condition_variable_destructor.o = debug.o exception.o filesystem/directory_iterator.o filesyste m/int128_builtins.o filesystem/operations.o functional.o future.o hash.o = ios.o iostream.o locale.o memory.o mutex.o mutex_destructor.o new.o = optional.o random.o random_shuffle.o regex.o shared_mutex.o stdexcept.o string.o strstream.o system_error.o thread.o typeinfo.o = utility.o valarray.o variant.o vector.o cxxrt_auxhelper.o = cxxrt_dynamic_cast.o cxxrt_exception.o cxxrt_guard.o = cxxrt_libelftc_dem_g nu3.o cxxrt_memory.o cxxrt_stdexcept.o cxxrt_terminate.o = cxxrt_typeinfo.o CWD = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++ TARGET libc++.a -- command output -- building static c++ library -- filemon acquired metadata -- # filemon version 5 # Target pid 22471 # Start 1611359217.214996 V 5 E 22961 /bin/sh R 22961 /etc/libmap.conf R 22961 /var/run/ld-elf.so.hints R 22961 /lib/libedit.so.7 R 22961 /lib/libc.so.7 R 22961 /lib/libncursesw.so.9 R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE F 22961 22962 E 22962 = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/us= r/sbin/rm . . . The vintage of . . ./tmp/legacy/usr/sbin/rm is not going to change the content of: /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++ in the above 3-"CMD" sequence. > one must compare the=20 > creation date of the installed /usr/bin/awk to every file it depends = on.=20 > If any of those files is newer, the whole chain must be recompiled. The dependency tracking builds up to the final things being built, not from the final things backwards. It frequently is the case that only some of the chain required for a final thing is rebuilt. As an example: A final link may reference a mix of changed and unchanged material, for example. But that final link is required if any of its inputs changed. Unnecessary changes to inputs might lead to a relink that otherwise would not be needed. > I > can readily see how this would avalanche, but it seems necessary. Have=20= > I got at least that part right?=20 The "avalanche" reference and implication fits my alternate wording as well as yours and so is good, whatever the other details of the avalanche structure may be. >>> The files (programs) were used during the activity that generated = the prior >>> build of the target. The worry is that the updated programs might = have >>> differing results from older ones and so the new timestamps lead to >>> rebuilding. The worry is just unlikely to be an actual problem for = many of >>> the particular programs. >>>=20 >>> It would be good if META_MODE could ignore those programs that are = in the >>> legacy area and are unlikely to cause the output to vary in some >>> significant way. >>>=20 >=20 > Is this to say that META_MODE is checking outside the dependency chain > (web?), reacting to files changed but not to be used again? I'm not sure of the reference/context-intended here. See the prior libc++.a example above for an example in which the timestamp on . . ./tmp/legacy/usr/sbin/rm is irrelevant to if the libc++.a actually needs to be rebuilt. >>>> More broadly, I've been surprised to see lots of files associated = with foreign >>>> architechtures reported in self-hosting buildworld logs on the = Pi2/3. Things >>>> with mips, ppc and i386 in the pathname. The log always reports = building "cross=20 >>>> tools" when it's compiling for itself, which puts an odd meaning on = the term "cross".=20 >>>> Are cross tools and object files for alien hardware really needed = in such a case? >>>=20 >>> Clang and the llvm tools are by default built to be cross >>> compile capable. You could compile targeting powerpc64 while >>> building on armv7, for example. (Gcc had had separate >>> compilers for such.) >>>=20 >>> In an earlier message I showed my src.conf like file that I >>> use that included turning off generating a clang that can >>> do so, limiting it to arm (including aarch64). This is >>> not the same as the "cross tools" stage issue but does get >>> rid of some of the build activity for clang. It also does >>> not change all llvm tools to completely avoid other processor >>> families. >>>=20 >=20 > Probably wisest if I leave artifice like that alone. Best to=20 > minimize the amount of tampering at the expense of wasting=20 > cpu cycles. The build systems is already immensely complicated=20 > and private variations just make it worse. =20 Up to you. "man src.conf" documents the likes of WITHOUT_LLVM_TARGET_MIPS=3D : it is part of a documented interface for controlling the builds. >>> I'll note that building main (14-CURRENT) armv7 from >>> stable/13 armv7 or 13-CURRENT armv7, is an example of >>> "cross tools" being involved. Cross-tools span more >>> issues than you might initially think of, such as some >>> differing defaults for the host targeted files (old >>> context) vs. the new context's files. Thus the: >>>=20 >>> "/usr/src/Makefile.inc1" line 335: SYSTEM_COMPILER: libclang will be = built for bootstrapping a cross-compiler. >>>=20 >>> that is involved. Example difference in context in an >>> explicit notation (amd64 context example, not armv7): >>>=20 >>> -target x86_64-unknown-freebsd13.0 >>> -target x86_64-unknown-freebsd14.0 >>>=20 >>> The processor family is not what is varying for those. >>>=20 >=20 > That puts a new light on the notion of "cross".=20 >=20 >>>>> I'll note that stable/13 now exists and git's main is now = 14-CURRENT. >>>>> Unfortunately, it seems that main (14-CURRENT) will not get the >>>>> weekly snapshot builds that are reported at any: >>>>>=20 >>>>> = https://lists.freebsd.org/pipermail/freebsd-snapshots/202*-*/thread.html >>>>>=20 >>>>> and go somewhere below (for example): >>>>>=20 >>>>> http://ftp3.freebsd.org/pub/FreeBSD/snapshots/ >>>>>=20 >>>>> until sometime after after 13.0-RELEASE : only stable/{11,12,13} >>>>> likely will, from what I'm told. Also, ci.freebsd.org is not >>>>> building main (14-CURRENT) yet. stable/13 builds are non-debug >>>>> builds. >>>>>=20 >>>>> https://artifact.ci.freebsd.org/snapshot/ being filling in has >>>>> not caught up with 13.0-STABLE , stable/13 , 14.0-CURRENT , or >>>>> main yet. >>>>=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)