From nobody Tue Feb 21 21:31:37 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PLstk1SqQz3tFrQ for ; Tue, 21 Feb 2023 21:31:58 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PLstj1Yb6z4H2Q for ; Tue, 21 Feb 2023 21:31:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=MzVam4n8; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1677015115; bh=FzE1I31FSnydEYZ21zzULxNRPUuG7CpqKVpSQ6RsRig=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=MzVam4n86fSy/D3PueFb/cdq94gEeghKXXRWDdxaVDqNiog+N2l7k1WfwoTXS0bLAJuJM3VnnwgLMPh0ej4LWAPekO1hX3HBD7eyIoekhZHIlOayT7uNA8GwgBGhCfbW/M3Sq/bqGpsd8FYav/y76tZJFIr/fTAsMbLEOPXwp2ttpZXhHu56btw+k8rI7DItqj7E9or9SSeAzQLrjELMogJr0JQgKJNxIDAlGERidsUEoLKrV5wcV0xYnT4M5sEcLHeyC9KvQM1zXahNMJtRHQlEyI7YjplaUXlK6JDjAJ/frs0N5UpjL00w83qmjmjAl9gBilsuYcOW+KOhYRVetw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1677015115; bh=Jt39m2N7oBPK0UNuxj2uCIUSVa3LrmYuJPuxzMesTaN=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=oH6BKISZPI2/18V/2SjQjSkXKE2VZSwStJ4hIxp+rihtoZIFZTdlHVYtNxydLppdWltz0TsliljubsFUFCaFcbCgou6B8JHdh4NblMXOSdA9SYcZHdtHK98sYZyciTc1gqLwxbaRr/nYs2PXhW69cMqhpe0MzWJNUz1N2/Qw5rAvq2UwNzpM9towsz+5zRzKz8HA8r0z+iVArRqffR3Tza4jH9OWtOMf2sFFpWD2L6mn4OiBJLqvYkxVr//3tx1hUzmc8oNiiwYG5/iBUsVejJILARxSmrPOocaXjsKZeFF6WAMHvE0vT7EXgyo7WkOvxX/xxuLOsr9jkKwoX8lUsg== X-YMail-OSG: lf8hBusVM1kYpZ7HvZiLJqbXMkvoWvv.ZRtklZ_uN17yzDUoWl.4H_l78GRyWW2 kcIOqAOghSZtt.hVgMldVvA3Q_I4L6V1nSoXluB9n7YuUQ6WBTR1LKWkzTF8NwGaF7jXTMjgbosf iiwlN9O64UpHESMiqgvgxytIjfgffmqzXH0IomLV6EnnDuzhnq4SVVuoIF4QDNI2zrryJRAC3PFZ kKQPKahFT827lScefhcB6xcEbQkv5s2H81SrWta2gOufe730qQZ41nQ3psnUUzHerNf3Csd.BxEO aTCkXkmWUp2v1_W6JSZE8kPHVHzzOscd4GVjWbMy_FwbLyt9lvkf1XClNnr_bOMve63.FTOoAAT2 nGg0eGjoc9SpqA4VrAEKchKYBuvZffDTpdVJV_XRCXvyicKVT780HgJJ7y.KlQTaBE97wb3IUydg DJW8MJ8Q2rk5JE4DO5Jt.sc08vGGNpHN7_P70o4ipKEKs5IHo0FAMEu2W.hzPCwfry_N3vfg5zpX Jsu_Yk1uNz4BnsJspt4YFjWdUbqwf4lgY0z8nu5_VP3MyeNZUVgfMbQFQ4aPn5s6NTPRxpeSjeGu BVINg0QYdh4.szix.yotkKgOskm2RbVCADLKdFHkUlezx8Oxa_D5JcdeotoXk3vcRor5_T2qQTjg Cwe7D8Lx7Or9Rvob_7EVG0f5Jn9nE0y6FlS1NBzahNin217dexZfX9YQwQGRooCgCXe95MCb6bBs UMursr5L5ptQZ3wi2Br8Y_Ih99lGLKOmB5DCTBut7N.SO0cyoSxoAp1qkxErGMndthRWlKNjjhxz J3BNi6F.iJUzwwnYYzqOsr4xQU1kNzeVNXofQeIyV3i5ZKym9Vx7u3JTPYHCnvuwCMKjQBZAizwY e9RLphcGQ56toLEFNj7jy_xjZx.Qs4p94qBASnbGPHP2__eYTdfOl_LMDZmv6twYBJ.8cllnzrrq FTrwDdGg0yqk8hiTxAOUstKZL4XGsv_uj9Y8aVkiF54okPQq7BWrrYPfFVsCeJli5Z5COfv01TGo Vgf_1yh2MWD1Qva3kVtaIeiIMzY.nMisGUwgs75TTmnlAlB6NU7AgcI2RRR0ogAkyh6WbDEuCW3N Y10kxXM2g7lEVBZVHU4tfmygUhisVjPUTcinP99g8Xi4dnTrglCvkxPYIXIOT62i2evUyU2MOFtG 5XnpZjbGuKfjhZatwOmScWh1ra6NzawiAPfUfwdb9x4UNG7IHkTScCeiseO6uymed4g7IH..GfPt fUjkJ2ABNqal94dKh4Tkff9_E2iXzEW4c7Sauq1mZL_s.7vsYCeBSdTpfmvr4h6MOKFtTdoQ5M.M JcCuG_KdiuYuad7RZyqzZbjn8wiHxRNQywM98LjqnHz5XCnr39sz2pEwMz2EDuA7PKMB_4IYESDK WAO4f1DZjN_lRkMHm2h92jjFhYdfz_mg8VrXv7mCwj9vE9SVCTMDYqBkiBrPT8L.xPWo4LjrknH1 jravLYwJzACS7JyBuppRYw6yULgfxWl27mQ2_kQLPc8XZzmZZFjK92nA.DCeAJ3wk1W85GkNiND6 htyd2BR489IS3r0IPGFNLTCdg4hzl5iJCK31MgCAQbfG7nxYNZU_TyF7BBxnw0s3JGd9lUON3D9D RR3tMb2IQZdkKbZ9DgYJf2jOqpwwAzC.Hf0jp0IWmqf4QfN79eBQDdqcX4L52qLg16DCAxGW51ZQ DW8O.QkVOCYCinCoRSMKEnnK7GT9VcMC1.VoExXcnbKhl5E5zvLbewSJGjLljasBbLl6d3jwEWNj RDQnETIEDcY7RKAt00AJk1epHpaNE7UTYcMOWRyQGpxpfIO7BvSs0qcfABYdvdqrgucEwwY6eLij VV0xyCsTtEXZepBR_sc1H45XxEZvAGpGalQSpZsGPAlalQ.Pfx6H_xC0oouxjGaO1asToXZf7TBI fXncEUpngsfi5CworD.30NhSQ23snp24FEFhivTnAgzPyWp0nRk7w9zJ8nyyouVt7rz61hXujAjG H08YuCmtTltf10iKvYqG3tj9Ntcm8i90v_O8OAHU9FW7NvQEW.0XLwPFLZ_AiNh0YLkAF92Vi6pp t5mQrAhWS_V5d6xPUsdWhU5NAWfYrHzIqei7y4N0hSVnnrJ5pO9xAamXukEQxfkqqHpZflBcuQVt mLs_1kU8z.JIXOWiIG2fBNQq3L_M95QWKnOlNIXxcKapdRsjEuXfM8CV0gIHU0V4Cvs4SvBP3xiO j7SJzkTHYifehdS28738aHmlLsqznZgUF66oHHvGAVY0SnmfYL.uG_8q4NG5c0MOCANsmXp18XA5 .HEo- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Tue, 21 Feb 2023 21:31:55 +0000 Received: by hermes--production-bf1-57c96c66f6-kqcsw (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 410228c329f9ac3fba9d215bc5da8fc1; Tue, 21 Feb 2023 21:31:49 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) From: Mark Millard In-Reply-To: <73088.1611797582@kaos.jnpr.net> Date: Tue, 21 Feb 2023 13:31:37 -0800 Cc: Bryan Drewery , Current FreeBSD , Peter Content-Transfer-Encoding: quoted-printable Message-Id: References: <3345EBA5-A09C-4E3F-B94D-39F57F56BDBB@yahoo.com> <73088.1611797582@kaos.jnpr.net> To: "Simon J. Gerraty" X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spamd-Result: default: False [-3.47 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.97)[-0.968]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-current@FreeBSD.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from] X-Rspamd-Queue-Id: 4PLstj1Yb6z4H2Q X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N [This will be a question relative to the old material below.] On Jan 27, 2021, at 17:33, Simon J. Gerraty wrote: > Mark Millard via freebsd-current wrote: >>>> Given an already built, installed and booted system version, I've >>>> noted a big difference for META_MODE in 2 rebuild contexts (no >>>> source updates involved): >>>>=20 >>>> *) Prior buildworld buildkernel, installkernel, installworld, boot >>>> presumed before (A) and before (B) below. >=20 > Yes installworld is going to cause problems unless you tell meta mode = to > ignore everything outside of the src/obj tree. > But even that isn't enough as you note below. >=20 >>>> So I used make -dM for the commented buildworld buildkernel lines, = logging >>>> the build output and later diff'ing them. >>>>=20 >>>> Result that I noticed? Lots of lines uniquely from (B)'s case, = ending with >>>> one of: >>>>=20 >>>> file = '/usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/u= sr/sbin/awk' is newer than the target... >=20 > Yes. That would be expected. > I think Bryan added some knobs to mitigate some of this. >=20 > grep -n META.*IGNORE share/mk/*mk >=20 > might be instructive. >=20 >>>> Many/most/all(?) of these seem to me to be unlikely to actually = need to >>>> contribute to what needs to be rebuilt (just based on being newer). = So >>>> the option to ignore (some of?) them could be useful in making = META_MODE >>>> builds quicker. >=20 > Yes on all counts. That's why bmake provides a number of knobs to > ignore such things. > They are listed in the man page in increasing order of expense. >=20 > One of the risks of the sort of introspection meta mode does, is = delving > too deep (like the dwarfs ;-) >=20 > The .MAKE.META.IGNORE_* and .MAKE.META.BAILIWICK variables help to > contain the potential insanity. >=20 >>> The following from one of the .meta files makes the point that rm = use >>> in the example is unlikely to be important to needing to rebuild, >>> despite it actually causing a file rebuild. Nor is the specific echo >>> command listed relevant. Only the "ar" command is: >>>=20 >>> # Meta data file = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/lib/libc++/li= bc++.a.meta >>> CMD @echo building static c++ library >>> CMD @rm -f libc++.a >>> CMD ar -crsD libc++.a algorithm.o any.o atomic.o barrier.o bind.o >=20 >>> -- filemon acquired metadata -- >>> # filemon version 5 >>> # Target pid 22471 >>> # Start 1611359217.214996 >>> V 5 >>> E 22961 /bin/sh >>> R 22961 /etc/libmap.conf >>> R 22961 /var/run/ld-elf.so.hints >>> R 22961 /lib/libedit.so.7 >>> R 22961 /lib/libc.so.7 >>> R 22961 /lib/libncursesw.so.9 >>> R 22961 /usr/share/locale/C.UTF-8/LC_CTYPE >>> F 22961 22962 >>> E 22962 = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/us= r/sbin/rm >=20 >>> The timestamp on . . ./tmp/legacy/usr/sbin/rm is not >>> actually relevant to if libc++.a needs to be rebuilt. >=20 > True.=20 > If there is nothing under .../tmp/legacy that should be counted you = can > just: >=20 > .MAKE.META_IGNORE_PATHS +=3D that path >=20 > which would be the cheapest option. >=20 > If however there are things that should be considered you'd have to > use a much more specific list of .MAKE.META_IGNORE_PATHS or > perhaps something like: >=20 > .MAKE.META.IGNORE_PATTERNS +=3D */rm >=20 > would ignore an rm binary regardless of where found. >=20 > worst case you might need something like: >=20 > # realpath > .MAKE.META.IGNORE_FILTER +=3D tA > # ignore anything here > .MAKE.META.IGNORE_FILTER +=3D N*/tmp/legacy/usr/bin/* >=20 > Unlike .MAKE.META.IGNORE_PATTERNS which is a list of patterns to match > the goal with .MAKE.META.IGNORE_FILTER is to reduced to an empty = string > paths to be ignored. >=20 >>> Of course, the structure also point out the judgment >>> is specific to understanding the sequence of CMD's >>> listed above. Only a hack of ignoring, not recording, >>> or commenting out the filemon lines ending in >>> /tmp/legacy/usr/sbin/rm would seem to avoid the @rm >>> handling issue. Such might well have its own risks. >=20 > No hacking needed, there are a range of knobs to help. >=20 >> Just for reference for more about the sequencing involved: >>=20 >> Looks like in my example various . . ./tmp/legacy/. . ./*bin/ >> actually are links to files in: >>=20 >> = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bi= n/ >>=20 >> and the later after-install buildworld "Rebuilding the >> temporary build tree" step leads to the updated dates for >> files in that area, updated via the code that reports: >>=20 >> Linking host tools into = /usr/obj/amd64_clang/amd64.amd64/usr/fbsd/mm-src/amd64.amd64/tmp/legacy/bi= n >>=20 >> So the prior installworld leaves updated dates and the >> linking to those installed files makes >> . . ./tmp/legacy/. . ./*bin/* have the newer dates show >> up for the legacy paths as well. >>=20 >> In turn the dependency tracking via META_MODE uses the new >> dates on . . ./tmp/legacy/. . ./*bin/* files to cause >> rebuilds of more materials than if installworld had not >> been done. >>=20 >> It is not obvious if Bryan D. would find the effort to avoid >> this worthwhile for the contexts that drive FreeBSD's build >> environment choices. >=20 > For people that use installworld and also want META_MODE > it would seem some additional IGNORE work may be beneficial. Since some of the paths reported ended up being links (symbolic, as I remember), what are the principles for which form of paths should be the basis for paths in the likes of: .MAKE.META_IGNORE_PATHS .MAKE.META.IGNORE_PATTERNS .MAKE.META.IGNORE_FILTER Target of link? Path to the link itself, not the target? Both? (Something had interrupted my explorations back then and I'd forgotten about your note and have never done the exploration. Trying to partially answer a question on the lists lead to me reviewing the old thread and running into your note again.) =3D=3D=3D Mark Millard marklmi at yahoo.com