Date: Tue, 21 Feb 2023 21:53:59 -0800 From: Mark Millard <marklmi@yahoo.com> To: Peter <pmc@citylink.dinoex.sub.org> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org> Subject: Re: 13.2 BETA2: how do debug META_MODE? Message-ID: <F68956DB-38B8-45D7-ACF9-1D76644BFED7@yahoo.com> In-Reply-To: <A6A8EA3C-7D14-4467-B00E-388685C4A7F4@yahoo.com> References: <41B536B0-DA66-449E-96BB-E11A8750471A.ref@yahoo.com> <41B536B0-DA66-449E-96BB-E11A8750471A@yahoo.com> <Y/S/RfZaWXXYPx2w@disp.intra.daemon.contact> <ED41CCB8-7A32-4924-A756-099D8E819C4D@yahoo.com> <Y/V5ozYzwL67qDOP@disp.intra.daemon.contact> <F9AE0FD8-5E94-484A-A67A-8D4FAB967F59@yahoo.com> <Y/WH9icO%2BgWPBybo@disp.intra.daemon.contact> <A6A8EA3C-7D14-4467-B00E-388685C4A7F4@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Feb 21, 2023, at 20:51, Mark Millard <marklmi@yahoo.com> wrote:
> On Feb 21, 2023, at 19:11, Peter <pmc@citylink.dinoex.sub.org> wrote:
>=20
>> On Tue, Feb 21, 2023 at 06:44:09PM -0800, Mark Millard wrote:
>> ! On Feb 21, 2023, at 18:10, Peter <pmc@citylink.dinoex.sub.org> =
wrote:
>> !=20
>> ! > On Tue, Feb 21, 2023 at 11:56:13AM -0800, Mark Millard wrote:
>> ! > ! On Feb 21, 2023, at 04:55, Peter <pmc@citylink.dinoex.sub.org> =
wrote:
>> ! > !=20
>> ! > ! > ! # cd /usr/src/
>> ! > ! > ! # env WITH_META_MODE=3Dyes make buildworld
>> ! > ! > ! # env WITH_META_MODE=3Dyes make installworld
>> ! > ! > ! # env WITH_META_MODE=3Dyes make buildworld (again #0)
>> ! > ! > ! ## no more rebuilds below?
>> ! > ! > ! # env WITH_META_MODE=3Dyes make buildworld (again #1)
>> ! > ! > ! # env WITH_META_MODE=3Dyes make buildworld (again #2)
>> ! > ! >=20
>> ! > ! > But what is the difference between #0 and #1?
>> ! > !=20
>> ! > ! awk, cp, ln, rm, sed, and many more from
>> ! > ! . . ./tmp/legacy/usr/sbin/have new dates
>> ! > ! for rebuilds after installworld (that targets
>> ! > ! the running system). Not true for #1 and #2.
>> ! > !=20
>> ! > ! The dates on these tools being more recent than
>> ! > ! the files that they were involved in producing
>> ! > ! leads to rebuilding those files. That in turn
>> ! > ! leads to other files being rebuilt.
>> ! > !=20
>> ! > ! make with -dM reports the likes of:
>> ! > !=20
>> ! > ! file '. . ./tmp/legacy/usr/sbin/awk' is newer than the =
target...
>> ! > !=20
>> ! > ! explicitly as it goes. As I remember tmp/legacy/usr/sbin/
>> ! > ! was always part of the path for what I found.
>> ! >=20
>> ! > Mark, thanks a lot for the proper input at the right time!
>> ! >=20
>> ! > This put me on the right track and I mananged to analyze and
>> ! > understand what is actually happening.
>> ! >=20
>> ! > It looks like my issue does resolve itself somehow, and things
>> ! > start to behave as expected again after four builds.
>> !=20
>> ! Intersting.
>> !=20
>> ! > ! I did not do the analysis of how (e.g.) tmp/legacy/usr/sbin/awk
>> ! > ! ended up being newer than such a target and, so, causing a
>> ! > ! rebuild of that target. I was going the direction: that
>> ! > ! it is newer really is unlikely to justify the rebuild for
>> ! > ! the target(s) in question. The other direction about how
>> ! > ! it got to be newer is also relevant.
>> ! >=20
>> ! > I have now analyzed some parts of it. META_MODE typically finds =
some
>> ! > build-tools to rebuild, but then if the result is not different
>> ! > from what was there before, then "install" will not copy it to =
the
>> ! > bin-dir, and so the avalanche gets usually avoided.
>> ! >=20
>> !=20
>> ! The implication is that "install -C" is in use, quoting the
>> ! man page:
>> !=20
>> ! -C Copy the file. If the target file already exists and =
the files
>> ! are the same, then do not change the modification time =
of the
>> ! target. If the target's file flags and mode need not =
to be
>> ! changed, the target's inode change time is also =
unchanged.
>> !=20
>> ! -c Copy the file. This is actually the default. The -c =
option is
>> ! only included for backwards compatibility.
>> !=20
>> ! -C might have more of an effect in a reproducible-build
>> ! style build process than on a non-reproducible-build
>> ! style one.
>>=20
>> Yepp. "install -p" is used, see /usr/src/tools/install.sh
That may be incorrect about what is happening for
_bootstap_tools_links and other things. Why do I
say that? Several points . . .
I do not see "tools" in any PATH=3D so far, making implicit
use unlikely.
/usr/main-src/share/mk/sys.mk:INSTALL ?=3D =
${INSTALL_CMD:Uinstall}
/usr/main-src/share/mk/src.tools.mk:INSTALL_CMD?=3D install
vs.
/usr/main-src/Makefile: INSTALL=3D"sh =
${.CURDIR}/tools/install.sh"
/usr/main-src/Makefile.inc1:BMAKEENV=3D INSTALL=3D"sh =
${.CURDIR}/tools/install.sh" \
/usr/main-src/Makefile.inc1:KTMAKEENV=3D INSTALL=3D"sh =
${.CURDIR}/tools/install.sh" \
Also:
# kernel-tools stage
KTMAKEENV=3D INSTALL=3D"sh ${.CURDIR}/tools/install.sh" \
vs.
# world stage
WMAKEENV=3D ${CROSSENV} \
INSTALL=3D"${INSTALL_CMD} -U" \
and:
.if defined(DB_FROM_SRC) || defined(NO_ROOT)
IMAKE_INSTALL=3D INSTALL=3D"${INSTALL_CMD} ${INSTALLFLAGS}"
So: explicitly varying styles for various contexts.
It looks to me like the make output would show:
install . . .=20
vs.
sh . . ./tools/install.sh . . .
based on if the script is not-in-use vs. is-in-use
(respectively).
Looking at some of my logs I see the likes of:
install -o root -g wheel -m 555 cap_mkdb =
/FBSDFSSDroot-mnt//usr/bin/cap_mkdb
which looks to not be using the script and not using
either -C or -p as well. cap_mkdb is an example from
the _bootstap_tools_links list.
Some of the below do have -C use, others do not.
# grep -r '\-o.*-g.*-m' /usr/main-src/Makefile* /usr/main-src/share/ | =
more
/usr/main-src/share/sendmail/Makefile: ${INSTALL} -T =
package=3D${PACKAGE:Usendmail} ${TAGS_ARGS} -o ${BINOWN} -g ${BINGRP} -m =
755 -d ${DDIR}/${dir}
/usr/main-src/share/sendmail/Makefile: ${INSTALL} -T =
package=3D${PACKAGE:Usendmail} ${TAGS_ARGS} -o ${BINOWN} -g ${BINGRP} -m =
444 ${SENDMAIL_DIR}/${file} ${DDIR}/${file}
/usr/main-src/share/mk/bsd.info.mk: ${INSTALL} -o ${INFOOWN} -g =
${INFOGRP} -m ${INFOMODE} \
/usr/main-src/share/mk/bsd.info.mk: ${INSTALL} -o ${INFOOWN} -g =
${INFOGRP} -m ${INFOMODE} \
/usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},dev} -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \
/usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},dev} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} =
\
/usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},dev} -C -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} =
\
/usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} ${TAG_ARGS} ${STRIP} =
-o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \
/usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},dbg} -o ${LIBOWN} -g ${LIBGRP} -m ${DEBUGMODE} \
/usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},dev} -S -C -o ${LIBOWN} -g ${LIBGRP} -m =
${LIBMODE} \
/usr/main-src/share/mk/bsd.lib.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},dev} -o ${LIBOWN} -g ${LIBGRP} -m ${LIBMODE} \
/usr/main-src/share/mk/bsd.doc.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},docs} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \
/usr/main-src/share/mk/bsd.doc.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},docs} -o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \
/usr/main-src/share/mk/bsd.incs.mk: ${INSTALL} ${${group}TAG_ARGS} =
-C -o ${${group}OWN} -g ${${group}GRP} -m ${${group}MODE} \
/usr/main-src/share/mk/bsd.incs.mk: ${INSTALL} ${${group}TAG_ARGS} =
-C -o ${${group}OWN} -g ${${group}GRP} -m ${${group}MODE} \
/usr/main-src/share/mk/bsd.man.mk:MINSTALL?=3D ${INSTALL} ${TAG_ARGS} =
-o ${MANOWN} -g ${MANGRP} -m ${MANMODE}
/usr/main-src/share/mk/bsd.man.mk:MINSTALL?=3D ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},man} -o ${MANOWN} -g ${MANGRP} -m ${MANMODE}
/usr/main-src/share/mk/bsd.prog.mk: ${INSTALL} ${TAG_ARGS} ${STRIP} =
-o ${BINOWN} -g ${BINGRP} -m ${BINMODE} \
/usr/main-src/share/mk/bsd.prog.mk: ${INSTALL} =
${TAG_ARGS:D${TAG_ARGS},dbg} -o ${BINOWN} -g ${BINGRP} -m ${DEBUGMODE} \
/usr/main-src/share/mk/bsd.own.mk:HRDLINK?=3D -l h -o ${_LINKOWN} -g =
${_LINKGRP} -m ${_LINKMODE}
/usr/main-src/share/mk/bsd.own.mk:MANHRDLINK?=3D -l h -o ${MANOWN} -g =
${MANGRP} -m ${MANMODE}
/usr/main-src/share/mk/bsd.own.mk:SYMLINK?=3D -l s -o ${_SYMLINKOWN} =
-g ${_SYMLINKGRP} -m ${_SYMLINKMODE}
/usr/main-src/share/mk/bsd.own.mk:LSYMLINK?=3D -l s -o ${LIBOWN} -g =
${LIBGRP} -m ${LIBMODE}
/usr/main-src/share/mk/bsd.own.mk:RSYMLINK?=3D -l rs -o =
${_SYMLINKOWN} -g ${_SYMLINKGRP} -m ${_SYMLINKMODE}
/usr/main-src/share/zoneinfo/Makefile: -o ${BINOWN} -g ${BINGRP} -m =
${NOBINMODE} \
/usr/main-src/share/zoneinfo/Makefile: ${INSTALL} ${TAG_ARGS} -o =
${BINOWN} -g ${BINGRP} -m ${NOBINMODE} \
/usr/main-src/share/zoneinfo/Makefile: ${INSTALL} ${TAG_ARGS} -o =
${BINOWN} -g ${BINGRP} -m ${NOBINMODE} \
Thus it looks to me like installworld produces a new
date for cap_mkdb and, in the material below,
${WORLDTMP}/legacy/bin/cap_mkdb ends up having a
copy of that new date in the next buildworld .
> The code for the _bootstap_tools_links uses "cp -pf",
> not install, to establish part of . . ./tmp/legacy/bin/ .
> (Note: . . ./tmp/legacy/sbin -> ../bin so is a via a
> symbolic link.) Before the "cp -pf" there is a "rm -f"
> deleting the target file before the copy: the prior
> file in . . ./tmp/legacy/bin/ is never directly
> preserved. (The new copy might still be identical
> to the old one: the source path one might happen to
> be identical as well.)
>=20
> # Link the tools that we need for building but don't need to bootstrap =
because
> # the host version is known to be compatible into ${WORLDTMP}/legacy
> # We do this before building any of the bootstrap tools in case they =
depend on
> # the presence of any of the links (e.g. as m4/lex/awk)
> ${_bt}-links: .PHONY
>=20
> .for _tool in ${_bootstrap_tools_links}
> ${_bt}-link-${_tool}: .PHONY
> @rm -f "${WORLDTMP}/legacy/bin/${_tool}"; \
> source_path=3D`which ${_tool}`; \
> if [ ! -e "$${source_path}" ] ; then \
> echo "Cannot find host tool '${_tool}'"; false; \
> fi; \
> cp -pf "$${source_path}" "${WORLDTMP}/legacy/bin/${_tool}"
> ${_bt}-links: ${_bt}-link-${_tool}
> .endfor
>=20
> Note: This is for the !defined(BOOTSTRAP_ALL_TOOLS) case.
> Note: the code uses the abbreviation: _bt=3D =
_bootstrap-tools
>=20
> _bootstrap_tools_links is built mostly in terms of
> _basic_bootstrap_tools and _basic_bootstrap_tools_multilink from
> earlier logic. For reference, showing what ends up handled
> this way:
>=20
> # grep -r "_bootstrap_tools" /usr/main-src/Makefile* =
/usr/main-src/share/ | more
> /usr/main-src/Makefile.inc1:# _bootstrap_tools_links variable.
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3Dm4 lex
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3Dmtree
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3Dcat
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3Dcrunchide
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3Dcrunchgen
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3Dmkimg
> /usr/main-src/Makefile.inc1:_kerberos5_bootstrap_tools=3D \
> /usr/main-src/Makefile.inc1:.ORDER: =
${_kerberos5_bootstrap_tools:C/^/${_bt}-/g}
> /usr/main-src/Makefile.inc1:.for _tool in =
${_kerberos5_bootstrap_tools}
> /usr/main-src/Makefile.inc1:# The tools listed in =
_basic_bootstrap_tools will generally not be
> /usr/main-src/Makefile.inc1:# case we use the =
_basic_bootstrap_tools_multilink variable which is a list of
> =
/usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink=3Dusr.bin/gre=
p grep,egrep,fgrep
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink+=3Dbin/test=
test,[
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.bin/cut =
bin/expr usr.bin/gencat usr.bin/join \
> =
/usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink+=3Dusr.bin/aw=
k awk,nawk
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.bin/file2c
> =
/usr/main-src/Makefile.inc1:_basic_bootstrap_tools_multilink+=3Dusr.bin/bi=
ntrans uuencode,uudecode
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.bin/xargs
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.bin/cap_mkdb
> =
/usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.sbin/services_mk=
db usr.sbin/pwd_mkdb
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.bin/ldd
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dbin/chflags
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3Dsysctl
> =
/usr/main-src/Makefile.inc1:_other_bootstrap_tools+=3Dtools/build/cross-bu=
ild/fake_chflags
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.bin/mkfifo
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.bin/jot
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dsbin/md5
> /usr/main-src/Makefile.inc1:_basic_bootstrap_tools+=3Dusr.sbin/tzsetup
> =
/usr/main-src/Makefile.inc1:_other_bootstrap_tools+=3D${_basic_bootstrap_t=
ools}
> /usr/main-src/Makefile.inc1:.for _subdir _links in =
${_basic_bootstrap_tools_multilink}
> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=3D${_subdir}
> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=3Dusr.bin/bmake
> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=3Dlib/libbz2
> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=3Dlib/libz
> /usr/main-src/Makefile.inc1:_other_bootstrap_tools+=3Dlib/libcrypt
> /usr/main-src/Makefile.inc1:# All tools in _basic_bootstrap_tools have =
the same name as the subdirectory
> =
/usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3D${_basic_bootstrap_t=
ools:T}
> /usr/main-src/Makefile.inc1:.for _subdir _links in =
${_basic_bootstrap_tools_multilink}
> /usr/main-src/Makefile.inc1:_bootstrap_tools_links+=3D${_links:S/,/ =
/g}
> /usr/main-src/Makefile.inc1:.for _tool in ${_bootstrap_tools_links}
> /usr/main-src/Makefile.inc1: ${_kerberos5_bootstrap_tools} \
> /usr/main-src/Makefile.inc1: ${_other_bootstrap_tools} \
>=20
> What the prior installworld did for the analogous files is
> a separate issue.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F68956DB-38B8-45D7-ACF9-1D76644BFED7>