Date: Thu, 23 Feb 2023 19:31:07 -0800 From: Mark Millard <marklmi@yahoo.com> To: "Simon J. Gerraty" <sjg@juniper.net> Cc: Bryan Drewery <bdrewery@FreeBSD.org>, Current FreeBSD <freebsd-current@FreeBSD.org>, Peter <pmc@citylink.dinoex.sub.org> Subject: Re: FYI: Why META_MODE rebuilds so much for building again after installworld (no source changes) Message-ID: <F02F01EE-9866-4F37-884B-74A2665A5F08@yahoo.com> In-Reply-To: <93460.1677202939@kaos.jnpr.net> References: <B74790D9-FBC2-4818-BEAF-34E5B705C460@yahoo.com> <3345EBA5-A09C-4E3F-B94D-39F57F56BDBB@yahoo.com> <DB0C7B41-2101-4C5C-BFC8-3C95CC0B9F6F@yahoo.com> <73088.1611797582@kaos.jnpr.net> <CB7040D0-3BF4-496F-A54F-87E5378016E0@yahoo.com> <F6BF110D-7855-4A10-A53F-52B34282234F@yahoo.com> <10819.1677108389@kaos.jnpr.net> <76FA98EF-6184-4D7E-A01F-0EE8117D0D10@yahoo.com> <29887.1677115125@kaos.jnpr.net> <27790339-240F-4C97-97C7-38AFD8DE03D5@yahoo.com> <7FB6F619-6E71-4075-8A6C-573564371DD5@yahoo.com> <2655.1677134606@kaos.jnpr.net> <242BB478-B2FE-4BCC-A56E-098F3FEB3EE1@yahoo.com> <42586.1677183334@kaos.jnpr.net> <AE95CF5D-0B7E-4DA3-8777-5FA47E1751D8@yahoo.com> <30.1677189836@kaos.jnpr.net> <1B5FCF8A-0DFD-4246-8464-65A44A40529F@yahoo.com> <93460.1677202939@kaos.jnpr.net>
index | next in thread | previous in thread | raw e-mail
[I'm back at this again after some time on other subjects.]
On Feb 23, 2023, at 17:42, Simon J. Gerraty <sjg@juniper.net> wrote:
> Mark Millard <marklmi@yahoo.com> wrote:
>
>> Simplifying context . . .
>> . . .
>>> As I mentioned previously, there is no variablity of OBJTOP within the
>>> context of a single make instance - at least not once it starts running
>>> targets.
>>>
>>>> . . .
>>>
>>> .MAKE.META.IGNORE_PATHS += ${OBJTOP}/tmp/legacy/usr
>>
>> I'll use that definition line for the below.
>
> Ok.
>
>>> should result in nothing under ${OBJTOP}/tmp/legacy/usr causing a target
>>> to be out of date - just because it is newer.
>>
>> I'll ignore there that that is skipping too much
>> and just show what happens for the 2nd buildkernel
>> of 2 in a row when I use that exact line for both
>> make runs.
>>
>> First counts of the "is newer than" lines, counting
>> separate program names separately:
>>
>> # cat /usr/obj/BUILDs/main-amd64-nodbg-clang/sys-typescripts/typescript-make-amd64-nodbg-clang-amd64-host-2023-02-23:16:15:18 | grep "is newer than the target" | sed -
>> e "s@^.*: file '@file '@" | sort | uniq -c | sort -rn | more
>> 2553 file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath' is newer than the target...
>> 1001 file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln' is newer than the target...
>>
>
> It is of course critical to know what OBJDIR is at this point.
> also can you show me the line in the meta file that is matching.
So, using specific "is newer than" notice examples instead of
the above counts . . .
First some output log lines around a few sbin/realpath "is newer than"
related Building lines, with the .info lines in place now (I've
got both kmod.mk and kern.mk with the .info line, likely producing
redundant output but I did not know up front for sure):
make[4]: "/usr/main-src/sys/conf/kmod.mk" line 72:
.CURDIR=/usr/main-src/sys/modules/aac
.OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
.MAKE.META.IGNORE_PATHS=/bin/sh /bin /lib /rescue /sbin /usr/bin /usr/lib /usr/sbin /usr/share /usr/include /usr/local/etc/libmap.d /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.am
d64/sys/GENERIC-NODBG/modules/usr/main-src/tmp/legacy/usr
make[4]: "/usr/main-src/sys/conf/kern.mk" line 3: .CURDIR=/usr/main-src/sys/modules/aac
.OBJDIR=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
OBJTOP=/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src
.MAKE.META.IGNORE_PATHS=/bin/sh /bin /lib /rescue /sbin /usr/bin /usr/lib /usr/sbin /usr/share /usr/include /usr/local/etc/libmap.d /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.am
d64/sys/GENERIC-NODBG/modules/usr/main-src/tmp/legacy/usr
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine.meta: 23: file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath' is newer than the target...
Building /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine
machine -> /usr/main-src/sys/amd64/include
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/x86.meta: 23: file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath' is newer than the target...
Building /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/x86
x86 -> /usr/main-src/sys/x86/include
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/i386.meta: 23: file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath' is newer than the target...
Building /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/i386
i386 -> /usr/main-src/sys/i386/include
Some lines just following the above are some sbin/ln "is newer than"
related Building lines:
Skipping meta for beforedepend: .PHONY
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h.meta: 12: file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln' is newer than the target...
Building /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_cam.h.meta: 12: file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln' is newer than the target...
Building /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_cam.h
/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_aac.h.meta: 12: file '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln' is newer than the target...
Building /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_aac.h
For reference for a sbin/realpath "is newer than" example,
the sys/modules/aac/machine.meta :
# Meta data file /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/machine.meta
CMD @case machine in machine) path=/usr/main-src/sys/amd64/include ;; *) path=/usr/main-src/sys/machine/include ;; esac ; path=`realpath $path`; echo machine "->" $path ; ln -fns $path machine
CWD /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
TARGET machine
-- command output --
machine -> /usr/main-src/sys/amd64/include
-- filemon acquired metadata --
# filemon version 5
# Target pid 84728
# Start 1677207916.433304
V 5
E 84729 /bin/sh
R 84729 /etc/libmap.conf
R 84729 /usr/local/etc/libmap.d
R 84729 /usr/local/etc/libmap.d/mesa.conf
R 84729 /var/run/ld-elf.so.hints
R 84729 /lib/libedit.so.8
R 84729 /lib/libc.so.7
R 84729 /lib/libtinfow.so.9
R 84729 /usr/share/locale/C.UTF-8/LC_CTYPE
F 84729 84730
E 84730 /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/realpath
R 84730 /etc/libmap.conf
R 84730 /usr/local/etc/libmap.d
R 84730 /usr/local/etc/libmap.d/mesa.conf
R 84730 /var/run/ld-elf.so.hints
R 84730 /lib/libc.so.7
X 84730 0 0
E 84729 /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
R 84729 /etc/libmap.conf
R 84729 /usr/local/etc/libmap.d
R 84729 /usr/local/etc/libmap.d/mesa.conf
R 84729 /var/run/ld-elf.so.hints
R 84729 /lib/libc.so.7
D 84729 machine
L 84729 '/usr/main-src/sys/amd64/include' 'machine'
X 84729 0 0
# Stop 1677207916.435305
# Bye bye
For reference for a sbin/ln "is newer than" example,
the sys/modules/aac/opt_scsi.h.meta :
# Meta data file /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac/opt_scsi.h.meta
CMD ln -sf /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/opt_scsi.h opt_scsi.h
CWD /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/modules/usr/main-src/sys/modules/aac
TARGET opt_scsi.h
-- command output --
-- filemon acquired metadata --
# filemon version 5
# Target pid 84728
# Start 1677207916.439304
V 5
E 84735 /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/tmp/legacy/usr/sbin/ln
R 84735 /etc/libmap.conf
R 84735 /usr/local/etc/libmap.d
R 84735 /usr/local/etc/libmap.d/mesa.conf
R 84735 /var/run/ld-elf.so.hints
R 84735 /lib/libc.so.7
D 84735 opt_scsi.h
L 84735 '/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64/sys/GENERIC-NODBG/opt_scsi.h' 'opt_scsi.h'
X 84735 0 0
# Stop 1677207916.440304
# Bye bye
> If you add the .info line I suggested in kern.mk or kmod.mk
> you should get some useful info.
Added to both for now. The material above has the output
from them from just before a "is newer than" notice.
>> Thousands of rebuilt things based on:
>>
>> . . ./tmp/legacy/usr/sbin/realpath
>> . . ./tmp/legacy/usr/sbin/ln
>>
>> It appears that buildkernel does not use an OBJTOP definition
>> that references:
>
> It is possible it does not have a "normal" value for it.
> Kernel builds start in the objdir so .CURDIR is actually under
> OBJTOP, so unless the makefiles have appropriate logic for that case,
> and depending on how OBJTOP is derrived, you may have a bogus value.
>
> The fix in that case would be the makefile setting OBJTOP.
>
>>
>> /usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.amd64
>>
>> in my context.
>>
>> For reference of Build lines paired with a few of those "is newer
>> than" lines:
>>
>>
>> I've still no clue of a notation that avoids this for
>> my choice to use personal MAKEOBJDIRPREFIX paths:
>
> That should work, looking at share/mk/src.sys.obj.mk
> though looks like you might get
>
> OBJROOT?= ${_default_makeobjdirprefix}${SRCTOP}/
>
> which may not be valid for a kernel build.
> At least ${.OBJDIR} will not have that ${OBJROOT} as a prefix
> and some of the settings for OBJTOP look just wrong to me:
>
> OBJTOP:= ${OBJROOT:H}
>
> is the opposite of what the relationship b/w OBJROOT and OBJTOP are in
> DIRDEPS_BUILD but it looks like
>
> OBJTOP:= ${OBJROOT}${MACHINE}.${MACHINE_ARCH}
>
> is more likely to be used, but given the above default for OBJROOT
> that is unlikely to work for a kernel build.
>
> Looks like share/mk/src.sys.obj.mk grok's SB (the setup I/we use) so you
> can take control by setting SB and SB_OBJROOT to anything you like and
> it should be used for OBJROOT and presumably hence OBJTOP even for a
> kernel build. Though I normally use MAKEOBJDIR (similar to the way it
> is set in _default_makeobjdir), I don't know how well that works with
> legacy targets though - there's a lot of baked in assumptions about
> using MAKEOBJDIRPREFIX.
>
> Again that .info line I gave you would provide some useful clues.
>
===
Mark Millard
marklmi at yahoo.com
help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F02F01EE-9866-4F37-884B-74A2665A5F08>