Date: Tue, 4 May 2021 00:10:16 -0700 From: Mark Millard <marklmi@yahoo.com> To: Ed Maste <emaste@freebsd.org> Cc: FreeBSD-STABLE Mailing List <freebsd-stable@freebsd.org>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files? Message-ID: <7D83A5C8-FDB4-46D6-A148-15011606B3BE@yahoo.com> In-Reply-To: <F5D8E009-E6E1-42DA-856C-A6D91DA74A40@yahoo.com> References: <35482701-95A3-48B2-9A8E-B7E0092119B1.ref@yahoo.com> <35482701-95A3-48B2-9A8E-B7E0092119B1@yahoo.com> <CAPyFy2DQRn1CeiF=yaF6NgpLZ1zU8psbV0xiz-DWiSUy=DM8NA@mail.gmail.com> <43F20589-A7C7-42FF-9020-09CEE037D1CD@yahoo.com> <91F820A1-8940-4246-A20A-E62685F50079@yahoo.com> <F5D8E009-E6E1-42DA-856C-A6D91DA74A40@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2021-May-3, at 21:27, Mark Millard <marklmi at yahoo.com> wrote:
> On 2021-May-3, at 19:26, Mark Millard <marklmi at yahoo.com> wrote:
>=20
>> On 2021-May-3, at 10:51, Mark Millard <marklmi at yahoo.com> wrote:
>>=20
>>> On 2021-May-3, at 07:47, Ed Maste <emaste at freebsd.org> wrote:
>>>=20
>>>> On Thu, 29 Apr 2021 at 02:50, Mark Millard via freebsd-current
>>>> <freebsd-current@freebsd.org> wrote:
>>>>>=20
>>>>> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping and =
/usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping differ
>>>>> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/sbin/ping6 and =
/usr/obj/DESTDIRs/13_0R-CA7-poud/sbin/ping6 differ
>>>>> Files /usr/obj/DESTDIRs/13_0R-CA7-chroot/usr/bin/ntpq and =
/usr/obj/DESTDIRs/13_0R-CA7-poud/usr/bin/ntpq differ...
>>>>=20
>>>> This is unexpected. Unfortunately I haven't looked at =
reproducibility
>>>> in a while, and my work was all on x86. This could be a regression =
or
>>>> a longstanding issue with arm64.
>>>>=20
>>>> If you install the diffoscope package (py37-diffoscope) and run it =
on
>>>> the two directories / files it should give a more convenient view =
of
>>>> the differences. (Or, if you can make a tarball of the differing =
files
>>>> I can take a look.)
>>>=20
>>> I no longer have the same content in those directory
>>> trees: newer rebuild and the same buildworld used to
>>> installworld to both places, instead of 2 different
>>> buildworld's. I'm also unsure how reproducible getting
>>> differences was.
>>>=20
>>> I can eventually do experiments to test multiple separate
>>> buildworld's and installworld's, but the machine is busy
>>> building ports and the llvm builds involved means it
>>> will be some time before I'd switch activities. And the
>>> buildworld's involve llvm builds as well and take notable
>>> time themselves. So my next comparison will not be any
>>> time soon.
>>>=20
>>> I'll let you know if I manage to generate another example,
>>> this time being sure to keep the data. If I try multiple
>>> times without finding any differences, I'll eventually
>>> decide "enough is enough" and let you know.
>>=20
>> I've still got a long ways to go to do the first
>> actual comparison of builds.
>>=20
>> But I'll note that I've built and stalled py37-diffoscope
>> (new to me). A basic quick test showed that it reports:
>>=20
>> W: diffoscope.main: Fuzzy-matching is currently disabled as the =
"tlsh" module is unavailable.
>>=20
>> As I'm not familiar with the tool, you might need to send
>> notes about how you want me to use the tool to get the
>> output that you would want. (And, so, I get to learn . . .)
>=20
> I've tried another experiment (* in the path matches "28" and "30"):
>=20
> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh
> $<3/>2021-05-03 21:08:48 W: diffoscope.main: Fuzzy-matching is =
currently disabled as the "tlsh" module is unavailable.
> $<3/>Traceback (most recent call last):
> File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", =
line 745, in main
> sys.exit(run_diffoscope(parsed_args))
> File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", =
line 677, in run_diffoscope
> difference =3D load_diff_from_path(path1)
> File =
"/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", =
line 31, in load_diff_from_path
> return load_diff(codecs.getreader("utf-8")(fp), path)
> File =
"/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", =
line 35, in load_diff
> return JSONReaderV1().load(fp, path)
> File =
"/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", =
line 33, in load
> raw =3D json.load(fp)
> File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load
> return loads(fp.read(),
> File "/usr/local/lib/python3.7/codecs.py", line 504, in read
> newchars, decodedbytes =3D self.decode(data, self.errors)
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position =
18: invalid start byte
>=20
> The two older snapshots of a Boot Environment have
> bin/sh files that compare equal. But every program I
> tried the above sort of thing against on got the same
> UnicodeDecodeError result from diffoscope, byte value
> and position matching.
>=20
> These snapshots have more than an installworld in them
> and so are messy to compare overall. But the
> installworld (and installkernel) content show similar
> differences to what I reported before as far as
> example files with differences go. But this is aarch64,
> not armv7.
>=20
> It will still be notable time before I have simple
> installworld tree's to compare.
While waiting for the 2nd buildworld to complete, I used
the 1st buildworld's materials to installworld twice (to
different, empty directory trees) and then did a diff.
The diff reported:
# diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ =
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm2/ | more
diff: /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local: No =
such file or directory
diff: =
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/sys/pjdfstest/tests/t=
ests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tes=
ts/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests/tests=
/tests/tests/tests/tests/tests/tests/tests: Too many levels of symbolic =
links
FYI:
# ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local
lrwxr-xr-x 1 root wheel 14 May 3 23:18:22 2021 =
/usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/usr/tests/local -> =
../local/tests
So this looks like the expected result and suggests that
the problem(s) are at the buildworld stage. (No surprise.)
I should be sleeping by the time the 2nd buildworld for
cortex-a72 (aarch64) is done. (That is what I decided
to test initially.) So it will be more hours until I have
basic comparison results from 2 distinct buildworlds.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7D83A5C8-FDB4-46D6-A148-15011606B3BE>