From owner-freebsd-stable@freebsd.org Tue May 4 17:17:42 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6EA415FAC30 for ; Tue, 4 May 2021 17:17:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic304-24.consmr.mail.gq1.yahoo.com (sonic304-24.consmr.mail.gq1.yahoo.com [98.137.68.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FZRNw4BSlz3nFm for ; Tue, 4 May 2021 17:17:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1620148654; bh=5hPBqD1eQ4r3oEFTQXB4jZ+ruJJjDNzv3EnxgpC8AxD=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=JDGeFUZmtnZPWcq5PjQ/zWtytZOXvtgjBjnCx8pHb+kxdI9idJQEAsFrEKViietcW0OdKHkhssLQWbgOqOsi9nNprQ1qySXqEc1iyoOKRoHUCRbg1482dkSW+FcVoqvdwoJLs7mhexKSWjuSkAWz4mKEY5Sa658lThZ4FENlxbpqCzVksDFurHdbKRHc437C+OnMQUTyq/PtcAiXOYPNWbctj9MWbRfkF8iC3kl4/yJ2qpi1oLsZYJ5kvK3LqR24820qQhL1Gc3ArHyCywBgW8cfY8oWqJ6pE+ELKBAw7RMA8fpQ1iJuNgRG3FyjsC44hhIPXc9i9yxYHPX22KYyRg== X-YMail-OSG: jab7ix8VM1kUz_1QVWzmsslDaw.he.zwmC1cgpNUdVfFLIGPbOT4GOAyCIo9J8v vfHqo1GN4HqHm2jA4Zsae2XMTmHGMveWZ2lVQ3FQvctlbCOcJQFbi.Ggjlv4y0BMHIZnTNGLbC4t oOfISLjDLHdfRz7S2uK3Ay.Gvd0wLllS1s58aKkzrJ647WZMZkf3cC9YSugPH0JVgnw8xjuhvuKO fEo1P0K._Uwjb.f3HhNHTpKQ1J9vstozA855DU3VpcUJLdc59NGIrVUFdI9mvEUEQhell._lHiMi 9p6_sN0.zzAFMrliE0mKeqpMirndwXFi60IDqZNVb15ysVc_ygxS2OS6KsBqPOSJBWOFgUWKvuB1 nMRJKxmUEO2zp4Wf1hjGr2yBOgReqwErXL07dMKX2i9s.O8.R6m2jwGkwAb9tCyEa0pfLHMyDv5q KeGrc85GZqVHRqlkFQnJ3yCiGlftmQlgDOygnM.c7tGx8WhAoLgKHZ_gA931RkflI6p1Y6aCk3cE boRyh2JxKghqyVhceFV8ulVg1Ws4HErccxE.Uk9wVWpg0O1hc3LK6hDKnThhoW_W5qCDXZcLOZhA UB2S8I7USm.LcAs39jsFegNmtqh2oNUi4sy0jk59KAoQATJ01tkJvbcHt5eYsrdviNTpINIFUXaJ X23LiGK37PrE5I2mE85yVDNEYVRwaBe8hQV1z3g__nYRCHg63MmOUMQvYVX9fsHJt94VeYsiL59S z_x.tqcpxtQL0_xXMX9oFCuq4c4S5o0uBz_9GIKRr7IE6_0qX8uOHaCF6bXh1MjtkFwRbbbJi4cP jQwUNqWIm_MDPoYbyGk2UvWLRL252IFwwHXDNyS3hzWQDR5re3UOWVvWR7tuO0y1SN5EhEuwIqzB VpJpc0SHlrt2OLHOCTxBC9jeHGWd72KwVezR9z28ahTfx2wo0e58ohfCDJdllg2TPXXMHphSAsjg qnwdJM_zLc6eY9hjxtoEyZnsHXh4E7tZjwpGb8NznMxeRQrN1IqiRjzlB2CfAZlYW6dA.gTBmNJL 8jmvmQov0zh8jrgSKChh_fU2tYDLrvLo7iFVjgE0ySrINlltNvmVHs0AiTkW7GtueaHc8fSapd30 2bgy9JVIhtbQ19YhwQMVqWm.bjIrinVifrJV5XzcU7szUkaBs6sXeDjdYpLgYY4A7tSdHK2QKbqr C3zRGuV_0dCV3u45j0dMG.qI1upHqo8PLzDjT5zmzEkHLnz06j5U62MM9QQjuYUgMRVrm20NaD5k BkFSokCsXkYAlzTDhNhOWeaU1IbCVUZBYk.sw5xiqy858RId5yXbWXCsQjRJEUu4va9CalYuQyai p08nKOcl9LHV1Ag2p6noFXYuY9Wnuroffa_qaWYch3y6D3tJ_mM33iRDJGyNXPSgkXHCiEUgRd0P ROsAB4I_dqB1SsdHeYjL8TuA2RRQwS3ul0dSQ4JnJyXT11DvNEGRD6oQXYEo02Ysjn01mTsL71qC Ki82fUXWZ4rkR6TBklKKITouuPXV6ZT1.jqegkV0k3WQYn2tS_gG_UwnKQg4cNoOydUzR0b2K.K6 Pj09JjM8AfI8eym4pmTlLN5h.tfhLs5R.DnZt25f1JDApoNYO8DZsvgu6_x9FW.Dmn4nXWjMIulW Cdtp4ruSTBvli94fI_DSnIrCXXjqIPivMt4MYuhVqoAQXsdNq8S9bFj_AKfw5FIUxtPtrNPcGUTP O6xFCAqP0cz5AX3rziiBxdfaBiz875x09rIRw8mFIRjVGx5Tr9Bsm0d_LNTAyyBvtVpOlo.8USjO YHZYHeZdlkhmNL1dyyGy7BktDNQanCu5nF463V5iZ6ApyrHvW91rShRPPJlGDyqAVJFrknZ6uNjI VLCdKCJbQrQP_KnE_U7cjN79hHz1Lc9qbLuWvAx8k1DhWc1d.KcDKV7J7NgsttN0Bkhtx2sugwBp 4f7Cl6.9Icd9Wh9KKbWZcq6lCWRJIXCe31qRluX8roI_k1fjbS3qsDke9kGJmUDSNVz3HkONkIa5 CNgQ8M4jXEEmrhveILjhz3AFGZQw4na4nnCfgJK_DROFPFe9oVTQm.aNSVbxp3HcU3cO2V6gLMRw xUSOoR0Ghp07T6yq9ZRbJXGLiZREYRH6URsWzUvjUzoWIrbbIPupWdWXL88PbQv4cTcZyGXcjpOd LvFPQW5cdGKUwBipgTqEFSBKHhzDpsTMbSRoC8lK5vzD8abZkj7k4GC4.PAMrmFZb.j1TDDwhhuU _aj5ubZARUjJi3mFjcNp3OTY_0OWvctLIF13eCmzV41EaWXAHekCOzcYreDsjPLb689xpn3qzb0Q p_vypr9uyQmq9gRx.cRrbDU4QEhR11cEAKG6zl6dc41tMD7daJ4FKcpdUb9aGbHgEPNTnl2oK9rT X_GB_UV64r.I4IDacd.KfbkeNhPAXPmVzRC1ehTtlrbMTENialJc48Maag7iW_pnwzGooUu8oJ3Y amGfi182KfS8wD32ZW3D5zUYLZeOR77znfGsn9GEcpBdYFkhCvdtUUlIQKJWr1BKWN66TIPXPTZl LehESH8xxCHeNIWkcebaLK8LtsamsK2bGtteV1tIDBlgdkHUToy_BJO1JP0C3CPsVPsJS6GW0LFl 2TaMrYfGBcaqhzM9muCCTnfM4kj4JDxFGne_Eb3pAPNcybP7WYiEoLfRDcof0eb0MEg-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic304.consmr.mail.gq1.yahoo.com with HTTP; Tue, 4 May 2021 17:17:34 +0000 Received: by kubenode542.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 3916381615388a11c064df3a50eecc99; Tue, 04 May 2021 17:17:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files? From: Mark Millard In-Reply-To: <3FC6BCDD-5865-4B5D-8238-3EC38AD4E78C@yahoo.com> Date: Tue, 4 May 2021 10:17:27 -0700 Cc: FreeBSD-STABLE Mailing List , freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: References: <35482701-95A3-48B2-9A8E-B7E0092119B1.ref@yahoo.com> <35482701-95A3-48B2-9A8E-B7E0092119B1@yahoo.com> <43F20589-A7C7-42FF-9020-09CEE037D1CD@yahoo.com> <91F820A1-8940-4246-A20A-E62685F50079@yahoo.com> <27B5086A-7C98-4AE6-885A-CC7C7BD2F64B@yahoo.com> <3FC6BCDD-5865-4B5D-8238-3EC38AD4E78C@yahoo.com> To: Ed Maste X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 4FZRNw4BSlz3nFm X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.205:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.68.205:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; BLOCKLISTDE_FAIL(0.00)[98.137.68.205:query timed out]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.205:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.205:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 May 2021 17:17:42 -0000 [Just adding readelf -S info since it seems to show more.] On 2021-May-4, at 10:01, Mark Millard wrote: > On 2021-May-4, at 08:51, Mark Millard wrote: >=20 >> On 2021-May-4, at 06:01, Ed Maste wrote: >>=20 >>> On Mon, 3 May 2021 at 22:26, Mark Millard wrote: >>>>=20 >>>> But I'll note that I've built and stalled py37-diffoscope >>>> (new to me). A basic quick test showed that it reports: >>>>=20 >>>> W: diffoscope.main: Fuzzy-matching is currently disabled as the = "tlsh" module is unavailable. >>>=20 >>> I just looked up tlsh - its "A Locality Sensitive Hash"; I presume >>> diffoscope uses it to infer file renames. I believe the warning >>> emitted here should have no impact on the output we're looking for. >>=20 >> Okay. >>=20 >>> As far as the utf-8 issues go, diffoscope requires a utf-8 locale = and >>> I suspect that is the issue. If you don't have LANG set already, try >>> setting LANG=3DC.UTF-8 in your environment. >>=20 >> That is not the issue for the UnicodeDecodeError: >>=20 >> # echo $LANG >> C.UTF-8 >>=20 >> # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh >> $<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is = currently disabled as the "tlsh" module is unavailable. >> $<3/>Traceback (most recent call last): >> File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", = line 745, in main >> sys.exit(run_diffoscope(parsed_args)) >> File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", = line 677, in run_diffoscope >> difference =3D load_diff_from_path(path1) >> File = "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", = line 31, in load_diff_from_path >> return load_diff(codecs.getreader("utf-8")(fp), path) >> File = "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", = line 35, in load_diff >> return JSONReaderV1().load(fp, path) >> File = "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", = line 33, in load >> raw =3D json.load(fp) >> File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load >> return loads(fp.read(), >> File "/usr/local/lib/python3.7/codecs.py", line 504, in read >> newchars, decodedbytes =3D self.decode(data, self.errors) >> UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position = 18: invalid start byte >>=20 >=20 > Well, the list of differing files is huge. But this seems to > be .gnu_debuglink content for the area it is in. Specifically: the last 4 bytes of the .gnu_debuglink section. > I'll note > that I did installworld but not the likes of distrib-dirs > or distribution this time. >=20 > This test did buildworld to two distinct directories: >=20 > zroot/BUILDs/13_0R-CA72-nodbg-clang 5.13G 118G 5.13G = /usr/obj/BUILDs/13_0R-CA72-nodbg-clang > zroot/BUILDs/13_0R-CA72-nodbg-clang-alt 4.28G 118G 4.28G = /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt >=20 > and installworld to 2 distinct directories: >=20 > zroot/DESTDIRs/13_0R-CA72-instwrld-alt 1.44G 118G 1.44G = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt > zroot/DESTDIRs/13_0R-CA72-instwrld-norm 1.44G 118G 1.44G = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm >=20 > Previously (armv7 target) I had built, installed, rebuilt > to same directory (after clean-out) and installed to an > alternate directory. That had gotten only a few files > different but I do not know (yet) if it was the procedural > difference that made the difference. >=20 > Prefix of the list of different files this time: >=20 > # diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/ | more > Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/[ and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/[ differ > Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat differ > Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags differ > Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio differ > . . . >=20 > Looking, aarch64 seems to typically get a back-to-back > sequence of 4 bytes different in native programs in my > builds: >=20 > # cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat > 00003bd4 1d 65 > 00003bd5 eb a3 > 00003bd6 bb ca > 00003bd7 8e 1a >=20 > # ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat > -r-xr-xr-x 1 root wheel 18448 May 4 08:55:01 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat > -r-xr-xr-x 1 root wheel 18448 May 3 23:16:36 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat >=20 > Sections: > Idx Name Size VMA LMA File = off Algn > . . . > 25 .gnu_debuglink 00000010 0000000000000000 0000000000000000 = 00003bc8 2**0 > CONTENTS, READONLY Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align . . . [25] .comment PROGBITS 0000000000000000 00002c70 00000000000000b2 0000000000000001 MS 0 0 1 [26] .gnu_debuglink PROGBITS 0000000000000000 00003bc8 0000000000000010 0000000000000000 0 0 1 [27] .shstrtab STRTAB 0000000000000000 00003bd8 0000000000000100 0000000000000000 0 0 1 [28] .symtab SYMTAB 0000000000000000 00002d28 0000000000000ea0 0000000000000018 29 96 8 [29] .strtab STRTAB 0000000000000000 00003cd8 00000000000003b3 0000000000000000 0 0 1 > 00003bd4-00003bc8 =3D=3D 0xC Note: 0xC+0x4 =3D=3D 0x10 (the size), so the last 4 bytes of .gnu_debuglink > # cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags > 00002208 88 a1 > 00002209 e6 40 > 0000220a 60 94 > 0000220b bf ce >=20 > # ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags > -r-xr-xr-x 1 root wheel 11440 May 4 08:55:01 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags > -r-xr-xr-x 1 root wheel 11440 May 3 23:16:36 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags >=20 > Sections: > Idx Name Size VMA LMA File = off Algn > . . . > 25 .gnu_debuglink 00000014 0000000000000000 0000000000000000 = 000021f8 2**0 > CONTENTS, READONLY Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align . . . [25] .comment PROGBITS 0000000000000000 000016d8 00000000000000b2 0000000000000001 MS 0 0 1 [26] .gnu_debuglink PROGBITS 0000000000000000 000021f8 0000000000000014 0000000000000000 0 0 1 [27] .shstrtab STRTAB 0000000000000000 0000220c 0000000000000100 0000000000000000 0 0 1 [28] .symtab SYMTAB 0000000000000000 00001790 0000000000000a68 0000000000000018 29 83 8 [29] .strtab STRTAB 0000000000000000 0000230c 000000000000021f 0000000000000000 0 0 1 > 00002208-000021f8 =3D=3D 0x10 Note: 0x10+0x4 =3D=3D 0x14 (the size), so the last 4 bytes of .gnu_debuglink > # cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio > 000050c4 6b 3e > 000050c5 08 ca > 000050c6 7a 2f > 000050c7 5d 64 >=20 > # ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio > -r-xr-xr-x 1 root wheel 23728 May 4 08:55:01 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio > -r-xr-xr-x 1 root wheel 23728 May 3 23:16:37 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio >=20 > Sections: > Idx Name Size VMA LMA File = off Algn > . . . > 25 .gnu_debuglink 00000010 0000000000000000 0000000000000000 = 000050b8 2**0 > CONTENTS, READONLY Section Headers: [Nr] Name Type Address Offset Size EntSize Flags Link Info Align . . . [25] .comment PROGBITS 0000000000000000 00004298 00000000000000b2 0000000000000001 MS 0 0 1 [26] .gnu_debuglink PROGBITS 0000000000000000 000050b8 0000000000000010 0000000000000000 0 0 1 [27] .shstrtab STRTAB 0000000000000000 000050c8 0000000000000100 0000000000000000 0 0 1 [28] .symtab SYMTAB 0000000000000000 00004350 0000000000000d68 0000000000000018 29 100 8 [29] .strtab STRTAB 0000000000000000 000051c8 0000000000000363 0000000000000000 0 0 1 > 000050c4-000050b8 =3D=3D 0xC Note: 0xC+0x4 =3D=3D 0x10 (the size), so the last 4 bytes of .gnu_debuglink > For all I know, some individual byte(s) in the 4 might accidentally > match sometimes. The addition offset after .gnu_debuglink's file > offset does vary (0xC and 0x10 above). Specifically: the last 4 bytes of the .gnu_debuglink section. > The content of those differences do not look like > file path components, for example the 0x08 byte. >=20 > I do build with: >=20 > # Avoid stripping but do not control host -g status as well: > DEBUG_FLAGS+=3D > # > WITH_REPRODUCIBLE_BUILD=3D > WITH_DEBUG_FILES=3D >=20 > But that was true for the earlier armv7 target example > that I reported that only got a few files with > differences. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)