From owner-freebsd-stable@freebsd.org Tue May 4 17:01:25 2021 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 54B875FA0BB for ; Tue, 4 May 2021 17:01:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-19.consmr.mail.gq1.yahoo.com (sonic314-19.consmr.mail.gq1.yahoo.com [98.137.69.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4FZR2D2xzhz3mQd for ; Tue, 4 May 2021 17:01:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1620147682; bh=7BIiPqAM/LhnxKPot5GI2Bflf4HIDLcquhABCiIfFUv=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=s7VMdMiOJqMeJiQ8xlHc0dLIvap39FPGG7XIgeSi3Flh0LsCIoU05zjzJIJ5HmIiOf8JyJNUSsL0cHnjlI9vy6RmvmOuuTikzH6EvyTUul/igqDrAYHJhtLbuGRsfDvX3i2oESWnnfDcaB/9viXlFIxdB/bFLf58KfCA67CevQCCyMbhz5/WQAjTgqTL3T03mKCd4lTS74ZInUxRQnSDB+tT+MerWaxnL5d/99p0KWOAPeBn4dQ0u5ItH+jgGMn+GfMO12gxZwVeyzv+AOllYvWbvcFhHgU/Imo+n+MNLwltbMc4+hcfBZKnFqvJqhcMV6mFe+S9e2UbM9MpxbZNSA== X-YMail-OSG: oslOhY8VM1lsaDBsmMAnZPpJ60kzzIGgsY6UKYYTArp2w2mzorUW9pZcWgOOxpp QUsOkyumH5E5q9GE1A_4B..vweaZMa2c_ZcNb4isKhJExyoDUtD_lThKQewFu5XKodirLPUqgQch REGW0buuMAh1CjiH0Du8z35GUktdvhSZbQ7s9hUD5cJbC2CapO_IJKABSiK6xh2zmtaFAiIsHjMX AoU_J2b2byqS4taqatCAqVVf.VkaFprgcpPgVVykRoF907YtNgD7JE2jjA0Y5Uwl7jU2Vv6Hx6_N ieCX4VXrasBoNRkPUaa7Kg13R7KbAOgZygblBHX1ZxdmIQQFM44sB6pKETTuoaJ2NQHmakG8Kzix 8sihZWjORFNra2qeFxBmRjF_Du28G1rt9.gz38vzSUmyaIICaYaeQJ.xI_QditrmY93NZG3aoteU CsP9iyvyh4H8c6UOOe6T9cguzGMKPusAsA7FpFKMSDGHpIwUZlgz9D5qHa_IFmghsV2mFaTEkrK9 tP.ssaSUKugVwNvu2GmFD.mvxsU1f15WoxjF.rlXFbkSTQFeOKAChQdO.KmrEgwed4XwuaOabBPx qCgIHxl7OIKjocqWiqZhshbGw_cJo0zGfS3Grk56QJFQUh8haZ7Ir_4lme.7eztBTkot0WL_UoOI h6NJ41Kou2IJSS_auBtgtJePFKQgTaZRBytWUJPoJ2bXXp.XK12x0XW6Rr7GVk.3oh6rTi2eHgzm ygR5a0jwtPmCiLpdfzROfdC2fgiKgAuxXXrbgZ.dvyfQIcx3iCIOjrajX8mVATnVCD4VdMd0JHn. na5ilpcomsKx8Zhlm4fL_vmGmFYOc6qwu3apS1Vzu5gWe8vOtHi6OexoHHcv_NCjAJ9kYj_aCSVz eoWFAlmE52j5rKAdkYlz_t6lKGtKg_QfDfAMkU6lQ.itW3X7km2oHFqaMyJicMmKkojt.eYpYhXY hJeNAA4fO_9YeSer9DrJEB2FkT2Df0pVuhyvEF49bHWUW.1bwzG_CCuMScNnxs3xoQ_NWtBBm6xY FdhOpdtI06vDz5pjbk0iTvSjys4zGH4pHwPXzlq_GKJyOKxtTD9a7KpTl_JoGOpe0MawnQhfC_OG zV.t8hG8HkV4huGihHwGP5Wytzj0be073i35_HNrCoFP71DTqH1j_Cd1BZWpnQJW97Nunf2K1G9N OlYCYKL6Fsxhh6oDY4b8vcQAzFN7suNZBu6Hm5TK4Gyq.SWr7yuB2iLqm3ImMCGqP6qhb9eI1mJE M.bcO7haqbbhZDqmamR8eZ2UjApheISJhMvkOHad7C9AI7LufNSGU_w2rdqVlbFjWYFvSGA2d4bY CCgBMAPHtZPgaq_KQ1Fp5EyU0pCijQk6zeUTltQaiB.PkBg0i0qkM25GaYaX7qYimiEntLfOMHyd kTH4.ougDx4IfzKHtoDd7eJsCJA4aZ1RTQxFN1Qojp7yQRpe62S5f.4lPKl2p9G3Ez_pi34QXX6r cnkcLjqjvJ3Af9Aw3wWsZXSDnx_Xmt8Apph1_2dFktM6qvfiaCVqv_BBdWRHe2YtisKVlYqS3xj5 NJOQTVAi5CBaT5FitHF9WHTXbKWeGwWeEcutLcL3Fo2R2eXTSCpjuYwqyPEdCYSx87IEfH9.sDa0 N2ZLW_upJQOk1XFSo06O6_04qDYOC.iflYNNVsv73s6hd9j9Za_Xw59xKFsk.sb6tVd.u9NDLF.c UpiFAxBNx0utVXK6Ak.FcMsz4mIcFz3foT7LFJ9z3qpVW2aDwBJx2YmZ8fq2wQDa9NXmm2Q5DrPz 1SwSJP27V9RwjvuPV_VXY.3npH9ckYARycZSl6Y._LhI.tnpK2Vc6N4N9c6TYmV3AFAhcKo36Xly NYA5fA.DkhugqMeNmqr_gEotJAEBladGsMwkblfg_PsFV8kRdB22YIyYcAUX68jJfCbgseHILhJ4 WTMB0x1xjybfAbTRy.6YCga2dQ5z28ZJ434ce_xfYpBJDje8OKV_rtpnLmeYpYpdaTY8TTvNjA2s LAhIylF_R1989uHiE3z0oIkXzEkmn.HRF8WdWC75JWznepZshGZlDMfQ60a7w2UQZt4FdbuAnPAX 2vShHJ2fmAzdhJmZI8EmEbLeKvjM4I.JRE1biXUR.jmAj34kmFkKPSmdeXigIpnWB2w62o.WGLoj YTuxucLLl1TBnJKQeA2_KqMKRcXZ4z0UOk4THsPltNMDTNrUVe.WAoTkBArNWPrvbFMAg6CpidO6 OkeN.clAh8BMD8rTFu93zC77Q77hSR_ORoeuNxOyGwa8KY.LRPWTOdmOD7KmHDV9ahK2wxLranND Tsdp1fh0zY4PHKHyBGMyPxseydfzUBvaYqraLvx1oq4sKCH2IhpyzIISrfeARiLOybAM6xgA7cti BJyYNMQ4i96u1KTHsc9L5EOgp2HJLyGyFJ3k0eyzrbu5l2vIOqOT3.SdMuoRu_D4gBNQU5WmJIse tOKTwpeQA0Eb8bJQPKYHQokGSPTsxd12CQfJ9aBTA8Fo73MuRBPf3MBhR0aXSTtCMcNTJ9L49uTT l69hRqSI3fwwP4RD2a4rBE0o0knfcUTGMZt9IzG.Ho8a23Z4ZC6GHvyHSajzGhOXqjSMwmynjg6r gChPhQ1byk7uX.1IraMHOy90QNIkV1woal71kUPUi7TRCzDmOBwMB65oIvFgh5lZG7ixEQtH.m9E bPS1iEsVhNfLWnzhNo58wIXH211kSmFm1YltEoWq00fx7_g-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Tue, 4 May 2021 17:01:22 +0000 Received: by kubenode538.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID c435762641fda1a4169dcdc43b265b30; Tue, 04 May 2021 17:01:17 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: FYI: WITH_REPRODUCIBLE_BUILD= problem for some files? From: Mark Millard In-Reply-To: <27B5086A-7C98-4AE6-885A-CC7C7BD2F64B@yahoo.com> Date: Tue, 4 May 2021 10:01:15 -0700 Cc: FreeBSD-STABLE Mailing List , freebsd-current Content-Transfer-Encoding: quoted-printable Message-Id: <3FC6BCDD-5865-4B5D-8238-3EC38AD4E78C@yahoo.com> References: <35482701-95A3-48B2-9A8E-B7E0092119B1.ref@yahoo.com> <35482701-95A3-48B2-9A8E-B7E0092119B1@yahoo.com> <43F20589-A7C7-42FF-9020-09CEE037D1CD@yahoo.com> <91F820A1-8940-4246-A20A-E62685F50079@yahoo.com> <27B5086A-7C98-4AE6-885A-CC7C7BD2F64B@yahoo.com> To: Ed Maste X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 4FZR2D2xzhz3mQd X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.69.82:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.69.82:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.82:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.82:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 May 2021 17:01:25 -0000 On 2021-May-4, at 08:51, Mark Millard wrote: > On 2021-May-4, at 06:01, Ed Maste wrote: >=20 >> On Mon, 3 May 2021 at 22:26, Mark Millard wrote: >>>=20 >>> But I'll note that I've built and stalled py37-diffoscope >>> (new to me). A basic quick test showed that it reports: >>>=20 >>> W: diffoscope.main: Fuzzy-matching is currently disabled as the = "tlsh" module is unavailable. >>=20 >> I just looked up tlsh - its "A Locality Sensitive Hash"; I presume >> diffoscope uses it to infer file renames. I believe the warning >> emitted here should have no impact on the output we're looking for. >=20 > Okay. >=20 >> As far as the utf-8 issues go, diffoscope requires a utf-8 locale and >> I suspect that is the issue. If you don't have LANG set already, try >> setting LANG=3DC.UTF-8 in your environment. >=20 > That is not the issue for the UnicodeDecodeError: >=20 > # echo $LANG > C.UTF-8 >=20 > # diffoscope /.zfs/snapshot/2021-04-*-01:40:48-0/bin/sh > $<3/>2021-05-04 08:49:21 W: diffoscope.main: Fuzzy-matching is = currently disabled as the "tlsh" module is unavailable. > $<3/>Traceback (most recent call last): > File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", = line 745, in main > sys.exit(run_diffoscope(parsed_args)) > File "/usr/local/lib/python3.7/site-packages/diffoscope/main.py", = line 677, in run_diffoscope > difference =3D load_diff_from_path(path1) > File = "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", = line 31, in load_diff_from_path > return load_diff(codecs.getreader("utf-8")(fp), path) > File = "/usr/local/lib/python3.7/site-packages/diffoscope/readers/__init__.py", = line 35, in load_diff > return JSONReaderV1().load(fp, path) > File = "/usr/local/lib/python3.7/site-packages/diffoscope/readers/json.py", = line 33, in load > raw =3D json.load(fp) > File "/usr/local/lib/python3.7/json/__init__.py", line 293, in load > return loads(fp.read(), > File "/usr/local/lib/python3.7/codecs.py", line 504, in read > newchars, decodedbytes =3D self.decode(data, self.errors) > UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position = 18: invalid start byte >=20 Well, the list of differing files is huge. But this seems to be .gnu_debuglink content for the area it is in. I'll note that I did installworld but not the likes of distrib-dirs or distribution this time. This test did buildworld to two distinct directories: zroot/BUILDs/13_0R-CA72-nodbg-clang 5.13G 118G 5.13G = /usr/obj/BUILDs/13_0R-CA72-nodbg-clang zroot/BUILDs/13_0R-CA72-nodbg-clang-alt 4.28G 118G 4.28G = /usr/obj/BUILDs/13_0R-CA72-nodbg-clang-alt and installworld to 2 distinct directories: zroot/DESTDIRs/13_0R-CA72-instwrld-alt 1.44G 118G 1.44G = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt zroot/DESTDIRs/13_0R-CA72-instwrld-norm 1.44G 118G 1.44G = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm Previously (armv7 target) I had built, installed, rebuilt to same directory (after clean-out) and installed to an alternate directory. That had gotten only a few files different but I do not know (yet) if it was the procedural difference that made the difference. Prefix of the list of different files this time: # diff -rq /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/ = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/ | more Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/[ and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/[ differ Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat differ Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags differ Files /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio and = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio differ . . . Looking, aarch64 seems to typically get a back-to-back sequence of 4 bytes different in native programs in my builds: # cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat 00003bd4 1d 65 00003bd5 eb a3 00003bd6 bb ca 00003bd7 8e 1a # ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat -r-xr-xr-x 1 root wheel 18448 May 4 08:55:01 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/cat -r-xr-xr-x 1 root wheel 18448 May 3 23:16:36 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/cat Sections: Idx Name Size VMA LMA File off = Algn . . . 25 .gnu_debuglink 00000010 0000000000000000 0000000000000000 = 00003bc8 2**0 CONTENTS, READONLY 00003bd4-00003bc8 =3D=3D 0xC # cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags 00002208 88 a1 00002209 e6 40 0000220a 60 94 0000220b bf ce # ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags -r-xr-xr-x 1 root wheel 11440 May 4 08:55:01 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chflags -r-xr-xr-x 1 root wheel 11440 May 3 23:16:36 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chflags Sections: Idx Name Size VMA LMA File off = Algn . . . 25 .gnu_debuglink 00000014 0000000000000000 0000000000000000 = 000021f8 2**0 CONTENTS, READONLY 00002208-000021f8 =3D=3D 0x10 # cmp -x /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio 000050c4 6b 3e 000050c5 08 ca 000050c6 7a 2f 000050c7 5d 64 # ls -Tld /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio -r-xr-xr-x 1 root wheel 23728 May 4 08:55:01 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-alt/bin/chio -r-xr-xr-x 1 root wheel 23728 May 3 23:16:37 2021 = /usr/obj/DESTDIRs/13_0R-CA72-instwrld-norm/bin/chio Sections: Idx Name Size VMA LMA File off = Algn . . . 25 .gnu_debuglink 00000010 0000000000000000 0000000000000000 = 000050b8 2**0 CONTENTS, READONLY 000050c4-000050b8 =3D=3D 0xC For all I know, some individual byte(s) in the 4 might accidentally match sometimes. The addition offset after .gnu_debuglink's file offset does vary (0xC and 0x10 above). The content of those differences do not look like file path components, for example the 0x08 byte. I do build with: # Avoid stripping but do not control host -g status as well: DEBUG_FLAGS+=3D # WITH_REPRODUCIBLE_BUILD=3D WITH_DEBUG_FILES=3D But that was true for the earlier armv7 target example that I reported that only got a few files with differences. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)