From nobody Sun Nov 21 15:50:47 2021 X-Original-To: arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 290AF189263F for ; Sun, 21 Nov 2021 15:51:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hxvz43SXyz3P0R for ; Sun, 21 Nov 2021 15:51:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1637509897; bh=QZSEXVV7FCogL5NbOUTECsNErnqrYYObbNZQsklWEuA=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=r/T5YcS8y840HIuty/HiQvPzqNyjbk5hspQ5sSgeuyYCM3j0OkhgytCmk8strx3ALKBuwwIzfi63Ke1Qox74gnXT9AuNFLeH/VsRPscVBXcpzePghgOiuyeE7tVyYxfe/tYdQUS9mKVOodMhMqsyNIMZA1Zl8eYIDq0GXOiJoDi5UiL4zvxn1uwjkEioJCjTIIP+LHoS5UKCbj20Zc/75tzv6P40jx+mCZGMzHS5WAA0usYdX7JsuG1Shv4iAuoHyBg5T5vxp3igE9tgNq1/CIy4YmEIrvzOI8oJHtqCUcQjMKI23DKglDhRVBwQgXY+l/xXFcNAllTslkrHh5pwCQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1637509897; bh=RFWsQzl2v2I2nXC4DJ3ikpMeKzH5HZFYZZySPTNUrZz=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Wb0T74S72scCFiulqYyK43FjLU46Fcm2veWZbipoPWgr1d4Vg7gItB3hPcPRBGMgN/sDfN3wtPXV7fyvJQFEHzRwIOr/IDBVH6p2K+g27dXyFCctBFRFVbJ2cQZ2J85olsSBkAyNSeYyHgQLpPr+slVMSe0u/KPUf+qowKGtrE/MhAnil8mxejaa/ZSNnj8HlIjHMyxl87NEkLDehTXjt3HKS+Biu9koVU2nPeCUK5yOiBCuuAOTQNy9qAe2oIDHJwNLO9+n/OYO8CHGKrEdVNDGk6JhQ2jW148qitisI3QLIXTsBHiAm69wM2TeVU4+BSXBRWUdSjbiOcIuGruFKQ== X-YMail-OSG: JKJw9FIVM1m3C82aTQft.Nz141yIMMdHHvkIjLLEbYUqu_jjBC1D.46UhvIqaBD 76teelPRC95TjqqurKg1GvJQWKsK3A.m5X5aMNaWTt1EX44QOeGqjjgrYdIsBFE0PhzVYh8ZUGWc yob_vrPYMYSTa7GyL.9s7SyAcfaudn7.FNX8py6vaChfmi72C8x424aMBAqzLjI_4.RckifbrWQM QyolScqF.uFFHdH68oQcUGVeMJ1X49bKXLSU0bpzwOHOkAj2kxtelrcVLL6PLF.JOnkhwIKZvheH UpNQw3yMm17bqDleVCunGrHHvFtv3vGyVZXXjnpVQIF5T6Ow2pexjDizpdcToKige2GtP4Tkfe_V FXf0R6AuGu.ohMpE0F2damb3qpI41czmbzG8UoX6erGLULRV1hZ6o4pBL1qw8duUae3GZ45qK7Zb PhF80whT92a5OXrjGSGYqCEpdfpzSem3qFGZDDxq75ui63X1g.h9VFhfbLSnoVyYCmtlkrE0_1sb 5pSzL6pKeepSxhM1lwfk.mowX9WdPmUf0w8vk.7BU2bmpQklT4NWHRl5JcZqZT49Vs4tj_rfbeVF QRewH2j1iscJUBes5eD4RTR7YF4.i.1NyCZzd6iksI4lA7lvJZ9aOCfYhM.TdqJlKU9XWi.bdlcm x9j7SbSnEjEh4Xze_XB..zjWJ.8SQL62BD.5bnhbu5OrhjXvooSJTqsckBXQxUcWlq8WqntLxJ0E 8MAfgMDTfKJcSyplM9RQ4l67qZWG_D1WrKwyICrHlpu9kdcakOtJKN_axqZEfy4GaqlymHCy62Zv 1ReeAS1Ig149ba122MtHK9rnw1yYnUTZM8thTSE2JT0ehuo0_frtKvHZbS0Fe8jbZmTSthDwe2M7 jKg.54nvYVgP2y2FD9fGaapr6pWfSUlXiN9GUbAgeLXKPpPto6paQLJJui7ypprVg0SUjRS0l8AT J5QU63NTNklXLDHUeJKVTMhkwkVILd3mNYY8rkNqunfdfmMpo5DgSB_t2UJUO856.Is7w_.1utri nAyk_4mBAAVikslN19T7wHpHDwOaSOU1n7zwcyho6Sk.Ag5IqTElxrfbjn4EfgHbYD.y0KtFauVT K4g7dB21PO74b7bSMwoGPosW6EzmrV7q7KvCmrqc0TtJPo4yBkkA7iMXVC13adzYMhtff4eDzCz9 5Xv7cOCIb3BCeCbtmI.bDArEHWhdc1V7o__Z38q60T5RnRD4TzGLt8xBIDZdH2s79WtScFcesgPe 5EPAfurxpXw9cxI7w8wL2kaONGctNUBHNp5nyYp1j7cWnmgvSJ9eyKcY7BHPHn3nlPg7K6ASQ5V1 I8MZzfdaNJFL_Ymu9bPQHMMVLl4bY_Knvjo24b0dKyVT.zW8yzK5cZmgbqFfPnS07riSExuvowcX .AQUwjyu2shJGVIVLNnYV2rqX6TQDepTBI.j9WSOfSLXORKLsYWnJ9l1XxGl0ewK71unWnhKBppy TQlNZ1cvOqpP_DXdab8ahsoOdNWmCfsenBD_6GPirQA7UaMuvPbxC7PhGe.2NCWiFbBzhf_c0.nm hTGP5Ca0n2NheBOuGzWkrfka5yl9P_q.XIXJVIG9NhsyY.y2IFuoWOENE4LWyqDAvbTVL6mIvwg5 EYqNs0s99_ZkH60bHwlZNY6aOTmP0zEJqInNPLmjTDjTQiOMccI.pDOFXl00f4Mpa2_dK6XNB9u3 qJPW91RMvFt2IHAaLOyPBP9t59ji8IVhcYcKTU17gTHdqjehxOPEBJr3QzhV34qh4cn50xEeeAiJ keuPM7S5BDtoh01Bnv.MXgn.TO9KRA5CqwMzXXpnikc6yDgJGikF3NWf5s8ZIoCCWRuhD9Mfr_6r KCTyqYFwUlGQZzlwmnr7AMIeWAwjwA3utCAcS7GGJ_RuA8_ErErkELPuVFXEzi617qn2zdvzGSSM p9SPc4OuCi2w3aBoPVwWLZtkfrBlbbcidFMjQ_3YwAkA.9RjFAZmoV9eBYiT7.YNDtDDpuxVHP.y z3Jn0pQYDZfC5v2IhHVtOEpwOaex3IIIGk.Xo6OOCtVaj.FKTGaPJ13pr5GXDFuAMTSAmsBS4xPl vQ3v2qMR4.FKBDq8MSm.mT6Ybal_faZbr57erYz1DXaa7.WYVCpPQbjRRzbzKBVIyr6VvrsIN5M8 VELf7NwXZkTJbKkyrU.WAIYvIcBENs4y1kTxI0vYUrd39eXTN8aEbtiUr2fACFNIFA1_KwCaNi4r s7dFCuLUNzW84eujv9AV.t0uOLOLKfZWHS4oYo.b.8g-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sun, 21 Nov 2021 15:51:37 +0000 Received: by kubenode527.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 200c7f179f811ed8c8d4f0075a2965ea; Sun, 21 Nov 2021 15:50:50 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: aarch64(?) poudiere-devel based builds seem to get fairly-rare corrupted files after recent system update(s?) Date: Sun, 21 Nov 2021 07:50:47 -0800 References: <2CA61249-321C-45AA-9755-597146AB8E9F@yahoo.com> <65AA4BCD-EC4B-4A19-B750-C7FC6E5ADDF5@yahoo.com> <9BF4F65B-6437-4D88-AF34-9BCFBF90D6F3@yahoo.com> <4B591638-4693-4403-8549-88D7A1D9D669@yahoo.com> <0006EB30-B9F9-465A-8B9A-A0C03899CEFC@yahoo.com> To: freebsd-current , "freebsd-arm@freebsd.org" In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Hxvz43SXyz3P0R X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="r/T5YcS8"; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.147:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.147:from]; RCVD_COUNT_TWO(0.00)[2] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-current X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Nov-20, at 11:54, Mark Millard wrote: > On 2021-Nov-19, at 22:20, Mark Millard wrote: >=20 >> On 2021-Nov-18, at 12:15, Mark Millard wrote: >>=20 >>> On 2021-Nov-17, at 11:17, Mark Millard wrote: >>>=20 >>>> On 2021-Nov-15, at 15:43, Mark Millard wrote: >>>>=20 >>>>> On 2021-Nov-15, at 13:13, Mark Millard wrote: >>>>>=20 >>>>>> On 2021-Nov-15, at 12:51, Mark Millard wrote: >>>>>>=20 >>>>>>> On 2021-Nov-15, at 11:31, Mark Millard = wrote: >>>>>>>=20 >>>>>>>> I updated from (shown a system that I've not updated yet): >>>>>>>>=20 >>>>>>>> # uname -apKU >>>>>>>> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #18 = main-n250455-890cae197737-dirty: Thu Nov 4 13:43:17 PDT 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64=20 >>>>>>>> 1400040 1400040 >>>>>>>>=20 >>>>>>>> to: >>>>>>>>=20 >>>>>>>> # uname -apKU >>>>>>>> FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #19 = main-n250667-20aa359773be-dirty: Sun Nov 14 02:57:32 PST 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400042 1400042 >>>>>>>>=20 >>>>>>>> and then updated /usr/ports/ and started poudriere-devel based = builds of >>>>>>>> the ports I's set up to use. However my last round of port = builds from >>>>>>>> a general update of /usr/ports/ were on 2021-10-23 before = either of the >>>>>>>> above. >>>>>>>>=20 >>>>>>>> I've had at least two files that seem to be corrupted, where a = later part >>>>>>>> of the build hits problematical file(s) from earlier build = activity. For >>>>>>>> example: >>>>>>>>=20 >>>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:1: warning: null = character ignored [-Wnull-character] >>>>>>>> =20 >>>>>>>> ^ >>>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:2: warning: null = character ignored [-Wnull-character] >>>>>>>> >>>>>>>> ^ >>>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:3: warning: null = character ignored [-Wnull-character] >>>>>>>> =20 >>>>>>>> ^ =20 >>>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:4: warning: null = character ignored [-Wnull-character] >>>>>>>> >>>>>>>> ^ >>>>>>>> . . . >>>>>>>>=20 >>>>>>>> Removing the xorgproto-2021.4 package and rebuilding via >>>>>>>> poudiere-devel did not get a failure of any ports dependent >>>>>>>> on it. >>>>>>>>=20 >>>>>>>> This was from a use of: >>>>>>>>=20 >>>>>>>> # poudriere jail -j13_0R-CA7 -i >>>>>>>> Jail name: 13_0R-CA7 >>>>>>>> Jail version: 13.0-RELEASE-p5 >>>>>>>> Jail arch: arm.armv7 >>>>>>>> Jail method: null >>>>>>>> Jail mount: /usr/obj/DESTDIRs/13_0R-CA7-poud >>>>>>>> Jail fs: =20 >>>>>>>> Jail updated: 2021-11-04 01:48:49 >>>>>>>> Jail pkgbase: disabled >>>>>>>>=20 >>>>>>>> but another not-investigated example was from: >>>>>>>>=20 >>>>>>>> # poudriere jail -j13_0R-CA72 -i >>>>>>>> Jail name: 13_0R-CA72 >>>>>>>> Jail version: 13.0-RELEASE-p5 >>>>>>>> Jail arch: arm64.aarch64 >>>>>>>> Jail method: null >>>>>>>> Jail mount: /usr/obj/DESTDIRs/13_0R-CA72-poud >>>>>>>> Jail fs: =20 >>>>>>>> Jail updated: 2021-11-04 01:48:01 >>>>>>>> Jail pkgbase: disabled >>>>>>>>=20 >>>>>>>> (so no 32-bit COMPAT involved). The apparent corruption >>>>>>>> was in a different port (autoconfig, noticed by the >>>>>>>> build of automake failing via config reporting >>>>>>>> /usr/local/share/autoconf-2.69/autoconf/autoconf.m4f >>>>>>>> being rejected). >>>>>>>>=20 >>>>>>>> /usr/obj/DESTDIRs/13_0R-CA7-poud/ and >>>>>>>> /usr/obj/DESTDIRs/13_0R-CA72-poud/ and the like track the >>>>>>>> system versions. >>>>>>>>=20 >>>>>>>> The media is an Optane 960 in the PCIe slot of a HoneyComb >>>>>>>> (16 Cortex-A72's). The context is a root on ZFS one, ZFS >>>>>>>> used in order to have bectl, not redundancy. >>>>>>>>=20 >>>>>>>> The ThreadRipper 1950X (so amd64) port builds did not give >>>>>>>> evidence of such problems based on the updated system. (Also >>>>>>>> Optane media in a PCIe slot, also root on ZFS.) But the >>>>>>>> errors seem rare enough to not be able to conclude much. >>>>>>>=20 >>>>>>> For aarch64 targeting aarch64 there was also this >>>>>>> explicit corruption notice during the poudriere(-devel) >>>>>>> bulk build: >>>>>>>=20 >>>>>>> . . . >>>>>>> [CA72_ZFS] Extracting arm-none-eabi-gcc-8.4.0_3: ......... >>>>>>> pkg-static: Fail to extract = /usr/local/libexec/gcc/arm-none-eabi/8.4.0/lto1 from package: Lzma = library error: Corrupted input data >>>>>>> [CA72_ZFS] Extracting arm-none-eabi-gcc-8.4.0_3... done >>>>>>>=20 >>>>>>> Failed to install the following 1 package(s): = /packages/All/arm-none-eabi-gcc-8.4.0_3.pkg >>>>>>> *** Error code 1 >>>>>>> Stop. >>>>>>> make: stopped in /usr/ports/sysutils/u-boot-orangepi-plus-2e >>>>>>>=20 >>>>>>> I'm not yet to the point of retrying after removing >>>>>>> arm-none-eabi-gcc-8.4.0_3 : other things are being built. >>>>>>=20 >>>>>>=20 >>>>>> Another context with my prior general update of /usr/ports/ >>>>>> and the matching port builds: Back then I used USE_TMPFS=3Dall >>>>>> but the failure is based on USE_TMPFS-"data" instead. So: >>>>>> lots more I/O. >>>>>>=20 >>>>>=20 >>>>> None of the 3 corruptions repeated during bulk builds that >>>>> retried the builds that generated the files. All of the >>>>> ports that failed by hitting the corruptions in what they >>>>> depended on, built fine in teh retries. >>>>>=20 >>>>> For reference: >>>>>=20 >>>>> I'll note that, back when I was using USE_TMPFS=3Dall , I also >>>>> did some separate bulk -a test runs, both aarch64 (Cortex-A72) >>>>> native and Cortext-A72 targeting Cortex-A7 (armv7). None of >>>>> those showed evidence of file corruptions. In general I've >>>>> not had previous file corruptions with this system. (There >>>>> was a little more than 245 GiBytes swap, which covered the >>>>> tmpfs needs when they were large.) >>>>=20 >>>>=20 >>>> I set up a contrasting test context and got no evidence of >>>> corruptions in that context. (Note: the 3 bulk builds >>>> total to around 24 hrs of activity for the 3 examples >>>> of 460+ ports building.) So, for the Cortex-A72 system, >>>=20 >>> I set up a UFS on Optane (U.2 via M.2 adapter) context and >>> also got no evidence of corruptions in that context (same >>> hardware and a copy of the USB3 SSD based system). The >>> sequence of 3 bulks took somewhat over 18 hrs using the >>> Optane. >>>=20 >>>> root on UFS on portable USB3 SSD: no evidence of corruptions >>> Also: >>> root on UFS on Optane U.2 via M.2: no evidence of corruptions >>>> vs.: >>>> root on ZFS on optane in PCIe slot: solid evidence of 3 known = corruptions >>>>=20 >>>> Both had USE_TMPFS=3D"data" in use. The same system build >>>> had been installed and booted for both tests. >>>>=20 >>>> The evidence of corruptions is rare enough for this not to >>>> be determinative, but it is suggestive. >>>>=20 >>>> Unfortunately, ZFS vs. UFS and Optane-in-PCIe vs. USB3 are >>>> not differentiated by this test result. >>>>=20 >>>> There is also the result that I've not seen evidence of >>>> corruptions on the ThreadRipper 1950 X (amd64) system. >>>> Again, not determinative, but suggestive, given how rare >>>> the corruptions seem to be. >>>=20 >>> So far the only things unique to the observed corruptions are: >>>=20 >>> root on ZFS context (vs. root on UFS) >>> and: >>> Optane in a PCIe slot (but no contrasting ZFS case tested) >>>=20 >>> The PCIe slot does not seem to me to be likely to be contributing. >>> So this seem to be suggestive of a ZFS problem. >>>=20 >>> A contributing point might be that the main [so: 14] system was >>> built via -mcpu=3Dcortex-a72 for execution on a Cortext-A72 system. >>>=20 >>> [I previously ran into a USB subsystem mishandling of keeping >>> things coherent for the week memory ordering in this sort of >>> context. That issue was fixed. But back then I was lucky enough >>> to be able to demonstrate fails vs. works by adding an >>> appropriate instruction to FreeBSD in a few specific places >>> (more than necessary as it turned out). Someone else determined >>> where the actual mishandling was that covered all required >>> places. My generating that much information in this context >>> seems unlikely.] >>=20 >>=20 >> I started a retry of root-on-ZFS with the Optane-in-PCIe-slot media >> and it got its first corruption (in a different place, 2nd bulk >> build this time). The use of the corrupted file reports: >>=20 >> configure:13269: cc -o conftest -Wall -Wextra -fsigned-char = -Wdeclaration-after-statement -O2 -pipe -mcpu=3Dcortex-a53 -g = -fstack-protector-strong -fno-strict-aliasing -DUSE_MEMORY_H = -I/usr/local/incl >> ude -mcpu=3Dcortex-a53 -fstack-protector-strong conftest.c = -L/usr/local/lib -logg >&5 >> In file included from conftest.c:27: >> In file included from /usr/local/include/ogg/ogg.h:24: >> In file included from /usr/local/include/ogg/os_types.h:154: >> /usr/local/include/ogg/config_types.h:1:1: warning: null character = ignored [-Wnull-character] >> >> ^ >> /usr/local/include/ogg/config_types.h:1:2: warning: null character = ignored [-Wnull-character] >> >> ^ >> /usr/local/include/ogg/config_types.h:1:3: warning: null character = ignored [-Wnull-character] >> >> ^ >> . . . >> /usr/local/include/ogg/config_types.h:1:538: warning: null character = ignored [-Wnull-character] >> . . . (nulls) . . . >>=20 >> So: 538 such null bytes. >>=20 >> Thus, another example of something like a page of nulls being >> written out when ZFS is in use. >>=20 >> audio/gstreamer1-plugins-ogg also failed via referencing the file >> during its build. >>=20 >> (The bulk run is still going and there is one more bulk run to go.) >>=20 >=20 > Well, 528 happened to be the size of config_types.h --and of > config_types.h from a build that did not get the corruption there. >=20 > So looking at the other (later) corruption, which was a bigger file > (looking via bulk -i and installing what contained the file but > looking from outside the jail): >=20 > # find /usr/local/ -name "libtextstyle.so*" -exec ls -Tld {} \; > -rwxr-xr-x 1 root wheel 2339104 Nov 20 01:05:05 2021 = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so.0.1.1 > lrwxr-xr-x 1 root wheel 21 Nov 20 01:05:05 2021 = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so.0 -> libtextstyle.so.0.1.1 > lrwxr-xr-x 1 root wheel 21 Nov 20 01:05:05 2021 = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so -> libtextstyle.so.0.1.1 >=20 > hd = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so.0.1.1 | more > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| > * > 0023b120 >=20 > So the whole, over 2 MiByte, the whole file ended up with just null = Bytes. >=20 > To cross check on live system caching vs. on disk, I rebooted and = redid the > bulk -i based install of libtextstyle and looked at = libtextstyle.so.0.1.1 : > still all zeros. >=20 > For reference, zpool scrub afterward resulted in: >=20 > # zpool status > pool: zopt0 > state: ONLINE > scan: scrub repaired 0B in 00:01:49 with 0 errors on Sat Nov 20 = 11:47:31 2021 > config: >=20 > NAME STATE READ WRITE CKSUM > zopt0 ONLINE 0 0 0 > nda1p3 ONLINE 0 0 0 >=20 > But it is not a ZFS redundancy context: ZFS used just to use bectl . Using bectl (on the root-on-ZFS Optane in PCIe slot), I booted stable/13 : # uname -apKU FreeBSD CA72_16Gp_ZFS 13.0-STABLE FreeBSD 13.0-STABLE #13 = stable/13-n248062-109330155000-dirty: Sat Nov 13 23:55:14 PST 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/13S-CA72-nodbg-clang/usr/13S-src/arm64.= aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1300520 1300520 and tried the sequence of 3 bulk runs: There was no evidence of corruptions, suggesting that the Optane in the PCIe slot is not the source of the problem of having some file(s) end up with all bytes being null bytes. So, overall, ending up with evidence of corruptions generated during bulk builds seem to be tied to main's [so: 14's] ZFS implementation in: # uname -apKU FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #18 = main-n250455-890cae197737-dirty: Thu Nov 4 13:43:17 PDT 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64=20 1400040 1400040 because that is all that is unique to having the evidence of corruptions. Since there have been ZFS updates in main since then, it seems that the next experiment would be to update main and try again under main. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)