From nobody Sat Nov 20 19:54:09 2021 X-Original-To: arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 6B53E188F1CD for ; Sat, 20 Nov 2021 19:54:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-22.consmr.mail.gq1.yahoo.com (sonic301-22.consmr.mail.gq1.yahoo.com [98.137.64.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4HxPPV0cmxz3k90 for ; Sat, 20 Nov 2021 19:54:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1637438053; bh=cYcKVeA8+06t3AFainxt+18bdXJU9Njlit5Ir6W06W0=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=O0XEz6/WBkJyU7csQ6ABwayX+R6J46Otje6GlZ35UUni5p6rPdTyF5ZaEQm+VgMl7Vvh1dvT7C6DH83YK+h0hdaZt8aC6dGHSTloWOnbOFcnHtCKsvIx691y9K7XNmOGUOLLzaNT9jwPzEzHyZfnUsIh86UP9rOnPEaPlupLeb1MyvKwGapH1c1jFghWQr8iRYLwnMEHsDCpFJO2Iv/YeEkdBLBxmU64TbyVgAnnoETl5WDLcPVguarn25K/pTFgpVKdDnKnJ+HgCnQdFqF6tzuvCgX/ebBPGyk1ao9TjGgQVOke2eEBrVnc/DMN1wAnCLhgZ06oKpEhf2MIPqS4Dw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1637438053; bh=7Yv/3tTt6GcbfAGgh5kvrhu3m6fRn3q8m4fwwzleaqM=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=tqkZDZM4IJuXHV6rjrrLEB5/7ZJJOF6kfw0Qzc0nGRq95osz6/Tvgm7kBmTcf4TcX5qy98U4GgG4DLgbRIeZCB2wvVtNEa77Vvn0o6uTjqwLkNwgVte+b14b6zYUONaRUaSvvM7JO2//Oc2oOqI5aK/eKRxn8MIZQEL8HeEBwZjqfnLAlj7ZWVtGq5Sl1AloA6cMwrjG5uiF6BvzQt44WTWcBD8FBtG0OhEqA422k0MfgbUSyzepLO8UtY+NfvlpQKxfsNEAH0VVZiv3wMshVOdIECbf6MTok/CDhacyU9oWKA/PXYXW3RW4SiNfoGnAVEc+jmEukQliuQk4OWWvbQ== X-YMail-OSG: KAF.tNQVM1lxJEOe981imz_mHXsdmEQEwWrjn497ze6soeJwkuKKd1EPOiENzka Kakc7uLv2j8Go9Cq1hWuuCPoWs3IbRP_vZQbGgHOnUB4z.VlEWGjLwEr7Xcihd16_txv34jzYJ.9 .WX0yFB74GYyRy9K.C6ksoiT0XVmnU.eVswbMac4Xcz.mljbHplZcq0JA0fODed_Rq9NL4T.ISDU flEX9QuiiTznjv_M8OlMGXHRK6JWWIgti5mcR2GUPN6ynNqbMuzLeYFUq2pHCTFLm0wDgl8h2ILg Z2WQfRQICwCYR0ReC966qFEELBpeNlJiqvDj27vlVOPy5LPvak_l660wBsuZPQJHh7lF.ZAbWfD5 nIVwI9jQULxx7Vb_F19le0XRbarcTyNgZntAEfS3YBAVTme4r_XHejBIlI12ztKuIC605C7eslh6 0S9nAJQ4gFHVDNGD2QbY4RvU8AFpZ.toDdrIrrXgxuWnc4.U03Q1ijaJ7xxjFY.Zio.iPZVJTXZd FkyyLtg_DmVZY..ixcsBypwOFlSCA8gUlPP_E7OSbv36sJzFfABRT7Ebu.hFb1o4vjxkImBujm9E qJCE8P5e1H8kMNz8jB3HVvLzHcEz.HYO2YXmhtwtOkaM9RpaUcrQ_ztajgHGuunzu_apiVfLd.XK I_O8NlUMj1zZzfEcLsQTeuAuHJnML0sagcZMq6_D7IFYrc8s2a_hicfPIHzlBPqQGlpvME1STbCC glfSHgTXLahingYgld4zoWiyI4Tvm.65rY5.dHGpji56B4E9nQD1P4eQ8CWIGKhwLBNDA7Cxtvr8 tw1NsAKRtw4.WzN4j2gbn_UeQ2nQ8JM_kWo1oeJu7m5pi6UH8I1H5j8knYMN06.NgJIdQWQBAB_e 2SoiOmy3UVrYGh0Fus6hwHlyIGk.wQiljijU8jG1N1khM.r16DBs6GNALQx8_L4Y2eQ4xlHe0Ylw CaDVyQm_9Rn8ZfNHctjEg7Ke6.JXsez95q0_vh1BaRGqHqDq4jLdzF7bIKGvwnF4zyaIO64HSMkH spOssem7pNVH.H6J7U1P0Jife5Jr0YJYDEmMFxmjteaVxGwyFWuMYcFewPs.6a4JWGWeHGNP4L8l JwqEWpcRL5yYgYdvAmfISaU_C6DevPflPALnBR6jUbmM5WXMl2SJngbJWb1xfDqSqVkoez8W8_MW WQXCdg3D7aCHebhI2x6fsLEb.GxHPgjmR5V3pbEHv07.w.Mi_CsLk_BoRnPIhmPdEQoauGPUmHdU k4Trajq.1yGpx74ZCWIeM_0DIRit8t4IKimdOF2nxGuEggYEap.05vovuGdj47IlLjJpcmyWblw2 A1x8R_k28wTr59VyIiqU4r7x8Hsxnb77XzFdxp2alrJIkyUJa5EaozaZ3LP2Sdh6GUh4f6E2G1nz PeexgoixxqqkoMafg_28uvKgKubHXSMKhSr5jhcS9wZZ2DbOAafVHvjF6SwPefUDqfK7aMUjp.fC y0Z232.fcj5BbBgcpz3chZH5Ga5a0TqGHCwbD2BYhsYcAjDuSdGdtu9aMacdAZfpZgr2VxW6A0yd KTXnbbRzpL17RELCF0PDMK9aFEV80r18c9siRg6eF..vDKaKKOXpvnxsmwt5gTfc.s5G.Mm4QZZj dZ4cVHkYNEpp05BjDEw6_pI8y7c6qFSSYQLZYP1KUyx.iY.X6UmIFi5_E76Kp0GvUcAd4Y4pWz0n KZ_QYl1vOTZRNIAtZRLcHwvE2gJm4PYdNFjcLHH70qp4RTxSU.gwJQN4QodOpgScgDVqAXDEF38g NLUJAE2slA89YbDoCRpoz4vjhzCNUK7O3FxcF7LJFt.Hxu6xiJ2GUPd2Tdix5KWtVabQpyQZL4pJ ENdgf7YcLS_.uKX_88c4JzyiLcuVSegUSZKJZe8UhsnTjzAkGLPCfvBRy2yjSTFCt7kokV5ho3ta kW_YyagubyD1bhNWV5EZlUJkE3ddA.Q2g5idyXjruHptlIxamk0n0xNaT0Sor_nhz6PT1vWD4AUO UpVsfMffUhhd4jC7DFHhcxdb0obJhp1N1RPz.ghlTWFmjSFkP.IsAHp78nYg.HOw64ppcJiX4S_h s7GV3BR2z5FWShdoxLpaEemnigFqAmXGIT2daQuhHYN066fqHG0j5YoLWe111aYBcTsTQ7gqxMSx vY5fiHJBQeTbtOYcVbG9lWTzEfVm1Jns262oF75dh.JDTEbTpAdr2ylnItA9mSpL6CdLe2oCKU7_ btqSGB6G0x2VYBBO6Twg0y3V_MAqJCXebGRnoZAbYnT5LloYnsJVOfuzXgbdvrG8HttXtcrkorod fDyQRvcl4Owvbzg4si8bZ X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Sat, 20 Nov 2021 19:54:13 +0000 Received: by kubenode511.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID a6d4bebea471a42359ddf873a13f06ab; Sat, 20 Nov 2021 19:54:10 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: aarch64(?) poudiere-devel based builds seem to get fairly-rare corrupted files after recent system update(s?) Date: Sat, 20 Nov 2021 11:54:09 -0800 References: <2CA61249-321C-45AA-9755-597146AB8E9F@yahoo.com> <65AA4BCD-EC4B-4A19-B750-C7FC6E5ADDF5@yahoo.com> <9BF4F65B-6437-4D88-AF34-9BCFBF90D6F3@yahoo.com> <4B591638-4693-4403-8549-88D7A1D9D669@yahoo.com> <0006EB30-B9F9-465A-8B9A-A0C03899CEFC@yahoo.com> To: freebsd-current , "freebsd-arm@freebsd.org" In-Reply-To: <0006EB30-B9F9-465A-8B9A-A0C03899CEFC@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4HxPPV0cmxz3k90 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="O0XEz6/W"; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-1.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(1.00)[1.000]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.148:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.148:from]; RCVD_COUNT_TWO(0.00)[2] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-current X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Nov-19, at 22:20, Mark Millard wrote: > On 2021-Nov-18, at 12:15, Mark Millard wrote: >=20 >> On 2021-Nov-17, at 11:17, Mark Millard wrote: >>=20 >>> On 2021-Nov-15, at 15:43, Mark Millard wrote: >>>=20 >>>> On 2021-Nov-15, at 13:13, Mark Millard wrote: >>>>=20 >>>>> On 2021-Nov-15, at 12:51, Mark Millard wrote: >>>>>=20 >>>>>> On 2021-Nov-15, at 11:31, Mark Millard wrote: >>>>>>=20 >>>>>>> I updated from (shown a system that I've not updated yet): >>>>>>>=20 >>>>>>> # uname -apKU >>>>>>> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #18 = main-n250455-890cae197737-dirty: Thu Nov 4 13:43:17 PDT 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64=20 >>>>>>> 1400040 1400040 >>>>>>>=20 >>>>>>> to: >>>>>>>=20 >>>>>>> # uname -apKU >>>>>>> FreeBSD CA72_16Gp_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #19 = main-n250667-20aa359773be-dirty: Sun Nov 14 02:57:32 PST 2021 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400042 1400042 >>>>>>>=20 >>>>>>> and then updated /usr/ports/ and started poudriere-devel based = builds of >>>>>>> the ports I's set up to use. However my last round of port = builds from >>>>>>> a general update of /usr/ports/ were on 2021-10-23 before either = of the >>>>>>> above. >>>>>>>=20 >>>>>>> I've had at least two files that seem to be corrupted, where a = later part >>>>>>> of the build hits problematical file(s) from earlier build = activity. For >>>>>>> example: >>>>>>>=20 >>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:1: warning: null = character ignored [-Wnull-character] >>>>>>> =20 >>>>>>> ^ >>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:2: warning: null = character ignored [-Wnull-character] >>>>>>> >>>>>>> ^ >>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:3: warning: null = character ignored [-Wnull-character] >>>>>>> =20 >>>>>>> ^ =20 >>>>>>> /usr/local/include/X11/extensions/XvMC.h:1:4: warning: null = character ignored [-Wnull-character] >>>>>>> >>>>>>> ^ >>>>>>> . . . >>>>>>>=20 >>>>>>> Removing the xorgproto-2021.4 package and rebuilding via >>>>>>> poudiere-devel did not get a failure of any ports dependent >>>>>>> on it. >>>>>>>=20 >>>>>>> This was from a use of: >>>>>>>=20 >>>>>>> # poudriere jail -j13_0R-CA7 -i >>>>>>> Jail name: 13_0R-CA7 >>>>>>> Jail version: 13.0-RELEASE-p5 >>>>>>> Jail arch: arm.armv7 >>>>>>> Jail method: null >>>>>>> Jail mount: /usr/obj/DESTDIRs/13_0R-CA7-poud >>>>>>> Jail fs: =20 >>>>>>> Jail updated: 2021-11-04 01:48:49 >>>>>>> Jail pkgbase: disabled >>>>>>>=20 >>>>>>> but another not-investigated example was from: >>>>>>>=20 >>>>>>> # poudriere jail -j13_0R-CA72 -i >>>>>>> Jail name: 13_0R-CA72 >>>>>>> Jail version: 13.0-RELEASE-p5 >>>>>>> Jail arch: arm64.aarch64 >>>>>>> Jail method: null >>>>>>> Jail mount: /usr/obj/DESTDIRs/13_0R-CA72-poud >>>>>>> Jail fs: =20 >>>>>>> Jail updated: 2021-11-04 01:48:01 >>>>>>> Jail pkgbase: disabled >>>>>>>=20 >>>>>>> (so no 32-bit COMPAT involved). The apparent corruption >>>>>>> was in a different port (autoconfig, noticed by the >>>>>>> build of automake failing via config reporting >>>>>>> /usr/local/share/autoconf-2.69/autoconf/autoconf.m4f >>>>>>> being rejected). >>>>>>>=20 >>>>>>> /usr/obj/DESTDIRs/13_0R-CA7-poud/ and >>>>>>> /usr/obj/DESTDIRs/13_0R-CA72-poud/ and the like track the >>>>>>> system versions. >>>>>>>=20 >>>>>>> The media is an Optane 960 in the PCIe slot of a HoneyComb >>>>>>> (16 Cortex-A72's). The context is a root on ZFS one, ZFS >>>>>>> used in order to have bectl, not redundancy. >>>>>>>=20 >>>>>>> The ThreadRipper 1950X (so amd64) port builds did not give >>>>>>> evidence of such problems based on the updated system. (Also >>>>>>> Optane media in a PCIe slot, also root on ZFS.) But the >>>>>>> errors seem rare enough to not be able to conclude much. >>>>>>=20 >>>>>> For aarch64 targeting aarch64 there was also this >>>>>> explicit corruption notice during the poudriere(-devel) >>>>>> bulk build: >>>>>>=20 >>>>>> . . . >>>>>> [CA72_ZFS] Extracting arm-none-eabi-gcc-8.4.0_3: ......... >>>>>> pkg-static: Fail to extract = /usr/local/libexec/gcc/arm-none-eabi/8.4.0/lto1 from package: Lzma = library error: Corrupted input data >>>>>> [CA72_ZFS] Extracting arm-none-eabi-gcc-8.4.0_3... done >>>>>>=20 >>>>>> Failed to install the following 1 package(s): = /packages/All/arm-none-eabi-gcc-8.4.0_3.pkg >>>>>> *** Error code 1 >>>>>> Stop. >>>>>> make: stopped in /usr/ports/sysutils/u-boot-orangepi-plus-2e >>>>>>=20 >>>>>> I'm not yet to the point of retrying after removing >>>>>> arm-none-eabi-gcc-8.4.0_3 : other things are being built. >>>>>=20 >>>>>=20 >>>>> Another context with my prior general update of /usr/ports/ >>>>> and the matching port builds: Back then I used USE_TMPFS=3Dall >>>>> but the failure is based on USE_TMPFS-"data" instead. So: >>>>> lots more I/O. >>>>>=20 >>>>=20 >>>> None of the 3 corruptions repeated during bulk builds that >>>> retried the builds that generated the files. All of the >>>> ports that failed by hitting the corruptions in what they >>>> depended on, built fine in teh retries. >>>>=20 >>>> For reference: >>>>=20 >>>> I'll note that, back when I was using USE_TMPFS=3Dall , I also >>>> did some separate bulk -a test runs, both aarch64 (Cortex-A72) >>>> native and Cortext-A72 targeting Cortex-A7 (armv7). None of >>>> those showed evidence of file corruptions. In general I've >>>> not had previous file corruptions with this system. (There >>>> was a little more than 245 GiBytes swap, which covered the >>>> tmpfs needs when they were large.) >>>=20 >>>=20 >>> I set up a contrasting test context and got no evidence of >>> corruptions in that context. (Note: the 3 bulk builds >>> total to around 24 hrs of activity for the 3 examples >>> of 460+ ports building.) So, for the Cortex-A72 system, >>=20 >> I set up a UFS on Optane (U.2 via M.2 adapter) context and >> also got no evidence of corruptions in that context (same >> hardware and a copy of the USB3 SSD based system). The >> sequence of 3 bulks took somewhat over 18 hrs using the >> Optane. >>=20 >>> root on UFS on portable USB3 SSD: no evidence of corruptions >> Also: >> root on UFS on Optane U.2 via M.2: no evidence of corruptions >>> vs.: >>> root on ZFS on optane in PCIe slot: solid evidence of 3 known = corruptions >>>=20 >>> Both had USE_TMPFS=3D"data" in use. The same system build >>> had been installed and booted for both tests. >>>=20 >>> The evidence of corruptions is rare enough for this not to >>> be determinative, but it is suggestive. >>>=20 >>> Unfortunately, ZFS vs. UFS and Optane-in-PCIe vs. USB3 are >>> not differentiated by this test result. >>>=20 >>> There is also the result that I've not seen evidence of >>> corruptions on the ThreadRipper 1950 X (amd64) system. >>> Again, not determinative, but suggestive, given how rare >>> the corruptions seem to be. >>=20 >> So far the only things unique to the observed corruptions are: >>=20 >> root on ZFS context (vs. root on UFS) >> and: >> Optane in a PCIe slot (but no contrasting ZFS case tested) >>=20 >> The PCIe slot does not seem to me to be likely to be contributing. >> So this seem to be suggestive of a ZFS problem. >>=20 >> A contributing point might be that the main [so: 14] system was >> built via -mcpu=3Dcortex-a72 for execution on a Cortext-A72 system. >>=20 >> [I previously ran into a USB subsystem mishandling of keeping >> things coherent for the week memory ordering in this sort of >> context. That issue was fixed. But back then I was lucky enough >> to be able to demonstrate fails vs. works by adding an >> appropriate instruction to FreeBSD in a few specific places >> (more than necessary as it turned out). Someone else determined >> where the actual mishandling was that covered all required >> places. My generating that much information in this context >> seems unlikely.] >=20 >=20 > I started a retry of root-on-ZFS with the Optane-in-PCIe-slot media > and it got its first corruption (in a different place, 2nd bulk > build this time). The use of the corrupted file reports: >=20 > configure:13269: cc -o conftest -Wall -Wextra -fsigned-char = -Wdeclaration-after-statement -O2 -pipe -mcpu=3Dcortex-a53 -g = -fstack-protector-strong -fno-strict-aliasing -DUSE_MEMORY_H = -I/usr/local/incl > ude -mcpu=3Dcortex-a53 -fstack-protector-strong conftest.c = -L/usr/local/lib -logg >&5 > In file included from conftest.c:27: > In file included from /usr/local/include/ogg/ogg.h:24: > In file included from /usr/local/include/ogg/os_types.h:154: > /usr/local/include/ogg/config_types.h:1:1: warning: null character = ignored [-Wnull-character] > > ^ > /usr/local/include/ogg/config_types.h:1:2: warning: null character = ignored [-Wnull-character] > > ^ > /usr/local/include/ogg/config_types.h:1:3: warning: null character = ignored [-Wnull-character] > > ^ > . . . > /usr/local/include/ogg/config_types.h:1:538: warning: null character = ignored [-Wnull-character] > . . . (nulls) . . . >=20 > So: 538 such null bytes. >=20 > Thus, another example of something like a page of nulls being > written out when ZFS is in use. >=20 > audio/gstreamer1-plugins-ogg also failed via referencing the file > during its build. >=20 > (The bulk run is still going and there is one more bulk run to go.) >=20 Well, 528 happened to be the size of config_types.h --and of config_types.h from a build that did not get the corruption there. So looking at the other (later) corruption, which was a bigger file (looking via bulk -i and installing what contained the file but looking from outside the jail): # find /usr/local/ -name "libtextstyle.so*" -exec ls -Tld {} \; -rwxr-xr-x 1 root wheel 2339104 Nov 20 01:05:05 2021 = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so.0.1.1 lrwxr-xr-x 1 root wheel 21 Nov 20 01:05:05 2021 = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so.0 -> libtextstyle.so.0.1.1 lrwxr-xr-x 1 root wheel 21 Nov 20 01:05:05 2021 = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so -> libtextstyle.so.0.1.1 hd = /usr/local/poudriere/data/.m/13_0R-CA7-default/ref/usr/local/lib/libtextst= yle.so.0.1.1 | more 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = |................| * 0023b120 So the whole, over 2 MiByte, the whole file ended up with just null = Bytes. To cross check on live system caching vs. on disk, I rebooted and redid = the bulk -i based install of libtextstyle and looked at = libtextstyle.so.0.1.1 : still all zeros. For reference, zpool scrub afterward resulted in: # zpool status pool: zopt0 state: ONLINE scan: scrub repaired 0B in 00:01:49 with 0 errors on Sat Nov 20 = 11:47:31 2021 config: NAME STATE READ WRITE CKSUM zopt0 ONLINE 0 0 0 nda1p3 ONLINE 0 0 0 But it is not a ZFS redundancy context: ZFS used just to use bectl . =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)