From nobody Thu Apr 13 14:05:04 2023 X-Original-To: dev-commits-src-main@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Py1Yc5rylz458mN for ; Thu, 13 Apr 2023 14:05:08 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-qt1-x82a.google.com (mail-qt1-x82a.google.com [IPv6:2607:f8b0:4864:20::82a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Py1Yc4S5Qz4QGZ for ; Thu, 13 Apr 2023 14:05:08 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x82a.google.com with SMTP id gb12so13364102qtb.6 for ; Thu, 13 Apr 2023 07:05:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; t=1681394707; x=1683986707; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=SwC4h0o6JTZ2nbjTjoaKtAalDE+xU9li87T6BCp0I7o=; b=HyUKMBNZsr0LFsouXEig+tpUec9Dr1KpKLfsJid76mXIKq+1I03HooJ0qVkDEnjiCw O7L2hfSyFDMYmCvLcvTKYjOIzlvQ+78y/TU8PamRnt7sHH7UyAHwIDcP/2ZggjMcKv+u LkHZoSI98vKY2V0EjMf4LBhhz+F++304ft8f2SKxiYTN7B7EE4EP2W3rEhqSX8a4/O5q +RFuxAbRY6O2Gd9kFlNXtIYhsqe57ssnNIIh69IeMnSwtlfhQkjesr3Uj8HWRa7eyHNV xiL0RvAfHgllQ4YbFnKQGDWMaRI0ZTZ7yO7qioG+/sIVtKaw39+6Lxo1wG3o3IvfL5SV Nn7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1681394707; x=1683986707; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=SwC4h0o6JTZ2nbjTjoaKtAalDE+xU9li87T6BCp0I7o=; b=QE0MUrbfxQVoRweRzPJ8ej+FgOnBxFJYHc7Qj6EI0nMN8Ep1X3MkWWYWDujl0g36RV wZCBpwbPsbrHVf4ki6warjKEQ37RG6Ahzzk4+QpgyWtEjcMhjqZHvmSsNsCZa2obhew9 TOzXfrB/JkGtrzT63aXQ/d+Doq+uLAKVO684sku/S0xmWUjOnwcHUXP43S/8ETMnM6xU lS3aE8m5SxR+opFmdJdVjxd7u+bRa5RZ6RwJUcbXGpI62fDZLBO4hiqTESOo90TWMHqS zBgNfB8L1o7kZ5MVON8NHghh99Wh99KDkmAzbDztpVdYeJrAPswBZSsznCB7Qj3HBZ38 4bGw== X-Gm-Message-State: AAQBX9fPQLVJPZYLJM+Gcn/vjzXiDRCytPXeUay20ZjWw2RI5nB18wJx p5oJ9IB+ifz96FTscJP8VcyUEg== X-Google-Smtp-Source: AKy350ZoY2H2SnOdOaZYTVwaZkahpkh+gkfEbXyyG5J2J+U7DsiIeqMM9eJGMX83MPbIOjPSurGblA== X-Received: by 2002:ac8:5896:0:b0:3b9:b6e3:c78e with SMTP id t22-20020ac85896000000b003b9b6e3c78emr3050711qta.8.1681394706608; Thu, 13 Apr 2023 07:05:06 -0700 (PDT) Received: from mutt-hbsd (pool-100-16-219-215.bltmmd.fios.verizon.net. [100.16.219.215]) by smtp.gmail.com with ESMTPSA id h21-20020ac85695000000b003e3c23dd2cfsm500327qta.84.2023.04.13.07.05.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 13 Apr 2023 07:05:05 -0700 (PDT) Date: Thu, 13 Apr 2023 10:05:04 -0400 From: Shawn Webb To: Cy Schubert Cc: Mateusz Guzik , Pawe? Jakub Dawidek , Mark Millard , vishwin@freebsd.org, dev-commits-src-main@freebsd.org, Current FreeBSD , pjd@freebsd.org Subject: Re: git: 2a58b312b62f - main - zfs: merge openzfs/zfs@431083f75 Message-ID: <20230413140504.nnm23cjjv65mwzjr@mutt-hbsd> X-Operating-System: FreeBSD mutt-hbsd 14.0-CURRENT-HBSD FreeBSD 14.0-CURRENT-HBSD X-PGP-Key: https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/blob/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc References: <20230413071032.18BFF31F@slippy.cwsent.com> <20230413063321.60344b1f@cschubert.com> <20230413135635.6B62F354@slippy.cwsent.com> List-Id: Commit messages for the main branch of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-main List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-main@freebsd.org X-BeenThere: dev-commits-src-main@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="of4regawd5rmtjji" Content-Disposition: inline In-Reply-To: <20230413135635.6B62F354@slippy.cwsent.com> X-Rspamd-Queue-Id: 4Py1Yc4S5Qz4QGZ X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --of4regawd5rmtjji Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 13, 2023 at 06:56:35AM -0700, Cy Schubert wrote: > In message om> > , Mateusz Guzik writes: > > On 4/13/23, Cy Schubert wrote: > > > On Thu, 13 Apr 2023 19:54:42 +0900 > > > Pawe=3DC5=3D82 Jakub Dawidek wrote: > > > > > >> On Apr 13, 2023, at 16:10, Cy Schubert w= rote=3D > > : > > >> > > > >> > =3DEF=3DBB=3DBFIn message <20230413070426.8A54F25A@slippy.cwsent.c= om>, Cy Sc=3D > > hubert > > >> > writes: > > >> > In message <20230413064252.1E5C1318@slippy.cwsent.com>, Cy Schubert > > >> > writes: > > >> >> In message , Mark > > >> >> Millard > > >> >>> write > > >> >>> s: > > >> >>> [This just puts my prior reply's material into Cy's > > >> >>>> adjusted resend of the original. The To/Cc should > > >> >>>> be coomplete this time.] > > >> >>>> > > >> >>>> On Apr 12, 2023, at 22:52, Cy Schubert =3D > > =3D3D > > >> >>>> wrote: > > >> >>>> > > >> >>>> In message , Ma= rk =3D > > =3D3D > > >> >>>>> Millard=3D3D20 > > >> >>>> write > > >> >>>>> s: > > >> >>>>> From: Charlie Li wrote on > > >> >>>>>> Date: Wed, 12 Apr 2023 20:11:16 UTC : > > >> >>>>>> =3D3D20 > > >> >>>>>> Charlie Li wrote: > > >> >>>>>>> Mateusz Guzik wrote: > > >> >>>>>>>> can you please test poudriere with > > >> >>>>>>>>> https://github.com/openzfs/zfs/pull/14739/files > > >> >>>>>>>>> =3D3D20 > > >> >>>>>>>>> After applying, on the md(4)-backed pool regardless of =3D= 3D3D > > >> >>>>>>>> block_cloning,=3D3D3D20 > > >> >>>>>> the cy@ `cp -R` test reports no differing (ie corrupted) file= s. =3D > > =3D3D > > >> >>>>>>>> Will=3D3D3D20=3D3D3D > > >> >>>> =3D3D20 > > >> >>>>>> report back on poudriere results (no block_cloning). > > >> >>>>>>>> =3D3D3D20 > > >> >>>>>>>> As for poudriere, build failures are still rolling in. Thes= e ar=3D > > e > > >> >>>>>>>> =3D3D > > >> >>>>>>> (and=3D3D3D20=3D3D3D > > >> >>>> =3D3D20 > > >> >>>>>> have been) entirely random on every run. Some examples from t= his =3D > > =3D3D > > >> >>>>>>> run: > > >> >>>> =3D3D3D20 > > >> >>>>>>> lang/php81: > > >> >>>>>>> - post-install: @${INSTALL_DATA} > > >> >>>>>>> ${WRKSRC}/php.ini-development=3D3D3D20 > > >> >>>>>>> ${WRKSRC}/php.ini-production ${WRKDIR}/php.conf =3D3D3D > > >> >>>>>>> ${STAGEDIR}/${PREFIX}/etc > > >> >>>>>> - consumers fail to build due to corrupted php.conf packaged > > >> >>>>>>> =3D3D3D20 > > >> >>>>>>> devel/ninja: > > >> >>>>>>> - phase: stage > > >> >>>>>>> - install -s -m 555=3D3D3D20 > > >> >>>>>>> /wrkdirs/usr/ports/devel/ninja/work/ninja-1.11.1/ninja=3D3D3= D20 > > >> >>>>>>> /wrkdirs/usr/ports/devel/ninja/work/stage/usr/local/bin > > >> >>>>>>> - consumers fail to build due to corrupted bin/ninja packaged > > >> >>>>>>> =3D3D3D20 > > >> >>>>>>> devel/netsurf-buildsystem: > > >> >>>>>>> - phase: stage > > >> >>>>>>> - mkdir -p=3D3D3D20 > > >> >>>>>>> =3D3D3D > > >> >>>>>>> =3D3D > > >> >>>>>> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/l= ocal=3D > > /share/n > > >> >>>> e=3D3D > > >> >> =3D3D3D > > >> >>>> tsurf-buildsystem/makefiles=3D3D3D20 > > >> >>>>>> =3D3D3D > > >> >>>>>>> =3D3D > > >> >>>>>> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/l= ocal=3D > > /share/n > > >> >>>> e=3D3D > > >> >> =3D3D3D > > >> >>>> tsurf-buildsystem/testtools > > >> >>>>>> for M in Makefile.top Makefile.tools Makefile.subdir =3D3D3D > > >> >>>>>>> Makefile.pkgconfig=3D3D3D20 > > >> >>>>>> Makefile.clang Makefile.gcc Makefile.norcroft Makefile.open64= ; do > > >> >>>>>> \ > > >> >>>>>>> cp makefiles/$M=3D3D3D20 > > >> >>>>>>> =3D3D3D > > >> >>>>>>> =3D3D > > >> >>>>>> /wrkdirs/usr/ports/devel/netsurf-buildsystem/work/stage/usr/l= ocal=3D > > /share/n > > >> >>>> e=3D3D > > >> >> =3D3D3D > > >> >>>> tsurf-buildsystem/makefiles/;=3D3D3D20 > > >> >>>>>> \ > > >> >>>>>>> done > > >> >>>>>>> - graphics/libnsgif fails to build due to NUL characters in= =3D3D3D=3D > > 20 > > >> >>>>>>> Makefile.{clang,subdir}, causing nothing to link > > >> >>>>>>> =3D3D20 > > >> >>>>>> Summary: I have problems building ports into packages > > >> >>>>>> via poudriere-devel use despite being fully updated/patched > > >> >>>>>> (as of when I started the experiment), never having enabled > > >> >>>>>> block_cloning ( still using openzfs-2.1-freebsd ). > > >> >>>>>> =3D3D20 > > >> >>>>>> In other words, I can confirm other reports that have > > >> >>>>>> been made. > > >> >>>>>> =3D3D20 > > >> >>>>>> The details follow. > > >> >>>>>> =3D3D20 > > >> >>>>>> =3D3D20 > > >> >>>>>> [Written as I was working on setting up for the experiments > > >> >>>>>> and then executing those experiments, adjusting as I went > > >> >>>>>> along.] > > >> >>>>>> =3D3D20 > > >> >>>>>> I've run my own tests in a context that has never had the > > >> >>>>>> zpool upgrade and that jump from before the openzfs import to > > >> >>>>>> after the existing commits for trying to fix openzfs on > > >> >>>>>> FreeBSD. I report on the sequence of activities getting to > > >> >>>>>> the point of testing as well. > > >> >>>>>> =3D3D20 > > >> >>>>>> By personal policy I keep my (non-temporary) pool's compatible > > >> >>>>>> with what the most recent ??.?-RELEASE supports, using > > >> >>>>>> openzfs-2.1-freebsd for now. The pools involved below have > > >> >>>>>> never had a zpool upgrade from where they started. (I've no > > >> >>>>>> pools that have ever had a zpool upgrade.) > > >> >>>>>> =3D3D20 > > >> >>>>>> (Temporary pools are rare for me, such as this investigation. > > >> >>>>>> But I'm not testing block_cloning or anything new this time.) > > >> >>>>>> =3D3D20 > > >> >>>>>> I'll note that I use zfs for bectl, not for redundancy. So > > >> >>>>>> my evidence is more limited in that respect. > > >> >>>>>> =3D3D20 > > >> >>>>>> The activities were done on a HoneyComb (16 Cortex-A72 cores). > > >> >>>>>> The system has and supports ECC RAM, 64 GiBytes of RAM are > > >> >>>>>> present. > > >> >>>>>> =3D3D20 > > >> >>>>>> I started by duplicating my normal zfs environment to an > > >> >>>>>> external USB3 NVMe drive and adjusting the host name and such > > >> >>>>>> to produce the below. (Non-debug, although I do not strip > > >> >>>>>> symbols.) : > > >> >>>>>> =3D3D20 > > >> >>>>>> # uname -apKU > > >> >>>>>> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #90 = =3D3D3D > > >> >>>>>> main-n261544-cee09bda03c8-dirty: Wed Mar 15 20:25:49 PDT 2023 > > >> >>>>>> =3D3D3D > > >> >>>>>> =3D3D > > >> >>>>>> root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/= main=3D > > -src/arm > > >> >>>> 6=3D3D > > >> >> =3D3D3D > > >> >>>> 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1400082 1400082 > > >> >>>>>> =3D3D20 > > >> >>>>>> I then did: git fetch, stash push ., merge --ff-only, stash a= pply=3D > > . > > >> >>>>>> : > > >> >>>>>> my normal procedure. I then also applied the patch from: > > >> >>>>>> =3D3D20 > > >> >>>>>> https://github.com/openzfs/zfs/pull/14739/files > > >> >>>>>> =3D3D20 > > >> >>>>>> Then I did: buildworld buildkernel, install them, and reboote= d. > > >> >>>>>> =3D3D20 > > >> >>>>>> The result was: > > >> >>>>>> =3D3D20 > > >> >>>>>> # uname -apKU > > >> >>>>>> FreeBSD CA72_4c8G_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #91 = =3D3D3D > > >> >>>>>> main-n262122-2ef2c26f3f13-dirty: Wed Apr 12 19:23:35 PDT 2023 > > >> >>>>>> =3D3D3D > > >> >>>>>> =3D3D > > >> >>>>>> root@CA72_4c8G_ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/= main=3D > > -src/arm > > >> >>>> 6=3D3D > > >> >> =3D3D3D > > >> >>>> 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarc> >> >>>>>> =3D3D20 > > >> >>>>>> The later poudriere-devel based build of packages from ports = is > > >> >>>>>> based on: > > >> >>>>>> =3D3D20 > > >> >>>>>> # ~/fbsd-based-on-what-commit.sh -C /usr/ports > > >> >>>>>> 4e94ac9eb97f (HEAD -> main, freebsd/main, freebsd/HEAD) =3D3D= 3D > > >> >>>>>> devel/freebsd-gcc12: Bump to 12.2.0. > > >> >>>>>> Author: John Baldwin > > >> >>>>>> Commit: John Baldwin > > >> >>>>>> CommitDate: 2023-03-25 00:06:40 +0000 > > >> >>>>>> branch: main > > >> >>>>>> merge-base: 4e94ac9eb97fab16510b74ebcaa9316613182a72 > > >> >>>>>> merge-base: CommitDate: 2023-03-25 00:06:40 +0000 > > >> >>>>>> n613214 (--first-parent --count for merge-base) > > >> >>>>>> =3D3D20 > > >> >>>>>> poudriere attempted to build 476 packages, starting > > >> >>>>>> with pkg (in order to build the 56 that I explicitly > > >> >>>>>> indicate that I want). It is my normal set of ports. > > >> >>>>>> The form of building is biased to allowing a high > > >> >>>>>> load average compared to the number of hardware > > >> >>>>>> threads (same as cores here): each builder is allowed > > >> >>>>>> to use the full count of hardware threads. The build > > >> >>>>>> =3DE2=3D82=3DAC=3DC3=3D8FL=3DE2=3D82=3DAC=3DE2=3D82=3DAC=3DE2= =3D82=3DAC=3DE2=3D82=3DAC=3DE2=3D80=3DB9 > =3D > > > >> used USE_TMPFS=3D3D3D3D"data" instead of the > > >> >>>>>> USE_TMPFS=3D3D3D3Dall I > > >> >> normally use on the build machine involved. > > >> >>>>>> =3D3D20 > > >> >>>>>> And it produced some random errors during the attempted > > >> >>>>>> builds. A type of example that is easy to interpret > > >> >>>>>> without further exploration is: > > >> >>>>>> =3D3D20 > > >> >>>>>> pkg_resources.extern.packaging.requirements.InvalidRequiremen= t: > > >> >>>>>> Parse > > >> >>>>>> =3D3D > > >> >> =3D3D3D > > >> >>>> error at "'\x00\x00\x00\x00\x00\x00\x00\x00'": Expected > > >> >>>> W:(0-9A-Za-z) > > >> >>>>>> 0 > > >> >> da0p8 ONLINE 0 0 0 > > >> >>>>>> =3D3D20 > > >> >>>>>> errors: No known data errors > > >> >>>>>> =3D3D20 > > >> >>>>>> =3D3D20 > > >> >>>>>> =3D3D3D3D=3D3D3D3D=3D3D3D3D > > >> >>>>>> Mark Millard > > >> >>>>>> marklmi at yahoo.com > > >> >>>>>> =3D3D20 > > >> >>>>> =3D3D20 > > >> >>>>> Let's try this again. Claws-mail didn't include the list addre= ss i=3D > > n > > >> >>>>> =3D3D > > >> >>>>> the=3D3D20 > > >> >>>> header. Trying to reply, again, using exmh instead. > > >> >>>>> =3D3D20 > > >> >>>>> =3D3D20 > > >> >>>>> Did your pools suffer the EXDEV problem? The EXDEV also corrup= ted =3D > > =3D3D > > >> >>>>> files. > > >> >>>> > > >> >>>> As I reported, this was a jump from before the import > > >> >>>> to as things are tonight (here). So: NO, unless the > > >> >>>> existing code as of tonight still has the EXDEV problem! > > >> >>>> > > >> >>>> Prior to this experiment I'd not progressed any media > > >> >>>> beyond: main-n261544-cee09bda03c8-dirty Wed Mar 15 20:25:49. > > >> >>>> > > >> >>>> I think, without sufficient investigation we risk jumping to > > >> >>>>> conclusions. I've taken an extremely cautious approach, rolling > > >> >>>>> back > > >> >>>>> snapshots (as much as possible, i.e. poudriere datasets) when = EXDE=3D > > V > > >> >>>>> corruption was encountered. > > >> >>>>> > > >> >>>> Again: nothing between main-n261544-cee09bda03c8-dirty and > > >> >>>> main-n262122-2ef2c26f3f13-dirty was involved at any stage. > > >> >>>> > > >> >>>> =3D3D20 > > >> >>>>> I did not rollback any snapshots in my MH mail directory. Roll= ing > > >> >>>>> back > > >> >>>>> snapshots of my MH maildir would result in loss of email. I ha= ve t=3D > > o > > >> >>>>> live with that corruption. Corrupted files in my outgoing sent > > >> >>>>> email > > >> >>>>> directory remain: > > >> >>>>> =3D3D20 > > >> >>>>> slippy$ ugrep -cPa '\x00' ~/.Mail/note | grep -c :1=3D3D20 > > >> >>>>> 53 > > >> >>>>> slippy$=3D3D20 > > >> >>>>> =3D3D20 > > >> >>>>> There are 53 corrupted files in my note log of 9913 emails. Th= ose =3D > > =3D3D > > >> >>>>> files > > >> >>>> will never be fixed. They were corrupted by the EXDEV bug. Any = new > > >> >>>> ZFS > > >> >>>>> or ZFS patches cannot retroactively remove the corruption from > > >> >>>>> those > > >> >>>>> files. > > >> >>>>> =3D3D20 > > >> >>>>> But my poudriere files, because the snapshots were rolled back, > > >> >>>>> were > > >> >>>>> "repaired" by the rolled back snapshots. > > >> >>>>> =3D3D20 > > >> >>>>> I'm not convinced that there is presently active corruption si= nce > > >> >>>>> the problem has been fixed. I am convinced that whatever corru= ptio=3D > > n > > >> >>>>> that was written at the time will remain forever or until those > > >> >>>>> files > > >> >>>>> are deleted or replaced -- just like my email files written to= dis=3D > > k > > >> >>>>> at > > >> >>>>> the time. > > >> >>>>> > > >> >>>> My test results and procedure just do not fit your conclusion > > >> >>>> that things are okay now if block_clonging is completely avoide= d. > > >> >>>> > > >> >>> Admitting I'm wrong: sending copies of my last reply to you back= to > > >> >>> myself, > > >> >>> > > >> >> again and again, three times, I've managed to reproduce the corru= ptio=3D > > n > > >> >> you > > >> >>> are talking about. > > >> >>> > > >> >> This email itself was also corrupted. Below is what was sent. Good > > >> >> thing > > >> >> multiple copies are saved by exmh. > > >> >> > > >> >> Admitting I'm wrong: sending copies of my last reply to you back = to > > >> >> myself, > > >> >> again and again, three times, I've managed to reproduce the corru= ptio=3D > > n > > >> >> you > > >> >> are talking about. > > >> >> > > >> > This email itself was also corrupted. Below is what was sent. Good > > >> > thing > > >> > multiple copies are saved by exmh. > > >> > > > >> > Admitting I'm wrong: sending copies of my last reply to you back to > > >> > myself, > > >> > again and again, three times, I've managed to reproduce the corrup= tion > > >> > you > > >> > are talking about. > > >> > > > >> > From my previous email to you. > > >> > > > >> > header. Trying to reply:::::::::, again, using exmh instead. > > >> > ^^^^^^^^^ > > >> > Here it is, nine additional bytes of garbage. I've replaced the ga= rbag=3D > > e > > >> > with colons because nulls mess up a lot of things, including cut&p= aste=3D > > . > > >> > > > >> > In another instance about 500 bytes were removed. I can reproduce = the > > >> > corruption at will now. > > >> > > > >> > The EXDEV patch is applied. Block_cloning is disabled. > > >> > > > >> > Somehow nulls and other garbage are inserted in the middle of emai= ls > > >> > after > > >> > the ZFS upgrade. > > >> > > > >> Can you please try this patch: > > >> > > >> github.com > > > > > > The patch was applied yesterday at noon (PDT). > > > > > >> > > >> > > >> > > >> Unfortunately I don=3DE2=3D80=3D99t see how this can happen with blo= ck cloning > > >> disabled. > > > > > > It does and it's reproducible. > > > > > > > There is corruption with the recent import, with the > > https://github.com/openzfs/zfs/pull/14739/files patch applied and > > block cloning disabled on the pool. >=20 > Same here. >=20 > > > > There is no corruption with top of main with zfs merge reverted altoget= her. >=20 > I'm in the process of building a branch reverting the merge altogether an= d=20 > will test it on my sandbox machine later today. >=20 > > > > Which commit results in said corruption remains to be seen, a variant > > of the tree with just block cloning support reverted just for testing > > purposes is about to be evaluated. I've learned over the years downstream that it's not really my place to tell upstream what to do or how to do it. However, I think given the seriousness of this, upstream might do well to revert the commit until a solid fix is in place. Upstream might want to consider the impacts this is having not just with downstream projects, but also regular users. Really bad timing to have a lot of new tax documentation that I really don't want to lose. I'd really like to have an up-to-date, security patched OS, but I guess I'll stay behind so that I don't risk losing critical financial documentation. Does the ZFS project have some sort of automated testing to catch data-gobbling, pool killing bugs? It seems like this would have been caught with some CI/CD stress testing automation scripts. Thanks, --=20 Shawn Webb Cofounder / Security Engineer HardenedBSD https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A= 4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc --of4regawd5rmtjji Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEA6TL67gupaZ9nzhT/y5nonf44foFAmQ4DAkACgkQ/y5nonf4 4fomgxAAn0eDUQVU6hZ4z9rC/NttvibQseTBVvr/0Re48+GLu+AFYz34BT10t/p/ vxINgpeoHMMEfC8k67dojIcElsDOUa9M2Y5XNnJI6rr1TjwIiIE3Hpn7etH59g8O +k9B715AkRwP+3o8N4xUPXIHnZ6c/auFdFZO3MUsYOhr1O0UrkYtH5IC1OjcapXl x2uENZj9FaEFsjgWpVyougkUsh04/9T+xbj7+s6J0npvDCVuXkqDKiyQEWfE/ntu b7LcU2c/fpLvwWdYNYBV6+vIGy99aRYdld9OW9IdkA0XZnkyurKXU9OPgmZgIBze f1dqkEZzGdM3K/6CqS9RuzHYIzMyFtt3ic+vZZB+VKOqP8O66chERGP+lZnG41Rz QESnEFn1H2QUBH6xTw+RKXHwL3Gsx/HUmQK0uQksqKY54AeTBMEKjlmsL05G1MGF 78vcaCcMlbHja7aHeAlaPpYfVchl6O+VRogz5FFoPF4xMldoKNYTOAv+qlCNE9qw y1qwDsg2f7vQCtrS0XbBZCQ28zF18v9DEzLxXF3xYfmE/lu0AqQgGICxwkOW4QZv Hw7kr2D5PdSLSzN4hlbsuGyvb4VVq7fGlw6JnVxLUIr41V5Y/XjY6uSysoQvZsdo 7zCHdyIMsbELZncZAzMXwhSUDpeArKQS0vS9WLZQHScsZEP3huQ= =mNE2 -----END PGP SIGNATURE----- --of4regawd5rmtjji--