Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 4 Nov 2023 22:34:48 +0100
From:      Martin Matuska <mm@FreeBSD.org>
To:        Alexander Leidinger <Alexander@Leidinger.net>
Cc:        John F Carr <jfc@mit.edu>, freebsd-fs@freebsd.org
Subject:   Re: ZFS txg rollback: expected timeframe?
Message-ID:  <2f93493b-a9cc-4c71-848a-efc55751a33e@FreeBSD.org>
In-Reply-To: <e48bfe90cd5e44c28d9656c1ca817bb2@Leidinger.net>
References:  <CAJg7qzHONfMeLUm20OE6Jo5uFLt9bY5VVhbY8z%2BoEVcHYwyoXw@mail.gmail.com> <18B0B6B6-9237-42D0-9FB2-D55CE72E1CCA@mit.edu> <CAJg7qzEnQFUS4v=zYjch3KTySxuErnwC2E5zYTwMB5UkfoTV6g@mail.gmail.com> <e48bfe90cd5e44c28d9656c1ca817bb2@Leidinger.net>

next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format.
--------------6M70tl1GHxx17km1YMm0xKtr
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Alexander,

I am already running block cloning with stable/14 on many production 
servers and also on my Ubuntu 23.10 notebook. The referenced issue 
appeared way before block cloning was even introduced so that is 
unrelated to me. Block cloning is not the same as deduplication. When 
copying a file with copy_file_range(2) between two datasets on the same 
pool, blocks of data are only referenced instead of doing a real copy. 
That is all magic that is being done. Did you check "zpool get 
bcloneused,bclonesaved poolname" before re-creating the pool? That would 
tell you if any blocks were cloned using block cloning at all.

Cheers,
Martin

On 04/11/2023 18:35, Alexander Leidinger wrote:
>
> Am 2023-10-31 15:27, schrieb Alexander Leidinger:
>
>> On Tue, Oct 31, 2023 at 1:15 PM John F Carr <jfc@mit.edu> wrote:
>>
>>
>>
>>     > On Oct 31, 2023, at 06:16, Alexander Leidinger
>>     <alexleidingerde@gmail.com> wrote:
>>     >
>>     > Issue: a overheating CPU may have corrupted a zpool (4 * 4TB in
>>     raidz2 setup) in a way that a normal import of the pool panics
>>     the machine with "VERIFY3(l->blk_birth == r->blk_birth) failed
>>     (101867360 == 101867222)".
>>     >
>>
>>     I disabled that assertion because it gives a false alarm with
>>     some combinaion
>>     of deduplication, cloning, and snapshotting on one of my systems.
>>
>>     See
>>
>>     https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261538
>>     https://github.com/openzfs/zfs/issues/11480
>>
>> I don't have deduplication on this pool. There are clones, and 
>> snapshots, and there could be recent ones if poudriere does some. Is 
>> it still a false alarm in this case? If yes, you say a kernel with 
>> this patch applied should let me import the pool without rollback?
>> The github issue is from 2022, I have my doubts that this is the same 
>> issue we see. I rather expect some issues around the 
>> copy_file_range(2) related code for ZFS which was re-enabled 20 days 
>> ago (maybe it is valid to remove this assert, or maybe the block 
>> cloning part needs some tweak). CC Martin for the block cloning part.
>
> So in the end I was at least able to import the pool read-only with 
> the patch to disable this VERIFY3 panic. After a final incremental 
> backup I re-created the pool (with vfs.zfs.bclone_enabled=0) and 
> restored all datasets. Now some checks (this VERIFY3 part is enabled 
> again) and then a full backup.
>
> There is still the question what caused it. With the above report from 
> John about some issues when dedup is enabled (which wasn't on this 
> pool until the default of bclone_enabled was changed to 1, which is 
> some kind of dedup internal to ZFS as far as I was understanding the 
> description of block cloning in the openzfs ticket of block cloning) I 
> have some reservations about enabling it again.
>
> Martin, maybe it's a good idea to disable block cloning again, until 
> someone with the corresponding OpenZFS knowledge has investigated this...
>
> Bye,
> Alexander.
>
> -- 
> http://www.Leidinger.net <http://www.Leidinger.net>; 
> Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF
> http://www.FreeBSD.org <http://www.FreeBSD.org>; netchild@FreeBSD.org 
>  : PGP 0x8F31830F9F2772BF
--------------6M70tl1GHxx17km1YMm0xKtr
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p>Hi Alexander,</p>
    <p>I am already running block cloning with stable/14 on many
      production servers and also on my Ubuntu 23.10 notebook. The
      referenced issue appeared way before block cloning was even
      introduced so that is unrelated to me. Block cloning is not the
      same as deduplication. When copying a file with copy_file_range(2)
      between two datasets on the same pool, blocks of data are only
      referenced instead of doing a real copy. That is all magic that is
      being done. Did you check "zpool get bcloneused,bclonesaved
      poolname" before re-creating the pool? That would tell you if any
      blocks were cloned using block cloning at all.</p>
    <p>Cheers,<br>
      Martin<br>
    </p>
    <div class="moz-cite-prefix">On 04/11/2023 18:35, Alexander
      Leidinger wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:e48bfe90cd5e44c28d9656c1ca817bb2@Leidinger.net">
      <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
      <p id="reply-intro">Am 2023-10-31 15:27, schrieb Alexander
        Leidinger:</p>
      <blockquote type="cite"
style="padding: 0 0.4em; border-left: #1010ff 2px solid; margin: 0">
        <div id="replybody1">
          <div dir="ltr">
            <div dir="ltr">On Tue, Oct 31, 2023 at 1:15 PM John F Carr
              &lt;<a href="mailto:jfc@mit.edu" rel="noreferrer"
                moz-do-not-send="true" class="moz-txt-link-freetext">jfc@mit.edu</a>&gt;
              wrote:</div>
            <div class="v1gmail_quote">
              <blockquote class="v1gmail_quote"
style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><br>
                <br>
                &gt; On Oct 31, 2023, at 06:16, Alexander Leidinger &lt;<a
                  href="mailto:alexleidingerde@gmail.com"
                  rel="noreferrer" moz-do-not-send="true"
                  class="moz-txt-link-freetext">alexleidingerde@gmail.com</a>&gt;
                wrote:<br>
                &gt; <br>
                &gt; Issue: a overheating CPU may have corrupted a zpool
                (4 * 4TB in raidz2 setup) in a way that a normal import
                of the pool panics the machine with
                "VERIFY3(l-&gt;blk_birth == r-&gt;blk_birth) failed
                (101867360 == 101867222)".<br>
                &gt; <br>
                <br>
                I disabled that assertion because it gives a false alarm
                with some combinaion<br>
                of deduplication, cloning, and snapshotting on one of my
                systems.<br>
                <br>
                See<br>
                <br>
                 <a
href="https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261538"
                  target="_blank" rel="noopener noreferrer"
                  moz-do-not-send="true" class="moz-txt-link-freetext">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=261538</a><br>;
                 <a href="https://github.com/openzfs/zfs/issues/11480"
                  target="_blank" rel="noopener noreferrer"
                  moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/openzfs/zfs/issues/11480</a></blockquote>;
              <div> </div>
              <div>I don't have deduplication on this pool. There are
                clones, and snapshots, and there could be recent ones if
                poudriere does some. Is it still a false alarm in this
                case? If yes, you say a kernel with this patch applied
                should let me import the pool without rollback?</div>
              <div> </div>
              <div>The github issue is from 2022, I have my doubts that
                this is the same issue we see. I rather expect some
                issues around the copy_file_range(2) related code for
                ZFS which was re-enabled 20 days ago (maybe it is valid
                to remove this assert, or maybe the block cloning part
                needs some tweak). CC Martin for the block cloning part.</div>
            </div>
          </div>
        </div>
      </blockquote>
      <p>So in the end I was at least able to import the pool read-only
        with the patch to disable this VERIFY3 panic. After a final
        incremental backup I re-created the pool (with
        vfs.zfs.bclone_enabled=0) and restored all datasets. Now some
        checks (this VERIFY3 part is enabled again) and then a full
        backup.</p>
      <p>There is still the question what caused it. With the above
        report from John about some issues when dedup is enabled (which
        wasn't on this pool until the default of bclone_enabled was
        changed to 1, which is some kind of dedup internal to ZFS as far
        as I was understanding the description of block cloning in the
        openzfs ticket of block cloning) I have some reservations about
        enabling it again.</p>
      <p>Martin, maybe it's a good idea to disable block cloning again,
        until someone with the corresponding OpenZFS knowledge has
        investigated this...</p>
      <p>Bye,<br>
        Alexander.</p>
      <div id="signature">-- <br>
        <div class="pre"
          style="margin: 0; padding: 0; font-family: monospace"><a
            href="http://www.Leidinger.net" target="_blank"
            rel="noopener noreferrer" moz-do-not-send="true">http://www.Leidinger.net</a>;
          <a href="mailto:Alexander@Leidinger.net:"
            moz-do-not-send="true" class="moz-txt-link-freetext">Alexander@Leidinger.net:</a>
          PGP 0x8F31830F9F2772BF<br>
          <a href="http://www.FreeBSD.org" target="_blank"
            rel="noopener noreferrer" moz-do-not-send="true">http://www.FreeBSD.org</a>;
             <a href="mailto:netchild@FreeBSD.org"
            moz-do-not-send="true" class="moz-txt-link-freetext">netchild@FreeBSD.org</a>
           : PGP 0x8F31830F9F2772BF</div>
      </div>
    </blockquote>
  </body>
</html>

--------------6M70tl1GHxx17km1YMm0xKtr--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2f93493b-a9cc-4c71-848a-efc55751a33e>