From nobody Thu May 16 14:38:08 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VgCPr1jmkz5LLqd for ; Thu, 16 May 2024 14:38:24 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VgCPq27mMz43Zl for ; Thu, 16 May 2024 14:38:23 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x634.google.com with SMTP id a640c23a62f3a-a59b49162aeso310930566b.3 for ; Thu, 16 May 2024 07:38:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1715870300; x=1716475100; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=5ehfDzGxg+ZPoJnr1bHlqY97Wkf6tb3QqziP1cLsKF4=; b=HHZnQQWB4Uy/GWmazV6Y7BZROS00cCfvrgrZPDjla6zJtu8kj5SdqjDQ7rYKVhcx4g c1Fa8gS50yA03L55WJMHFPlyQlgCdTxJS/BLTsgxIQd48I+T2O0/7ujpOCsT7L003Rkd 8mVXLBwqRu+RL9MbtU/vzqVwhe+77IxWv8oBFWkWft3WKYDzMVvICp5uy0+RBCIPc1JK TgdLktfCbW7sU35zI0mEK754HwedvF0EruZa36N9ivVUUboRjSJCY/bIRfmEFYMxFPi8 nF/2xa4zKw9ALAXjx7+2KTQYMsk/C/C/S4Qsbd7ZROuKqUOmr9KuYhBDghhgqM3MDfvT Rrbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715870300; x=1716475100; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5ehfDzGxg+ZPoJnr1bHlqY97Wkf6tb3QqziP1cLsKF4=; b=khkfmhfea68ZC7jEjrsEIE+FmYr3BO5yB840MIi5qip6kIj5kvNGPJoE7V7EW/x29c I0RE3Y+zINtPytyfaGt7vv+nZqBVDoLfhUCceYuPTULJHCQed9FlrOaeKwASTeHkb3QA t8T6o0djPFqs+jR9ZSBp169GNj6HX453ODzX1npSFHszpzhvmLajOyWvL8f6YfvTWtue HonskimAmLaOo0I7B1E1ArgNnBHfKIg8HFpTx3k9ThzOHT/Q3JP54sNNBg3jiWM4DoMA tj9dy73r0I+QzvIiylOZH5J27dK+hI3lqv79vIlbny4tCO+nJu7x9HiJzUSgcLLuu6Hl wNuQ== X-Gm-Message-State: AOJu0YzST8wvOgnCnJarZVnLSlotclJtFxXWEofvJD9ychftm/+BuwBd N3ZELaBu6kOSGEt+2oH2MYj+0CZVpl6Q8CeNthXeMnkK1s/z7Qizfp9yMN0pHPxuUqeL2j5dqN3 a1zqlsnjXgeiwWiyshzLK2p7Gz08Dooy9cU4uhIDXF8yvDTdPHNM= X-Google-Smtp-Source: AGHT+IER0ZGT8Xwa3UFDnoGB6lUVA3tc6rTfr3UzrKMNUKtK9KJCIqF1WUFiOj2rFTXPcTqAZxIgkcDAFL14xiICeXI= X-Received: by 2002:a17:906:7196:b0:a55:5ba7:2889 with SMTP id a640c23a62f3a-a5a2d5c9f69mr1257882866b.42.1715870300248; Thu, 16 May 2024 07:38:20 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <4c331e4f-75c0-4124-bb11-84568e91ca61@sentex.net> In-Reply-To: <4c331e4f-75c0-4124-bb11-84568e91ca61@sentex.net> From: Warner Losh Date: Thu, 16 May 2024 08:38:08 -0600 Message-ID: Subject: Re: Open ZFS vs FreeBSD ZFS boot issues To: mike tancsa Cc: FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="000000000000ba385d0618932fa9" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4VgCPq27mMz43Zl --000000000000ba385d0618932fa9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, May 16, 2024 at 8:14=E2=80=AFAM mike tancsa wrote= : > I have a strange edge case I am trying to work around. I have a > customer's legacy VM which is RELENG_11 on ZFS. There is some > corruption that wont clear on a bunch of directories, so I want to > re-create it from backups. I have done this many times in the past but > this one is giving me grief. Normally I do something like this on my > backup server (RELENG_13) > > truncate -s 100G file.raw > mdconfig -f file.raw > gpart create -s gpt md0 > gpart add -t freebsd-boot -s 512k md0 > gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 md0 > gpart add -t freebsd-swap -s 2G md0 > gpart add -t freebsd-zfs md0 > zpool create -d -f -o altroot=3D/mnt2 -o feature@lz4_compress=3Denabled -= o > cachefile=3D/var/tmp/zpool.cache myZFSPool /dev/md0p3 > I'm surprised you don't specifically create compatibility with some older standard and then maybe add compression. But I'd start there: create one that doesn't use lz4_compress (it's not read-only compatible, meaning the old boot loader has to 100% implement it faithfully). You might also look at disabling one or more of hold_birth and embedded_dat= a as well. Those aren't 'read-only' compatible. But all of that is kinda speculative. I'm not aware of any bugs in this area that were fixed, and all these options are in the features_for_read list that is in the boot loader. > Then zfs send -r backuppool | zfs recv myZFSPool > > I can then export / import the myZFSPool without issue. I can even > import and examine myZFSPool on the original RELENG_11 VM that is > currently running. A checksum of all the files under /boot are > identical. But every time I try to boot it (KVM), it panics early > > FreeBSD/x86 ZFS enabled bootstrap loader, Revision 1.1 > > (Tues Oct 10:24:17 EDT 2018 user@hostname) > panic: free: guard2 fail @ 0xbf153040 + 2061 from unknown:0 > --> Press a key on the console to reboot <-- > This is a memory corruption bug. You'll need to find what's corrupting memory and make it stop. I imagine this might be a small incompatibility with OpenZFS or just a bug in what openZFS is generating on the releng13 server. > Through a bunch of pf rdrs and nfs mounts, I was able to do the same > above steps on the live RELENG_11 image and do the zfs send/recv and the > image boots up no problem. Any ideas on how to work around this or > what the problem might be I am running into ? The issue seems to be that > I do the zfs recv on a RELENG_13 box. If I do the zfs recv on RELENG_11 > (which takes a LOT longer) it takes forever. zdb differences [1] below. > > The kernel is r339251 11.2-STABLE. I know this is a crazy old issue, > but hoping to at least learn something about ZFS as a result of going > down this rabbit hole. I will I think just do the send|recv via a > RELENG_11 just to get them up and running. They dont have the $ to get > me to upgrade it all for them and this is partly a favor to them to help > them limp along a bit more... > What version is the boot loader? There's been like 6 years of fixes and churn since the date above? Maybe the latest on RELENG_11 for it if you are still running 11.2-stable. Any chance you can use the stable/13 or stable/14 loaders? 11 is really not supported anymore and hasn't been for quite some time. I have no time for it beyond this quick peek. Warner > ---Mike > > > 1 zdb live pool > > ns9zroot: > version: 5000 > name: 'livezroot' > state: 0 > txg: 26872926 > pool_guid: 15183996218106005646 > hostid: 2054190969 > hostname: 'customer-hostname' > com.delphix:has_per_vdev_zaps > vdev_children: 1 > vdev_tree: > type: 'root' > id: 0 > guid: 15183996218106005646 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 15258031439924457243 > path: '/dev/vtbd0p3' > whole_disk: 1 > metaslab_array: 256 > metaslab_shift: 32 > ashift: 12 > asize: 580889083904 > is_log: 0 > DTL: 865260 > create_txg: 4 > com.delphix:vdev_zap_leaf: 129 > com.delphix:vdev_zap_top: 130 > features_for_read: > com.delphix:hole_birth > com.delphix:embedded_data > > > MOS Configuration: > version: 5000 > name: 'fromBackupPool' > state: 0 > txg: 2838 > pool_guid: 1150606583960632990 > hostid: 2054190969 > hostname: 'customer-hostname' > com.delphix:has_per_vdev_zaps > vdev_children: 1 > vdev_tree: > type: 'root' > id: 0 > guid: 1150606583960632990 > create_txg: 4 > children[0]: > type: 'disk' > id: 0 > guid: 4164348845485675975 > path: '/dev/md0p3' > whole_disk: 1 > metaslab_array: 256 > metaslab_shift: 29 > ashift: 12 > asize: 105221193728 > is_log: 0 > create_txg: 4 > com.delphix:vdev_zap_leaf: 129 > com.delphix:vdev_zap_top: 130 > features_for_read: > com.delphix:hole_birth > com.delphix:embedded_data > Neither of these --000000000000ba385d0618932fa9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, May 16, 2024 at 8:14=E2=80=AF= AM mike tancsa <mike@sentex.net&g= t; wrote:
I have= a strange edge case I am trying to work around.=C2=A0 I have a
customer's legacy VM which is RELENG_11 on ZFS.=C2=A0 There is some corruption that wont clear on a bunch of directories, so I want to
re-create it from backups. I have done this many times in the past but
this one is giving me grief. Normally I do something like this on my
backup server (RELENG_13)

truncate -s 100G file.raw
mdconfig -f file.raw
gpart create -s gpt md0
gpart add -t freebsd-boot -s 512k md0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 md0
gpart add -t freebsd-swap -s 2G md0
gpart add -t freebsd-zfs md0
zpool create -d -f -o altroot=3D/mnt2 -o feature@lz4_compress=3Denabled -o =
cachefile=3D/var/tmp/zpool.cache myZFSPool /dev/md0p3
=
I'm surprised you don't specifically create compatib= ility with some older
standard and then maybe add compression. Bu= t I'd start there: create
one that doesn't use lz4_compre= ss (it's not read-only compatible,
meaning the old boot loade= r has to 100% implement it faithfully).

You might = also look at disabling one or more of hold_birth and embedded_data
as well. Those aren't 'read-only' compatible.

<= /div>
But all of that is kinda speculative. I'm not aware of any bu= gs in this area that
were fixed, and all these options are in the= features_for_read list that is in the boot
loader.
=C2=A0
Then zfs send -r backuppool | zfs recv myZFSPool

I can then export / import the myZFSPool without issue. I can even
import and examine myZFSPool on the original RELENG_11 VM that is
currently running. A checksum of all the files under /boot are
identical.=C2=A0 But every time I try to boot it (KVM), it panics early

FreeBSD/x86 ZFS enabled bootstrap loader, Revision 1.1

(Tues Oct 10:24:17 EDT 2018 user@hostname)
panic: free: guard2 fail @ 0xbf153040 + 2061 from unknown:0
--> Press a key on the console to reboot <--
This is a memory corruption bug. You'll need to find what&#= 39;s corrupting
memory and make it stop.

I imagine this might be a small incompatibility with OpenZFS or just
=
a bug in what openZFS is generating on the releng13 server.
<= div>=C2=A0
Through a bunch of pf rdrs and nfs mounts, I was able to do the same
above steps on the live RELENG_11 image and do the zfs send/recv and the image boots up no problem.=C2=A0=C2=A0 Any ideas on how to work around this= or
what the problem might be I am running into ? The issue seems to be that I do the zfs recv on a RELENG_13 box. If I do the zfs recv on RELENG_11 (which takes a LOT longer) it takes forever. zdb differences [1] below.

The kernel is r339251 11.2-STABLE.=C2=A0 I know this is a crazy old issue, =
but hoping to at least learn something about ZFS as a result of going
down this rabbit hole.=C2=A0 I will I think just do the send|recv via a RELENG_11 just to get them up and running.=C2=A0 They dont have the $ to ge= t
me to upgrade it all for them and this is partly a favor to them to help them limp along a bit more...

What vers= ion is the boot loader? There's been like 6 years of fixes and
churn since the date above? Maybe the latest on RELENG_11 for itif you are still running 11.2-stable.

Any chance= you can use the stable/13 or stable/14 loaders? 11 is really
not= supported anymore and hasn't been for quite some time. I have no
=
time for it beyond this quick peek.

Warne= r
=C2=A0
=C2=A0=C2=A0=C2=A0=C2=A0 ---Mike


1 zdb live pool

ns9zroot:
=C2=A0=C2=A0=C2=A0=C2=A0 version: 5000
=C2=A0=C2=A0=C2=A0=C2=A0 name: 'livezroot'
=C2=A0=C2=A0=C2=A0=C2=A0 state: 0
=C2=A0=C2=A0=C2=A0=C2=A0 txg: 26872926
=C2=A0=C2=A0=C2=A0=C2=A0 pool_guid: 15183996218106005646
=C2=A0=C2=A0=C2=A0=C2=A0 hostid: 2054190969
=C2=A0=C2=A0=C2=A0=C2=A0 hostname: 'customer-hostname'
=C2=A0=C2=A0=C2=A0=C2=A0 com.delphix:has_per_vdev_zaps
=C2=A0=C2=A0=C2=A0=C2=A0 vdev_children: 1
=C2=A0=C2=A0=C2=A0=C2=A0 vdev_tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 type: 'root'
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 id: 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 guid: 15183996218106005646=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 create_txg: 4
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 children[0]:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ty= pe: 'disk'
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 id= : 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gu= id: 15258031439924457243
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pa= th: '/dev/vtbd0p3'
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wh= ole_disk: 1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 me= taslab_array: 256
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 me= taslab_shift: 32
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 as= hift: 12
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 as= ize: 580889083904
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 is= _log: 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 DT= L: 865260
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cr= eate_txg: 4
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 co= m.delphix:vdev_zap_leaf: 129
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 co= m.delphix:vdev_zap_top: 130
=C2=A0=C2=A0=C2=A0=C2=A0 features_for_read:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 com.delphix:hole_birth
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 com.delphix:embedded_data<= br>

MOS Configuration:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 version: 5000
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 name: 'fromBackupPool&= #39;
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 state: 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 txg: 2838
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pool_guid: 115060658396063= 2990
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hostid: 2054190969
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 hostname: 'customer-ho= stname'
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 com.delphix:has_per_vdev_z= aps
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 vdev_children: 1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 vdev_tree:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ty= pe: 'root'
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 id= : 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gu= id: 1150606583960632990
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 cr= eate_txg: 4
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ch= ildren[0]:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 type: 'disk'
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 id: 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 guid: 4164348845485675975
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 path: '/dev/md0p3'
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 whole_disk: 1
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 metaslab_array: 256
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 metaslab_shift: 29
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 ashift: 12
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 asize: 105221193728
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 is_log: 0
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 create_txg: 4
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 com.delphix:vdev_zap_leaf: 129
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 com.delphix:vdev_zap_top: 130
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 features_for_read:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 co= m.delphix:hole_birth
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 co= m.delphix:embedded_data

Neither of thes= e=C2=A0
--000000000000ba385d0618932fa9--