From nobody Fri Nov 8 22:11:23 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4XlY7Y24ywz5csTW for ; Fri, 08 Nov 2024 22:11:37 +0000 (UTC) (envelope-from girgen@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4XlY7Y1nmjz4S0w; Fri, 8 Nov 2024 22:11:37 +0000 (UTC) (envelope-from girgen@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731103897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=9c5aMOhvpS8lgbbOUg2UzpDwzwNg+YSf33P+P5GmR3g=; b=tpnviEGFUtFKNjeGda61PjAu83M7AFmwOI8jfdJE1t54shVnS7d8wR342xZ9LMSRoSYswk /qirg5aAaxQazx3oA1QK+DKxGyvBxsE7LSasctW49suCBZ+VCJ8Rm7ywkukWCugjn7Km1Y yJv+13ll2mSm39EU/wgUC39JxwUvdpitXJa34cqAXgqGq4OBbC2omgevEIZpdplBcKuoiA 1fRoGAdRkw9Mm9Gm0h8NQLVzQeBx1CHN0sLfaRWiwMTckpMt7n6HThSG9EkFnb/7w9U2YF 5ZBc7AavHut/sSy4DBTyNn5NFzGjA0+938d8qgF4sIqBzql40OxnJ+VuRbsHug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1731103897; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=9c5aMOhvpS8lgbbOUg2UzpDwzwNg+YSf33P+P5GmR3g=; b=tGu2sLHsZcD1NAcOSf9w854s533QJuLWa21BrBLzAwLfTFI/6FaooxooNfeAELFmKSb93X FVvbm65vceYO5ka+YabcWrl+b9Zeg+oN6ceH3VC4y5SA62SRyV5ag2To7rimKHShmQkGcx 0QaZ+NQ+XaiXHhn8Niqx7640XNxiHBio7M6ZC4f+mf385fmYFdtSq9hgaJmfI5KbCMW55n hJzrn69pvXgR4OSXW8C+3BSMK+ezTZWJMO+ezRpvFnYWVtAbQHikZNpKxrD6FnK65rD1WC Vm8O3l6P+uFwQ7+oh9zCZhlAu5hN7LpqL00VF/VJ2VC41lXRqyvvluHIDrsFpA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1731103897; a=rsa-sha256; cv=none; b=u7d446mLLnXz9wyPR+BWSL8k+H2voWVicwdz/SfTx7QOLyQz3Z4R6Ln0Hog42rltrIHtFp CPBdaSS+DFNuTFxRt8+409yabUV6KFAU91gi52fHNLNPKeB0R0XoU1aAoE0VCkOzutdIy4 KeaS8y723nE1cFW+wgbM691vuPfffmNYKRm1zZM7i2B8AehSJrmuqWeHMOuytdcBzUTId4 2q/+8iTmHHMe0Tj9I1S22R+WJ1sgTPaUs+1R6dk68FBrKB5U5+ZNU9MjhAj6KEd0DWxmCk 2EthKQ5KPYqQzldQNdgJlORl568oCxZ0f4gcemfs8MBzpdrt6lxTs9iY3liaUA== Received: from smtpclient.apple (h-213-164-202-49.A498.priv.bahnhof.se [213.164.202.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: girgen/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4XlY7X5cPtz155R; Fri, 8 Nov 2024 22:11:36 +0000 (UTC) (envelope-from girgen@FreeBSD.org) From: Palle Girgensohn Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973" List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.200.121\)) Subject: Re: zfs snapshot corruption when using encryption Date: Fri, 8 Nov 2024 23:11:23 +0100 In-Reply-To: Cc: freebsd-fs@freebsd.org To: void References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> X-Mailer: Apple Mail (2.3826.200.121) --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > 8 nov. 2024 kl. 20:20 skrev void : >=20 > On Fri, Nov 08, 2024 at 03:29:22PM +0100, Palle Girgensohn wrote: >=20 >> I cannot run `zfs send -I fs@previous_snap fs@problematic_snap`, I = get >> warning: cannot send 's@problematic_snap': Input/output error` >> Removing the snapshot fixes the problem. >=20 > What an odd problem. Like, it can write but not read. What zfs = version? > On 13.3 I'm using: > zfs-2.1.14-FreeBSD_gd99134be8 > zfs-kmod-2.1.14-FreeBSD_gd99134be8 >=20 > on 14-stable: > zfs-2.2.6-FreeBSD_g33174af15 > zfs-kmod-2.2.6-FreeBSD_g33174af15 >=20 > on 15-current: > zfs-2.3.99-31-FreeBSD_gb2f6de7b5 > zfs-kmod-2.3.99-31-FreeBSD_gb2f6de7b5 >=20 $ freebsd-version -ku 14.0-RELEASE-p10 14.0-RELEASE-p10 dunno how to present in the above fashion? Binary install, not from = source. > Is the encryption you're using is the GELI > based whole-disk one (which has been around iirc for a few years) or > the relatively recent zfs encryption that works per-filesystem. This = is why I'm asking the zfs version. I thought the latter is relatively = quite new, and I've never heard of it working on 14.0. The latter, zfs per-filesystem zfs encryption. I haven't heard any = horror stories about it, albeit it's quite fresh code. > The way I'd go about trying to address the issue would be to start = from the lowest layer and work up. On a raidz2 array (the card in JBOD = mode),=20 yes, JBOD > I can identify all the drives as da0-7, so can use smartctl -x to = query them all directly, looking for, in particular: >=20 > Reallocated_Sector_Ct > Reported_Uncorrect > Current_Pending_Sector > Offline_Uncorrectable >=20 > If your disk arrangement is connected via something like HP Smart = Array > or similar, you'll need to look at the man page for smartctl for the = exact > syntax to query the disks behind the card. Na, it's JBOD alright. I see nothing weird with the disks. Palle= --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

8 nov. 2024 kl. 20:20 skrev void <void@f-m.fm>:

On = Fri, Nov 08, 2024 at 03:29:22PM +0100, Palle Girgensohn = wrote:

I cannot run `zfs send -I = fs@previous_snap fs@problematic_snap`, I get
warning: cannot send = 's@problematic_snap': Input/output error`
Removing the snapshot fixes = the problem.

What an odd problem. Like, it can write = but not read. What zfs version?
On 13.3 I'm = using:
zfs-2.1.14-FreeBSD_gd99134be8
zfs-kmod-2.1.14-FreeBSD_gd99134= be8

on = 14-stable:
zfs-2.2.6-FreeBSD_g33174af15
zfs-kmod-2.2.6-FreeBSD_g3317= 4af15

on = 15-current:
zfs-2.3.99-31-FreeBSD_gb2f6de7b5
zfs-kmod-2.3.99-31-Free= BSD_gb2f6de7b5


$ freebsd-version -ku
14.0-RELEASE-p10
14.0-RELEASE-p10

dunno = how to present in the above fashion? Binary install, not from = source.

Is the encryption you're using is the = GELI
based whole-disk one (which has been around iirc for a few = years) or
the relatively recent zfs encryption that works = per-filesystem. This is why I'm asking the zfs version. I thought the = latter is relatively quite new, and I've never heard of it working on = 14.0.

The = latter, zfs per-filesystem zfs encryption. I haven't heard any horror = stories about it, albeit it's quite fresh code.


The way I'd go about trying to address the issue = would be to start from the lowest layer and work up. On a raidz2 array = (the card in JBOD mode), 

yes, JBOD

I can identify all the drives as da0-7, so can use smartctl -x to = query them all directly, looking for, in = particular:

Reallocated_Sector_Ct
Reported_Uncorrect
Current_= Pending_Sector
Offline_Uncorrectable

If your disk arrangement = is connected via something like HP Smart Array
or similar, you'll = need to look at the man page for smartctl for the exact
syntax to = query the disks behind the card.

Na, it's JBOD alright. I see nothing = weird with the disks.

Palle= --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973--