Date: Fri, 8 Nov 2024 23:11:23 +0100 From: Palle Girgensohn <girgen@FreeBSD.org> To: void <void@f-m.fm> Cc: freebsd-fs@freebsd.org Subject: Re: zfs snapshot corruption when using encryption Message-ID: <F328561D-AD0A-475D-8E67-9DDD93468301@FreeBSD.org> In-Reply-To: <Zy5kmpL_8dJh0AGZ@int21h> References: <03E4CCF5-0F9A-4B0E-A9DA-81C7C677860C@FreeBSD.org> <Zy4XHlKodLu7utBa@int21h> <3E85AAAE-8B1E-47C7-B581-E3D98AB03907@FreeBSD.org> <Zy5kmpL_8dJh0AGZ@int21h>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > 8 nov. 2024 kl. 20:20 skrev void <void@f-m.fm>: >=20 > On Fri, Nov 08, 2024 at 03:29:22PM +0100, Palle Girgensohn wrote: >=20 >> I cannot run `zfs send -I fs@previous_snap fs@problematic_snap`, I = get >> warning: cannot send 's@problematic_snap': Input/output error` >> Removing the snapshot fixes the problem. >=20 > What an odd problem. Like, it can write but not read. What zfs = version? > On 13.3 I'm using: > zfs-2.1.14-FreeBSD_gd99134be8 > zfs-kmod-2.1.14-FreeBSD_gd99134be8 >=20 > on 14-stable: > zfs-2.2.6-FreeBSD_g33174af15 > zfs-kmod-2.2.6-FreeBSD_g33174af15 >=20 > on 15-current: > zfs-2.3.99-31-FreeBSD_gb2f6de7b5 > zfs-kmod-2.3.99-31-FreeBSD_gb2f6de7b5 >=20 $ freebsd-version -ku 14.0-RELEASE-p10 14.0-RELEASE-p10 dunno how to present in the above fashion? Binary install, not from = source. > Is the encryption you're using is the GELI > based whole-disk one (which has been around iirc for a few years) or > the relatively recent zfs encryption that works per-filesystem. This = is why I'm asking the zfs version. I thought the latter is relatively = quite new, and I've never heard of it working on 14.0. The latter, zfs per-filesystem zfs encryption. I haven't heard any = horror stories about it, albeit it's quite fresh code. > The way I'd go about trying to address the issue would be to start = from the lowest layer and work up. On a raidz2 array (the card in JBOD = mode),=20 yes, JBOD > I can identify all the drives as da0-7, so can use smartctl -x to = query them all directly, looking for, in particular: >=20 > Reallocated_Sector_Ct > Reported_Uncorrect > Current_Pending_Sector > Offline_Uncorrectable >=20 > If your disk arrangement is connected via something like HP Smart = Array > or similar, you'll need to look at the man page for smartctl for the = exact > syntax to query the disks behind the card. Na, it's JBOD alright. I see nothing weird with the disks. Palle= --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta charset=3D"UTF-8"><meta http-equiv=3D"content-type" = content=3D"text/html; charset=3Dus-ascii"></head><body = style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;"><br style=3D"caret-color: rgb(0, 0, 0); = color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); = font-family: Menlo-Regular; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote = type=3D"cite" style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;">8 nov. 2024 kl. 20:20 skrev void <void@f-m.fm>:<br><br>On = Fri, Nov 08, 2024 at 03:29:22PM +0100, Palle Girgensohn = wrote:<br><br><blockquote type=3D"cite">I cannot run `zfs send -I = fs@previous_snap fs@problematic_snap`, I get<br>warning: cannot send = 's@problematic_snap': Input/output error`<br>Removing the snapshot fixes = the problem.<br></blockquote><br>What an odd problem. Like, it can write = but not read. What zfs version?<br>On 13.3 I'm = using:<br>zfs-2.1.14-FreeBSD_gd99134be8<br>zfs-kmod-2.1.14-FreeBSD_gd99134= be8<br><br>on = 14-stable:<br>zfs-2.2.6-FreeBSD_g33174af15<br>zfs-kmod-2.2.6-FreeBSD_g3317= 4af15<br><br>on = 15-current:<br>zfs-2.3.99-31-FreeBSD_gb2f6de7b5<br>zfs-kmod-2.3.99-31-Free= BSD_gb2f6de7b5<br><br></blockquote><br style=3D"caret-color: rgb(0, 0, = 0); color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); = font-family: Menlo-Regular; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; display: inline = !important; float: none;">$ freebsd-version -ku</span><br = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: = inline !important; float: none;">14.0-RELEASE-p10</span><br = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: = inline !important; float: none;">14.0-RELEASE-p10</span><br = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; display: inline !important; float: none;">dunno = how to present in the above fashion? Binary install, not from = source.</span><br style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0); font-family: Menlo-Regular; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><br = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><blockquote type=3D"cite" style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;">Is the encryption you're using is the = GELI<br>based whole-disk one (which has been around iirc for a few = years) or<br>the relatively recent zfs encryption that works = per-filesystem. This is why I'm asking the zfs version. I thought the = latter is relatively quite new, and I've never heard of it working on = 14.0.<br></blockquote><br style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; display: inline !important; float: none;">The = latter, zfs per-filesystem zfs encryption. I haven't heard any horror = stories about it, albeit it's quite fresh code.</span><br = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><blockquote type=3D"cite" style=3D"font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;">The way I'd go about trying to address the issue = would be to start from the lowest layer and work up. On a raidz2 array = (the card in JBOD mode),<span = class=3D"Apple-converted-space"> </span><br></blockquote><br = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: = inline !important; float: none;">yes, JBOD</span><br style=3D"caret-color:= rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 13px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><blockquote = type=3D"cite" style=3D"font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;">I can identify all the drives as da0-7, so can use smartctl -x to = query them all directly, looking for, in = particular:<br><br>Reallocated_Sector_Ct<br>Reported_Uncorrect<br>Current_= Pending_Sector<br>Offline_Uncorrectable<br><br>If your disk arrangement = is connected via something like HP Smart Array<br>or similar, you'll = need to look at the man page for smartctl for the exact<br>syntax to = query the disks behind the card.<br></blockquote><br style=3D"caret-color:= rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Menlo-Regular; = font-size: 13px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); color: = rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; orphans: auto; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; widows: auto; word-spacing: = 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: = inline !important; float: none;">Na, it's JBOD alright. I see nothing = weird with the disks.</span><br style=3D"caret-color: rgb(0, 0, 0); = color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 13px; = font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; orphans: auto; text-align: start; text-indent: = 0px; text-transform: none; white-space: normal; widows: auto; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><br style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); = font-family: Menlo-Regular; font-size: 13px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = orphans: auto; text-align: start; text-indent: 0px; text-transform: = none; white-space: normal; widows: auto; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: = Menlo-Regular; font-size: 13px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; display: inline !important; float: = none;">Palle</span></body></html>= --Apple-Mail=_59E36AA6-CDF3-40DA-93B9-25F2EB7D1973--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F328561D-AD0A-475D-8E67-9DDD93468301>