Date: Thu, 14 Dec 2023 14:35:07 -0800 From: Xin LI <delphij@gmail.com> To: Lexi Winter <lexi@le-fay.org> Cc: Pete Wright <pete@nomadlogic.org>, "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org> Subject: Re: unusual ZFS issue Message-ID: <CAGMYy3sXBd=BjcLS7xNeTTJkYgCq0k94N2c_ZJ3kAyOdx2JWSg@mail.gmail.com> In-Reply-To: <1656A349-9E7D-4798-89D1-ECE39126FBC8@le-fay.org> References: <787CB64A-1687-49C3-9063-2CE3B6F957EF@le-fay.org> <twivq3fadiotyfpblbqzfulx2wvel5povfic7626hktylqr4gg@uxipr5j7igs5> <C210159C-4747-45A1-9FB1-1708D6A21DE8@le-fay.org> <ec82ce34-6463-4200-960a-c820605d0cfd@nomadlogic.org> <CAGMYy3sp-6T0GYEBVG1WSkn4hzvQd15XuwD8bvx1qg1Qys%2BpCg@mail.gmail.com> <1656A349-9E7D-4798-89D1-ECE39126FBC8@le-fay.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000f44e6b060c7fe5bc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, Dec 14, 2023 at 2:29=E2=80=AFPM Lexi Winter <lexi@le-fay.org> wrote= : > On 14 Dec 2023, at 22:25, Xin LI <delphij@gmail.com> wrote: > > Try "zpool status -x" and see if it would show something useful? > > the output seems to be the same as =E2=80=98zpool status -v=E2=80=99: > > # zpool status -xv > pool: data > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A > scan: scrub in progress since Thu Dec 14 18:58:21 2023 > 11.5T / 18.8T scanned at 962M/s, 8.71T / 18.8T issued at 726M/s > 0B repaired, 46.41% done, 04:02:02 to go > config: > > NAME STATE READ WRITE CKSUM > data ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > da4p1 ONLINE 0 0 0 > da6p1 ONLINE 0 0 0 > da5p1 ONLINE 0 0 0 > da7p1 ONLINE 0 0 0 > da1p1 ONLINE 0 0 0 > da0p1 ONLINE 0 0 0 > da3p1 ONLINE 0 0 0 > da2p1 ONLINE 0 0 0 > logs > mirror-2 ONLINE 0 0 0 > ada0p4 ONLINE 0 0 0 > ada1p4 ONLINE 0 0 0 > cache > ada1p5 ONLINE 0 0 0 > ada0p5 ONLINE 0 0 0 > > errors: Permanent errors have been detected in the following files: > This is strange, I'd expect some non-zero values above; did you 'zpool clear' before this? Note that this is permanent damage (otherwise ZFS will automatically "heal" the pool by overwriting the bad data with good copies, and your applications shouldn't see these). You can delete these files or datasets and restore them from backup. If you are not using ECC memory, they may occasionally cause damage like this one if you are really unlucky, by the way. i think this is expected since -x just filters the output to show pools > with errors? > Yeah.... -x shows only pools that are not healthy. If you have only one pool and that's the pool you are seeing issues, the output should be identical. Cheers, --000000000000f44e6b060c7fe5bc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:monospace,monospace"><br></div></div><br><div class=3D"gmail_quote= "><div dir=3D"ltr" class=3D"gmail_attr">On Thu, Dec 14, 2023 at 2:29=E2=80= =AFPM Lexi Winter <<a href=3D"mailto:lexi@le-fay.org">lexi@le-fay.org</a= >> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On = 14 Dec 2023, at 22:25, Xin LI <<a href=3D"mailto:delphij@gmail.com" targ= et=3D"_blank">delphij@gmail.com</a>> wrote:<br> > Try "zpool status -x" and see if it would show something use= ful?<br> <br> the output seems to be the same as =E2=80=98zpool status -v=E2=80=99:<br> <br> # zpool status -xv<br> =C2=A0 pool: data<br> =C2=A0state: ONLINE<br> status: One or more devices has experienced an error resulting in data<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 corruption.=C2=A0 Applications may be affected.= <br> action: Restore the file in question if possible.=C2=A0 Otherwise restore t= he<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 entire pool from backup.<br> =C2=A0 =C2=A0see: <a href=3D"https://openzfs.github.io/openzfs-docs/msg/ZFS= -8000-8A" rel=3D"noreferrer" target=3D"_blank">https://openzfs.github.io/op= enzfs-docs/msg/ZFS-8000-8A</a><br> =C2=A0 scan: scrub in progress since Thu Dec 14 18:58:21 2023<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 11.5T / 18.8T scanned at 962M/s, 8.71T / 18.8T = issued at 726M/s<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 0B repaired, 46.41% done, 04:02:02 to go<br> config:<br> <br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 STATE=C2=A0 =C2= =A0 =C2=A0READ WRITE CKSUM<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 data=C2=A0 =C2=A0 =C2=A0 =C2=A0 ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 raidz2-0=C2=A0 ONLINE=C2=A0 =C2=A0 =C2= =A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da4p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da6p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da5p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da7p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da1p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da0p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da3p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 da2p1=C2=A0 =C2=A0ONLINE=C2=A0 = =C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 logs<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 mirror-2=C2=A0 ONLINE=C2=A0 =C2=A0 =C2= =A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada0p4=C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada1p4=C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 cache<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada1p5=C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ada0p5=C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 = =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00=C2=A0 =C2=A0 =C2=A00<br> <br> errors: Permanent errors have been detected in the following files:<br></bl= ockquote><div><br></div><div><div class=3D"gmail_default" style=3D"font-fam= ily:monospace,monospace">This is strange, I'd expect some non-zero valu= es above; did you 'zpool clear' before this?</div></div><div class= =3D"gmail_default" style=3D"font-family:monospace,monospace"><br></div><div= class=3D"gmail_default" style=3D"font-family:monospace,monospace">Note tha= t this is permanent damage (otherwise ZFS will automatically "heal&quo= t; the pool by overwriting the bad data with good copies, and your applicat= ions shouldn't see these).=C2=A0 You can delete these files or datasets= and restore them from backup.=C2=A0 If you are not using ECC memory, they = may occasionally cause damage like this one if you are really unlucky, by t= he way.</div><div><br></div><blockquote class=3D"gmail_quote" style=3D"marg= in:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1e= x"> i think this is expected since -x just filters the output to show pools wit= h errors?<br> </blockquote><div><br></div><div class=3D"gmail_default" style=3D"font-fami= ly:monospace,monospace">Yeah.... -x shows only pools that are not healthy.= =C2=A0 If you have only one pool and that's the pool you are seeing iss= ues, the output should be identical.</div><div class=3D"gmail_default" styl= e=3D"font-family:monospace,monospace"><br></div><div class=3D"gmail_default= " style=3D"font-family:monospace,monospace">Cheers,</div></div></div> --000000000000f44e6b060c7fe5bc--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGMYy3sXBd=BjcLS7xNeTTJkYgCq0k94N2c_ZJ3kAyOdx2JWSg>