Date: Mon, 20 Dec 2021 15:15:00 +0100 From: Fabian Keil <freebsd-listen@fabiankeil.de> To: freebsd-hackers@freebsd.org Subject: Re: Patches for GPT and geli recovery Message-ID: <20211220151500.5e57c1a6@fabiankeil.de> In-Reply-To: <67419422-5633-4e4b-870d-aec8762ec6a1@gmail.com> References: <20211219175011.3023a232@fabiankeil.de> <CAFPNf59bXZTEdYzSmM7qH5mwYSykRdXrpHUOqn-qiE9ND2d=xQ@mail.gmail.com> <67419422-5633-4e4b-870d-aec8762ec6a1@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Sig_/XQGc+8skOnRuNnHdVPt8VzF Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Jason Bacon <bacon4000@gmail.com> wrote on 2021-12-19 at 16:21:39: > On 12/19/21 13:40, Lee Brown wrote: > >=20 > >=20 > > On Sun, Dec 19, 2021 at 8:52 AM Fabian Keil=20 > > <freebsd-listen@fabiankeil.de <mailto:freebsd-listen@fabiankeil.de>> wr= ote: > >=20 > > [cut] > > BTW, I would also be interested to know if others have > > experienced similar data corruption and could figure > > out how it happened. > >=20 > > Sounds like bitrot.=C2=A0 Bits flip on disks all the time, it doesn't m= atter=20 > > if they are spinning rust or SSD, it happens.=C2=A0 Sometimes they are= =20 > > detected and corrected, in which case you won't know.=C2=A0 Sometimes t= hey=20 > > are detected and uncorrectable, you'll see that error propagated into=20 > > the driver.=C2=A0 And sometimes they are not detected at all and cause = no=20 > > errors that the OS can surmise.=C2=A0 The higher the density of bits, t= he=20 > > higher the probability of corruption.=C2=A0 SMART is not reliably=20 > > predictive.=C2=A0 How does it happen?=C2=A0 Cosmic rays and entropy.=C2= =A0 I've had=20 > > lighty written SSD's fail after a few months. > >=20 > > I don't use ZFS, but have GELI-Authentication under a GMIRROR, so=20 > > whenever a bad checksum is read, it breaks the mirror, which gets=20 > > attention (Iast I looked, there wasn't a simple userland hook for bad=20 > > GELI reads, but there was for GMIRROR add/remove events). =20 > How old was the corrupted filesystem? I just checked: fk@t520 /var/log/fk/2021-12-20 $grep "zpool create" *zpool-history* ssh-steffen-sudo-zpool-history--l-bpool-20211220T102957:2017-08-10.21:52:07= zpool create -f -o version=3D28 -O compression=3Dlzjb bpool /dev/ada0p2 [u= ser 0 (root) on kendra] ssh-steffen-sudo-zpool-history--l-dpool-20211220T103420:2015-03-17.18:46:42= zpool create dpool /dev/gpt/dpool-ada0.eli [user 0 (root) on kendra] ssh-steffen-sudo-zpool-history--l-rpool-ada1-20211220T103234:2017-04-11.12:= 33:47 zpool create -o version=3D28 -o failmode=3Dcontinue -O compression=3D= lzjb -O checksum=3Dsha256 rpool mirror /dev/ada0p3.eli /dev/da1p3.eli [user= 0 (root) on ElectroBSD-11.0-STABLE-amd64] sudo-zpool-history--l-cloudia2-20211220T103856:2017-04-12.14:45:07 zpool cr= eate -O recordsize=3D1m -O checksum=3Dsha512 cloudia2 /dev/label/cloudia2.e= li [user 0 (root) on t520.local] So it looks like the partially corrupted pool "dpool" on partition five was created on 2015-03-17 while the (former) root pool "rpool-ada1" which didn't show any signs of corruption was created on 2017-04-11 which indicates that I installed a new operating system with cloudiatr and kept the data pool unmodified. The boot pool "bpool" was created on 2017-08-10 but it gets recreated with each ElectroBSD kernel update anyway. > I habitually wipe my disks and do= =20 > a fresh install at least once every 2 years to avoid issues like this.=20 Do you read back the complete data after fresh installs to confirm that the rewritten data arrived on disk as expected? I prefer ZFS scrubs to confirm that the data is still reachable. It's not obvious to me that recreating the data is safer than keeping the old data but verifying checksums. > I have experienced unexplained, unrecoverable errors on old filesystems,= =20 > but fortunately nothing critical. I too have experienced various unrecoverable errors on disks but I never lost GPT partition data and geli meta data at the same time while most of the data on disk remained valid and without the disk reporting any problems. While the pools "dpool" and "cloudia2" contained a couple of corrupt blocks this could be completely unrelated to the corruption of the partition table and the geli meta data. > This to me serves as another reminder to maintain regular backups of=20 > important files and consider everything else expendable. Agreed. The problem disk mostly contained DVD rips and while some of them weren't available on other disks as well, they could be recreated by simply ripping the DVDs again. Of course it's conceivable that some of the source DVDs now contain corruption as well (I own many older DVDs that contain corrupt blocks), but I could probably buy them new or rent them if needed. I use zogftw for backups and my important data is backed up to multiple external pools and some of them are stored off-site. Fabian --Sig_/XQGc+8skOnRuNnHdVPt8VzF Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iF0EARECAB0WIQTKUNd6H/m3+ByGULIFiohV/3dUnQUCYcCP5QAKCRAFiohV/3dU nSZWAKCVKS3A32rTmr/6Ymq7QoQBiobNywCfUWgNungBQYNYqLdTGI77WTHKEw0= =JFP0 -----END PGP SIGNATURE----- --Sig_/XQGc+8skOnRuNnHdVPt8VzF--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20211220151500.5e57c1a6>