Date: Sun, 7 Jan 2024 11:34:06 -0700 From: Warner Losh <imp@bsdimp.com> To: lev@freebsd.org Cc: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs <freebsd-fs@freebsd.org>, freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. Message-ID: <CANCZdfrYCk7%2B6wCALvszmNZOcZeDxxNp%2Bk5PyH%2BTGJZ%2BovsU=Q@mail.gmail.com> In-Reply-To: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> References: <f97d80ee-0b01-4d68-beb5-53e905f0404c@FreeBSD.org> <e74464be-09b6-43e2-9365-7b0271b2d6eb@FreeBSD.org> <cc136316-f285-41bd-8d59-c5adce06e277@quip.cz> <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000036f5f2060e5f548d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Jan 7, 2024 at 10:57=E2=80=AFAM Lev Serebryakov <lev@freebsd.org> w= rote: > On 07.01.2024 16:38, Miroslav Lachman wrote: > > >>> ZFS: i/o error - all block copies unavailable > >>> ZFS: can't read MOS of pool zroot > >>> > >>> after that. > >> I've re-created pool from scratch > >> > >> zpool create znewroot ada0p3 && zfs send zroot | zfs receive znewroo= t > && zpool destroy zroot && zpool attach znewroot ada0p3 ada1p3 > >> > >> but gptzfsboot still can not boot from it with same diagnostics :-( > I must have missed it. What were the diagnostics? > > How large are your disks in a question? > 2TB > > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: <HGST HUS726020ALE610 APGNTD05> ACS-2 ATA SATA 3.x device > ada0: Serial Number K5HPZZLD > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada0: Command Queueing enabled > ada0: 1907729MB (3907029168 512 byte sectors) > ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 > ada1: <WDC WD2000FYYZ-01UL1B1 01.01K02> ATA8-ACS SATA 3.x device > ada1: Serial Number WD-WMC1P0504169 > ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada1: Command Queueing enabled > ada1: 1907729MB (3907029168 512 byte sectors) > < 4294967296 sectors should be good. So these drives shouldn't see this problem. the BIOS interfaces should have no trouble here. > > As far as I search the internet it is caused by the boot code (later > stage which is in a file in /boot directory) was moved too far from the > beginning of the disk and some old BIOS cannot allow the system to contin= ue > booting. > Oh, it is good hypothesis. It is Haswell-time MSI board (old Hetzner > EX40 instance)... > Yes. If the drives are > 2TB you lose. BIOS is not for you... Unless you make special partitions that are in the first 2TB of the drive and only boot off of those. Also, if the drives are 4k, you likely lose, though it's hit or miss. Those are the hard limits of the BIOS ABI. > It can also be avoided if your machine supports EFI boot, but my HP > Microserver Gen 8 does not support it. > I'll try to switch to EFI, but it needs some luck to get to BIOS with > provided KVM, it is very unstable :-) > BIOS booting is dying. It will be unsupportable in not too many more years and the code removed. The rapid proliferation of ZFS crypto and compression types is hastening the race to see who can use up the most space in the boot loader. We can do marginal things to make it better wrt the 640k limit, sure, but then we hit other limits like the 2TB address space, like not being able to reliably support 4k drives, etc. BIOS booting likely will support an increasingly small subset of all possible booting methods as we go forward. The current crazy mix of different alternative firmwares makes it hard to know what will survive, but as we hit these limitations, it will make it harder and harder to configure, deploy and manage these systems. The Linux on ZFS root pages, btw, recommend having two pools on two partitions on the disk. One that's a few GB that's the bool that has the kernel in it, and the other, rest of the disk, that's rpool for the root pool. If people want to continue to support BIOS booting (or rather, booting using the CSM interfaces), then somebody is going to need to step up to the plate and implement a similar option in bsdinstall, bectl, freebsd-update, etc. Warner --00000000000036f5f2060e5f548d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Sun, Jan 7, 2024 at 10:57=E2=80=AF= AM Lev Serebryakov <<a href=3D"mailto:lev@freebsd.org">lev@freebsd.org</= a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0p= x 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On= 07.01.2024 16:38, Miroslav Lachman wrote:<br> <br> >>> ZFS: i/o error - all block copies unavailable<br> >>> ZFS: can't read MOS of pool zroot<br> >>><br> >>> =C2=A0=C2=A0=C2=A0 after that.<br> >> =C2=A0=C2=A0I've re-created pool from scratch<br> >><br> >> =C2=A0=C2=A0zpool create znewroot ada0p3 && zfs send zroot= | zfs receive znewroot && zpool destroy zroot && zpool att= ach znewroot ada0p3 ada1p3<br> >><br> >> =C2=A0=C2=A0but gptzfsboot still can not boot from it with same di= agnostics :-(<br></blockquote><div><br></div><div>I must have missed it. Wh= at were the diagnostics?</div><div>=C2=A0</div><blockquote class=3D"gmail_q= uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex"> > How large are your disks in a question?<br> =C2=A0 =C2=A02TB<br> <br> ada0 at ahcich0 bus 0 scbus0 target 0 lun 0<br> ada0: <HGST HUS726020ALE610 APGNTD05> ACS-2 ATA SATA 3.x device<br> ada0: Serial Number K5HPZZLD<br> ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)<br> ada0: Command Queueing enabled<br> ada0: 1907729MB (3907029168 512 byte sectors)<br> ada1 at ahcich1 bus 0 scbus1 target 0 lun 0<br> ada1: <WDC WD2000FYYZ-01UL1B1 01.01K02> ATA8-ACS SATA 3.x device<br> ada1: Serial Number WD-WMC1P0504169<br> ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)<br> ada1: Command Queueing enabled<br> ada1: 1907729MB (3907029168 512 byte sectors)<br></blockquote><div><br></di= v><div><=C2=A04294967296 sectors should be good. So these drives shouldn= 't see this problem. the BIOS interfaces should have no trouble here.</= div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> > As far as I search the internet it is caused by the boot code (later s= tage which is in a file in /boot directory) was moved too far from the begi= nning of the disk and some old BIOS cannot allow the system to continue boo= ting.<br> =C2=A0 =C2=A0Oh, it is good hypothesis. It is Haswell-time MSI board (old H= etzner EX40 instance)...<br></blockquote><div><br></div><div>Yes. If the dr= ives are > 2TB you lose. BIOS is not for you...=C2=A0 Unless you make sp= ecial partitions that are in the first 2TB of the drive and only boot off o= f those. Also, if the drives are 4k, you likely lose, though it's hit o= r miss. Those are the hard limits of the BIOS ABI.</div><div><br></div><blo= ckquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left= :1px solid rgb(204,204,204);padding-left:1ex">> It can also be avoided i= f your machine supports EFI boot, but my HP Microserver Gen 8 does not supp= ort it.<br> =C2=A0 =C2=A0I'll try to switch to EFI, but it needs some luck to get t= o BIOS with provided KVM, it is very unstable :-)<br></blockquote><div><br>= </div><div>BIOS booting is dying. It will be unsupportable in not too many = more years and the code removed. The rapid proliferation of ZFS crypto and = compression types is hastening the race to see who can use up the most spac= e in the boot loader. We can do marginal things to make it better wrt the 6= 40k limit, sure, but then we hit other limits like the 2TB address space, l= ike not being able to reliably support 4k drives, etc. BIOS booting likely = will support an increasingly small subset of all possible booting methods a= s we go forward. The current crazy mix of different alternative firmwares m= akes it hard to know what will survive, but as we hit these limitations, it= will make it harder and harder to configure, deploy and manage these syste= ms.</div><div><br></div><div>The Linux on ZFS root pages, btw, recommend ha= ving two pools on two partitions on the disk. One that's a few GB that&= #39;s the bool that has the kernel in it, and the other, rest of the disk, = that's rpool for the root pool. If people want to continue to support B= IOS booting (or rather, booting using the CSM interfaces), then somebody is= going to need to step up to the plate and implement a similar option in bs= dinstall, bectl, freebsd-update, etc.</div><div><br></div><div>Warner<br></= div></div></div> --00000000000036f5f2060e5f548d--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfrYCk7%2B6wCALvszmNZOcZeDxxNp%2Bk5PyH%2BTGJZ%2BovsU=Q>