Date: Sun, 7 Jan 2024 14:06:26 -0700 From: Warner Losh <imp@bsdimp.com> To: Miroslav Lachman <000.fbsd@quip.cz> Cc: lev@freebsd.org, freebsd-fs <freebsd-fs@freebsd.org>, freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. Message-ID: <CANCZdfo=jEwJh_%2BX2uxPHOYobv%2BHHzNfeLTaOS2%2BDA3pF_MnYg@mail.gmail.com> In-Reply-To: <a137529e-fd63-41a2-9ab7-080013f4cb3e@quip.cz> References: <f97d80ee-0b01-4d68-beb5-53e905f0404c@FreeBSD.org> <e74464be-09b6-43e2-9365-7b0271b2d6eb@FreeBSD.org> <cc136316-f285-41bd-8d59-c5adce06e277@quip.cz> <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> <CANCZdfrYCk7%2B6wCALvszmNZOcZeDxxNp%2Bk5PyH%2BTGJZ%2BovsU=Q@mail.gmail.com> <a137529e-fd63-41a2-9ab7-080013f4cb3e@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000fdcd44060e61747d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Jan 7, 2024 at 12:01=E2=80=AFPM Miroslav Lachman <000.fbsd@quip.cz>= wrote: > On 07/01/2024 19:34, Warner Losh wrote: > > > < 4294967296 sectors should be good. So these drives shouldn't see this > > problem. the BIOS interfaces should have no trouble here. > > [...] > > > Yes. If the drives are > 2TB you lose. BIOS is not for you... Unless > > you make special partitions that are in the first 2TB of the drive and > > only boot off of those. Also, if the drives are 4k, you likely lose, > > though it's hit or miss. Those are the hard limits of the BIOS ABI. > > It is not always that simple math. As I wrote in my previous reply, my > pool was unbootable in one machine but boots fine in the other. Both > were Intel based amd64 with BIOS, not EFI. I think there are some buggy > BIOSes where it cannot boot even on smaller pools than 2TB. (or maybe > some improved BIOSes supporting larger boundaries than 2TB? I don't know > in what exact position bootloader / kernel was on my 4TB pool) > OK. If the problem is that int13 has only 32-bits in the ABI, the math is that simple. The limit is 2^32 blocks, and there's no reliable provision for 4k sector sizes (there's some BIOSes that will do it, others that won't... it's a bit muddled looking at the problem reports, though we do try to support that). There's no BIOS64 implementation that extends the int13 interfaces to do wider block sizes that I've seen... It's just that it's so close it's easy to gravitate to a known issue... If other weird things are happening, then that means that we may have a typ= e problem that's truncating the logical block size (which the BIOS doesn't care about) to 32-bit (or maybe sometimes) which then leads to weird things happening. But... UEFI should suffer this same problem and we should hear about it a lot I'd think (though maybe how gptzfsboot is compiled might be the culprit, since that's the only thing that's confined to the gpt boot blocks that's not common binary code (we #include the implementation to make two different binary things....)). It shouldn't care that the copy of /boot/loader is past the 2TB logical limit, because the drives are smaller than 2TB and so none of their LBAs will be > 2^32 and should all work. If that's indeed the issue, then there's something weird about how we build it for gptzfsloader. The other thing it could be, though, is that if there's a resilvering, there's some subtle state that's confusing the simple reimplementation of ZFS reading that's in the boot loader. Though I'd expect to have heard about that before now. Especially since this would hit UEFI booting as well. Warner --000000000000fdcd44060e61747d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Sun, Jan 7, 2024 at 12:01=E2=80=AF= PM Miroslav Lachman <<a href=3D"mailto:000.fbsd@quip.cz">000.fbsd@quip.c= z</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin= :0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"= >On 07/01/2024 19:34, Warner Losh wrote:<br> <br> > <=C2=A04294967296 sectors should be good. So these drives shouldn&#= 39;t see this <br> > problem. the BIOS interfaces should have no trouble here.<br> <br> [...]<br> <br> > Yes. If the drives are > 2TB you lose. BIOS is not for you...=C2=A0= Unless <br> > you make special partitions that are in the first 2TB of the drive and= <br> > only boot off of those. Also, if the drives are 4k, you likely lose, <= br> > though it's hit or miss. Those are the hard limits of the BIOS ABI= .<br> <br> It is not always that simple math. As I wrote in my previous reply, my <br> pool was unbootable in one machine but boots fine in the other. Both <br> were Intel based amd64 with BIOS, not EFI. I think there are some buggy <br= > BIOSes where it cannot boot even on smaller pools than 2TB. (or maybe <br> some improved BIOSes supporting larger boundaries than 2TB? I don't kno= w <br> in what exact position bootloader / kernel was on my 4TB pool)<br></blockqu= ote><div><br></div><div>OK. If the problem is that int13 has only 32-bits i= n the ABI, the math is that simple.</div><div>The limit is 2^32 blocks, and= there's no reliable provision for 4k sector sizes (there's</div><d= iv>some BIOSes that will do it, others that won't... it's a bit mud= dled looking at the problem</div><div>reports, though we do try to support = that). There's no BIOS64 implementation that</div><div>extends the int1= 3 interfaces to do wider block sizes that I've seen... It's just th= at it's</div><div>so close it's easy to gravitate to a known issue.= ..</div><div><br></div><div>If other weird things are happening, then that = means that we may have a type</div><div>problem that's truncating the l= ogical block size (which the BIOS doesn't care</div><div>about) to 32-b= it (or maybe sometimes) which then leads to weird things happening.</div><d= iv>But... UEFI should suffer this same problem and we should hear about it = a lot</div><div>I'd think (though maybe how gptzfsboot is compiled migh= t be the culprit, since</div><div>that's the only thing that's conf= ined to the gpt boot blocks that's not common</div><div>binary code (we= #include the implementation to make two different binary</div><div>things.= ...)). It shouldn't care that the copy of /boot/loader is past the 2TB<= /div><div>logical limit, because the drives are smaller than 2TB and so non= e of their</div><div>LBAs will be > 2^32 and should all work. If that= 9;s indeed the issue, then there's</div><div>something weird about how = we build it for gptzfsloader.</div><div><br></div><div>The other thing it c= ould be, though, is that if there's a resilvering, there's some</di= v><div>subtle state that's confusing the simple reimplementation of ZFS= reading that's</div><div>in the boot loader. Though I'd expect to = have heard about that before now. Especially</div><div>since this would hit= UEFI booting as well.</div><div><br></div><div>Warner</div><div><br></div>= </div></div> --000000000000fdcd44060e61747d--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfo=jEwJh_%2BX2uxPHOYobv%2BHHzNfeLTaOS2%2BDA3pF_MnYg>