From nobody Sun Jan 7 21:06:26 2024 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4T7V9r0cxzz55yC1 for ; Sun, 7 Jan 2024 21:06:40 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x634.google.com (mail-ej1-x634.google.com [IPv6:2a00:1450:4864:20::634]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4T7V9q4pBmz43Dg for ; Sun, 7 Jan 2024 21:06:39 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x634.google.com with SMTP id a640c23a62f3a-a293f2280c7so126763066b.1 for ; Sun, 07 Jan 2024 13:06:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1704661598; x=1705266398; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=r40KQBPRA3q4Kx8Augm8TyuFOoZXmCoz+NYIaTV/oq8=; b=IjvvrIqAraXbiMl5TkgSsiewdSV0Whb61keKgb3cVfBjuwf3qzhy1vPlacux+pDWFt 3MAiKKIwNbAo7roCkiRtCKdAQtKJ3PBz9sg1ojyrcH+gVpbJfTRIhw+5FL04AyXkOfM2 H8gAsGAxGOqTAQa2nDwkR7C/nAgQ8EmLWxPtialnAS5iuLoN/InVP00JDxW34feP1iMg Ue5XPxAmC9F233TAvrl7knCe6McUu67K9uoBwUTFedJIl5A7RBjMuqtz8DNWOvyjfuDA I285oviGP+Nvjnf5B3w4r+xmgjWZgP4LePMd4mgv7Ow50ngxWtxzN/QmzT+eguM/o0Fo 2//Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704661598; x=1705266398; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=r40KQBPRA3q4Kx8Augm8TyuFOoZXmCoz+NYIaTV/oq8=; b=XWoBKbrWQAwuJqi9T7o2YMJCkvIwHFdvI6CrhUJf8eY6t6UPm/hnAUTdMg10tMSW11 FbOoCf6ED5vrWWD1SNQY1Fwg9NZ9mMFfE3XKb3EEvI0AMxok/G9CtNV45UjRxBxtdtn6 DkKuo9PQmBO78TZwDUK2qtOYTwBjT9Drqww/WdAmy22Xl3sMNP6Mw9wKzlpCns2OGLlk PiU9JYuUYyA9S3hdWpxf3WUfwNcsyEza3h5LYhhwORxj8pvaDwHHcaw/rd+fImtHd/AW u1F6c7U9YZCxtvZoHtOG/p17NPnu5VvueODqWGJIv1iJFz6UWLwD8rah8i+6oMxZEWjd PcyA== X-Gm-Message-State: AOJu0YwIa1pnfZdcWnZQ9e46oGCP2y45AiXFZy/VLqbf8d1axrhH3dPc lsPXDjEXTYsM77aXcgk9xovrtXlvdmz2nyz4D9vEgVhY57kZEA== X-Google-Smtp-Source: AGHT+IFokWdN5h8upavvIYSj7qNcUk4QfxU2+Z0cR/bheYt68OSZCktpnmYeZyYss2dJtypP1Irr5HvXyNDdzTAtYlg= X-Received: by 2002:a17:906:310b:b0:a28:1916:6cc9 with SMTP id 11-20020a170906310b00b00a2819166cc9mr423605ejx.270.1704661597664; Sun, 07 Jan 2024 13:06:37 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <065f4f5c-f38b-45f4-b7e7-5248f871f7e6@FreeBSD.org> In-Reply-To: From: Warner Losh Date: Sun, 7 Jan 2024 14:06:26 -0700 Message-ID: Subject: Re: FreeBSD 13.2-STABLE can not boot from damaged mirror AND pool stuck in "resilver" state even without new devices. To: Miroslav Lachman <000.fbsd@quip.cz> Cc: lev@freebsd.org, freebsd-fs , freebsd-stable Content-Type: multipart/alternative; boundary="000000000000fdcd44060e61747d" X-Rspamd-Queue-Id: 4T7V9q4pBmz43Dg X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] --000000000000fdcd44060e61747d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, Jan 7, 2024 at 12:01=E2=80=AFPM Miroslav Lachman <000.fbsd@quip.cz>= wrote: > On 07/01/2024 19:34, Warner Losh wrote: > > > < 4294967296 sectors should be good. So these drives shouldn't see this > > problem. the BIOS interfaces should have no trouble here. > > [...] > > > Yes. If the drives are > 2TB you lose. BIOS is not for you... Unless > > you make special partitions that are in the first 2TB of the drive and > > only boot off of those. Also, if the drives are 4k, you likely lose, > > though it's hit or miss. Those are the hard limits of the BIOS ABI. > > It is not always that simple math. As I wrote in my previous reply, my > pool was unbootable in one machine but boots fine in the other. Both > were Intel based amd64 with BIOS, not EFI. I think there are some buggy > BIOSes where it cannot boot even on smaller pools than 2TB. (or maybe > some improved BIOSes supporting larger boundaries than 2TB? I don't know > in what exact position bootloader / kernel was on my 4TB pool) > OK. If the problem is that int13 has only 32-bits in the ABI, the math is that simple. The limit is 2^32 blocks, and there's no reliable provision for 4k sector sizes (there's some BIOSes that will do it, others that won't... it's a bit muddled looking at the problem reports, though we do try to support that). There's no BIOS64 implementation that extends the int13 interfaces to do wider block sizes that I've seen... It's just that it's so close it's easy to gravitate to a known issue... If other weird things are happening, then that means that we may have a typ= e problem that's truncating the logical block size (which the BIOS doesn't care about) to 32-bit (or maybe sometimes) which then leads to weird things happening. But... UEFI should suffer this same problem and we should hear about it a lot I'd think (though maybe how gptzfsboot is compiled might be the culprit, since that's the only thing that's confined to the gpt boot blocks that's not common binary code (we #include the implementation to make two different binary things....)). It shouldn't care that the copy of /boot/loader is past the 2TB logical limit, because the drives are smaller than 2TB and so none of their LBAs will be > 2^32 and should all work. If that's indeed the issue, then there's something weird about how we build it for gptzfsloader. The other thing it could be, though, is that if there's a resilvering, there's some subtle state that's confusing the simple reimplementation of ZFS reading that's in the boot loader. Though I'd expect to have heard about that before now. Especially since this would hit UEFI booting as well. Warner --000000000000fdcd44060e61747d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sun, Jan 7, 2024 at 12:01=E2=80=AF= PM Miroslav Lachman <000.fbsd@quip.c= z> wrote:
On 07/01/2024 19:34, Warner Losh wrote:

> <=C2=A04294967296 sectors should be good. So these drives shouldn&#= 39;t see this
> problem. the BIOS interfaces should have no trouble here.

[...]

> Yes. If the drives are > 2TB you lose. BIOS is not for you...=C2=A0= Unless
> you make special partitions that are in the first 2TB of the drive and=
> only boot off of those. Also, if the drives are 4k, you likely lose, <= br> > though it's hit or miss. Those are the hard limits of the BIOS ABI= .

It is not always that simple math. As I wrote in my previous reply, my
pool was unbootable in one machine but boots fine in the other. Both
were Intel based amd64 with BIOS, not EFI. I think there are some buggy BIOSes where it cannot boot even on smaller pools than 2TB. (or maybe
some improved BIOSes supporting larger boundaries than 2TB? I don't kno= w
in what exact position bootloader / kernel was on my 4TB pool)

OK. If the problem is that int13 has only 32-bits i= n the ABI, the math is that simple.
The limit is 2^32 blocks, and= there's no reliable provision for 4k sector sizes (there's
some BIOSes that will do it, others that won't... it's a bit mud= dled looking at the problem
reports, though we do try to support = that). There's no BIOS64 implementation that
extends the int1= 3 interfaces to do wider block sizes that I've seen... It's just th= at it's
so close it's easy to gravitate to a known issue.= ..

If other weird things are happening, then that = means that we may have a type
problem that's truncating the l= ogical block size (which the BIOS doesn't care
about) to 32-b= it (or maybe sometimes) which then leads to weird things happening.
But... UEFI should suffer this same problem and we should hear about it = a lot
I'd think (though maybe how gptzfsboot is compiled migh= t be the culprit, since
that's the only thing that's conf= ined to the gpt boot blocks that's not common
binary code (we= #include the implementation to make two different binary
things.= ...)). It shouldn't care that the copy of /boot/loader is past the 2TB<= /div>
logical limit, because the drives are smaller than 2TB and so non= e of their
LBAs will be > 2^32 and should all work. If that= 9;s indeed the issue, then there's
something weird about how = we build it for gptzfsloader.

The other thing it c= ould be, though, is that if there's a resilvering, there's some
subtle state that's confusing the simple reimplementation of ZFS= reading that's
in the boot loader. Though I'd expect to = have heard about that before now. Especially
since this would hit= UEFI booting as well.

Warner

=
--000000000000fdcd44060e61747d--