From nobody Thu Apr 28 04:37:51 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B3FC7199758B for ; Thu, 28 Apr 2022 04:38:04 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-ej1-x62a.google.com (mail-ej1-x62a.google.com [IPv6:2a00:1450:4864:20::62a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KpjXq6JY4z3nLH for ; Thu, 28 Apr 2022 04:38:03 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-ej1-x62a.google.com with SMTP id y3so7110606ejo.12 for ; Wed, 27 Apr 2022 21:38:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=+mknO/pbyw6Yxlz1P65Y8CyaHIFK74hKbgkUzpQZ5Es=; b=GPrwn68HFTtDmbj2adpSUn8fLFKd1j5nS1Utwj4rUVU4M722xQ/xU6KQIMZWEYHEKy uofXV3R2yV49rP9pmCQYocPN+4orNyERtYqo6gVak8k9opvCmOtZ9KEO0CYSVhFin9gG Zff10NIGgtg1cePUN2fFEOu9p1QKUgT1HANjNXJ/OA8WOgixFqo47qXCHB2GyvuqhPqK hMiSlU0a9dHAY1jpPoPxNpn2A8zB5v7rn77nV4ScsAUPZwQ/7w/GvH0/OGA3GnyNuZbh VfNcn8TUhh7tCdmQb1GwQuwNAkppVaifKGpNxTUNqsKhMDHu3p2Y8jI3S274K7gKVic1 whJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=+mknO/pbyw6Yxlz1P65Y8CyaHIFK74hKbgkUzpQZ5Es=; b=ElfYEEbgBckTkUQHLUk5KrF92N3Ll2oHEyhqDtoITmueo5AWAjiewp1v9/MCAj81Cs Bbi2TVvuU0FJQyYP39+y4UiWppP1+KkQ7Y/LHaAjnyVH1UX5mthb77cofmF01yik7oJE fGjrTdxaVf93dt209JXEqRyTSZq83m93ewAOfM83DVkOPvLVQa3PVte9r5lPa2Ohx+Pq 9ROnk6mWUHSCZayEd3qqt1O1rSZ020+MMLfe6OKJ0dU1uk0yQmq8BMQxH80qX/u2hC8w YKI/iZX8ZnZXtoifjWQLocAhpn0ALRpYNJFeqA3zFpRCzZKxsuyvpS5PEAYITMohbuIE h+Ng== X-Gm-Message-State: AOAM530qZA/vE7QRRQTDYFnJsql9XL83C2jM3DnUn2XSiacKXx2udVSM ZZWWEMC5T4HZhv2eYdncShLOhl3hw3EjpAzi4Ff+xEY= X-Google-Smtp-Source: ABdhPJzWS1PfS4KJkDGaYyno9ifNOt1vlprt1IaqhlETVHv/J6OK7OtRLrX9TG3XqbRcAtF2nMxxlPgKzpAdjxenYd0= X-Received: by 2002:a17:907:168a:b0:6da:9177:9fdd with SMTP id hc10-20020a170907168a00b006da91779fddmr29963850ejc.757.1651120682722; Wed, 27 Apr 2022 21:38:02 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <2c2d774c-3677-c3b7-d099-71318743a434@zhegan.in> <20220427214502.f9ef1317f2911fb5247d1757@dec.sakura.ne.jp> <5914a66877e4799a405901b7c2c16505@lerctr.org> In-Reply-To: From: Zaphod Beeblebrox Date: Thu, 28 Apr 2022 00:37:51 -0400 Message-ID: Subject: Re: FreeBSD boot from root pool larger than 1Tb/cannot read MOS/all block copies unavailable To: Warner Losh Cc: Larry Rosenman , Tomoaki AOKI , "Eugene M. Zheganin" , FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="000000000000c6694e05ddaf7d39" X-Rspamd-Queue-Id: 4KpjXq6JY4z3nLH X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=GPrwn68H; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2a00:1450:4864:20::62a as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-3.18 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62a:from]; MLMMJ_DEST(0.00)[freebsd-stable]; NEURAL_HAM_SHORT(-0.18)[-0.176]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --000000000000c6694e05ddaf7d39 Content-Type: text/plain; charset="UTF-8" The point I want to reiterate, however, is this error is tossed out when my setup fails and the answer is a full powerdown of the motherboard --- after which the disks are adequately described. On Wed, Apr 27, 2022 at 9:26 AM Warner Losh wrote: > > > On Wed, Apr 27, 2022 at 7:05 AM Larry Rosenman wrote: > >> On 04/27/2022 7:45 am, Tomoaki AOKI wrote: >> [snip] >> > I've booting with around 1.5T or above (but under 2T) Root-on-ZFS >> > installation via UEFI for years. >> > >> > Basically, UEFI requires GPT partitioning. So many of those would boot >> > from larger than 1TB boot pool. >> > >> > As, IIRC, it is optional, some UEFI firmware would be able to boot from >> > MBR-patritioned disk but there would be some risk that they have the >> > same limitations as legacy BIOS in such situations. >> > >> > Note that possibly I'm just lucky enough not to be bitten. >> >> I've been booting UEFI from 10T pools for literally years, and I have a >> HP server >> with a 5T pool that IIRC boots GPT with BIOS, and a different Dell that >> has a 6T >> pool also GPT and UEFI. >> > > Both bsdlabel and mbr have a 32-bit block count, which means for 512-byte > blocks, the largest they can describe is 2TB. That's likely the root cause > of this issue. The BIOS can't read beyond that limit on the disks. It's > hit or > miss what BIOSes do with 4k drives, though most of the later ones before > uefi was well established coped well enough. > > This isn't to say that the old code in FreeBSD 12's boot loader didn't have > other issues with LBAs > 32-bits for ZFS. And for MBR ZFS booting the > update of the boot blocks is, at best, super weird. If you've not done > that, > you'll continue to have the older boot loader, even if the latest just > works. > I've never encountered this problem, so I can't offer better words of > advice, > though. All my ZFS booting started after I transitioned to UEFI booting. > > Warner > --000000000000c6694e05ddaf7d39 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
The point I want to reiterate, however, is this error is t= ossed out when my setup fails and the answer is a full powerdown of the mot= herboard --- after which the disks are adequately described.

<= div class=3D"gmail_quote">
On Wed, Apr= 27, 2022 at 9:26 AM Warner Losh <imp@= bsdimp.com> wrote:


On Wed, Apr 27, 2022 at 7:= 05 AM Larry Rosenman <ler@lerctr.org> wrote:
On 04/27/2022 7:45 am, Tomoaki AOKI wrote:
[snip]
> I've booting with around 1.5T or above (but under 2T) Root-on-ZFS<= br> > installation via UEFI for years.
>
> Basically, UEFI requires GPT partitioning. So many of those would boot=
> from larger than 1TB boot pool.
>
> As, IIRC, it is optional, some UEFI firmware would be able to boot fro= m
> MBR-patritioned disk but there would be some risk that they have the > same limitations as legacy BIOS in such situations.
>
> Note that possibly I'm just lucky enough not to be bitten.

I've been booting UEFI from 10T pools for literally years, and I have a=
HP server
with a 5T pool that IIRC boots GPT with BIOS, and a different Dell that has a 6T
pool also GPT and UEFI.

Both bsdlabel a= nd mbr have a 32-bit block count, which means for 512-byte
blocks= , the largest they can describe is 2TB. That's likely the root cause
of this issue. The BIOS can't read beyond that limit on the dis= ks. It's hit or
miss what BIOSes do with 4k drives, though mo= st of the later ones before
uefi was well established coped well = enough.

This isn't to say that the old code in= FreeBSD 12's boot loader didn't have
other issues with L= BAs > 32-bits for ZFS. And for MBR ZFS booting the
update of t= he boot blocks is, at best, super weird. If you've not done that,
=
you'll continue to have the older boot loader, even if the latest = just works.
I've never encountered this problem, so I can'= ;t offer better words of advice,
though. All my ZFS booting start= ed after I transitioned to UEFI booting.

Warner
--000000000000c6694e05ddaf7d39--