From nobody Wed Apr 27 13:25:29 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 04AF81AB3D22 for ; Wed, 27 Apr 2022 13:25:42 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-vk1-xa2b.google.com (mail-vk1-xa2b.google.com [IPv6:2607:f8b0:4864:20::a2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KpKJ52rLFz4mpJ for ; Wed, 27 Apr 2022 13:25:41 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-vk1-xa2b.google.com with SMTP id bc42so881838vkb.12 for ; Wed, 27 Apr 2022 06:25:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=RhEJ82nc1UOU1r/n1qHUSLT2MicCudbq3sNzeR1kbHc=; b=EzlhDlqkKfevVy+bylTkzvv+XT35N3hZl0MKHzDjrRVPSfsIPW/0cCQb6pBZF5DFEi dTSzeHG6cHTagMgJRTTq0MiW6B3Ek4boVEkILz0QuYR1Ofd8+mOtBEOGdxeC0j45btu/ KPU3nBdqyySPwFMfrOooQV+3KiQuKGEUUsrb6J44IWtkjW7txMWoGTeyOsFXF+QrbtG5 dlGukzFXXIKPn0TL5H1mpd3DauRMKqIJ+VxvIxHlmCppqL5CHlHYdavvP80ikwGmRcvl Qp1UswDb+bDpernNUXDAocYyIxZH/BiK8xLaqTngntK+03hhsujTIfb92rIhaO815EDz K23Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=RhEJ82nc1UOU1r/n1qHUSLT2MicCudbq3sNzeR1kbHc=; b=fIqp/FCRYApQoe+MIBDVX6LtCBZo7kE4aOcpUtnzLMgcVtv4U6RVB8UN90IG3syq9r vg5ft/bI4Jh7Ta/Dd5zziqOchilZvfTNNw647+8IqvuUotRw90evlkUrmxqC7zMhXN1l Qy7gi+wiWqEpbvvwwF+wPPyA5bwfMU8mwk0g3g1sEl/GZJrGM7Le2deJU1ulfP4YbDrX GB3ObkcyigMTX84r+RvsFArxi3mM69m9/ssAzGwc/C/fd7M7Ub6zAo2xXQk6y4vuGoAH zER0+QLONaDkV11fpKbS9ilDWdUupJeh8r4OnKA1GH7n8EeykhGbZLLaJ0Y9ndWNfXeY ibsQ== X-Gm-Message-State: AOAM532Wh8XZUyh8A6wdF80IvT2tiMVPRlqsf3qMhrIyqlizd9H6bDvB /8N5PD7GY1DNt8EwZfjFqc3Ja/Fn/52q+osD1R174Dt4nBs= X-Google-Smtp-Source: ABdhPJzrZ024uEvcQJ2NV08c9Gejp1nhKCqgPYRqfdKszvbH7v8ubWjUYrvVF4u51qpk2PnvN5uNUrjrqGVBosON3Os= X-Received: by 2002:a05:6122:2229:b0:32d:1642:b58b with SMTP id bb41-20020a056122222900b0032d1642b58bmr8389365vkb.27.1651065940650; Wed, 27 Apr 2022 06:25:40 -0700 (PDT) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 References: <2c2d774c-3677-c3b7-d099-71318743a434@zhegan.in> <20220427214502.f9ef1317f2911fb5247d1757@dec.sakura.ne.jp> <5914a66877e4799a405901b7c2c16505@lerctr.org> In-Reply-To: <5914a66877e4799a405901b7c2c16505@lerctr.org> From: Warner Losh Date: Wed, 27 Apr 2022 07:25:29 -0600 Message-ID: Subject: Re: FreeBSD boot from root pool larger than 1Tb/cannot read MOS/all block copies unavailable To: Larry Rosenman Cc: Tomoaki AOKI , "Eugene M. Zheganin" , FreeBSD-STABLE Mailing List Content-Type: multipart/alternative; boundary="000000000000e4bf0305dda2be2b" X-Rspamd-Queue-Id: 4KpKJ52rLFz4mpJ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=EzlhDlqk; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::a2b) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-2.94 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.943]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::a2b:from]; MLMMJ_DEST(0.00)[freebsd-stable]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --000000000000e4bf0305dda2be2b Content-Type: text/plain; charset="UTF-8" On Wed, Apr 27, 2022 at 7:05 AM Larry Rosenman wrote: > On 04/27/2022 7:45 am, Tomoaki AOKI wrote: > [snip] > > I've booting with around 1.5T or above (but under 2T) Root-on-ZFS > > installation via UEFI for years. > > > > Basically, UEFI requires GPT partitioning. So many of those would boot > > from larger than 1TB boot pool. > > > > As, IIRC, it is optional, some UEFI firmware would be able to boot from > > MBR-patritioned disk but there would be some risk that they have the > > same limitations as legacy BIOS in such situations. > > > > Note that possibly I'm just lucky enough not to be bitten. > > I've been booting UEFI from 10T pools for literally years, and I have a > HP server > with a 5T pool that IIRC boots GPT with BIOS, and a different Dell that > has a 6T > pool also GPT and UEFI. > Both bsdlabel and mbr have a 32-bit block count, which means for 512-byte blocks, the largest they can describe is 2TB. That's likely the root cause of this issue. The BIOS can't read beyond that limit on the disks. It's hit or miss what BIOSes do with 4k drives, though most of the later ones before uefi was well established coped well enough. This isn't to say that the old code in FreeBSD 12's boot loader didn't have other issues with LBAs > 32-bits for ZFS. And for MBR ZFS booting the update of the boot blocks is, at best, super weird. If you've not done that, you'll continue to have the older boot loader, even if the latest just works. I've never encountered this problem, so I can't offer better words of advice, though. All my ZFS booting started after I transitioned to UEFI booting. Warner --000000000000e4bf0305dda2be2b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Apr 27, 2022 at 7:05 AM Larry= Rosenman <ler@lerctr.org> wrot= e:
On 04/27/2022= 7:45 am, Tomoaki AOKI wrote:
[snip]
> I've booting with around 1.5T or above (but under 2T) Root-on-ZFS<= br> > installation via UEFI for years.
>
> Basically, UEFI requires GPT partitioning. So many of those would boot=
> from larger than 1TB boot pool.
>
> As, IIRC, it is optional, some UEFI firmware would be able to boot fro= m
> MBR-patritioned disk but there would be some risk that they have the > same limitations as legacy BIOS in such situations.
>
> Note that possibly I'm just lucky enough not to be bitten.

I've been booting UEFI from 10T pools for literally years, and I have a=
HP server
with a 5T pool that IIRC boots GPT with BIOS, and a different Dell that has a 6T
pool also GPT and UEFI.

Both bsdlabel a= nd mbr have a 32-bit block count, which means for 512-byte
blocks= , the largest they can describe is 2TB. That's likely the root cause
of this issue. The BIOS can't read beyond that limit on the dis= ks. It's hit or
miss what BIOSes do with 4k drives, though mo= st of the later ones before
uefi was well established coped well = enough.

This isn't to say that the old code in= FreeBSD 12's boot loader didn't have
other issues with L= BAs > 32-bits for ZFS. And for MBR ZFS booting the
update of t= he boot blocks is, at best, super weird. If you've not done that,
=
you'll continue to have the older boot loader, even if the latest = just works.
I've never encountered this problem, so I can'= ;t offer better words of advice,
though. All my ZFS booting start= ed after I transitioned to UEFI booting.

Warner
--000000000000e4bf0305dda2be2b--