From nobody Wed Nov 23 02:01:50 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NH4BL20b1z4jDBZ for ; Wed, 23 Nov 2022 02:02:02 +0000 (UTC) (envelope-from ler@lerctr.org) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "*.lerctr.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NH4BK5JFCz3hCt; Wed, 23 Nov 2022 02:02:01 +0000 (UTC) (envelope-from ler@lerctr.org) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=ler2019; h=Content-Type:Message-ID:References:In-Reply-To:Subject:Cc:To: From:Date:MIME-Version:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=RhXu8pZIABzMi9CLz0YbHTqvnAJ9MhLVXRhSDyiGVOE=; b=E25VTLXF0vskCdXI+iYZrthEFk bUNrbib6O7qLLNMurAWtJj7+LIyY5C9yEEq0laJMN09/8Wx2bektgubkIjXYgLXL9pmAiwRpefPE2 3B1GbOhW5ltxB0yGqPkaw+9TFWID0zP7f2uwJa3PFHIPDB+ALxHQRMpDgwbMk2L+GWFz1J4STHU33 PfHNSpIDcDrR7Re2hPafYJvBfIHQQbbex3sJHFsqhyQge6hK0HF+7idhGtXvt1IHnSFC1pg932Fix 4ewNi2+5hB0b+IHL9jVPqfkhGEoiFgX+Yp3OXBzGhwCysz3E00RxEYG0DxEO7LiJQWL54E5WJfGwo rlInfngQ==; Received-SPF: pass (thebighonker.lerctr.org: domain of lerctr.org designates 2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4 as permitted sender) client-ip=2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4; envelope-from=ler@lerctr.org; helo=webmail.lerctr.org; Received: from thebighonker.lerctr.org ([2602:fcdb:0:10:7ae3:b5ff:fe1b:23b4]:48149 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLS1.3) tls TLS_AES_256_GCM_SHA384 (Exim 4.95 (FreeBSD)) (envelope-from ) id 1oxf5C-000IQ2-GX; Tue, 22 Nov 2022 20:01:50 -0600 Received: from 2600:1700:210:b18f:785a:29fa:3f63:a575 by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Tue, 22 Nov 2022 20:01:50 -0600 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 Date: Tue, 22 Nov 2022 20:01:50 -0600 From: Larry Rosenman To: Eric Borisch Cc: Warner Losh , Alan Somers , FreeBSD FS , John Kennedy , Nathan Whitehorn Subject: Re: UEFI question In-Reply-To: References: <20220531123221.n7ubs365ceeqo2d4@x1> Message-ID: <1e6a680f170f85563caba1d0842bfa46@lerctr.org> X-Sender: ler@lerctr.org Content-Type: multipart/alternative; boundary="=_7e89b74620cdf99cbfe4dd82a77f1676" X-Rspamd-Queue-Id: 4NH4BK5JFCz3hCt X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:55103, ipnet:2602:fcdb::/36, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --=_7e89b74620cdf99cbfe4dd82a77f1676 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8; format=flowed On 11/22/2022 5:50 pm, Eric Borisch wrote: > On Tue, Nov 22, 2022 at 1:50 PM Warner Losh wrote: > > On Tue, Nov 22, 2022, 11:42 AM Alan Somers wrote: > On Tue, May 31, 2022 at 9:27 AM John Kennedy > wrote: >> >> On Tue, May 31, 2022 at 02:32:21PM +0200, Julien Cigar wrote: >>> I have a Dell R340 server on which installed a 13.0-RELEASE and then >>> upgraded to 13.1-RELEASE (through freebsd-update). ZFS is used >>> (raidz2). >>> >>> The system has the following partitions: >>> https://gist.github.com/silenius/2defdd5480c5c1bc9ba2ff8940756466 >>> Some things regarding UEFI are not clear to me: >>> - as you can see in the partition list, the installer created an EFI >>> partition on all 4 drives, however it looks like only the first one >>> has >>> has been populated (an mounted): >>> https://gist.github.com/silenius/1220c953f905d868c1615fd0e7122bbf .. >>> why ? >>> - if I understand well if my da0 disk dies the system becomes >>> unbootable >>> (https://gist.github.com/silenius/51d202053295270eaaeb2c02316165ee).. >>> correct? >>> - what's the correct way to fix this? should I newfs_msdos on each >>> EFI >>> partition and copy /boot/loader.efi as /EFI/BOOT/BOOTX64.EFI? or >>> should I use efibootmgr? >>> >>> Thanks for enlightening me :) >> >> I use UEFI to multi-boot a lot myself, but not through efibootmgr (vs >> picking it via UEFI/BIOS on boot manually). I'm multi-booting windows >> and FreeBSD, so I'm being (justifiably) paranoid, but not based on >> being burnt in recent memory. >> >> Depending on how your system works (drives are enumerated at startup), >> in theory any of your disks could be da0. If you've only got a >> working >> EFI partition on one, that is probably hedging your bets a bit but >> yes, >> if that drive died, you'd probably be dead. >> >> In my dual-disk setups, I'd see an EFI and a freebsd-boot (BIOS) >> partition >> on both disks, I split my total swap between the two (not mirrored), >> and >> did ZFS "raiding" (mirroring-ish, usually) across the freebsd-zfs >> partitions. >> >> In my case, using the UEFI/BIOS from the motherboard, I could manually >> pick a disk to boot from which let me vet UEFI or ZFS bootblock >> changes >> for sanity before committing to both. But yes, I'd upgrade them both >> to >> the same thing once I was sure it would work. > > I just noticed the same problem. For mirrored installs, the > 13.1-RELEASE installer only formats the efi partition on one hard > disk. Shouldn't it do both? > > We go back and forth on this question.... I think we should, but others > got grumpy when I suggested it. Maybe just all the drives in the zpool > used for the BE? > > Warner Just a "yes, please." If you're installing zfs root on a mirror of disks, POLA suggests the required boot bits are on both disks, too. Thanks, - Eric > This is my common setup for every FreeBSD box I build: ❯ gpart show => 40 3905945520 mfid0 GPT (1.8T) 40 532480 1 efi (260M) 532520 16779224 3 freebsd-swap (8.0G) 17311744 3888633816 4 freebsd-zfs (1.8T) => 40 3905945520 mfid1 GPT (1.8T) 40 532480 1 efi (260M) 532520 16779224 3 freebsd-swap (8.0G) 17311744 3888633816 4 freebsd-zfs (1.8T) => 40 5859442608 mfid2 GPT (2.7T) 40 532480 1 efi (260M) 532520 16779224 3 freebsd-swap (8.0G) 17311744 5842130904 4 freebsd-zfs (2.7T) => 40 5859442608 mfid3 GPT (2.7T) 40 532480 1 efi (260M) 532520 16779224 3 freebsd-swap (8.0G) 17311744 5842130904 4 freebsd-zfs (2.7T) => 40 3905945520 mfid4 GPT (1.8T) 40 532480 1 efi (260M) 532520 16779224 3 freebsd-swap (8.0G) 17311744 3888633816 4 freebsd-zfs (1.8T) => 40 3905945520 mfid5 GPT (1.8T) 40 532480 1 efi (260M) 532520 16777216 3 freebsd-swap (8.0G) 17309736 3888635824 4 freebsd-zfs (1.8T) ler in 🌐 borg in ~ via C v14.0.5-clang on ☁️ (us-east-1) ❯ Disk number and sizes vary but the setup is the same EFI or BOOT or Both, swap, and ZFS on *ALL* disks -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106 --=_7e89b74620cdf99cbfe4dd82a77f1676 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=UTF-8

On 11/22/2022 5:50 pm, Eric Borisch wrote:

On Tue, Nov 22, 2022 at 1:50 PM Warner Losh <imp@bsdimp.com> wrote:=


On Tue, Nov 22, 2022, 11:42 AM Alan= Somers <asome= rs@freebsd.org> wrote:
On Tue, May 31, 2022 at 9:27 AM John Kennedy <warlock@phouka.net&g= t; wrote:
>
> On Tue, May 31, 2022 at 02:32:21PM +0200, Jul= ien Cigar wrote:
> > I have a Dell R340 server on which installe= d a 13.0-RELEASE and then
> > upgraded to 13.1-RELEASE (through = freebsd-update). ZFS is used (raidz2).
> >
> > The sy= stem has the following partitions: https://gist.github.com/silenius/2defdd5480c5c1bc9ba2ff8940756466<= /a>
> > Some things regarding UEFI are not clear to me:
>= ; > - as you can see in the partition list, the installer created an EFI=
> > partition on all 4 drives, however it looks like only the f= irst one has
> > has been populated (an mounted): https://gist.github.com/silenius/1220c95= 3f905d868c1615fd0e7122bbf .. why ?
> > - if I understand wel= l if my da0 disk dies the system becomes
> >   unboota= ble (https://gist.github.= com/silenius/51d202053295270eaaeb2c02316165ee).. correct?
> >= ; - what's the correct way to fix this? should I newfs_msdos on each EFI> >   partition and copy /boot/loader.efi as /EFI/BOOT/B= OOTX64.EFI? or
> >   should I use efibootmgr?
>= ; >
> > Thanks for enlightening me :)
>
> I u= se UEFI to multi-boot a lot myself, but not through efibootmgr (vs
>= ; picking it via UEFI/BIOS on boot manually).  I'm multi-booting windo= ws
> and FreeBSD, so I'm being (justifiably) paranoid, but not base= d on
> being burnt in recent memory.
>
> Depending = on how your system works (drives are enumerated at startup),
> in t= heory any of your disks could be da0.  If you've only got a working> EFI partition on one, that is probably hedging your bets a bit but = yes,
> if that drive died, you'd probably be dead.
>
&= gt; In my dual-disk setups, I'd see an EFI and a freebsd-boot (BIOS) partit= ion
> on both disks, I split my total swap between the two (not mir= rored), and
> did ZFS "raiding" (mirroring-ish, usually) across the= freebsd-zfs partitions.
>
> In my case, using the UEFI/BIO= S from the motherboard, I could manually
> pick a disk to boot from= which let me vet UEFI or ZFS bootblock changes
> for sanity before= committing to both.  But yes, I'd upgrade them both to
> the = same thing once I was sure it would work.

I just noticed the sam= e problem.  For mirrored installs, the
13.1-RELEASE installer onl= y formats the efi partition on one hard
disk.  Shouldn't it do bo= th?
 
 
We go back and forth on this question.... I think we shou= ld, but others got grumpy when I suggested it. Maybe just all the drives in= the zpool used for the BE?
 
Warner
 
Just a "yes, please." If you're installing zfs root on a = mirror of disks, POLA suggests the required boot bits are on both disks, to= o.
 
Thanks,
  - Eric
 
 

This is my common setup for every FreeBSD box I build:

=E2=9D=AF gpart show
=3D>        40  3905945520  mfi= d0  GPT  (1.8T)
          40      532480=       1  efi  (260M)
      532520    16779224    &= nbsp; 3  freebsd-swap  (8.0G)
    17311744  3888633816      4&nb= sp; freebsd-zfs  (1.8T)
 
=3D>        40  3905945520  mfi= d1  GPT  (1.8T)
          40      532480=       1  efi  (260M)
      532520    16779224    &= nbsp; 3  freebsd-swap  (8.0G)
    17311744  3888633816      4&nb= sp; freebsd-zfs  (1.8T)
 
=3D>        40  5859442608  mfi= d2  GPT  (2.7T)
          40      532480=       1  efi  (260M)
      532520    16779224    &= nbsp; 3  freebsd-swap  (8.0G)
    17311744  5842130904      4&nb= sp; freebsd-zfs  (2.7T)
 
=3D>        40  5859442608  mfi= d3  GPT  (2.7T)
          40      532480=       1  efi  (260M)
      532520    16779224    &= nbsp; 3  freebsd-swap  (8.0G)
    17311744  5842130904      4&nb= sp; freebsd-zfs  (2.7T)
 
=3D>        40  3905945520  mfi= d4  GPT  (1.8T)
          40      532480=       1  efi  (260M)
      532520    16779224    &= nbsp; 3  freebsd-swap  (8.0G)
    17311744  3888633816      4&nb= sp; freebsd-zfs  (1.8T)
 
=3D>        40  3905945520  mfi= d5  GPT  (1.8T)
          40      532480=       1  efi  (260M)
      532520    16777216    &= nbsp; 3  freebsd-swap  (8.0G)
    17309736  3888635824      4&nb= sp; freebsd-zfs  (1.8T)
 
 
ler in =F0=9F=8C=90 borg in ~ via C v14.0.5-clang on =E2=98=81= =EF=B8=8F  (us-east-1)
=E2=9D=AF
 
Disk number and sizes vary but the setup is the same EFI or BOOT= or Both, swap, and ZFS on *ALL* disks
= -- 
Larry Rosenman        = ;             http://www.lerctr.org= /~ler
Phone: +1 214-642-9640           &n= bsp;     E-Mail: ler@lerctr.org=
US Mail: 5708 Sabbia Dr, Round Rock, TX 78665-2106
--=_7e89b74620cdf99cbfe4dd82a77f1676--