Date: Sat, 16 Sep 2023 17:22:20 +0100 From: Warner Losh <imp@bsdimp.com> To: Toomas Soome <tsoome@me.com> Cc: Mark Millard <marklmi@yahoo.com>, void <void@f-m.fm>, Current FreeBSD <freebsd-current@freebsd.org> Subject: Re: CURRRENT snapshot won't boot due missing ZFS feature Message-ID: <CANCZdfqpmFALu0OsLkmEpLu4iOoazE45==9TFPd8HFjW3SNuWQ@mail.gmail.com> In-Reply-To: <4D51E8E6-8AF0-4773-A9BA-D53C08B744EA@me.com> References: <7EEF3435-064D-4C3C-98E4-2B27A788DB43.ref@yahoo.com> <7EEF3435-064D-4C3C-98E4-2B27A788DB43@yahoo.com> <4D51E8E6-8AF0-4773-A9BA-D53C08B744EA@me.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000e24f0206057c50cf Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Sep 16, 2023 at 5:11=E2=80=AFPM Toomas Soome <tsoome@me.com> wrote: > > > > On 16. Sep 2023, at 18:43, Mark Millard <marklmi@yahoo.com> wrote: > > > > void <void_at_f-m.fm> wrote on > > Date: Sat, 16 Sep 2023 12:12:02 UTC : > > > >> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote: > >> > >>> Yes. The boot loader comes from the host. It must know how to read > ZFS. > >> > >> It knows how to read zfs. > > > > I expect Warner was indicating: you have a (efi?) loader that knows > > how to deal with the features listed in: > > > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd > > > > being active but not with some new feature(s) listed in: > > > > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > > > > being active. > > > > The following are the "read-only-compatibile no" features > > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd : > > > > blake3 > > ednor > > head_errlog > > vdev_zaps_v2 > > > > So any of those being active leads to lack of even read-only > > activity being compatible. (Although, the loader's subset > > of the potential overall activity might allow ignoring some > > specific "read-only-compatibile no" status examples.) > > > > For reference: > > > > # diff -u99 > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-f= reebsd > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > > --- > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-f= reebsd > 2021-06-24 20:08:57.206621000 -0700 > > +++ > /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2 > 2023-06-10 15:59:25.354999000 -0700 > > @@ -1,34 +1,40 @@ > > -# Features supported by OpenZFS 2.1 on FreeBSD > > +# Features supported by OpenZFS 2.2 on Linux and FreeBSD > > allocation_classes > > async_destroy > > +blake3 > > +block_cloning > > bookmark_v2 > > bookmark_written > > bookmarks > > device_rebuild > > device_removal > > draid > > +edonr > > embedded_data > > empty_bpobj > > enabled_txg > > encryption > > extensible_dataset > > filesystem_limits > > +head_errlog > > hole_birth > > large_blocks > > large_dnode > > livelist > > log_spacemap > > lz4_compress > > multi_vdev_crash_dump > > obsolete_counts > > project_quota > > redacted_datasets > > redaction_bookmarks > > resilver_defer > > sha512 > > skein > > spacemap_histogram > > spacemap_v2 > > userobj_accounting > > +vdev_zaps_v2 > > +zilsaxattr > > zpool_checkpoint > > zstd_compress > > > > (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does > > not exist yet. Thus were I had the diff look.) > > > >> On the host in question, there are many guests, > >> some with zfs-boot, some not, just file-based. > > > > But with what openzfs features active vs. not active > > in each case? > > > >> What the host is not, is zfs-on-root. It boots from ssd (ada0). > >> The vdevs are on a sas disk array. > >> > >>> So either your bootable partitions must not have > com.klarasystems:vdev_zaps_v2 > >>> in your BEs or you must have a new user boot. I think you can just > install > >>> the one from 14, but haven't tried it. > >> > >> Can you briefly explain how I'd install the one from 14 please? > > > > > > I do not use bhyve so I do not even know if the > > context is using the efi loader from a msdosfs > > vs. not. For efi loaders, copying from one msdosfs > > with a sufficient vintage to the one with the wrong > > vintage (replacing) is sufficient. > > bhyve in freebsd is traditionally using /boot/userboot.so, I believe. Yes. We use the *HOSTS* (running FreeBSD 13) /boot/userboot.so to boot the FreeBSD 14 image. Since we're not using the boot loader from the target image to load it for bhyve, the loader we're using has to understand the ZFS dataset that it's booting off of. FreeBSD 13's userboot.so doesn't support all the bells and whistles that the ZFS folks have added to 14. So, either you have to turn off those features (which I got no clue how to do in the normal installer), or you have to update userboot.so to the FreeBSD 14 version (which I think had a good chance of actually running on FreeBSD 13 since it has no 'system' references, which are confined to bhyveload). Warner > > > > # find /boot/efi/EFI/ -print > > /boot/efi/EFI/ > > /boot/efi/EFI/FREEBSD > > /boot/efi/EFI/FREEBSD/loader.efi > > /boot/efi/EFI/BOOT > > /boot/efi/EFI/BOOT/bootaa64.efi > > > > There may well be only: > > > > EFI/BOOT/bootaa64.efi > > > > for all I know. > > > > From an amd64 context: > > > > # find /boot/efi/EFI/ -print > > /boot/efi/EFI/ > > /boot/efi/EFI/FREEBSD > > /boot/efi/EFI/FREEBSD/loader.efi > > /boot/efi/EFI/BOOT > > /boot/efi/EFI/BOOT/bootx64.efi > > > > There may well be only: > > > > EFI/BOOT/bootx64.efi > > > > for all I know. > > > > (I set things up to have the EFI capitalization > > so that referencing efi/ vs. EFI/ in my context > > is unique for the mount point. vs. the msdosfs > > directory.) > > > > =3D=3D=3D > > Mark Millard > > marklmi at yahoo.com > > > > > > > --000000000000e24f0206057c50cf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">= <div dir=3D"ltr" class=3D"gmail_attr">On Sat, Sep 16, 2023 at 5:11=E2=80=AF= PM Toomas Soome <<a href=3D"mailto:tsoome@me.com">tsoome@me.com</a>> = wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0= px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> <br> > On 16. Sep 2023, at 18:43, Mark Millard <<a href=3D"mailto:marklmi@= yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>> wrote:<br> > <br> > void <<a href=3D"http://void_at_f-m.fm" rel=3D"noreferrer" target= =3D"_blank">void_at_f-m.fm</a>> wrote on<br> > Date: Sat, 16 Sep 2023 12:12:02 UTC :<br> > <br> >> On Sat, Sep 16, 2023 at 12:55:19PM +0100, Warner Losh wrote:<br> >> <br> >>> Yes. The boot loader comes from the host. It must know how to = read ZFS. <br> >> <br> >> It knows how to read zfs.<br> > <br> > I expect Warner was indicating: you have a (efi?) loader that knows<br= > > how to deal with the features listed in:<br> > <br> > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.1-freebsd<br> > <br> > being active but not with some new feature(s) listed in:<br> > <br> > sys/contrib/openzfs/cmd/zpool/compatibility.d/openzfs-2.2<br> > <br> > being active.<br> > <br> > The following are the "read-only-compatibile no" features<br= > > that are new in openzfs-2.2 compared to openzfs-2.1-freebsd :<br> > <br> > blake3<br> > ednor<br> > head_errlog<br> > vdev_zaps_v2<br> > <br> > So any of those being active leads to lack of even read-only<br> > activity being compatible. (Although, the loader's subset<br> > of the potential overall activity might allow ignoring some<br> > specific "read-only-compatibile no" status examples.)<br> > <br> > For reference:<br> > <br> > # diff -u99 /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.= d/openzfs-2.1-freebsd /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibi= lity.d/openzfs-2.2<br> > --- /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzf= s-2.1-freebsd 2021-06-24 20:08:57.206621000 -0700<br> > +++ /usr/main-src/sys/contrib/openzfs/cmd/zpool/compatibility.d/openzf= s-2.2 2023-06-10 15:59:25.354999000 -0700<br> > @@ -1,34 +1,40 @@<br> > -# Features supported by OpenZFS 2.1 on FreeBSD<br> > +# Features supported by OpenZFS 2.2 on Linux and FreeBSD<br> > allocation_classes<br> > async_destroy<br> > +blake3<br> > +block_cloning<br> > bookmark_v2<br> > bookmark_written<br> > bookmarks<br> > device_rebuild<br> > device_removal<br> > draid<br> > +edonr<br> > embedded_data<br> > empty_bpobj<br> > enabled_txg<br> > encryption<br> > extensible_dataset<br> > filesystem_limits<br> > +head_errlog<br> > hole_birth<br> > large_blocks<br> > large_dnode<br> > livelist<br> > log_spacemap<br> > lz4_compress<br> > multi_vdev_crash_dump<br> > obsolete_counts<br> > project_quota<br> > redacted_datasets<br> > redaction_bookmarks<br> > resilver_defer<br> > sha512<br> > skein<br> > spacemap_histogram<br> > spacemap_v2<br> > userobj_accounting<br> > +vdev_zaps_v2<br> > +zilsaxattr<br> > zpool_checkpoint<br> > zstd_compress<br> > <br> > (Last I checked, /usr/share/zfs/compatibility.d/openzfs-2.2 does<br> > not exist yet. Thus were I had the diff look.)<br> > <br> >> On the host in question, there are many guests,<br> >> some with zfs-boot, some not, just file-based.<br> > <br> > But with what openzfs features active vs. not active<br> > in each case?<br> > <br> >> What the host is not, is zfs-on-root. It boots from ssd (ada0).<br= > >> The vdevs are on a sas disk array.<br> >> <br> >>> So either your bootable partitions must not have com.klarasyst= ems:vdev_zaps_v2<br> >>> in your BEs or you must have a new user boot. I think you can = just install<br> >>> the one from 14, but haven't tried it.<br> >> <br> >> Can you briefly explain how I'd install the one from 14 please= ?<br> > <br> > <br> > I do not use bhyve so I do not even know if the<br> > context is using the efi loader from a msdosfs<br> > vs. not. For efi loaders, copying from one msdosfs<br> > with a sufficient vintage to the one with the wrong<br> > vintage (replacing) is sufficient.<br> <br> bhyve in freebsd is traditionally using /boot/userboot.so, I believe.</bloc= kquote><div><br></div><div>Yes. We use the *HOSTS* (running FreeBSD 13) /bo= ot/userboot.so to boot the FreeBSD 14</div><div>image. Since we're not = using the boot loader from the target image to load it for bhyve,</div><div= >the loader we're using has to understand the ZFS dataset that it's= booting off of. FreeBSD</div><div>13's userboot.so doesn't support= all the bells and whistles that the ZFS folks have added</div><div>to 14.<= /div><div><br></div><div>So, either you have to turn off those features (wh= ich I got no clue how to do in the=C2=A0</div><div>normal installer), or yo= u have to update userboot.so to the FreeBSD 14 version (which</div><div>I t= hink had a good chance of actually running on FreeBSD 13 since it has no &#= 39;system'</div><div>references, which are confined to bhyveload).</div= ><div><br></div><div>Warner<br></div><div>=C2=A0</div><blockquote class=3D"= gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20= 4,204,204);padding-left:1ex">> <br> > # find /boot/efi/EFI/ -print<br> > /boot/efi/EFI/<br> > /boot/efi/EFI/FREEBSD<br> > /boot/efi/EFI/FREEBSD/loader.efi<br> > /boot/efi/EFI/BOOT<br> > /boot/efi/EFI/BOOT/bootaa64.efi<br> > <br> > There may well be only:<br> > <br> > EFI/BOOT/bootaa64.efi<br> > <br> > for all I know.<br> > <br> > From an amd64 context:<br> > <br> > # find /boot/efi/EFI/ -print<br> > /boot/efi/EFI/<br> > /boot/efi/EFI/FREEBSD<br> > /boot/efi/EFI/FREEBSD/loader.efi<br> > /boot/efi/EFI/BOOT<br> > /boot/efi/EFI/BOOT/bootx64.efi<br> > <br> > There may well be only:<br> > <br> > EFI/BOOT/bootx64.efi<br> > <br> > for all I know.<br> > <br> > (I set things up to have the EFI capitalization<br> > so that referencing efi/ vs. EFI/ in my context<br> > is unique for the mount point. vs. the msdosfs<br> > directory.)<br> > <br> > =3D=3D=3D<br> > Mark Millard<br> > marklmi at <a href=3D"http://yahoo.com" rel=3D"noreferrer" target=3D"_= blank">yahoo.com</a><br> > <br> > <br> <br> <br> </blockquote></div></div> --000000000000e24f0206057c50cf--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfqpmFALu0OsLkmEpLu4iOoazE45==9TFPd8HFjW3SNuWQ>