Date: Tue, 6 Sep 2022 23:41:47 +0200 (CEST) From: Ronald Klop <ronald-lists@klop.ws> To: freebsd-arm <freebsd-arm@freebsd.org>, Warner Losh <imp@bsdimp.com>, Mark Millard <marklmi@yahoo.com> Subject: Re: kernel update broke -current Message-ID: <1098569247.12297.1662500507113@localhost> In-Reply-To: <CANCZdfq5h-1H5Ug6kBVFtzVmWVVmOU31N6PUXSMHZ8ZTx5Fr_Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_12296_882797294.1662500507110
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: quoted-printable
Van: Warner Losh <imp@bsdimp.com>
Datum: 6 september 2022 18:13
Aan: Mark Millard <marklmi@yahoo.com>
CC: freebsd-arm <freebsd-arm@freebsd.org>
Onderwerp: Re: kernel update broke -current
>=20
>=20
>=20
>=20
>=20
>=20
> On Mon, Sep 5, 2022 at 4:21 PM Mark Millard <marklmi@yahoo.com> wrote:
>=20
>> Warner Losh <imp_at_bsdimp.com> wrote on
>> Date: Mon, 05 Sep 2022 20:07:33 UTC :
>>=20
>> > On Sun, Sep 4, 2022 at 4:51 PM void <void@f-m.fm> wrote:
>> >=20
>> > > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:
>> > > >
>> > > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:
>> > > >> You need a newer boot loader...
>> > > >
>> > > >I was thinking - getting latest -current image, booting to it then
>> > > >copying the contents of /boot from the image onto the mounted zpool=
...
>> > > >
>> > > >feasible?
>> > >
>> > > Seems only EFI/ needed replacing from a recent snapshot. I thought i=
t might
>> > > be all of /boot but I was wrong. Thank you Mark!
>> > >
>> >=20
>> > Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot is
>> > updated
>> > when you do an installworld.
>>=20
>> One of the oddities of the update sequence instructions is
>> the lack of coverage of the likes of:
>>=20
>> Load Path: /efi=08oot=08ootaa64.efi
>>=20
>> What step of the sequencing for the overall system update?
>> When is such an update required? (Here the example would
>> be: Before rebooting when the ZFS pool(s) possibly used to
>> boot gain new features?) When is it not required to update
>> the loader in the ESP (or analogous)? Even just knowing
>> the stage at which one should do the update indicates some
>> about when to figure out if an update is needed and so
>> prompts to be ready.
>=20
>=20
> Today:
>=20
> make installworld installkernel
> mount -t msdos /dev/<mumble-esp> /boot/efi
>=20
> and then one of three scenarios
>=20
> (1) You have the old boot1.efi. This will *always* be in ESP:EFIBOOTBOOT$=
{ARCH}.EFI.
> (see uefi(8) for the values of ARCH). You can automatically detect this f=
or most installations
> that were done since about FreeBSD 12:
> % sudo efivar | grep Boot1
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev
>=20
>=20
> In this case you would do:
>=20
> % sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and you are =
done.
>=20
> Please note: If your system was installed a long time ago, you should als=
o check to see if
> you have a LoaderPath variable set:
> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
>=20
> /boot/loader.efi
>=20
> When Boot1 isn't set and it is set to the above value (or something simil=
ar), then you are
> booting with a really old copy of boot1.efi. You will need to update it a=
nd may need to arrange
> for a larger ESP since FreeBSD's boot1.efifat was a tiny image that was h=
ard to grow. Steal
> space from swap for a larger ESP, etc. Documenting that is beyond the sco=
pe of this email.
>=20
> (2) You are booting with loader.efi and it is in the bug-compatibility pl=
ace. Some UEFI BIOSes
> don't respect or can't set BootXXXX variables set by efibootmgr(8). In th=
at case, many people
> are booting from the 'compatibiltiy' location for removable devices. This=
will almost always be
> a single boot scenario because you can't easily boot other systems like t=
his (well, unless 'quit'
> works from the OK prompt to go to the next one on the list, but I digress=
). You will see something like:
>=20
> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> EFIBOOTBOOTX64.EFI
> (or AA64)
>=20
> when you are booting this way. In this case the update command is:
>=20
> % sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi
>=20
> to upgrade. Some people may be doing this already on systems that can sup=
port efibootmgr(8) and
> they can of course upgrade to using that, though that's also beyond the s=
cope of this email.
>=20
> (3) You are booting on a working system with a FreeBSD installed by the b=
oot loader, or some other
> custom arrangement. In that case, you'll see something like:
> sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> EFIFREEBSDLOADER.EFI
>=20
> (or some other place for custom setups: I assume if you are doing that yo=
u know enough to
> update my instructions as appropriate). To update, you'd do
>=20
> % sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
>=20
> Automating all these cases is, needless to say, tricky.
>=20
> =20
>> Part of the sequencing gets into having needed to do a
>> installworld to have a from-same-build content replacement
>> for the likes of /efi=08oot=08ootaa64.efi . So installkernel
>> already done, a reboot already done, and an installworld
>> having been done as well in order to have something to
>> copy over. (There are no pointers to alternate places to
>> get copies that I know of. One can find copies in the
>> build tree when one builds from source locally. So waiting
>> is not really required for that context.)
>=20
>=20
> Yea, I have a branch that has an 'installboot' target that updates the bo=
ot
> loader regardless, but given the 50-odd combinations of ways we can
> boot, it's a bit bogged down.
> =20
>> This also make it seem that updating ZFS pool features
>> should wait until after a system upgrade that spans both
>> the loader and kernel being ready for the new features,
>> even if compatibility with other systems is not a worry.
>=20
>=20
> Yes. ALWAYS update your boot blocks before zpool update.
> =20
>> Do any of the system upgrade instructions cover such
>> relationships? Do any of the ZFS pool upgrade instructions
>> cover such? Does zpool or the like suggest such issues
>> when it reports there are new features that could be
>> enabled?
>=20
>=20
> See above. :)
> =20
>> Part of what I expect happened here was contributed to by
>> a lack of being prompted to even think about the relevant
>> issues, leading to a pool feature upgrade that had not
>> been prepared for.
>=20
>=20
> Yea, so far 100% of the 'help, I upgraded and now the boot loader
> can't see zfs pool' issues have been 'I didn't upgrade my loader.efi
> properly after zpool upgrade.' It was mandatory when we had the
> OpenZFS rebase, but it is also necessary every few OpenZFS
> updates from upstream as nnew features are enabled.
>=20
> I'd love for someone to add this information to the handbook, and I'd
> happily review such an effort. I have time to do a brain dump, but not
> to make it pretty / formatted for asciidoctor and won't for some time.
>=20
> Warner
> =20
>> =3D=3D=3D
>> Mark Millard
>> marklmi at yahoo.com
>>=20
>=20
Interesting.=20
Does the loader pass a hint to the kernel about which loader was used to lo=
ad the kernel?
Regards,
Ronald
------=_Part_12296_882797294.1662500507110
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
<html><head></head><body><br><p><small><strong>Van:</strong> Warner Losh &l=
t;imp@bsdimp.com><br><strong>Datum:</strong> 6 september 2022 18:13<br><=
strong>Aan:</strong> Mark Millard <marklmi@yahoo.com><br><strong>CC:<=
/strong> freebsd-arm <freebsd-arm@freebsd.org><br><strong>Onderwerp:<=
/strong> Re: kernel update broke -current<br></small></p><blockquote style=
=3D"margin-left: 5px; border-left: 3px solid #ccc; margin-right: 0px; paddi=
ng-left: 5px;"><div class=3D"MessageRFC822Viewer do_not_remove" id=3D"P"><!=
-- P -->
<!-- processMimeMessage --><div class=3D"MultipartAlternativeViewer do_not_=
remove"><!-- P.P -->
<div class=3D"TextHTMLViewer do_not_remove" id=3D"P.P.P"><!-- P.P.P -->
<div class=3D"do_not_remove"><div class=3D"do_not_remove"><br></div><br><di=
v class=3D"gmail_quote do_not_remove"><div class=3D"gmail_attr do_not_remov=
e">On Mon, Sep 5, 2022 at 4:21 PM Mark Millard <<a href=3D"mailto:marklm=
i@yahoo.com" onclick=3D"composeMailto('marklmi@yahoo.com'); return false;">=
marklmi@yahoo.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote"=
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);p=
adding-left:1ex">Warner Losh <<a href=3D"http://imp_at_bsdimp.com" targe=
t=3D"_blank">imp_at_bsdimp.com</a>> wrote on<br>
Date: Mon, 05 Sep 2022 20:07:33 UTC :<br>
<br>
> On Sun, Sep 4, 2022 at 4:51 PM void <<a href=3D"mailto:void@f-m.fm"=
onclick=3D"composeMailto('void@f-m.fm'); return false;" target=3D"_blank">=
void@f-m.fm</a>> wrote:<br>
> <br>
> > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:<br>
> > ><br>
> > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:<br>
> > >> You need a newer boot loader...<br>
> > ><br>
> > >I was thinking - getting latest -current image, booting to it=
then<br>
> > >copying the contents of /boot from the image onto the mounted=
zpool ...<br>
> > ><br>
> > >feasible?<br>
> ><br>
> > Seems only EFI/ needed replacing from a recent snapshot. I though=
t it might<br>
> > be all of /boot but I was wrong. Thank you Mark!<br>
> ><br>
> <br>
> Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot is<=
br>
> updated<br>
> when you do an installworld.<br>
<br>
One of the oddities of the update sequence instructions is<br>
the lack of coverage of the likes of:<br>
<br>
Load Path: /efiootootaa64.efi<br>
<br>
What step of the sequencing for the overall system update?<br>
When is such an update required? (Here the example would<br>
be: Before rebooting when the ZFS pool(s) possibly used to<br>
boot gain new features?) When is it not required to update<br>
the loader in the ESP (or analogous)? Even just knowing<br>
the stage at which one should do the update indicates some<br>
about when to figure out if an update is needed and so<br>
prompts to be ready.<br></blockquote><div class=3D"do_not_remove"><br></div=
><div class=3D"do_not_remove">Today:</div><div class=3D"do_not_remove"><br>=
</div><div class=3D"do_not_remove">make installworld installkernel</div><di=
v class=3D"do_not_remove">mount -t msdos /dev/<mumble-esp> /boot/efi<=
/div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_remove">an=
d then one of three scenarios</div><div class=3D"do_not_remove"><br></div><=
div class=3D"do_not_remove">(1) You have the old boot1.efi. This will *alwa=
ys* be in ESP:EFIBOOTBOOT${ARCH}.EFI.</div><div class=3D"do_not_remove">(se=
e uefi(8) for the values of ARCH). You can automatically detect this for mo=
st installations</div><div class=3D"do_not_remove">that were done since abo=
ut FreeBSD 12:</div><div class=3D"do_not_remove">% sudo efivar | grep Boot1=
</div><div class=3D"do_not_remove">cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boo=
t1Path<br>cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev<br></div><div class=
=3D"do_not_remove"><br></div><div class=3D"do_not_remove">In this case you =
would do:</div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_=
remove">% sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and yo=
u are done.</div><div class=3D"do_not_remove"><br></div><div class=3D"do_no=
t_remove">Please note: If your system was installed a long time ago, you sh=
ould also check to see if</div><div class=3D"do_not_remove">you have a Load=
erPath variable set:</div><div class=3D"do_not_remove">% sudo efivar --utf =
cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br></div><div class=3D"do_n=
ot_remove">/boot/loader.efi</div><div class=3D"do_not_remove"><br></div><di=
v class=3D"do_not_remove">When Boot1 isn't set and it is set to the above v=
alue (or something similar), then you are</div><div class=3D"do_not_remove"=
>booting with a really old copy of boot1.efi. You will need to update it an=
d may need to arrange</div><div class=3D"do_not_remove">for a larger ESP si=
nce FreeBSD's boot1.efifat was a tiny image that was hard to grow. Steal</d=
iv><div class=3D"do_not_remove">space from swap for a larger ESP, etc. Docu=
menting that is beyond the scope of this email.</div><div class=3D"do_not_r=
emove"><br></div><div class=3D"do_not_remove">(2) You are booting with load=
er.efi and it is in the bug-compatibility place. Some UEFI BIOSes</div><div=
class=3D"do_not_remove">don't respect or can't set BootXXXX variables set =
by efibootmgr(8). In that case, many people</div><div class=3D"do_not_remov=
e">are booting from the 'compatibiltiy' location for removable devices=
. This will almost always be</div><div class=3D"do_not_remove">a single boo=
t scenario because you can't easily boot other systems like this (well, unl=
ess 'quit'</div><div class=3D"do_not_remove">works from the OK prompt to go=
to the next one on the list, but I digress). You will see something like:<=
/div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_remove">% =
sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath</div>cfee=
69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>EFIBOOTBOOTX64.EFI</div><div=
class=3D"gmail_quote do_not_remove">(or AA64)</div><div class=3D"gmail_quo=
te do_not_remove"><br></div><div class=3D"gmail_quote do_not_remove">when y=
ou are booting this way. In this case the update command is:</div><div clas=
s=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote do_not_=
remove">% sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi</div>=
<div class=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quot=
e do_not_remove">to upgrade. Some people may be doing this already on syste=
ms that can support efibootmgr(8) and</div><div class=3D"gmail_quote do_not=
_remove">they can of course upgrade to using that, though that's also beyon=
d the scope of this email.</div><div class=3D"gmail_quote do_not_remove"><b=
r></div><div class=3D"gmail_quote do_not_remove">(3) You are booting on a w=
orking system with a FreeBSD installed by the boot loader, or some other</d=
iv><div class=3D"gmail_quote do_not_remove">custom arrangement. In that cas=
e, you'll see something like:</div><div class=3D"gmail_quote do_not_remove"=
>sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>cfee6=
9ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>EFIFREEBSDLOADER.EFI</div><di=
v class=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote d=
o_not_remove">(or some other place for custom setups: I assume if you are d=
oing that you know enough to</div><div class=3D"gmail_quote do_not_remove">=
update my instructions as appropriate). To update, you'd do</div><div class=
=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote do_not_r=
emove">% sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi</div><di=
v class=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote d=
o_not_remove">Automating all these cases is, needless to say, tricky.<br><d=
iv class=3D"do_not_remove"> </div><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd=
ing-left:1ex">
Part of the sequencing gets into having needed to do a<br>
installworld to have a from-same-build content replacement<br>
for the likes of /efiootootaa64.efi . So installkernel<br>
already done, a reboot already done, and an installworld<br>
having been done as well in order to have something to<br>
copy over. (There are no pointers to alternate places to<br>
get copies that I know of. One can find copies in the<br>
build tree when one builds from source locally. So waiting<br>
is not really required for that context.)<br></blockquote><div class=3D"do_=
not_remove"><br></div><div class=3D"do_not_remove">Yea, I have a branch tha=
t has an 'installboot' target that updates the boot</div><div class=3D"do_n=
ot_remove">loader regardless, but given the 50-odd combinations of ways we =
can</div><div class=3D"do_not_remove">boot, it's a bit bogged down.</div><d=
iv class=3D"do_not_remove"> </div><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd=
ing-left:1ex">
This also make it seem that updating ZFS pool features<br>
should wait until after a system upgrade that spans both<br>
the loader and kernel being ready for the new features,<br>
even if compatibility with other systems is not a worry.<br></blockquote><d=
iv class=3D"do_not_remove"><br></div><div class=3D"do_not_remove">Yes. ALWA=
YS update your boot blocks before zpool update.</div><div class=3D"do_not_r=
emove"> </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p=
x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Do any of the system upgrade instructions cover such<br>
relationships? Do any of the ZFS pool upgrade instructions<br>
cover such? Does zpool or the like suggest such issues<br>
when it reports there are new features that could be<br>
enabled?<br></blockquote><div class=3D"do_not_remove"><br></div><div class=
=3D"do_not_remove">See above. :)</div><div class=3D"do_not_remove"> </=
div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left:1px solid rgb(204,204,204);padding-left:1ex">
Part of what I expect happened here was contributed to by<br>
a lack of being prompted to even think about the relevant<br>
issues, leading to a pool feature upgrade that had not<br>
been prepared for.<br></blockquote><div class=3D"do_not_remove"><br></div><=
div class=3D"do_not_remove">Yea, so far 100% of the 'help, I upgraded and n=
ow the boot loader</div><div class=3D"do_not_remove">can't see zfs pool' is=
sues have been 'I didn't upgrade my loader.efi</div><div class=3D"do_not_re=
move">properly after zpool upgrade.' It was mandatory when we had the</div>=
<div class=3D"do_not_remove">OpenZFS rebase, but it is also necessary every=
few OpenZFS</div><div class=3D"do_not_remove">updates from upstream as nne=
w features are enabled.</div><div class=3D"do_not_remove"><br></div><div cl=
ass=3D"do_not_remove">I'd love for someone to add this information to the h=
andbook, and I'd</div><div class=3D"do_not_remove">happily review such=
an effort. I have time to do a brain dump, but not</div><div class=3D"do_n=
ot_remove">to make it pretty / formatted for asciidoctor and won't for some=
time.</div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_rem=
ove">Warner</div><div class=3D"do_not_remove"> </div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">
=3D=3D=3D<br>
Mark Millard<br>
marklmi at <a href=3D"http://yahoo.com" target=3D"_blank">yahoo.com</a><br>
<br>
</blockquote></div></div>
</div></div></div><!-- TextHTMLViewer -->
<!-- MultipartAlternativeViewer -->
<!-- MessageRFC822Viewer -->
</blockquote><br><br>Interesting. <br>Does the loader pass a hint to t=
he kernel about which loader was used to load the kernel?<div><br></div><di=
v>Regards,</div><div>Ronald</div><div><br><br></div></body></html>
------=_Part_12296_882797294.1662500507110--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1098569247.12297.1662500507113>