Date: Wed, 7 Sep 2022 00:31:03 +0200 (CEST) From: Ronald Klop <ronald-lists@klop.ws> To: freebsd-arm <freebsd-arm@freebsd.org>, Mark Millard <marklmi@yahoo.com>, Warner Losh <imp@bsdimp.com> Subject: Re: kernel update broke -current Message-ID: <1146974959.11767.1662503463234@localhost> In-Reply-To: <1098569247.12297.1662500507113@localhost>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_11766_902161602.1662503463230
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit
Van: Ronald Klop <ronald-lists@klop.ws>
Datum: 6 september 2022 23:42
Aan: freebsd-arm <freebsd-arm@freebsd.org>, Warner Losh <imp@bsdimp.com>, Mark Millard <marklmi@yahoo.com>
Onderwerp: Re: kernel update broke -current
>
>
>
>
>
> Van: Warner Losh <imp@bsdimp.com>
> Datum: 6 september 2022 18:13
> Aan: Mark Millard <marklmi@yahoo.com>
> CC: freebsd-arm <freebsd-arm@freebsd.org>
> Onderwerp: Re: kernel update broke -current
>
>>
>>
>>
>> On Mon, Sep 5, 2022 at 4:21 PM Mark Millard <marklmi@yahoo.com> wrote:
>>
>>> Warner Losh <imp_at_bsdimp.com> wrote on
>>> Date: Mon, 05 Sep 2022 20:07:33 UTC :
>>>
>>> > On Sun, Sep 4, 2022 at 4:51 PM void <void@f-m.fm> wrote:
>>> >
>>> > > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:
>>> > > >
>>> > > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:
>>> > > >> You need a newer boot loader...
>>> > > >
>>> > > >I was thinking - getting latest -current image, booting to it then
>>> > > >copying the contents of /boot from the image onto the mounted zpool ...
>>> > > >
>>> > > >feasible?
>>> > >
>>> > > Seems only EFI/ needed replacing from a recent snapshot. I thought it might
>>> > > be all of /boot but I was wrong. Thank you Mark!
>>> > >
>>> >
>>> > Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot is
>>> > updated
>>> > when you do an installworld.
>>>
>>> One of the oddities of the update sequence instructions is
>>> the lack of coverage of the likes of:
>>>
>>> Load Path: /efiootootaa64.efi
>>>
>>> What step of the sequencing for the overall system update?
>>> When is such an update required? (Here the example would
>>> be: Before rebooting when the ZFS pool(s) possibly used to
>>> boot gain new features?) When is it not required to update
>>> the loader in the ESP (or analogous)? Even just knowing
>>> the stage at which one should do the update indicates some
>>> about when to figure out if an update is needed and so
>>> prompts to be ready.
>>
>>
>> Today:
>>
>> make installworld installkernel
>> mount -t msdos /dev/<mumble-esp> /boot/efi
>>
>> and then one of three scenarios
>>
>> (1) You have the old boot1.efi. This will *always* be in ESP:EFIBOOTBOOT${ARCH}.EFI.
>> (see uefi(8) for the values of ARCH). You can automatically detect this for most installations
>> that were done since about FreeBSD 12:
>> % sudo efivar | grep Boot1
>> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path
>> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev
>>
>>
>> In this case you would do:
>>
>> % sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and you are done.
>>
>> Please note: If your system was installed a long time ago, you should also check to see if
>> you have a LoaderPath variable set:
>> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
>>
>> /boot/loader.efi
>>
>> When Boot1 isn't set and it is set to the above value (or something similar), then you are
>> booting with a really old copy of boot1.efi. You will need to update it and may need to arrange
>> for a larger ESP since FreeBSD's boot1.efifat was a tiny image that was hard to grow. Steal
>> space from swap for a larger ESP, etc. Documenting that is beyond the scope of this email.
>>
>> (2) You are booting with loader.efi and it is in the bug-compatibility place. Some UEFI BIOSes
>> don't respect or can't set BootXXXX variables set by efibootmgr(8). In that case, many people
>> are booting from the 'compatibiltiy' location for removable devices. This will almost always be
>> a single boot scenario because you can't easily boot other systems like this (well, unless 'quit'
>> works from the OK prompt to go to the next one on the list, but I digress). You will see something like:
>>
>> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
>> cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
>> EFIBOOTBOOTX64.EFI
>> (or AA64)
>>
>> when you are booting this way. In this case the update command is:
>>
>> % sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi
>>
>> to upgrade. Some people may be doing this already on systems that can support efibootmgr(8) and
>> they can of course upgrade to using that, though that's also beyond the scope of this email.
>>
>> (3) You are booting on a working system with a FreeBSD installed by the boot loader, or some other
>> custom arrangement. In that case, you'll see something like:
>> sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
>> cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
>> EFIFREEBSDLOADER.EFI
>>
>> (or some other place for custom setups: I assume if you are doing that you know enough to
>> update my instructions as appropriate). To update, you'd do
>>
>> % sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
>>
>> Automating all these cases is, needless to say, tricky.
>>
>>
>>> Part of the sequencing gets into having needed to do a
>>> installworld to have a from-same-build content replacement
>>> for the likes of /efiootootaa64.efi . So installkernel
>>> already done, a reboot already done, and an installworld
>>> having been done as well in order to have something to
>>> copy over. (There are no pointers to alternate places to
>>> get copies that I know of. One can find copies in the
>>> build tree when one builds from source locally. So waiting
>>> is not really required for that context.)
>>
>>
>> Yea, I have a branch that has an 'installboot' target that updates the boot
>> loader regardless, but given the 50-odd combinations of ways we can
>> boot, it's a bit bogged down.
>>
>>> This also make it seem that updating ZFS pool features
>>> should wait until after a system upgrade that spans both
>>> the loader and kernel being ready for the new features,
>>> even if compatibility with other systems is not a worry.
>>
>>
>> Yes. ALWAYS update your boot blocks before zpool update.
>>
>>> Do any of the system upgrade instructions cover such
>>> relationships? Do any of the ZFS pool upgrade instructions
>>> cover such? Does zpool or the like suggest such issues
>>> when it reports there are new features that could be
>>> enabled?
>>
>>
>> See above. :)
>>
>>> Part of what I expect happened here was contributed to by
>>> a lack of being prompted to even think about the relevant
>>> issues, leading to a pool feature upgrade that had not
>>> been prepared for.
>>
>>
>> Yea, so far 100% of the 'help, I upgraded and now the boot loader
>> can't see zfs pool' issues have been 'I didn't upgrade my loader.efi
>> properly after zpool upgrade.' It was mandatory when we had the
>> OpenZFS rebase, but it is also necessary every few OpenZFS
>> updates from upstream as nnew features are enabled.
>>
>> I'd love for someone to add this information to the handbook, and I'd
>> happily review such an effort. I have time to do a brain dump, but not
>> to make it pretty / formatted for asciidoctor and won't for some time.
>>
>> Warner
>>
>>> ===
>>> Mark Millard
>>> marklmi at yahoo.com
>>>
>>
>
>
> Interesting.
> Does the loader pass a hint to the kernel about which loader was used to load the kernel?
>
> Regards,
> Ronald
>
>
Oh I pressed send too soon. I now understand all the tools you already mentioned.
Sorry for the noise.
Regards
------=_Part_11766_902161602.1662503463230
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
<html><head></head><body><br><p><small><strong>Van:</strong> Ronald Klop &l=
t;ronald-lists@klop.ws><br><strong>Datum:</strong> 6 september 2022 23:4=
2<br><strong>Aan:</strong> freebsd-arm <freebsd-arm@freebsd.org>, War=
ner Losh <imp@bsdimp.com>, Mark Millard <marklmi@yahoo.com><br>=
<strong>Onderwerp:</strong> Re: kernel update broke -current<br></small></p=
><blockquote style=3D"margin-left: 5px; border-left: 3px solid #ccc; margin=
-right: 0px; padding-left: 5px;"><div class=3D"MessageRFC822Viewer do_not_r=
emove" id=3D"P"><!-- P -->
<!-- processMimeMessage --><div class=3D"MultipartAlternativeViewer do_not_=
remove"><!-- P.P -->
<div class=3D"TextHTMLViewer do_not_remove" id=3D"P.P.P"><!-- P.P.P -->
<br><p><small><strong>Van:</strong> Warner Losh <imp@bsdimp.com><br><=
strong>Datum:</strong> 6 september 2022 18:13<br><strong>Aan:</strong> Mark=
Millard <marklmi@yahoo.com><br><strong>CC:</strong> freebsd-arm <=
freebsd-arm@freebsd.org><br><strong>Onderwerp:</strong> Re: kernel updat=
e broke -current<br></small></p><blockquote style=3D"margin-left: 5px; bord=
er-left: 3px solid #ccc; margin-right: 0px; padding-left: 5px;"><div class=
=3D"MessageRFC822Viewer do_not_remove" id=3D"P">
<div class=3D"MultipartAlternativeViewer do_not_remove">
<div class=3D"TextHTMLViewer do_not_remove" id=3D"P.P.P">
<div class=3D"do_not_remove"><div class=3D"do_not_remove"><br></div><br><di=
v class=3D"gmail_quote do_not_remove"><div class=3D"gmail_attr do_not_remov=
e">On Mon, Sep 5, 2022 at 4:21 PM Mark Millard <<a href=3D"mailto:marklm=
i@yahoo.com" onclick=3D"composeMailto('marklmi@yahoo.com'); return false;">=
marklmi@yahoo.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote"=
style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);p=
adding-left:1ex">Warner Losh <<a href=3D"http://imp_at_bsdimp.com" targe=
t=3D"_blank">imp_at_bsdimp.com</a>> wrote on<br>
Date: Mon, 05 Sep 2022 20:07:33 UTC :<br>
<br>
> On Sun, Sep 4, 2022 at 4:51 PM void <<a href=3D"mailto:void@f-m.fm"=
onclick=3D"composeMailto('void@f-m.fm'); return false;" target=3D"_blank">=
void@f-m.fm</a>> wrote:<br>
> <br>
> > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:<br>
> > ><br>
> > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:<br>
> > >> You need a newer boot loader...<br>
> > ><br>
> > >I was thinking - getting latest -current image, booting to it=
then<br>
> > >copying the contents of /boot from the image onto the mounted=
zpool ...<br>
> > ><br>
> > >feasible?<br>
> ><br>
> > Seems only EFI/ needed replacing from a recent snapshot. I though=
t it might<br>
> > be all of /boot but I was wrong. Thank you Mark!<br>
> ><br>
> <br>
> Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot is<=
br>
> updated<br>
> when you do an installworld.<br>
<br>
One of the oddities of the update sequence instructions is<br>
the lack of coverage of the likes of:<br>
<br>
Load Path: /efiootootaa64.efi<br>
<br>
What step of the sequencing for the overall system update?<br>
When is such an update required? (Here the example would<br>
be: Before rebooting when the ZFS pool(s) possibly used to<br>
boot gain new features?) When is it not required to update<br>
the loader in the ESP (or analogous)? Even just knowing<br>
the stage at which one should do the update indicates some<br>
about when to figure out if an update is needed and so<br>
prompts to be ready.<br></blockquote><div class=3D"do_not_remove"><br></div=
><div class=3D"do_not_remove">Today:</div><div class=3D"do_not_remove"><br>=
</div><div class=3D"do_not_remove">make installworld installkernel</div><di=
v class=3D"do_not_remove">mount -t msdos /dev/<mumble-esp> /boot/efi<=
/div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_remove">an=
d then one of three scenarios</div><div class=3D"do_not_remove"><br></div><=
div class=3D"do_not_remove">(1) You have the old boot1.efi. This will *alwa=
ys* be in ESP:EFIBOOTBOOT${ARCH}.EFI.</div><div class=3D"do_not_remove">(se=
e uefi(8) for the values of ARCH). You can automatically detect this for mo=
st installations</div><div class=3D"do_not_remove">that were done since abo=
ut FreeBSD 12:</div><div class=3D"do_not_remove">% sudo efivar | grep Boot1=
</div><div class=3D"do_not_remove">cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boo=
t1Path<br>cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev<br></div><div class=
=3D"do_not_remove"><br></div><div class=3D"do_not_remove">In this case you =
would do:</div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_=
remove">% sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and yo=
u are done.</div><div class=3D"do_not_remove"><br></div><div class=3D"do_no=
t_remove">Please note: If your system was installed a long time ago, you sh=
ould also check to see if</div><div class=3D"do_not_remove">you have a Load=
erPath variable set:</div><div class=3D"do_not_remove">% sudo efivar --utf =
cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br></div><div class=3D"do_n=
ot_remove">/boot/loader.efi</div><div class=3D"do_not_remove"><br></div><di=
v class=3D"do_not_remove">When Boot1 isn't set and it is set to the above v=
alue (or something similar), then you are</div><div class=3D"do_not_remove"=
>booting with a really old copy of boot1.efi. You will need to update it an=
d may need to arrange</div><div class=3D"do_not_remove">for a larger ESP si=
nce FreeBSD's boot1.efifat was a tiny image that was hard to grow. Steal</d=
iv><div class=3D"do_not_remove">space from swap for a larger ESP, etc. Docu=
menting that is beyond the scope of this email.</div><div class=3D"do_not_r=
emove"><br></div><div class=3D"do_not_remove">(2) You are booting with load=
er.efi and it is in the bug-compatibility place. Some UEFI BIOSes</div><div=
class=3D"do_not_remove">don't respect or can't set BootXXXX variables set =
by efibootmgr(8). In that case, many people</div><div class=3D"do_not_remov=
e">are booting from the 'compatibiltiy' location for removable devices=
. This will almost always be</div><div class=3D"do_not_remove">a single boo=
t scenario because you can't easily boot other systems like this (well, unl=
ess 'quit'</div><div class=3D"do_not_remove">works from the OK prompt to go=
to the next one on the list, but I digress). You will see something like:<=
/div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_remove">% =
sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath</div>cfee=
69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>EFIBOOTBOOTX64.EFI</div><div=
class=3D"gmail_quote do_not_remove">(or AA64)</div><div class=3D"gmail_quo=
te do_not_remove"><br></div><div class=3D"gmail_quote do_not_remove">when y=
ou are booting this way. In this case the update command is:</div><div clas=
s=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote do_not_=
remove">% sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi</div>=
<div class=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quot=
e do_not_remove">to upgrade. Some people may be doing this already on syste=
ms that can support efibootmgr(8) and</div><div class=3D"gmail_quote do_not=
_remove">they can of course upgrade to using that, though that's also beyon=
d the scope of this email.</div><div class=3D"gmail_quote do_not_remove"><b=
r></div><div class=3D"gmail_quote do_not_remove">(3) You are booting on a w=
orking system with a FreeBSD installed by the boot loader, or some other</d=
iv><div class=3D"gmail_quote do_not_remove">custom arrangement. In that cas=
e, you'll see something like:</div><div class=3D"gmail_quote do_not_remove"=
>sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>cfee6=
9ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>EFIFREEBSDLOADER.EFI</div><di=
v class=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote d=
o_not_remove">(or some other place for custom setups: I assume if you are d=
oing that you know enough to</div><div class=3D"gmail_quote do_not_remove">=
update my instructions as appropriate). To update, you'd do</div><div class=
=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote do_not_r=
emove">% sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi</div><di=
v class=3D"gmail_quote do_not_remove"><br></div><div class=3D"gmail_quote d=
o_not_remove">Automating all these cases is, needless to say, tricky.<br><d=
iv class=3D"do_not_remove"> </div><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd=
ing-left:1ex">
Part of the sequencing gets into having needed to do a<br>
installworld to have a from-same-build content replacement<br>
for the likes of /efiootootaa64.efi . So installkernel<br>
already done, a reboot already done, and an installworld<br>
having been done as well in order to have something to<br>
copy over. (There are no pointers to alternate places to<br>
get copies that I know of. One can find copies in the<br>
build tree when one builds from source locally. So waiting<br>
is not really required for that context.)<br></blockquote><div class=3D"do_=
not_remove"><br></div><div class=3D"do_not_remove">Yea, I have a branch tha=
t has an 'installboot' target that updates the boot</div><div class=3D"do_n=
ot_remove">loader regardless, but given the 50-odd combinations of ways we =
can</div><div class=3D"do_not_remove">boot, it's a bit bogged down.</div><d=
iv class=3D"do_not_remove"> </div><blockquote class=3D"gmail_quote" st=
yle=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padd=
ing-left:1ex">
This also make it seem that updating ZFS pool features<br>
should wait until after a system upgrade that spans both<br>
the loader and kernel being ready for the new features,<br>
even if compatibility with other systems is not a worry.<br></blockquote><d=
iv class=3D"do_not_remove"><br></div><div class=3D"do_not_remove">Yes. ALWA=
YS update your boot blocks before zpool update.</div><div class=3D"do_not_r=
emove"> </div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p=
x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Do any of the system upgrade instructions cover such<br>
relationships? Do any of the ZFS pool upgrade instructions<br>
cover such? Does zpool or the like suggest such issues<br>
when it reports there are new features that could be<br>
enabled?<br></blockquote><div class=3D"do_not_remove"><br></div><div class=
=3D"do_not_remove">See above. :)</div><div class=3D"do_not_remove"> </=
div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left:1px solid rgb(204,204,204);padding-left:1ex">
Part of what I expect happened here was contributed to by<br>
a lack of being prompted to even think about the relevant<br>
issues, leading to a pool feature upgrade that had not<br>
been prepared for.<br></blockquote><div class=3D"do_not_remove"><br></div><=
div class=3D"do_not_remove">Yea, so far 100% of the 'help, I upgraded and n=
ow the boot loader</div><div class=3D"do_not_remove">can't see zfs pool' is=
sues have been 'I didn't upgrade my loader.efi</div><div class=3D"do_not_re=
move">properly after zpool upgrade.' It was mandatory when we had the</div>=
<div class=3D"do_not_remove">OpenZFS rebase, but it is also necessary every=
few OpenZFS</div><div class=3D"do_not_remove">updates from upstream as nne=
w features are enabled.</div><div class=3D"do_not_remove"><br></div><div cl=
ass=3D"do_not_remove">I'd love for someone to add this information to the h=
andbook, and I'd</div><div class=3D"do_not_remove">happily review such=
an effort. I have time to do a brain dump, but not</div><div class=3D"do_n=
ot_remove">to make it pretty / formatted for asciidoctor and won't for some=
time.</div><div class=3D"do_not_remove"><br></div><div class=3D"do_not_rem=
ove">Warner</div><div class=3D"do_not_remove"> </div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex">
=3D=3D=3D<br>
Mark Millard<br>
marklmi at <a href=3D"http://yahoo.com" target=3D"_blank">yahoo.com</a><br>
<br>
</blockquote></div></div>
</div></div></div>
</blockquote><br><br>Interesting. <br>Does the loader pass a hint to t=
he kernel about which loader was used to load the kernel?<div class=3D"do_n=
ot_remove"><br></div><div class=3D"do_not_remove">Regards,</div><div class=
=3D"do_not_remove">Ronald</div><div class=3D"do_not_remove"><br><br></div><=
/div></div></div><!-- TextHTMLViewer -->
<!-- MultipartAlternativeViewer -->
<!-- MessageRFC822Viewer -->
</blockquote><br>Oh I pressed send too soon. I now understand all the tools=
you already mentioned.<div>Sorry for the noise. </div><div><br></div><div>=
Regards</div><div><br><br></div></body></html>
------=_Part_11766_902161602.1662503463230--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1146974959.11767.1662503463234>