Date: Tue, 6 Sep 2022 18:13:29 -0600 From: Warner Losh <imp@bsdimp.com> To: Ronald Klop <ronald-lists@klop.ws> Cc: freebsd-arm <freebsd-arm@freebsd.org>, Mark Millard <marklmi@yahoo.com> Subject: Re: kernel update broke -current Message-ID: <CANCZdfr5nej_Gj%2BqhOdHJW%2BU9K4kof3P-axygnFOkX448UGbnQ@mail.gmail.com> In-Reply-To: <1146974959.11767.1662503463234@localhost> References: <1098569247.12297.1662500507113@localhost> <1146974959.11767.1662503463234@localhost>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000062b97b05e80b2f52
Content-Type: text/plain; charset="UTF-8"
On Tue, Sep 6, 2022 at 4:31 PM Ronald Klop <ronald-lists@klop.ws> wrote:
>
> *Van:* Ronald Klop <ronald-lists@klop.ws>
> *Datum:* 6 september 2022 23:42
> *Aan:* freebsd-arm <freebsd-arm@freebsd.org>, Warner Losh <imp@bsdimp.com>,
> Mark Millard <marklmi@yahoo.com>
> *Onderwerp:* Re: kernel update broke -current
>
>
> *Van:* Warner Losh <imp@bsdimp.com>
> *Datum:* 6 september 2022 18:13
> *Aan:* Mark Millard <marklmi@yahoo.com>
> *CC:* freebsd-arm <freebsd-arm@freebsd.org>
> *Onderwerp:* Re: kernel update broke -current
>
>
>
> On Mon, Sep 5, 2022 at 4:21 PM Mark Millard <marklmi@yahoo.com> wrote:
>
>> Warner Losh <imp_at_bsdimp.com> wrote on
>> Date: Mon, 05 Sep 2022 20:07:33 UTC :
>>
>> > On Sun, Sep 4, 2022 at 4:51 PM void <void@f-m.fm> wrote:
>> >
>> > > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:
>> > > >
>> > > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:
>> > > >> You need a newer boot loader...
>> > > >
>> > > >I was thinking - getting latest -current image, booting to it then
>> > > >copying the contents of /boot from the image onto the mounted zpool
>> ...
>> > > >
>> > > >feasible?
>> > >
>> > > Seems only EFI/ needed replacing from a recent snapshot. I thought it
>> might
>> > > be all of /boot but I was wrong. Thank you Mark!
>> > >
>> >
>> > Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot is
>> > updated
>> > when you do an installworld.
>>
>> One of the oddities of the update sequence instructions is
>> the lack of coverage of the likes of:
>>
>> Load Path: /efiootootaa64.efi
>>
>> What step of the sequencing for the overall system update?
>> When is such an update required? (Here the example would
>> be: Before rebooting when the ZFS pool(s) possibly used to
>> boot gain new features?) When is it not required to update
>> the loader in the ESP (or analogous)? Even just knowing
>> the stage at which one should do the update indicates some
>> about when to figure out if an update is needed and so
>> prompts to be ready.
>>
>
> Today:
>
> make installworld installkernel
> mount -t msdos /dev/<mumble-esp> /boot/efi
>
> and then one of three scenarios
>
> (1) You have the old boot1.efi. This will *always* be in
> ESP:EFIBOOTBOOT${ARCH}.EFI.
> (see uefi(8) for the values of ARCH). You can automatically detect this
> for most installations
> that were done since about FreeBSD 12:
> % sudo efivar | grep Boot1
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev
>
> In this case you would do:
>
> % sudo cp /boot/boot1.efi /boot/efi/efi/boot/boot${ARCH}.efi and you are
> done.
>
> Please note: If your system was installed a long time ago, you should also
> check to see if
> you have a LoaderPath variable set:
> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> /boot/loader.efi
>
> When Boot1 isn't set and it is set to the above value (or something
> similar), then you are
> booting with a really old copy of boot1.efi. You will need to update it
> and may need to arrange
> for a larger ESP since FreeBSD's boot1.efifat was a tiny image that was
> hard to grow. Steal
> space from swap for a larger ESP, etc. Documenting that is beyond the
> scope of this email.
>
> (2) You are booting with loader.efi and it is in the bug-compatibility
> place. Some UEFI BIOSes
> don't respect or can't set BootXXXX variables set by efibootmgr(8). In
> that case, many people
> are booting from the 'compatibiltiy' location for removable devices. This
> will almost always be
> a single boot scenario because you can't easily boot other systems like
> this (well, unless 'quit'
> works from the OK prompt to go to the next one on the list, but I
> digress). You will see something like:
>
> % sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> EFIBOOTBOOTX64.EFI
> (or AA64)
>
> when you are booting this way. In this case the update command is:
>
> % sudo cp /boot/loader.efi /boot/efi/efi/boot/boot${ARCH}.efi
>
> to upgrade. Some people may be doing this already on systems that can
> support efibootmgr(8) and
> they can of course upgrade to using that, though that's also beyond the
> scope of this email.
>
> (3) You are booting on a working system with a FreeBSD installed by the
> boot loader, or some other
> custom arrangement. In that case, you'll see something like:
> sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath
> EFIFREEBSDLOADER.EFI
>
> (or some other place for custom setups: I assume if you are doing that you
> know enough to
> update my instructions as appropriate). To update, you'd do
>
> % sudo cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
>
> Automating all these cases is, needless to say, tricky.
>
>
>> Part of the sequencing gets into having needed to do a
>> installworld to have a from-same-build content replacement
>> for the likes of /efiootootaa64.efi . So installkernel
>> already done, a reboot already done, and an installworld
>> having been done as well in order to have something to
>> copy over. (There are no pointers to alternate places to
>> get copies that I know of. One can find copies in the
>> build tree when one builds from source locally. So waiting
>> is not really required for that context.)
>>
>
> Yea, I have a branch that has an 'installboot' target that updates the boot
> loader regardless, but given the 50-odd combinations of ways we can
> boot, it's a bit bogged down.
>
>
>> This also make it seem that updating ZFS pool features
>> should wait until after a system upgrade that spans both
>> the loader and kernel being ready for the new features,
>> even if compatibility with other systems is not a worry.
>>
>
> Yes. ALWAYS update your boot blocks before zpool update.
>
>
>> Do any of the system upgrade instructions cover such
>> relationships? Do any of the ZFS pool upgrade instructions
>> cover such? Does zpool or the like suggest such issues
>> when it reports there are new features that could be
>> enabled?
>>
>
> See above. :)
>
>
>> Part of what I expect happened here was contributed to by
>> a lack of being prompted to even think about the relevant
>> issues, leading to a pool feature upgrade that had not
>> been prepared for.
>>
>
> Yea, so far 100% of the 'help, I upgraded and now the boot loader
> can't see zfs pool' issues have been 'I didn't upgrade my loader.efi
> properly after zpool upgrade.' It was mandatory when we had the
> OpenZFS rebase, but it is also necessary every few OpenZFS
> updates from upstream as nnew features are enabled.
>
> I'd love for someone to add this information to the handbook, and I'd
> happily review such an effort. I have time to do a brain dump, but not
> to make it pretty / formatted for asciidoctor and won't for some time.
>
> Warner
>
>
>> ===
>> Mark Millard
>> marklmi at yahoo.com
>>
>>
>
> Interesting.
> Does the loader pass a hint to the kernel about which loader was used to
> load the kernel?
>
> Regards,
> Ronald
>
>
>
> Oh I pressed send too soon. I now understand all the tools you already
> mentioned.
> Sorry for the noise.
>
boot1.efi and loader.efi set the UEFI environment variables I used above.
No further hints
are passed into the kernel...
Warner
--00000000000062b97b05e80b2f52
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Tue, Sep 6, 2022 at 4:31 PM Ronald=
Klop <<a href=3D"mailto:ronald-lists@klop.ws">ronald-lists@klop.ws</a>&=
gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0=
px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br><=
p><small><strong>Van:</strong> Ronald Klop <<a href=3D"mailto:ronald-lis=
ts@klop.ws" target=3D"_blank">ronald-lists@klop.ws</a>><br><strong>Datum=
:</strong> 6 september 2022 23:42<br><strong>Aan:</strong> freebsd-arm <=
<a href=3D"mailto:freebsd-arm@freebsd.org" target=3D"_blank">freebsd-arm@fr=
eebsd.org</a>>, Warner Losh <<a href=3D"mailto:imp@bsdimp.com" target=
=3D"_blank">imp@bsdimp.com</a>>, Mark Millard <<a href=3D"mailto:mark=
lmi@yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>><br><strong>Onder=
werp:</strong> Re: kernel update broke -current<br></small></p><blockquote =
style=3D"margin-left:5px;border-left:3px solid rgb(204,204,204);margin-righ=
t:0px;padding-left:5px"><div id=3D"m_-4272484444305459964P">
<div>
<div id=3D"m_-4272484444305459964P.P.P">
<br><p><small><strong>Van:</strong> Warner Losh <<a href=3D"mailto:imp@b=
sdimp.com" target=3D"_blank">imp@bsdimp.com</a>><br><strong>Datum:</stro=
ng> 6 september 2022 18:13<br><strong>Aan:</strong> Mark Millard <<a hre=
f=3D"mailto:marklmi@yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>><=
br><strong>CC:</strong> freebsd-arm <<a href=3D"mailto:freebsd-arm@freeb=
sd.org" target=3D"_blank">freebsd-arm@freebsd.org</a>><br><strong>Onderw=
erp:</strong> Re: kernel update broke -current<br></small></p><blockquote s=
tyle=3D"margin-left:5px;border-left:3px solid rgb(204,204,204);margin-right=
:0px;padding-left:5px"><div id=3D"m_-4272484444305459964P">
<div>
<div id=3D"m_-4272484444305459964P.P.P">
<div><div><br></div><br><div class=3D"gmail_quote"><div class=3D"gmail_attr=
">On Mon, Sep 5, 2022 at 4:21 PM Mark Millard <<a href=3D"mailto:marklmi=
@yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>> wrote:<br></div><bl=
ockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-lef=
t:1px solid rgb(204,204,204);padding-left:1ex">Warner Losh <<a href=3D"h=
ttp://imp_at_bsdimp.com" target=3D"_blank">imp_at_bsdimp.com</a>> wrote =
on<br>
Date: Mon, 05 Sep 2022 20:07:33 UTC :<br>
<br>
> On Sun, Sep 4, 2022 at 4:51 PM void <<a href=3D"mailto:void@f-m.fm"=
target=3D"_blank">void@f-m.fm</a>> wrote:<br>
> <br>
> > On Sun, Sep 04, 2022 at 07:31:59PM +0000, void wrote:<br>
> > ><br>
> > >On Sun, 4 Sep 2022, at 19:07, Warner Losh wrote:<br>
> > >> You need a newer boot loader...<br>
> > ><br>
> > >I was thinking - getting latest -current image, booting to it=
then<br>
> > >copying the contents of /boot from the image onto the mounted=
zpool ...<br>
> > ><br>
> > >feasible?<br>
> ><br>
> > Seems only EFI/ needed replacing from a recent snapshot. I though=
t it might<br>
> > be all of /boot but I was wrong. Thank you Mark!<br>
> ><br>
> <br>
> Yes. You'll need to update EFI/BOOT on your ESP. The rest of /boot=
is<br>
> updated<br>
> when you do an installworld.<br>
<br>
One of the oddities of the update sequence instructions is<br>
the lack of coverage of the likes of:<br>
<br>
Load Path: /efiootootaa64.efi<br>
<br>
What step of the sequencing for the overall system update?<br>
When is such an update required? (Here the example would<br>
be: Before rebooting when the ZFS pool(s) possibly used to<br>
boot gain new features?) When is it not required to update<br>
the loader in the ESP (or analogous)? Even just knowing<br>
the stage at which one should do the update indicates some<br>
about when to figure out if an update is needed and so<br>
prompts to be ready.<br></blockquote><div><br></div><div>Today:</div><div><=
br></div><div>make installworld installkernel</div><div>mount -t msdos /dev=
/<mumble-esp> /boot/efi</div><div><br></div><div>and then one of thre=
e scenarios</div><div><br></div><div>(1) You have the old boot1.efi. This w=
ill *always* be in ESP:EFIBOOTBOOT${ARCH}.EFI.</div><div>(see uefi(8) for t=
he values of ARCH). You can automatically detect this for most installation=
s</div><div>that were done since about FreeBSD 12:</div><div>% sudo efivar =
| grep Boot1</div><div>cfee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Path<br>cf=
ee69ad-a0de-47a9-93a8-f63106f8ae99-Boot1Dev<br></div><div><br></div><div>In=
this case you would do:</div><div><br></div><div>% sudo cp /boot/boot1.efi=
/boot/efi/efi/boot/boot${ARCH}.efi and you are done.</div><div><br></div><=
div>Please note: If your system was installed a long time ago, you should a=
lso check to see if</div><div>you have a LoaderPath variable set:</div><div=
>% sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br></d=
iv><div>/boot/loader.efi</div><div><br></div><div>When Boot1 isn't set =
and it is set to the above value (or something similar), then you are</div>=
<div>booting with a really old copy of boot1.efi. You will need to update i=
t and may need to arrange</div><div>for a larger ESP since FreeBSD's bo=
ot1.efifat was a tiny image that was hard to grow. Steal</div><div>space fr=
om swap for a larger ESP, etc. Documenting that is beyond the scope of this=
email.</div><div><br></div><div>(2) You are booting with loader.efi and it=
is in the bug-compatibility place. Some UEFI BIOSes</div><div>don't re=
spect or can't set BootXXXX variables set by efibootmgr(8). In that cas=
e, many people</div><div>are booting from the 'compatibiltiy' locat=
ion for removable=C2=A0devices. This will almost always be</div><div>a sing=
le boot scenario because you can't easily boot other systems like this =
(well, unless 'quit'</div><div>works from the OK prompt to go to th=
e next one on the list, but I digress). You will see something like:</div><=
div><br></div><div>% sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99=
-LoaderPath</div>cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>EFIBOOT=
BOOTX64.EFI</div><div class=3D"gmail_quote">(or AA64)</div><div class=3D"gm=
ail_quote"><br></div><div class=3D"gmail_quote">when you are booting this w=
ay. In this case the update command is:</div><div class=3D"gmail_quote"><br=
></div><div class=3D"gmail_quote">% sudo cp /boot/loader.efi /boot/efi/efi/=
boot/boot${ARCH}.efi</div><div class=3D"gmail_quote"><br></div><div class=
=3D"gmail_quote">to upgrade. Some people may be doing this already on syste=
ms that can support efibootmgr(8) and</div><div class=3D"gmail_quote">they =
can of course upgrade to using that, though that's also beyond the scop=
e of this email.</div><div class=3D"gmail_quote"><br></div><div class=3D"gm=
ail_quote">(3) You are booting on a working system with a FreeBSD installed=
by the boot loader, or some other</div><div class=3D"gmail_quote">custom a=
rrangement. In that case, you'll see something like:</div><div class=3D=
"gmail_quote">sudo efivar --utf cfee69ad-a0de-47a9-93a8-f63106f8ae99-Loader=
Path<br>cfee69ad-a0de-47a9-93a8-f63106f8ae99-LoaderPath<br>EFIFREEBSDLOADER=
.EFI</div><div class=3D"gmail_quote"><br></div><div class=3D"gmail_quote">(=
or some other place for custom setups: I assume if you are doing that you k=
now enough to</div><div class=3D"gmail_quote">update my instructions as app=
ropriate). To update, you'd do</div><div class=3D"gmail_quote"><br></di=
v><div class=3D"gmail_quote">% sudo cp /boot/loader.efi /boot/efi/efi/freeb=
sd/loader.efi</div><div class=3D"gmail_quote"><br></div><div class=3D"gmail=
_quote">Automating all these cases is, needless to say, tricky.<br><div>=C2=
=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Part of the sequencing gets into having needed to do a<br>
installworld to have a from-same-build content replacement<br>
for the likes of /efiootootaa64.efi . So installkernel<br>
already done, a reboot already done, and an installworld<br>
having been done as well in order to have something to<br>
copy over. (There are no pointers to alternate places to<br>
get copies that I know of. One can find copies in the<br>
build tree when one builds from source locally. So waiting<br>
is not really required for that context.)<br></blockquote><div><br></div><d=
iv>Yea, I have a branch that has an 'installboot' target that updat=
es the boot</div><div>loader regardless, but given the 50-odd combinations =
of ways we can</div><div>boot, it's a bit bogged down.</div><div>=C2=A0=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b=
order-left:1px solid rgb(204,204,204);padding-left:1ex">
This also make it seem that updating ZFS pool features<br>
should wait until after a system upgrade that spans both<br>
the loader and kernel being ready for the new features,<br>
even if compatibility with other systems is not a worry.<br></blockquote><d=
iv><br></div><div>Yes. ALWAYS update your boot blocks before zpool update.<=
/div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px=
0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Do any of the system upgrade instructions cover such<br>
relationships? Do any of the ZFS pool upgrade instructions<br>
cover such? Does zpool or the like suggest such issues<br>
when it reports there are new features that could be<br>
enabled?<br></blockquote><div><br></div><div>See above. :)</div><div>=C2=A0=
</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;b=
order-left:1px solid rgb(204,204,204);padding-left:1ex">
Part of what I expect happened here was contributed to by<br>
a lack of being prompted to even think about the relevant<br>
issues, leading to a pool feature upgrade that had not<br>
been prepared for.<br></blockquote><div><br></div><div>Yea, so far 100% of =
the 'help, I upgraded and now the boot loader</div><div>can't see z=
fs pool' issues have been 'I didn't upgrade my loader.efi</div>=
<div>properly after zpool upgrade.' It was mandatory when we had the</d=
iv><div>OpenZFS rebase, but it is also necessary every few OpenZFS</div><di=
v>updates from upstream as nnew features are enabled.</div><div><br></div><=
div>I'd love for someone to add this information to the handbook, and I=
'd</div><div>happily=C2=A0review such an effort. I have time to do a br=
ain dump, but not</div><div>to make it pretty / formatted for asciidoctor a=
nd won't for some time.</div><div><br></div><div>Warner</div><div>=C2=
=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left:1px solid rgb(204,204,204);padding-left:1ex">
=3D=3D=3D<br>
Mark Millard<br>
marklmi at <a href=3D"http://yahoo.com" target=3D"_blank">yahoo.com</a><br>
<br>
</blockquote></div></div>
</div></div></div>
</blockquote><br><br>Interesting.=C2=A0<br>Does the loader pass a hint to t=
he kernel about which loader was used to load the kernel?<div><br></div><di=
v>Regards,</div><div>Ronald</div><div><br><br></div></div></div></div>
</blockquote><br>Oh I pressed send too soon. I now understand all the tools=
you already mentioned.<div>Sorry for the noise.</div></blockquote><div><br=
></div><div>boot1.efi and loader.efi set the UEFI=C2=A0environment variable=
s I used above. No further hints</div><div>are passed into the kernel...</d=
iv><div><br></div><div>Warner</div></div></div>
--00000000000062b97b05e80b2f52--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfr5nej_Gj%2BqhOdHJW%2BU9K4kof3P-axygnFOkX448UGbnQ>