Date: Sat, 3 Jun 2023 14:46:06 -0600 From: Warner Losh <imp@bsdimp.com> To: FreeBSD User <freebsd@walstatt-de.de> Cc: FreeBSD CURRENT <freebsd-current@freebsd.org> Subject: Re: WITH_BEARSSL: -8112 bytes available Message-ID: <CANCZdfp%2BNB%2Bhy_ofKWq%2B8=rrgycYvnP9e3XSxcjRfBoUFWT5Gw@mail.gmail.com> In-Reply-To: <20230603170316.74ea78f1@thor.intern.walstatt.dynvpn.de> References: <20230529105854.1903226d@thor.intern.walstatt.dynvpn.de> <CANCZdfo-PojzQrL5ppxSLLfp%2B5f-ToqQD_N3p3aYtEKBodQ7xQ@mail.gmail.com> <20230603170316.74ea78f1@thor.intern.walstatt.dynvpn.de>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000d945db05fd3fc245
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Sat, Jun 3, 2023 at 9:03=E2=80=AFAM FreeBSD User <freebsd@walstatt-de.de=
> wrote:
> Am Wed, 31 May 2023 12:15:12 -0600
> Warner Losh <imp@bsdimp.com> schrieb:
>
> Sorry for the late response.
>
>
> > On Mon, May 29, 2023 at 2:59=E2=80=AFAM FreeBSD User <freebsd@walstatt-=
de.de>
> wrote:
> >
> > > Hello,
> > >
> > > on CURRENT, enabling in /etc/src.conf
> > >
> > > WITH_BEARSSL=3D
> > >
> > > seems to result in a slightly enlarged loader binary, which seems to
> have
> > > a fixed size
> > > supposed on the error I get. See below.
> > >
> > > The system is amd64 (64 bit), for the record.
> > >
> > > Somewhere in the past developers mentioned this upcoming problem and
> > > provided a knob to adjust
> > > the used size - I forgot about that knob and I couldn't find it even =
in
> > > the loader docs - or
> > > looked at the wrong places.
> > >
> > > Can someone help me out here?
> > >
> > > The first error stops compileing world/kernel, but taking a second ru=
n,
> > > the error goes away.
> > >
> > > Kind regards and thanks in advance,
> > >
> > > oh
> > >
> > >
> > >
> > > [...]
> > > --- all_subdir_stand/efi ---
> > > SOURCE_DATE_EPOCH=3D1451606400 objcopy -j .peheader -j .text -j .sda=
ta
> -j
> > > .data -j .dynamic -j
> > > .dynsym -j .rel.dyn -j .rela.dyn -j .reloc -j .eh_frame -j
> > > set_Xcommand_set -j
> > > set_Xficl_compile_set --output-target=3Defi-app-x86_64 loader_4th.sy=
m
> > > loader_4th.efi ---
> > > all_subdir_stand/i386 ---
> > >
> > > -8112 bytes available 7.71 real 12.86 user 3.08 sys
> >
> >
> > bummer. I hate it when it's that close.
> >
> > You can try setting LOADERSIZE=3D560000 in your environment. We current=
ly
> set
> > the maximum to 550,000
>
> I tried to find find anything related to LOADER or SIZE in the docs. I
> remember you mentioned
> the existence of that variable months ago, but with no clue what to look
> after, it is almost
> impossible yo figure out as a non developer what the right knob might be.
>
Fair point.
> A grep -r on /usr/src shows up only in
>
> [...]
> ./stand/i386/loader/Makefile:LOADERSIZE?=3D 550000 # Large=
st
> known safe size for
> loader.bin
> ./stand/i386/loader/Makefile: @set -- `ls -l ${.TARGET}` ;
> x=3D$$((${LOADERSIZE}-$$5)); \
> [...]
>
> There is no sign/trace of it in any man page related to loader and
> sibblings. I found the
> variable rather quickly after knowing what to look after.
>
To be fair, this is a very under documented area, semi on purpose... But
that's a good point that it is
under documented. And now that the size is creeping up and we have options
that explode things,
maybe it should be better documented.
> > bytes because that's the most conservative number due to variation in t=
he
> > available BIOS space available.
> > This likely can be set even higher if you don't have add-in cards that
> are
> > consuming space in the lower 640k
> > of memory. 640k is the absolute limit, but you need 20-30k of stack for
> the
> > loader so pushing this much past
> > 625,000 or maybe 630,000 increases the risk of run-time crashes as the
> > stack smashes through the top of
> > the loader program. You may also have to disable the lua build, since i=
t
> > uses more stack and is just a smidge
> > larger than the forth build. _simp will be the smallest of them all. On
> my
> > system, without bearssl, I see:
> > -r-xr-xr-x 3 root wheel 503808 May 22 15:25 /boot/loader_lua
> > -r-xr-xr-x 1 root wheel 446464 May 22 15:25 /boot/loader_4th
> > -r-xr-xr-x 1 root wheel 385024 May 22 15:25 /boot/loader_simp
>
> In my case, with supposedly blewn up loader size by BEARSSL enables, it i=
s:
>
> -r-xr-xr-x 1 root wheel - 503808 3 Juni 12:33 /boot/loader_4th*
> -r-xr-xr-x 1 root wheel - 643584 3 Juni 12:33 /boot/loader_4th.efi*
> -r-xr-xr-x 1 root wheel - 503808 3 Juni 07:45 /boot/loader_4th.old*
> -r-xr-xr-x 3 root wheel - 569344 3 Juni 12:33 /boot/loader_lua*
> -r-xr-xr-x 2 root wheel - 737280 3 Juni 12:33 /boot/loader_lua.efi*
> -r-xr-xr-x 1 root wheel - 569344 3 Juni 07:45 /boot/loader_lua.old*
> -r-xr-xr-x 1 root wheel - 446464 3 Juni 12:33 /boot/loader_simp*
> -r-xr-xr-x 1 root wheel - 589312 3 Juni 12:33 /boot/loader_simp.efi*
> -r-xr-xr-x 1 root wheel - 446464 3 Juni 07:45 /boot/loader_simp.old*
>
> on FreeBSD 14.0-CURRENT #58 main-n263387-556b43492297: Fri Jun 2 20:19:5=
5
> CEST 2023 amd64.
> > which suggests a ~60k bump for adding forth and ~115k bump for lua. So
> the
> > 560,000 may need to be 625,000
> > which is living life on the edge for 4th, and simply too big for lua.
> >
> > I'd be open to adding docs on this, since I don't think this option is
> > currently documented since I added it
> > to experiment around with a good value.
>
> See above, personally I'd like to see some hints on that variable, even i=
f
> I do not fiddle
> around with it.
>
OK. that makes sense. In fact, it may make sense to disable the build of
the BIOS
loader entirely sometimes, which we can't easily do today.
> >
> > And no, I really do not want to support 'loadable modules'. BIOS bootin=
g
> is
> > on the way out, and people
> > that want to do complex stuff in the boot loader will simply have to do
> > that in UEFI or maybe kboot/LinuxBoot.
> > There's low RoI on adding this complexity, imho. We'd be better off,
> imho,
> > making things like the graphics
> > console optional since the fonts and code for that free up about 30k in
> > stupid experiments that I've done
> > (it's hard since vidconsole has a lot of calls into the graphics system
> > that aren't optional and easy to disable,
> > so I've had to do hack and slash to produce a super ugly result that is
> > only suggestive of the final savings):
> > -rw-r--r-- 1 imp imp 352256 May 31 12:04 loader_simp
> > I don't know if I slashed too much, or not enough since the code is
> rather
> > hard to separate out, so if you
> > really wanted to go down this path, it would take a lot of work and
> patient
> > understanding to make it so with
> > the low end of savings 20k and the high end on the order of maybe 40k.
> >
> > There's likely other ways to conserve space. We've not had space issues
> > with loader, et al, in the past,
> > so it's not well setup for subsetting. Though the different filesystem
> > support might also net you a fair amount:
> > LOADER_NET_SUPPORT?=3D yes
> > LOADER_NFS_SUPPORT?=3D yes
> > LOADER_TFTP_SUPPORT?=3D yes
> > LOADER_CD9660_SUPPORT?=3D yes
> > LOADER_EXT2FS_SUPPORT?=3D yes
> > LOADER_MSDOS_SUPPORT?=3D yes
> > LOADER_UFS_SUPPORT?=3D yes
> > LOADER_GZIP_SUPPORT?=3D yes
> > LOADER_BZIP2_SUPPORT?=3D yes
> > as would compiling w/o ZFS, which uses its own method (eg
> > WITHOUT_LOADER_ZFS). Tuning the loader
> > at this level does start to get into the weeds a bit, but can offer ~40=
k
> > savings turning off all but NET and UFS:
> > -rw-r--r-- 1 imp imp 344064 May 31 12:11 loader_simp
> > you get even about ~100k when you disable ZFS support with
> > -DWITHOUT_LOADER_ZFS:
> > -rw-r--r-- 1 imp imp 241664 May 31 12:12 loader_simp
> > (both of these are with the graphics console enabled without the silly
> > hacks to see how much that takes up).
> > Without the extras and ZFS, you might have bearssl and lua together
> even...
> >
> > Hope this helps.
>
> Very much appreciated, thanks.
>
> I'm not so much concerned by the space sonsumption or a corrupted boot
> process, I don't even
> know whether I really need this i386 stuff mentioned here, since all of m=
y
> boxes are UEFI,
> except the APU2, which are driven by SeaBIOS and bootin non UEFI.
>
Yea, I'm starting to think that the best way forward should include some
way to turn off
the BIOS loader if you know you won't need it and it's too big.
> The cosmetic issue is that whenever anything reagrding the loader gets
> touched by the
> compiler, out build system drops out with an error - a second run then
> performs clean (which
> is a kind of weird I guess).
>
Yea, the check should cause it to be deleted when it fails, but doesn't.
Warner
> >
> > Warner
>
> Thank you very much,
>
> Kind regards
> Oliver
>
>
> --
> O. Hartmann
>
--000000000000d945db05fd3fc245
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote">=
<div dir=3D"ltr" class=3D"gmail_attr">On Sat, Jun 3, 2023 at 9:03=E2=80=AFA=
M FreeBSD User <<a href=3D"mailto:freebsd@walstatt-de.de">freebsd@walsta=
tt-de.de</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D=
"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le=
ft:1ex">Am Wed, 31 May 2023 12:15:12 -0600<br>
Warner Losh <<a href=3D"mailto:imp@bsdimp.com" target=3D"_blank">imp@bsd=
imp.com</a>> schrieb:<br>
<br>
Sorry for the late response.<br>
<br>
<br>
> On Mon, May 29, 2023 at 2:59=E2=80=AFAM FreeBSD User <<a href=3D"ma=
ilto:freebsd@walstatt-de.de" target=3D"_blank">freebsd@walstatt-de.de</a>&g=
t; wrote:<br>
> <br>
> > Hello,<br>
> ><br>
> > on CURRENT, enabling in /etc/src.conf<br>
> ><br>
> > WITH_BEARSSL=3D<br>
> ><br>
> > seems to result in a slightly enlarged loader binary, which seems=
to have<br>
> > a fixed size<br>
> > supposed on the error I get. See below.<br>
> ><br>
> > The system is amd64 (64 bit), for the record.<br>
> ><br>
> > Somewhere in the past developers mentioned this upcoming problem =
and<br>
> > provided a knob to adjust<br>
> > the used size - I forgot about that knob and I couldn't find =
it even in<br>
> > the loader docs - or<br>
> > looked at the wrong places.<br>
> ><br>
> > Can someone help me out here?<br>
> ><br>
> > The first error stops compileing world/kernel, but taking a secon=
d run,<br>
> > the error goes away.<br>
> ><br>
> > Kind regards and thanks in advance,<br>
> ><br>
> > oh<br>
> ><br>
> ><br>
> ><br>
> > [...]<br>
> > --- all_subdir_stand/efi ---<br>
> > SOURCE_DATE_EPOCH=3D1451606400=C2=A0 objcopy -j .peheader -j .tex=
t -j .sdata -j<br>
> > .data=C2=A0 -j .dynamic -j<br>
> > .dynsym -j .rel.dyn=C2=A0 -j .rela.dyn -j .reloc -j .eh_frame -j<=
br>
> > set_Xcommand_set=C2=A0 -j<br>
> > set_Xficl_compile_set=C2=A0 --output-target=3Defi-app-x86_64 load=
er_4th.sym<br>
> > loader_4th.efi ---<br>
> > all_subdir_stand/i386 ---<br>
> ><br>
> > -8112 bytes available 7.71 real=C2=A0 =C2=A0 =C2=A0 =C2=A0 12.86 =
user=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03.08 sys=C2=A0 <br>
> <br>
> <br>
> bummer. I hate it when it's that close.<br>
> <br>
> You can try setting LOADERSIZE=3D560000 in your environment. We curren=
tly set<br>
> the maximum to 550,000<br>
<br>
I tried to find find anything related to LOADER or SIZE in the docs. I reme=
mber you mentioned<br>
the existence of that variable months ago, but with no clue what to look af=
ter, it is almost<br>
impossible yo figure out as a non developer what the right knob might be.<b=
r></blockquote><div><br></div><div>Fair point.</div><div>=C2=A0</div><block=
quote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1=
px solid rgb(204,204,204);padding-left:1ex">
A grep -r on /usr/src shows up only in <br>
<br>
[...]<br>
./stand/i386/loader/Makefile:LOADERSIZE?=3D=C2=A0 =C2=A0 =C2=A0 =C2=A055000=
0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Largest known safe size for<br>
loader.bin<br>
./stand/i386/loader/Makefile:=C2=A0 =C2=A0@set -- `ls -l ${.TARGET}` ; x=3D=
$$((${LOADERSIZE}-$$5)); \<br>
[...]<br>
<br>
There is no sign/trace of it in any man page related to loader and sibbling=
s. I found the<br>
variable rather quickly after knowing what to look after.<br></blockquote><=
div><br></div><div>To be fair, this is a very under documented area, semi o=
n purpose...=C2=A0 But that's a good point that it is</div><div>under d=
ocumented. And now that the size is creeping=C2=A0up and we have options th=
at explode things,</div><div>maybe it should be better documented.</div><di=
v>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> bytes because that's the most conservative number due to variation=
in the<br>
> available BIOS space available.<br>
> This likely can be set even higher if you don't have add-in cards =
that are<br>
> consuming space in the lower 640k<br>
> of memory. 640k is the absolute limit, but you need 20-30k of stack fo=
r the<br>
> loader so pushing this much past<br>
> 625,000 or maybe 630,000 increases the risk of run-time crashes as the=
<br>
> stack smashes through the top of<br>
> the loader program. You may also have to disable the lua build, since =
it<br>
> uses more stack and is just a smidge<br>
> larger than the forth build. _simp will be the smallest of them all. O=
n my<br>
> system, without bearssl, I see:<br>
> -r-xr-xr-x=C2=A0 3 root=C2=A0 wheel=C2=A0 503808 May 22 15:25 /boot/lo=
ader_lua<br>
> -r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 446464 May 22 15:25 /boot/lo=
ader_4th<br>
> -r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 385024 May 22 15:25 /boot/lo=
ader_simp<br>
<br>
In my case, with supposedly blewn up loader size by BEARSSL enables, it is:=
<br>
<br>
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 503808=C2=A0 3 Juni 12:33 /boot=
/loader_4th*<br>
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 643584=C2=A0 3 Juni 12:33 /boot=
/loader_4th.efi*<br>
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 503808=C2=A0 3 Juni 07:45 /boot=
/loader_4th.old*<br>
-r-xr-xr-x=C2=A0 3 root=C2=A0 wheel=C2=A0 - 569344=C2=A0 3 Juni 12:33 /boot=
/loader_lua*<br>
-r-xr-xr-x=C2=A0 2 root=C2=A0 wheel=C2=A0 - 737280=C2=A0 3 Juni 12:33 /boot=
/loader_lua.efi*<br>
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 569344=C2=A0 3 Juni 07:45 /boot=
/loader_lua.old*<br>
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 446464=C2=A0 3 Juni 12:33 /boot=
/loader_simp*<br>
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 589312=C2=A0 3 Juni 12:33 /boot=
/loader_simp.efi*<br>
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 446464=C2=A0 3 Juni 07:45 /boot=
/loader_simp.old*<br>
<br>
on FreeBSD 14.0-CURRENT #58 main-n263387-556b43492297: Fri Jun=C2=A0 2 20:1=
9:55 CEST 2023 amd64.<br>
> which suggests a ~60k bump for adding forth and ~115k bump for lua. So=
the<br>
> 560,000 may need to be 625,000<br>
> which is living life on the edge for 4th, and simply too big for lua.<=
br>
> <br>
> I'd be open to adding docs on this, since I don't think this o=
ption is<br>
> currently documented since I added it<br>
> to experiment around with a good value.<br>
<br>
See above, personally I'd like to see some hints on that variable, even=
if I do not fiddle<br>
around with it.<br></blockquote><div><br></div><div>OK. that makes sense. I=
n fact, it may make sense to disable the build of the BIOS</div><div>loader=
entirely sometimes, which we can't easily do today.</div><div>=C2=A0</=
div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bor=
der-left:1px solid rgb(204,204,204);padding-left:1ex">
> <br>
> And no, I really do not want to support 'loadable modules'. BI=
OS booting is<br>
> on the way out, and people<br>
> that want to do complex stuff in the boot loader will simply have to d=
o<br>
> that in UEFI or maybe kboot/LinuxBoot.<br>
> There's low RoI on adding this complexity, imho. We'd be bette=
r off, imho,<br>
> making things like the graphics<br>
> console optional since the fonts and code for that free up about 30k i=
n<br>
> stupid experiments that I've done<br>
> (it's hard since vidconsole has a lot of calls into the graphics s=
ystem<br>
> that aren't optional and easy to disable,<br>
> so I've had to do hack and slash to produce a super ugly result th=
at is<br>
> only suggestive of the final savings):<br>
> -rw-r--r--=C2=A0 1 imp=C2=A0 imp=C2=A0 352256 May 31 12:04 loader_simp=
<br>
> I don't know if I slashed too much, or not enough since the code i=
s rather<br>
> hard to separate out, so if you<br>
> really wanted to go down this path, it would take a lot of work and pa=
tient<br>
> understanding to make it so with<br>
> the low end of savings 20k and the high end on the order of maybe 40k.=
<br>
> <br>
> There's likely other ways to conserve space. We've not had spa=
ce issues<br>
> with loader, et al, in the past,<br>
> so it's not well setup for subsetting. Though the different filesy=
stem<br>
> support might also net you a fair amount:<br>
> LOADER_NET_SUPPORT?=3D=C2=A0 =C2=A0 yes<br>
> LOADER_NFS_SUPPORT?=3D=C2=A0 =C2=A0 yes<br>
> LOADER_TFTP_SUPPORT?=3D=C2=A0 =C2=A0yes<br>
> LOADER_CD9660_SUPPORT?=3D yes<br>
> LOADER_EXT2FS_SUPPORT?=3D yes<br>
> LOADER_MSDOS_SUPPORT?=3D=C2=A0 yes<br>
> LOADER_UFS_SUPPORT?=3D=C2=A0 =C2=A0 yes<br>
> LOADER_GZIP_SUPPORT?=3D=C2=A0 =C2=A0yes<br>
> LOADER_BZIP2_SUPPORT?=3D=C2=A0 yes<br>
> as would compiling w/o ZFS, which uses its own method (eg<br>
> WITHOUT_LOADER_ZFS). Tuning the loader<br>
> at this level does start to get into the weeds a bit, but can offer ~4=
0k<br>
> savings turning off all but NET and UFS:<br>
> -rw-r--r--=C2=A0 1 imp=C2=A0 imp=C2=A0 344064 May 31 12:11 loader_simp=
<br>
> you get even about ~100k when you disable ZFS support with<br>
> -DWITHOUT_LOADER_ZFS:<br>
> -rw-r--r--=C2=A0 1 imp=C2=A0 imp=C2=A0 241664 May 31 12:12 loader_simp=
<br>
> (both of these are with the graphics console enabled without the silly=
<br>
> hacks to see how much that takes up).<br>
> Without the extras and ZFS, you might have bearssl and lua together ev=
en...<br>
> <br>
> Hope this helps.<br>
<br>
Very much appreciated, thanks.<br>
<br>
I'm not so much concerned by the space sonsumption or a corrupted boot =
process, I don't even<br>
know whether I really need this i386 stuff mentioned here, since all of my =
boxes are UEFI,<br>
except the APU2, which are driven by SeaBIOS and bootin=C2=A0 non UEFI.<br>=
</blockquote><div><br></div><div>Yea,=C2=A0 I'm starting to think that =
the best way forward should include some way to turn off</div><div>the BIOS=
loader if you know you won't need it and it's too big.</div><div>=
=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0=
.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The cosmetic issue is that whenever anything reagrding the loader gets touc=
hed by the<br>
compiler, out build system drops out with=C2=A0 an error - a second run the=
n performs clean (which<br>
is a kind of weird I guess).<br></blockquote><div><br></div><div>Yea, the c=
heck should cause it to be deleted when it fails, but doesn't.</div><di=
v><br></div><div>Warner</div><div>=C2=A0</div><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20=
4);padding-left:1ex">
> <br>
> Warner<br>
<br>
Thank you very much,<br>
<br>
Kind regards<br>
Oliver<br>
<br>
<br>
-- <br>
O. Hartmann<br>
</blockquote></div></div>
--000000000000d945db05fd3fc245--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfp%2BNB%2Bhy_ofKWq%2B8=rrgycYvnP9e3XSxcjRfBoUFWT5Gw>