From nobody Sat Jun 3 20:46:06 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QYX2z3ZgXz4YVD4 for ; Sat, 3 Jun 2023 20:46:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QYX2z1Y9Sz44tC for ; Sat, 3 Jun 2023 20:46:19 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x536.google.com with SMTP id 4fb4d7f45d1cf-5149c76f4dbso5033232a12.1 for ; Sat, 03 Jun 2023 13:46:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20221208.gappssmtp.com; s=20221208; t=1685825177; x=1688417177; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=9gUQG10ntlK5mtZ7D2uVzokC0LtBpwoNs89BZ1rIrr0=; b=ENvycvpahhaQrX+92dPSTY0zmyR/abX1MT3j7aWg4bjT7ExowIoUhswzeEH6isBwIl X9lF3zxYRoyzRUNCwX4o3oabMNAWtMRkOUXS3q2UrzbwXkWiLyUntO/eZXXIjGBkaO2F nfXHuQ1+SIAVYMrzl+3wgPwiarg1JyJPp3XgC4AO34+SrDV7RqpbTxehcGkD5krd/pFc NpTItZmy7MEOAUU1s6fqpuaP3rXaTNLwqM4l98DaKDhTmWOB6lNiDlzLtXJy7/rpBkzL R5Um37QkU/3kKS+wnkCYzdM6eVYK7/n5+ZO4x+SzrbDJP+fSbNGI1RvbvDX/+Eon/H05 dBXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685825177; x=1688417177; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=9gUQG10ntlK5mtZ7D2uVzokC0LtBpwoNs89BZ1rIrr0=; b=ISNG2g42sb6Ka9GMr2B4dStQTOazpuVlGNIJeYvJV10GNp4wDNRWC++bQKkpR0RNdI uIQ6GiPi08FBqWI/i0PEP/rfd2oG18QcJopWDS5A66WroXN29cBIeuIlkccd6DYJAvjb Tk4Rxi5yb7zJQ5bTJ3vCqouQKWOPJqUTG+XSb9N/3oHe7WPjEQnjh0QEYWIykVpXMUG6 WNSvD4tt6wBYtjwtJuKxiactw4hqCBzRr0hZNfzhTUIn+vPPxZkHBc1+kaX3H3jAuQ5n mMrBaowU64Vy1N60msIqqa0u7pjQzsf/grxdSJ60jc4OIpDqVytE7jdj10Zb4c7vpJhR XHew== X-Gm-Message-State: AC+VfDxHrjN7LotkpQoLZ0fKoGAa96JzSzEW/WkJDrowDFMBY+OIPkXS K5zUt4PUt5Wg1L/7zuuNACo8cVu2SQSvNd4NjwiXLwk3MuuiLzgo X-Google-Smtp-Source: ACHHUZ49uzNAhYE5cn58soaFi5X6cWXIID3MKplxiLy/aaDd1sfOlP4CZHwL4X5BlQRuikpn2LnJxZZAIX37uwHxgGM= X-Received: by 2002:aa7:c2c7:0:b0:50b:caae:784 with SMTP id m7-20020aa7c2c7000000b0050bcaae0784mr4302747edp.20.1685825177333; Sat, 03 Jun 2023 13:46:17 -0700 (PDT) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 References: <20230529105854.1903226d@thor.intern.walstatt.dynvpn.de> <20230603170316.74ea78f1@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20230603170316.74ea78f1@thor.intern.walstatt.dynvpn.de> From: Warner Losh Date: Sat, 3 Jun 2023 14:46:06 -0600 Message-ID: Subject: Re: WITH_BEARSSL: -8112 bytes available To: FreeBSD User Cc: FreeBSD CURRENT Content-Type: multipart/alternative; boundary="000000000000d945db05fd3fc245" X-Rspamd-Queue-Id: 4QYX2z1Y9Sz44tC X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000d945db05fd3fc245 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Jun 3, 2023 at 9:03=E2=80=AFAM FreeBSD User wrote: > Am Wed, 31 May 2023 12:15:12 -0600 > Warner Losh schrieb: > > Sorry for the late response. > > > > On Mon, May 29, 2023 at 2:59=E2=80=AFAM FreeBSD User > wrote: > > > > > Hello, > > > > > > on CURRENT, enabling in /etc/src.conf > > > > > > WITH_BEARSSL=3D > > > > > > seems to result in a slightly enlarged loader binary, which seems to > have > > > a fixed size > > > supposed on the error I get. See below. > > > > > > The system is amd64 (64 bit), for the record. > > > > > > Somewhere in the past developers mentioned this upcoming problem and > > > provided a knob to adjust > > > the used size - I forgot about that knob and I couldn't find it even = in > > > the loader docs - or > > > looked at the wrong places. > > > > > > Can someone help me out here? > > > > > > The first error stops compileing world/kernel, but taking a second ru= n, > > > the error goes away. > > > > > > Kind regards and thanks in advance, > > > > > > oh > > > > > > > > > > > > [...] > > > --- all_subdir_stand/efi --- > > > SOURCE_DATE_EPOCH=3D1451606400 objcopy -j .peheader -j .text -j .sda= ta > -j > > > .data -j .dynamic -j > > > .dynsym -j .rel.dyn -j .rela.dyn -j .reloc -j .eh_frame -j > > > set_Xcommand_set -j > > > set_Xficl_compile_set --output-target=3Defi-app-x86_64 loader_4th.sy= m > > > loader_4th.efi --- > > > all_subdir_stand/i386 --- > > > > > > -8112 bytes available 7.71 real 12.86 user 3.08 sys > > > > > > bummer. I hate it when it's that close. > > > > You can try setting LOADERSIZE=3D560000 in your environment. We current= ly > set > > the maximum to 550,000 > > I tried to find find anything related to LOADER or SIZE in the docs. I > remember you mentioned > the existence of that variable months ago, but with no clue what to look > after, it is almost > impossible yo figure out as a non developer what the right knob might be. > Fair point. > A grep -r on /usr/src shows up only in > > [...] > ./stand/i386/loader/Makefile:LOADERSIZE?=3D 550000 # Large= st > known safe size for > loader.bin > ./stand/i386/loader/Makefile: @set -- `ls -l ${.TARGET}` ; > x=3D$$((${LOADERSIZE}-$$5)); \ > [...] > > There is no sign/trace of it in any man page related to loader and > sibblings. I found the > variable rather quickly after knowing what to look after. > To be fair, this is a very under documented area, semi on purpose... But that's a good point that it is under documented. And now that the size is creeping up and we have options that explode things, maybe it should be better documented. > > bytes because that's the most conservative number due to variation in t= he > > available BIOS space available. > > This likely can be set even higher if you don't have add-in cards that > are > > consuming space in the lower 640k > > of memory. 640k is the absolute limit, but you need 20-30k of stack for > the > > loader so pushing this much past > > 625,000 or maybe 630,000 increases the risk of run-time crashes as the > > stack smashes through the top of > > the loader program. You may also have to disable the lua build, since i= t > > uses more stack and is just a smidge > > larger than the forth build. _simp will be the smallest of them all. On > my > > system, without bearssl, I see: > > -r-xr-xr-x 3 root wheel 503808 May 22 15:25 /boot/loader_lua > > -r-xr-xr-x 1 root wheel 446464 May 22 15:25 /boot/loader_4th > > -r-xr-xr-x 1 root wheel 385024 May 22 15:25 /boot/loader_simp > > In my case, with supposedly blewn up loader size by BEARSSL enables, it i= s: > > -r-xr-xr-x 1 root wheel - 503808 3 Juni 12:33 /boot/loader_4th* > -r-xr-xr-x 1 root wheel - 643584 3 Juni 12:33 /boot/loader_4th.efi* > -r-xr-xr-x 1 root wheel - 503808 3 Juni 07:45 /boot/loader_4th.old* > -r-xr-xr-x 3 root wheel - 569344 3 Juni 12:33 /boot/loader_lua* > -r-xr-xr-x 2 root wheel - 737280 3 Juni 12:33 /boot/loader_lua.efi* > -r-xr-xr-x 1 root wheel - 569344 3 Juni 07:45 /boot/loader_lua.old* > -r-xr-xr-x 1 root wheel - 446464 3 Juni 12:33 /boot/loader_simp* > -r-xr-xr-x 1 root wheel - 589312 3 Juni 12:33 /boot/loader_simp.efi* > -r-xr-xr-x 1 root wheel - 446464 3 Juni 07:45 /boot/loader_simp.old* > > on FreeBSD 14.0-CURRENT #58 main-n263387-556b43492297: Fri Jun 2 20:19:5= 5 > CEST 2023 amd64. > > which suggests a ~60k bump for adding forth and ~115k bump for lua. So > the > > 560,000 may need to be 625,000 > > which is living life on the edge for 4th, and simply too big for lua. > > > > I'd be open to adding docs on this, since I don't think this option is > > currently documented since I added it > > to experiment around with a good value. > > See above, personally I'd like to see some hints on that variable, even i= f > I do not fiddle > around with it. > OK. that makes sense. In fact, it may make sense to disable the build of the BIOS loader entirely sometimes, which we can't easily do today. > > > > And no, I really do not want to support 'loadable modules'. BIOS bootin= g > is > > on the way out, and people > > that want to do complex stuff in the boot loader will simply have to do > > that in UEFI or maybe kboot/LinuxBoot. > > There's low RoI on adding this complexity, imho. We'd be better off, > imho, > > making things like the graphics > > console optional since the fonts and code for that free up about 30k in > > stupid experiments that I've done > > (it's hard since vidconsole has a lot of calls into the graphics system > > that aren't optional and easy to disable, > > so I've had to do hack and slash to produce a super ugly result that is > > only suggestive of the final savings): > > -rw-r--r-- 1 imp imp 352256 May 31 12:04 loader_simp > > I don't know if I slashed too much, or not enough since the code is > rather > > hard to separate out, so if you > > really wanted to go down this path, it would take a lot of work and > patient > > understanding to make it so with > > the low end of savings 20k and the high end on the order of maybe 40k. > > > > There's likely other ways to conserve space. We've not had space issues > > with loader, et al, in the past, > > so it's not well setup for subsetting. Though the different filesystem > > support might also net you a fair amount: > > LOADER_NET_SUPPORT?=3D yes > > LOADER_NFS_SUPPORT?=3D yes > > LOADER_TFTP_SUPPORT?=3D yes > > LOADER_CD9660_SUPPORT?=3D yes > > LOADER_EXT2FS_SUPPORT?=3D yes > > LOADER_MSDOS_SUPPORT?=3D yes > > LOADER_UFS_SUPPORT?=3D yes > > LOADER_GZIP_SUPPORT?=3D yes > > LOADER_BZIP2_SUPPORT?=3D yes > > as would compiling w/o ZFS, which uses its own method (eg > > WITHOUT_LOADER_ZFS). Tuning the loader > > at this level does start to get into the weeds a bit, but can offer ~40= k > > savings turning off all but NET and UFS: > > -rw-r--r-- 1 imp imp 344064 May 31 12:11 loader_simp > > you get even about ~100k when you disable ZFS support with > > -DWITHOUT_LOADER_ZFS: > > -rw-r--r-- 1 imp imp 241664 May 31 12:12 loader_simp > > (both of these are with the graphics console enabled without the silly > > hacks to see how much that takes up). > > Without the extras and ZFS, you might have bearssl and lua together > even... > > > > Hope this helps. > > Very much appreciated, thanks. > > I'm not so much concerned by the space sonsumption or a corrupted boot > process, I don't even > know whether I really need this i386 stuff mentioned here, since all of m= y > boxes are UEFI, > except the APU2, which are driven by SeaBIOS and bootin non UEFI. > Yea, I'm starting to think that the best way forward should include some way to turn off the BIOS loader if you know you won't need it and it's too big. > The cosmetic issue is that whenever anything reagrding the loader gets > touched by the > compiler, out build system drops out with an error - a second run then > performs clean (which > is a kind of weird I guess). > Yea, the check should cause it to be deleted when it fails, but doesn't. Warner > > > > Warner > > Thank you very much, > > Kind regards > Oliver > > > -- > O. Hartmann > --000000000000d945db05fd3fc245 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Sat, Jun 3, 2023 at 9:03=E2=80=AFA= M FreeBSD User <freebsd@walsta= tt-de.de> wrote:
Am Wed, 31 May 2023 12:15:12 -0600
Warner Losh <imp@bsd= imp.com> schrieb:

Sorry for the late response.


> On Mon, May 29, 2023 at 2:59=E2=80=AFAM FreeBSD User <freebsd@walstatt-de.de&g= t; wrote:
>
> > Hello,
> >
> > on CURRENT, enabling in /etc/src.conf
> >
> > WITH_BEARSSL=3D
> >
> > seems to result in a slightly enlarged loader binary, which seems= to have
> > a fixed size
> > supposed on the error I get. See below.
> >
> > The system is amd64 (64 bit), for the record.
> >
> > Somewhere in the past developers mentioned this upcoming problem = and
> > provided a knob to adjust
> > the used size - I forgot about that knob and I couldn't find = it even in
> > the loader docs - or
> > looked at the wrong places.
> >
> > Can someone help me out here?
> >
> > The first error stops compileing world/kernel, but taking a secon= d run,
> > the error goes away.
> >
> > Kind regards and thanks in advance,
> >
> > oh
> >
> >
> >
> > [...]
> > --- all_subdir_stand/efi ---
> > SOURCE_DATE_EPOCH=3D1451606400=C2=A0 objcopy -j .peheader -j .tex= t -j .sdata -j
> > .data=C2=A0 -j .dynamic -j
> > .dynsym -j .rel.dyn=C2=A0 -j .rela.dyn -j .reloc -j .eh_frame -j<= br> > > set_Xcommand_set=C2=A0 -j
> > set_Xficl_compile_set=C2=A0 --output-target=3Defi-app-x86_64 load= er_4th.sym
> > loader_4th.efi ---
> > all_subdir_stand/i386 ---
> >
> > -8112 bytes available 7.71 real=C2=A0 =C2=A0 =C2=A0 =C2=A0 12.86 = user=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A03.08 sys=C2=A0
>
>
> bummer. I hate it when it's that close.
>
> You can try setting LOADERSIZE=3D560000 in your environment. We curren= tly set
> the maximum to 550,000

I tried to find find anything related to LOADER or SIZE in the docs. I reme= mber you mentioned
the existence of that variable months ago, but with no clue what to look af= ter, it is almost
impossible yo figure out as a non developer what the right knob might be.

Fair point.
=C2=A0
A grep -r on /usr/src shows up only in

[...]
./stand/i386/loader/Makefile:LOADERSIZE?=3D=C2=A0 =C2=A0 =C2=A0 =C2=A055000= 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 # Largest known safe size for
loader.bin
./stand/i386/loader/Makefile:=C2=A0 =C2=A0@set -- `ls -l ${.TARGET}` ; x=3D= $$((${LOADERSIZE}-$$5)); \
[...]

There is no sign/trace of it in any man page related to loader and sibbling= s. I found the
variable rather quickly after knowing what to look after.
<= div>
To be fair, this is a very under documented area, semi o= n purpose...=C2=A0 But that's a good point that it is
under d= ocumented. And now that the size is creeping=C2=A0up and we have options th= at explode things,
maybe it should be better documented.
=C2=A0
> bytes because that's the most conservative number due to variation= in the
> available BIOS space available.
> This likely can be set even higher if you don't have add-in cards = that are
> consuming space in the lower 640k
> of memory. 640k is the absolute limit, but you need 20-30k of stack fo= r the
> loader so pushing this much past
> 625,000 or maybe 630,000 increases the risk of run-time crashes as the=
> stack smashes through the top of
> the loader program. You may also have to disable the lua build, since = it
> uses more stack and is just a smidge
> larger than the forth build. _simp will be the smallest of them all. O= n my
> system, without bearssl, I see:
> -r-xr-xr-x=C2=A0 3 root=C2=A0 wheel=C2=A0 503808 May 22 15:25 /boot/lo= ader_lua
> -r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 446464 May 22 15:25 /boot/lo= ader_4th
> -r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 385024 May 22 15:25 /boot/lo= ader_simp

In my case, with supposedly blewn up loader size by BEARSSL enables, it is:=

-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 503808=C2=A0 3 Juni 12:33 /boot= /loader_4th*
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 643584=C2=A0 3 Juni 12:33 /boot= /loader_4th.efi*
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 503808=C2=A0 3 Juni 07:45 /boot= /loader_4th.old*
-r-xr-xr-x=C2=A0 3 root=C2=A0 wheel=C2=A0 - 569344=C2=A0 3 Juni 12:33 /boot= /loader_lua*
-r-xr-xr-x=C2=A0 2 root=C2=A0 wheel=C2=A0 - 737280=C2=A0 3 Juni 12:33 /boot= /loader_lua.efi*
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 569344=C2=A0 3 Juni 07:45 /boot= /loader_lua.old*
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 446464=C2=A0 3 Juni 12:33 /boot= /loader_simp*
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 589312=C2=A0 3 Juni 12:33 /boot= /loader_simp.efi*
-r-xr-xr-x=C2=A0 1 root=C2=A0 wheel=C2=A0 - 446464=C2=A0 3 Juni 07:45 /boot= /loader_simp.old*

on FreeBSD 14.0-CURRENT #58 main-n263387-556b43492297: Fri Jun=C2=A0 2 20:1= 9:55 CEST 2023 amd64.
> which suggests a ~60k bump for adding forth and ~115k bump for lua. So= the
> 560,000 may need to be 625,000
> which is living life on the edge for 4th, and simply too big for lua.<= br> >
> I'd be open to adding docs on this, since I don't think this o= ption is
> currently documented since I added it
> to experiment around with a good value.

See above, personally I'd like to see some hints on that variable, even= if I do not fiddle
around with it.

OK. that makes sense. I= n fact, it may make sense to disable the build of the BIOS
loader= entirely sometimes, which we can't easily do today.
=C2=A0
>
> And no, I really do not want to support 'loadable modules'. BI= OS booting is
> on the way out, and people
> that want to do complex stuff in the boot loader will simply have to d= o
> that in UEFI or maybe kboot/LinuxBoot.
> There's low RoI on adding this complexity, imho. We'd be bette= r off, imho,
> making things like the graphics
> console optional since the fonts and code for that free up about 30k i= n
> stupid experiments that I've done
> (it's hard since vidconsole has a lot of calls into the graphics s= ystem
> that aren't optional and easy to disable,
> so I've had to do hack and slash to produce a super ugly result th= at is
> only suggestive of the final savings):
> -rw-r--r--=C2=A0 1 imp=C2=A0 imp=C2=A0 352256 May 31 12:04 loader_simp=
> I don't know if I slashed too much, or not enough since the code i= s rather
> hard to separate out, so if you
> really wanted to go down this path, it would take a lot of work and pa= tient
> understanding to make it so with
> the low end of savings 20k and the high end on the order of maybe 40k.=
>
> There's likely other ways to conserve space. We've not had spa= ce issues
> with loader, et al, in the past,
> so it's not well setup for subsetting. Though the different filesy= stem
> support might also net you a fair amount:
> LOADER_NET_SUPPORT?=3D=C2=A0 =C2=A0 yes
> LOADER_NFS_SUPPORT?=3D=C2=A0 =C2=A0 yes
> LOADER_TFTP_SUPPORT?=3D=C2=A0 =C2=A0yes
> LOADER_CD9660_SUPPORT?=3D yes
> LOADER_EXT2FS_SUPPORT?=3D yes
> LOADER_MSDOS_SUPPORT?=3D=C2=A0 yes
> LOADER_UFS_SUPPORT?=3D=C2=A0 =C2=A0 yes
> LOADER_GZIP_SUPPORT?=3D=C2=A0 =C2=A0yes
> LOADER_BZIP2_SUPPORT?=3D=C2=A0 yes
> as would compiling w/o ZFS, which uses its own method (eg
> WITHOUT_LOADER_ZFS). Tuning the loader
> at this level does start to get into the weeds a bit, but can offer ~4= 0k
> savings turning off all but NET and UFS:
> -rw-r--r--=C2=A0 1 imp=C2=A0 imp=C2=A0 344064 May 31 12:11 loader_simp=
> you get even about ~100k when you disable ZFS support with
> -DWITHOUT_LOADER_ZFS:
> -rw-r--r--=C2=A0 1 imp=C2=A0 imp=C2=A0 241664 May 31 12:12 loader_simp=
> (both of these are with the graphics console enabled without the silly=
> hacks to see how much that takes up).
> Without the extras and ZFS, you might have bearssl and lua together ev= en...
>
> Hope this helps.

Very much appreciated, thanks.

I'm not so much concerned by the space sonsumption or a corrupted boot = process, I don't even
know whether I really need this i386 stuff mentioned here, since all of my = boxes are UEFI,
except the APU2, which are driven by SeaBIOS and bootin=C2=A0 non UEFI.
=

Yea,=C2=A0 I'm starting to think that = the best way forward should include some way to turn off
the BIOS= loader if you know you won't need it and it's too big.
= =C2=A0
The cosmetic issue is that whenever anything reagrding the loader gets touc= hed by the
compiler, out build system drops out with=C2=A0 an error - a second run the= n performs clean (which
is a kind of weird I guess).

Yea, the c= heck should cause it to be deleted when it fails, but doesn't.

Warner
=C2=A0
>
> Warner

Thank you very much,

Kind regards
Oliver


--
O. Hartmann
--000000000000d945db05fd3fc245--