From nobody Thu Aug 11 03:54:24 2022 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M3CcF5m9kz4YY5D for ; Thu, 11 Aug 2022 03:54:37 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ua1-x931.google.com (mail-ua1-x931.google.com [IPv6:2607:f8b0:4864:20::931]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4M3CcD5pnFz3ycW for ; Thu, 11 Aug 2022 03:54:36 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-ua1-x931.google.com with SMTP id f15so6531536uao.12 for ; Wed, 10 Aug 2022 20:54:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc; bh=VOOO0iuYvWTxwpcJ+jAaKta2xFQriAacjZDFuaFtzn8=; b=jbT3eMA1A3TjUsU0j/O5qv4eZjzbOcoOs3i/+fLFgmivi1gf0KDEXHT4eHdv16BF5N 2jjHNXIhvVeEegaRn2EvtFoqMUFDiqGZo6oqgVJ1TgEe6dI/+WtBFQAaM7ZNLWg/xktA zEnOXVTdCPNRMt+VKhUG9h7cHMa4Kt6lwgpHtdpdmKZq5MYjJkDe5wcwe1PFVTH0Y2pB 4DZQYvSW9Ydav/vCFiz46kLjVF0Ktm1rVo15C2CWqN2aScMVAUXfN6iT643yBJyxRc8G VlkMvH3o4bmZoKUwJD/4Say625/7Hn6BCSAhvbwxTxpC3zoshlgzkT/cILYp5GindMk9 npJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc; bh=VOOO0iuYvWTxwpcJ+jAaKta2xFQriAacjZDFuaFtzn8=; b=QQT2kNuxZWM0nhUuuk05xfXIT9gqWv7s17Bxwa2XrgAUTgyQXInBpY9i3lJMGz37B4 qZIxsoWZUWF1TxFt4BLQYdjnjzVQs5ohUmrb6risqEWYuMq9mbbGx75mKacd7zsfT1vt mzSXb9+vBb+w9MGDtE++r5MdwhPJn1i+ymkZbR2O0ibOv8FimGNPqXCvpVWUtfOjGc9s 13hfvHqOODGufsRb5qfLNQZy59dA6a6R7euSd9QRMNnHGnHC1BwWYvPJe8DZpd+QRwWo R0l2QJlwww+h2T2NNCRWoicUBNdnUxJSEN0wAs6+go83pTYTNEXjw7VxesN00zm8mnUI 7fXg== X-Gm-Message-State: ACgBeo3ZPbpMD4zve6BL+5OY4hrHUw8x2gI+nswj6h6L6o1/+L5qxKE3 GhLZ5eduksWcCenrUb4pB9Djn7MkchQGTUjPeUo6VHLsJNs= X-Google-Smtp-Source: AA6agR6QPCj4iHwEeuwuVYF9SnHIGi6pt+lvNh3vba2gQRExoHOGaRp2zVpmMhsieV7yO0Vz3UHCF0jW4aDM5pIW338= X-Received: by 2002:a05:6130:112:b0:38c:4226:62f2 with SMTP id h18-20020a056130011200b0038c422662f2mr13441046uag.82.1660190075391; Wed, 10 Aug 2022 20:54:35 -0700 (PDT) List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org MIME-Version: 1.0 From: Warner Losh Date: Wed, 10 Aug 2022 21:54:24 -0600 Message-ID: Subject: BIOS /boot/loader is full, time to make some hard choices To: "freebsd-arch@freebsd.org" Content-Type: multipart/alternative; boundary="000000000000b4247305e5ef1fca" X-Rspamd-Queue-Id: 4M3CcD5pnFz3ycW X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20210112.gappssmtp.com header.s=20210112 header.b=jbT3eMA1; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::931) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20210112.gappssmtp.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MLMMJ_DEST(0.00)[freebsd-arch@freebsd.org]; R_SPF_NA(0.00)[no SPF record]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20210112.gappssmtp.com:+]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::931:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[bsdimp.com]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N --000000000000b4247305e5ef1fca Content-Type: text/plain; charset="UTF-8" Greetings, I just wasted a day chasing a problem with /boot/loader being too large. Due to a number of limitations in our infrastructure, we have a hard limit of 640k hard limit of total memory usage (though we can use upper memory for things like cache). 640k includes all the 'text' of the loader, plus btx, plus data, bss and stack. The BIOS calls we make are limited to 1MB due to calling them in 16-bit mode and needing to use SEGMENT:OFFSET addressing for various addresses (which on the IBM-PC means 640k). So, we're stuck with a limit of about 510,000 bytes for /boot/loader. Up through at last FreeBSD/12 (2020) /boot/loader was about 330kB. Now we're over 500kB due to a lot of extra things that have happened since then: frame buffer, OpenZFS being larger and pulling in more crypto and decompression software, and new filesystems. Between them all, we're pushing the hard limits. Now, we could in theory allow loading in high memory, have loadable modules in the loader, etc. I don't like this idea because we're going to kill BIOS in a few years and all these things add extra hair to a poorly tested, less well supported path. We need to subset what's in the BIOS loader. We need to make more things optional, and we need to set a policy to know how to choose between this and that when (not if) we run out of space again. I plan on making it easier to compile out certain file systems globally, as well as certain large features (possibly also some crypto / decompression algorithms). The policy I'm proposing is to include as many filesystems, crypto and compression in the boot loader that fit, in that order. And then other features as space allows. This means that /boot/loader.efi will have all these things (since there's no limits there), but for /boot/loader (BIOS) we'll have a subset. If we still want to have the graphics support in the BIOS loader for the installer, we'll have to leave out filesystem support, crypto support, etc. For the cd image and/or the memstick image this is fine: we only need to boot off of UFS/CD9660 (since we don't build ZFS images yet), so we can have the graphics support, but since we install so many different filesystems, with different features for crypto and/or decompression, we may need to skip graphics for the default /boot/loader we install for BIOS booting. So, since we still have a little time, we have time to talk about the future we want, but *I* also have only a limited amount of time to do tooling here. Doing build stuff is in scope. Reworking things, say, so we can have loadable modules likely isn't. Warner --000000000000b4247305e5ef1fca Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Greetings,

I just wasted a day chasing = a problem with /boot/loader being too large.

Due t= o a number of limitations in our infrastructure, we have a hard limit of 64= 0k hard limit of total memory usage (though we can use upper memory for thi= ngs like cache). 640k includes all the 'text' of the loader, plus b= tx, plus data, bss and stack. The BIOS calls we make are limited to 1MB due= to calling them in 16-bit mode and needing to use SEGMENT:OFFSET addressin= g for various addresses (which on the IBM-PC means 640k).

So, we're stuck with a limit of about 510,000 bytes for /boot/l= oader. Up through at last FreeBSD/12 (2020) /boot/loader was about 330kB. N= ow we're over 500kB due to a lot of extra things that have happened sin= ce then: frame buffer, OpenZFS being larger and pulling in more crypto and = decompression software, and new filesystems. Between them all, we're pu= shing the hard limits.

Now, we could in theory all= ow loading in high memory, have loadable modules in the loader, etc. I don&= #39;t like this idea because we're going to kill BIOS in a few years an= d all these things add extra hair to a poorly tested, less well supported p= ath.

We need to subset what's in the BIOS load= er. We need to make more things optional, and we need to set a policy to kn= ow how to choose between this and that when (not if) we run out of space ag= ain. I plan on making it easier to compile out certain file systems globall= y, as well as certain large features (possibly also some crypto / decompres= sion algorithms).

The policy I'm proposing is = to include as many filesystems, crypto and compression in the boot loader t= hat fit, in that order. And then other features as space allows. This means= that /boot/loader.efi will have all these things (since there's no lim= its there), but for /boot/loader (BIOS) we'll have a subset. If we stil= l want to have the graphics support in the BIOS loader for the installer, w= e'll have to leave out filesystem support, crypto support, etc. For the= cd image and/or the memstick image this is fine: we only need to boot off = of UFS/CD9660 (since we don't build ZFS images yet), so we can have the= graphics support, but since we install so many different filesystems, with= different features for crypto and/or decompression, we may need to skip gr= aphics for the default /boot/loader we install for BIOS booting.
=
So, since we still have a little time, we have time to talk = about the future we want, but *I* also have only a limited amount of time t= o do tooling here. Doing build stuff is in scope. Reworking things, say, so= we can have loadable modules likely isn't.

Wa= rner

--000000000000b4247305e5ef1fca--