Date: Sat, 21 Mar 2026 09:30:22 -0600 From: Warner Losh <imp@bsdimp.com> To: Shawn Webb <shawn.webb@hardenedbsd.org> Cc: A FreeBSD User <freebsd@walstatt-de.de>, Charlie Li <vishwin@freebsd.org>, Martin Matuska <mm@freebsd.org>, src-committers <src-committers@freebsd.org>, "<dev-commits-src-all@freebsd.org>" <dev-commits-src-all@freebsd.org>, "<dev-commits-src-main@freebsd.org>" <dev-commits-src-main@freebsd.org> Subject: Re: git: 8a62a2a5659d - main - zfs: merge openzfs/zfs@f8e5af53e Message-ID: <CANCZdfrEppttf%2Btu%2BvpHKGd8ikSBpPJ=38cpE%2BsRzD1AjRpFhQ@mail.gmail.com> In-Reply-To: <5eaaokkdmmkcojky7362jvkandaard6eu26n77f5z5q7lqutu5@kywp5prvxozj> References: <69b561ff.39ea9.b797d91@gitrepo.freebsd.org> <2jb2vg6baofimu5xkxf62o5ogaq7fu5pk4o3vzhpegy446bppf@fzwtj6wtwk53> <CANCZdfpaKyzMDL0%2BMk3tSEeqsSU_C-OoWo1Dwwjg%2BG%2BrxiV2Eg@mail.gmail.com> <hqup3oiorbdgvtputen566fcajjoucurqpk7je6qchixat3i6c@zmfpxouvy7vs> <wryftbwxvsc3mhm6iglbx3ikzentaoewo3ldiw4knyqkckmhmo@hnmn3nosgawc> <becfc71c-2ea1-4a13-a3b1-3235f056badf@freebsd.org> <20260321090444.6f81da15@thor.sb211.local> <qeang5o4tlutdyxu2zfksbpljrrvf22eebf6r7b62umffzaymn@or5qwqnmbtbx> <CANCZdfq7YQuzgavRWm=DMKEvxQE9cYOFAEMpgk6faQwOL2tC2Q@mail.gmail.com> <5eaaokkdmmkcojky7362jvkandaard6eu26n77f5z5q7lqutu5@kywp5prvxozj>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On Sat, Mar 21, 2026 at 9:23 AM Shawn Webb <shawn.webb@hardenedbsd.org> wrote: > On Sat, Mar 21, 2026 at 09:18:20AM -0600, Warner Losh wrote: > > On Sat, Mar 21, 2026, 9:08 AM Shawn Webb <shawn.webb@hardenedbsd.org> > wrote: > > > > > On Sat, Mar 21, 2026 at 09:04:17AM +0100, A FreeBSD User wrote: > > > > Am Tage des Herren Fri, 20 Mar 2026 23:27:20 -0400 > > > > Charlie Li <vishwin@freebsd.org> schrieb: > > > > > > > > > Shawn Webb wrote: > > > > > > On Tue, Mar 17, 2026 at 04:52:16PM +0000, Shawn Webb wrote: > > > > > >> On Tue, Mar 17, 2026 at 10:44:59AM -0600, Warner Losh wrote: > > > > > >>> On Tue, Mar 17, 2026 at 10:36 AM Shawn Webb < > > > shawn.webb@hardenedbsd.org> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Hey Martin, > > > > > >>>> > > > > > >>>> On Sat, Mar 14, 2026 at 01:26:23PM +0000, Martin Matuska > wrote: > > > > > >>>>> The branch main has been updated by mm: > > > > > >>>>> > > > > > >>>>> URL: > > > > > >>>> > > > > https://cgit.FreeBSD.org/src/commit/?id=8a62a2a5659d1839d8799b4274c04469d7f17c78 > > > > > > > > >>>>> > > > > > >>>>> commit 8a62a2a5659d1839d8799b4274c04469d7f17c78 > > > > > >>>>> Merge: f91464171d61 f8e5af53e92f > > > > > >>>>> Author: Martin Matuska <mm@FreeBSD.org> > > > > > >>>>> AuthorDate: 2026-03-14 12:14:56 +0000 > > > > > >>>>> Commit: Martin Matuska <mm@FreeBSD.org> > > > > > >>>>> CommitDate: 2026-03-14 12:14:56 +0000 > > > > > >>>>> > > > > > >>>>> [snip for brevity] > > > > > >>>>> > > > > > >>>>> Obtained from: OpenZFS > > > > > >>>>> OpenZFS commit: f8e5af53e92fa7c03393fbd4922cb9c1d0c15920 > > > > > >>>> > > > > > >>>> This commit seems to cause issues when building boot loader > > > related > > > > > >>>> code: > > > > > >>>> > > > > > >>>> ==== BEGIN LOG ==== > > > > > >>>> 114232 bytes available > > > > > >>>> btxld -v -f aout -e 0x200000 -o loader_simp -l > > > > > >>>> /usr/obj/usr/src/amd64.amd64/stand/i386/btx/btxldr/btxldr -b > > > > > >>>> /usr/obj/usr/src/amd64.amd64/stand/i386/btx/btx/btx > > > loader_simp.bin > > > > > >>>> kernel: ver=1.02 size=690 load=9000 entry=9010 map=16M > pgctl=0:58 > > > > > >>>> client: fmt=elf size=5e2e8 text=57930 data=514c bss=7470 > entry=0 > > > > > >>>> output: fmt=aout size=61000 text=1000 data=5f000 org=200000 > > > entry=200000 > > > > > >>>> ===> stand/i386/pxeldr (all) > > > > > >>>> -560 bytes available > > > > > >>>> *** Error code 1 > > > > > >>>> > > > > > >>> > > > > > >>> What all do you have enabled? The defaults aren't even close to > > > running out > > > > > >>> of space (though I've not looked at this). > > > > > >> > > > > > >> Hey Warner, > > > > > >> > > > > > >> Thanks for reaching out! I've uploaded `make showconfig` here: > > > > > >> https://hardenedbsd.org/~shawn/2026-03-17_srcconf-r01.txt > > > > > >> > > > > > >> The following options are specific to HardenedBSD (in no > particular > > > > > >> order): > > > > > >> > > > > > >> 1. MK_HBSD_UPDATE > > > > > >> 2. MK_HBSDCONTROL > > > > > >> 3. MK_PIE > > > > > >> 4. MK_RELRO > > > > > >> 5. MK_SHLIBRANDOM > > > > > >> 6. MK_ZERO_REGS > > > > > >> 7. MK_SPECTREV1_FIX > > > > > >> 8. MK_SAFESTACK > > > > > >> 9. MK_RETPOLINE > > > > > >> 10. MK_LTOLIB > > > > > >> 11. MK_CFI > > > > > > > > > > > > MK_RETPOLINE was the culprit. Something about this ZFS commit > causes > > > > > > LLVM to emit more retpoline entries than before--too many for a > > > little > > > > > > bootloader. That might be something to investigate later, but > only to > > > > > > satisfy a curious mind, not to actuall fix anything (since > nothing's > > > > > > actually broken.) > > > > > > > > > > > > Since it doesn't really make sense to apply speculative execution > > > > > > mitigations to a bootloader, I disabled retpoline for a > components > > > > > > in stand/. > > > > > > > > > > > > Good to go. > > > > > > > > > > > Also just got bit by this, albeit during the lua loader, since I > have > > > > > WITH_RETPOLINE in my src.conf. > > > > > > > > > > > > > Hello, > > > > > > > > I do not have WITH_RETPOLINE in my /etc/src.conf, but since I got > this > > > mysterious error about > > > > not enough bytes left, I use WITHOUT_LOADER_PXEBOOT= YES (due to > issues > > > with WITH_BEARSSL=YES > > > > also used). > > > > Despite not using any WITH_RETPOLINE I also catch the error ... > > > > > > Something about this ZFS commit causes the boot laoder to be too big. > > > I guess the first sign of trouble was with retpolines, but there seem > > > to now be additional signs. > > > > > > What's the process for filing a bug report against OpenzFS for > > > something like this? (Not asking you directly, just a general question > > > for the thread.) > > > > > > > That's a good question. We are rapidly aporoaching the day we will have > to > > freeze the set of zfs feature that we can boot with the BIOS path. > There's > > only so much space. > > > > Right now, there's little to no bootloader testing, let alone analysis > > done and that will need to change. > > Another question might also be: is it worth the time and effort? Are > people really using a ZFS-enabled traditional MBR BIOS bootloader on > FreeBSD 16-CURRENT (or eventual 16.0-RELEASE)? From my rather naive > (and ignorant, since I don't know FreeBSD's userbase well), it seems > like folks use either loader.efi or u-boot (or both!) > Nobody uses MBR BIOS bootloader. It's totally unsupported. They all use the GPT one, which is used for all kinds of things, most recently several light-weight VM systems that don't have UEFI. BIOS is what imposes the limit, not MBR. But there's a huge number of systems that are already in the field that need to be upgradeable, but we will eventually need to draw a line in the sand for that. > Not a question I can answer, of course, because everyone likely has a > different take on what "worth the time and effort" means to them. :-) > Yea, the BIOS path is stubbornly hanging around. Warner > Thanks, > > -- > Shawn Webb > Cofounder / Security Engineer > HardenedBSD > > Signal Username: shawn_webb.74 > Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50 > > https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc > [-- Attachment #2 --] <div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sat, Mar 21, 2026 at 9:23 AM Shawn Webb <<a href="mailto:shawn.webb@hardenedbsd.org">shawn.webb@hardenedbsd.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sat, Mar 21, 2026 at 09:18:20AM -0600, Warner Losh wrote:<br> > On Sat, Mar 21, 2026, 9:08 AM Shawn Webb <<a href="mailto:shawn.webb@hardenedbsd.org" target="_blank">shawn.webb@hardenedbsd.org</a>> wrote:<br> > <br> > > On Sat, Mar 21, 2026 at 09:04:17AM +0100, A FreeBSD User wrote:<br> > > > Am Tage des Herren Fri, 20 Mar 2026 23:27:20 -0400<br> > > > Charlie Li <<a href="mailto:vishwin@freebsd.org" target="_blank">vishwin@freebsd.org</a>> schrieb:<br> > > ><br> > > > > Shawn Webb wrote:<br> > > > > > On Tue, Mar 17, 2026 at 04:52:16PM +0000, Shawn Webb wrote:<br> > > > > >> On Tue, Mar 17, 2026 at 10:44:59AM -0600, Warner Losh wrote:<br> > > > > >>> On Tue, Mar 17, 2026 at 10:36 AM Shawn Webb <<br> > > <a href="mailto:shawn.webb@hardenedbsd.org" target="_blank">shawn.webb@hardenedbsd.org</a>><br> > > > > >>> wrote:<br> > > > > >>><br> > > > > >>>> Hey Martin,<br> > > > > >>>><br> > > > > >>>> On Sat, Mar 14, 2026 at 01:26:23PM +0000, Martin Matuska wrote:<br> > > > > >>>>> The branch main has been updated by mm:<br> > > > > >>>>><br> > > > > >>>>> URL:<br> > > > > >>>><br> > > <a href="https://cgit.FreeBSD.org/src/commit/?id=8a62a2a5659d1839d8799b4274c04469d7f17c78" rel="noreferrer" target="_blank">https://cgit.FreeBSD.org/src/commit/?id=8a62a2a5659d1839d8799b4274c04469d7f17c78</a><br>; > ><br> > > > > >>>>><br> > > > > >>>>> commit 8a62a2a5659d1839d8799b4274c04469d7f17c78<br> > > > > >>>>> Merge: f91464171d61 f8e5af53e92f<br> > > > > >>>>> Author: Martin Matuska <mm@FreeBSD.org><br> > > > > >>>>> AuthorDate: 2026-03-14 12:14:56 +0000<br> > > > > >>>>> Commit: Martin Matuska <mm@FreeBSD.org><br> > > > > >>>>> CommitDate: 2026-03-14 12:14:56 +0000<br> > > > > >>>>><br> > > > > >>>>> [snip for brevity]<br> > > > > >>>>><br> > > > > >>>>> Obtained from: OpenZFS<br> > > > > >>>>> OpenZFS commit: f8e5af53e92fa7c03393fbd4922cb9c1d0c15920<br> > > > > >>>><br> > > > > >>>> This commit seems to cause issues when building boot loader<br> > > related<br> > > > > >>>> code:<br> > > > > >>>><br> > > > > >>>> ==== BEGIN LOG ====<br> > > > > >>>> 114232 bytes available<br> > > > > >>>> btxld -v -f aout -e 0x200000 -o loader_simp -l<br> > > > > >>>> /usr/obj/usr/src/amd64.amd64/stand/i386/btx/btxldr/btxldr -b<br> > > > > >>>> /usr/obj/usr/src/amd64.amd64/stand/i386/btx/btx/btx<br> > > loader_simp.bin<br> > > > > >>>> kernel: ver=1.02 size=690 load=9000 entry=9010 map=16M pgctl=0:58<br> > > > > >>>> client: fmt=elf size=5e2e8 text=57930 data=514c bss=7470 entry=0<br> > > > > >>>> output: fmt=aout size=61000 text=1000 data=5f000 org=200000<br> > > entry=200000<br> > > > > >>>> ===> stand/i386/pxeldr (all)<br> > > > > >>>> -560 bytes available<br> > > > > >>>> *** Error code 1<br> > > > > >>>><br> > > > > >>><br> > > > > >>> What all do you have enabled? The defaults aren't even close to<br> > > running out<br> > > > > >>> of space (though I've not looked at this).<br> > > > > >><br> > > > > >> Hey Warner,<br> > > > > >><br> > > > > >> Thanks for reaching out! I've uploaded `make showconfig` here:<br> > > > > >> <a href="https://hardenedbsd.org/~shawn/2026-03-17_srcconf-r01.txt" rel="noreferrer" target="_blank">https://hardenedbsd.org/~shawn/2026-03-17_srcconf-r01.txt</a><br>; > > > > >><br> > > > > >> The following options are specific to HardenedBSD (in no particular<br> > > > > >> order):<br> > > > > >><br> > > > > >> 1. MK_HBSD_UPDATE<br> > > > > >> 2. MK_HBSDCONTROL<br> > > > > >> 3. MK_PIE<br> > > > > >> 4. MK_RELRO<br> > > > > >> 5. MK_SHLIBRANDOM<br> > > > > >> 6. MK_ZERO_REGS<br> > > > > >> 7. MK_SPECTREV1_FIX<br> > > > > >> 8. MK_SAFESTACK<br> > > > > >> 9. MK_RETPOLINE<br> > > > > >> 10. MK_LTOLIB<br> > > > > >> 11. MK_CFI<br> > > > > ><br> > > > > > MK_RETPOLINE was the culprit. Something about this ZFS commit causes<br> > > > > > LLVM to emit more retpoline entries than before--too many for a<br> > > little<br> > > > > > bootloader. That might be something to investigate later, but only to<br> > > > > > satisfy a curious mind, not to actuall fix anything (since nothing's<br> > > > > > actually broken.)<br> > > > > ><br> > > > > > Since it doesn't really make sense to apply speculative execution<br> > > > > > mitigations to a bootloader, I disabled retpoline for a components<br> > > > > > in stand/.<br> > > > > ><br> > > > > > Good to go.<br> > > > > ><br> > > > > Also just got bit by this, albeit during the lua loader, since I have<br> > > > > WITH_RETPOLINE in my src.conf.<br> > > > ><br> > > ><br> > > > Hello,<br> > > ><br> > > > I do not have WITH_RETPOLINE in my /etc/src.conf, but since I got this<br> > > mysterious error about<br> > > > not enough bytes left, I use WITHOUT_LOADER_PXEBOOT= YES (due to issues<br> > > with WITH_BEARSSL=YES<br> > > > also used).<br> > > > Despite not using any WITH_RETPOLINE I also catch the error ...<br> > ><br> > > Something about this ZFS commit causes the boot laoder to be too big.<br> > > I guess the first sign of trouble was with retpolines, but there seem<br> > > to now be additional signs.<br> > ><br> > > What's the process for filing a bug report against OpenzFS for<br> > > something like this? (Not asking you directly, just a general question<br> > > for the thread.)<br> > ><br> > <br> > That's a good question. We are rapidly aporoaching the day we will have to<br> > freeze the set of zfs feature that we can boot with the BIOS path. There's<br> > only so much space.<br> > <br> > Right now, there's little to no bootloader testing, let alone analysis<br> > done and that will need to change.<br> <br> Another question might also be: is it worth the time and effort? Are<br> people really using a ZFS-enabled traditional MBR BIOS bootloader on<br> FreeBSD 16-CURRENT (or eventual 16.0-RELEASE)? From my rather naive<br> (and ignorant, since I don't know FreeBSD's userbase well), it seems<br> like folks use either loader.efi or u-boot (or both!)<br></blockquote><div><br></div><div>Nobody uses MBR BIOS bootloader. It's totally unsupported. They all use the</div><div>GPT one, which is used for all kinds of things, most recently several light-weight</div><div>VM systems that don't have UEFI. BIOS is what imposes the limit, not MBR.</div><div><br></div><div>But there's a huge number of systems that are already in the field that need</div><div>to be upgradeable, but we will eventually need to draw a line in the sand for</div><div>that.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Not a question I can answer, of course, because everyone likely has a<br> different take on what "worth the time and effort" means to them. :-)<br></blockquote><div><br></div><div>Yea, the BIOS path is stubbornly hanging around.</div><div><br></div><div>Warner</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> Thanks,<br> <br> -- <br> Shawn Webb<br> Cofounder / Security Engineer<br> HardenedBSD<br> <br> Signal Username: shawn_webb.74<br> Tor-ified Signal: +1 303-901-1600 / shawn_webb_opsec.50<br> <a href="https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc" rel="noreferrer" target="_blank">https://git.hardenedbsd.org/hardenedbsd/pubkeys/-/raw/master/Shawn_Webb/03A4CBEBB82EA5A67D9F3853FF2E67A277F8E1FA.pub.asc</a><br>; </blockquote></div></div>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CANCZdfrEppttf%2Btu%2BvpHKGd8ikSBpPJ=38cpE%2BsRzD1AjRpFhQ>