Date: Sun, 16 Nov 2025 10:29:00 +0100 From: Thomas Schweikle <tschweikle@gmail.com> To: Sulev-Madis Silber <freebsd-current-freebsd-org111@ketas.si.pri.ee> Cc: freebsd-current@freebsd.org Subject: Re: "etcupdate extract" -- Failed to build new tree. Message-ID: <CA%2BrGx5e44JhSLTwjLJt=LXT47e-%2B2n=uicdqAbZgx6raXJg=Ow@mail.gmail.com> In-Reply-To: <B7D6F208-6435-49C3-91B3-77CB7B334E55@ketas.si.pri.ee> References: <CA%2BrGx5ekwmBNUJKoCOk1c-ts_Q_VXuRoQ4Krkuqey_c-KZbaXQ@mail.gmail.com> <86qzuo1ab1.fsf@ltc.des.dev> <CA%2BrGx5cF%2B-QTh95Jw1hs59Dsyk7TSAn%2BYBfWw_suNY98Q7i56w@mail.gmail.com> <868qgw14xj.fsf@ltc.des.dev> <CA%2BrGx5d3ZOXZM6yw4eyO-CbhjiCGu%2BZz1COmLiRDOOQEiOa1Qg@mail.gmail.com> <864irj1ett.fsf@ltc.des.dev> <CA%2BrGx5eKUx2KzfN=81pKrTkC6-3yhKiBn_2xgLqC5oADBuddSg@mail.gmail.com> <86jz0fyzjj.fsf@ltc.des.dev> <CA%2BrGx5dZgdMkW7zWsrbwLast8t_BhBZt6CF75H8VjW_Lez7cYA@mail.gmail.com> <31a80d2d-65b3-4604-ac5f-1440d23fb85a@plan-b.pwste.edu.pl> <CA%2BrGx5cUAo-93t1fTDhEeLxqrJ4ewb%2BriLfodOEhKAwNM6koig@mail.gmail.com> <43c4ae93-71e2-4b2e-b265-b84b96a70666@plan-b.pwste.edu.pl> <4570E827-C9ED-4473-AE60-C895DBD9C2BF@ketas.si.pri.ee> <CA%2BrGx5cw9zd8e-uwB0v7WHwzm6Dco%2BUtPWRtYikUW3aZ1fz8jw@mail.gmail.com> <2121B7E6-0DB6-427A-9C72-640A3AEE26AA@ketas.si.pri.ee> <CA%2BrGx5fOJS5SwywFfe3Eo%2BFVgbAQ-hCLih-tA49AwP=cgkzwhw@mail.gmail.com> <B7D6F208-6435-49C3-91B3-77CB7B334E55@ketas.si.pri.ee>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] Seems like someone "optimized" "./mk_cmds" by changing permissions to "chmod 0755 m/mk_cmds*", then directly calling "./mk_cmds". Within older versions of "etcupdate" it was executed by "/bin/sh ./mk_cmds". "/bin/sh ./mk_cmds" circumvents "noexec". But it makes it possible to compile sources without executing anything within /usr/src directly. But: while /usr/src is mounted zroot/usr/src on /usr/src (zfs, local, noexec, nosuid, nfsv4acls) /usr/obj is mounted zroot/usr on /usr (zfs, local, nfsv4acls) And both mounts are what FreeBSD on zfs with GPT ( https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot) expects (copied right from the handbook pages): zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp zfs create -o canmount=off -o mountpoint=/usr zroot/usr zfs create zroot/usr/home zfs create -o exec=off -o setuid=off zroot/usr/src zfs create zroot/usr/obj zfs create -o mountpoint=/usr/ports -o setuid=off zroot/usr/ports zfs create -o exec=off -o setuid=off zroot/usr/ports/distfiles zfs create -o exec=off -o setuid=off zroot/usr/ports/packages zfs create -o canmount=off -o mountpoint=/var zroot/var zfs create -o exec=off -o setuid=off zroot/var/audit zfs create -o exec=off -o setuid=off zroot/var/crash zfs create -o exec=off -o setuid=off zroot/var/log zfs create -o atime=on -o exec=off -o setuid=off zroot/var/mail zfs create -o exec=on -o setuid=off zroot/var/tmp Since "etcupdate" worked about four months ago, then it started to stop working, something has changed with how it works. For me it looks like there creating mk_cmds within /usr/src, instead within /usr/obj. This was ok as long as mk_cmds was called with "/bin/sh ./mk_cmds". As soon as this was changed to "chmod 0755 mk_cmds*; ./mk_cmds" it broke. I've tested a lot last days. It is not "noexec" set for "/usr/src". It is something changed just before forking "stable/15". Going back to this makes it working again (as it does copying "etcupdate" from "stable/14". On Sun, Nov 9, 2025 at 2:49 AM Sulev-Madis Silber < freebsd-current-freebsd-org111@ketas.si.pri.ee> wrote: > > > On November 8, 2025 10:36:43 PM GMT+02:00, Thomas Schweikle < > tschweikle@gmail.com> wrote: > >On Sat, Nov 8, 2025 at 8:00 PM Sulev-Madis Silber < > >freebsd-current-freebsd-org111@ketas.si.pri.ee> wrote: > > > >> first thing that comes to my mind... > >> > >> is there any other fs where noexec is used? > >> > >The other systems are set up the same way, mounting /usr/src > >zroot/usr/src on /usr/src (zfs, local, noexec, nosuid, nfsv4acls) > > > > > >> especially on which the etcupdate actually runs on? > >> > >It runs on three other systems: > >- stable/13 > >- stable/14 > >and a pre stable/15 (before ALPHA was out and the branch done). > > > >It does not run if executed on a > >- stable/15 > >- current > > > >in both cases "make buildetc" works as expected if not called from inside > >"etcupdate". > > > > > >> i haven't so far studied the whole thing but if you have put noexec > >> everywhere, including tmp dirs or whatever etcupdate uses, it could be > it! > >> > >noexec does not touch "root", but standard users. Does etcupdate switch to > >some other user while executing? Does it do so before branching stable/15? > > unfortunately noexec don't even let root execute anything directly > > read works, shell scripts, makefiles > > if you have other fses in that very same machine mounted noexec then this > code will fail. and it indeed does give permission denied > > i don't know where mk_cmds is. perhaps in obj? i'm currently not debugging > this > > whether it should be able to run noexec, no idea, not everything can be > > noexec is there by purpose i assume, but maybe it could be temporary > disabled > > btw noexec as a security measure can be also bypassed > > tell us if disabling noexec on fses helped > > that's also fun thing, that not many add, or if they do, they know what > happens. many building type tasks like to execute something in dirs > > you didn't tell that you had noexec on and nobody asked either so maybe > it's noexec for all that time and you had to do manual work > > i hope it's fixed now!!! > > > > >And just switching into /usr/src, then executing "make etcupdate" does > call > >"./mk_cmds" and it does not fail because it is forbidden to run -- it > runs. > > > >just a wild guess. noexec is not always used and it often blows up random > >> thing unexpectedly. others have it disabled and therefore don't see it > >> > >> > >> > >> On November 8, 2025 8:21:22 PM GMT+02:00, Thomas Schweikle < > >> tschweikle@gmail.com> wrote: > >> >Looking at the log from etcupdate I found it failing with > >> > > >> >chmod 755 mk_cmds > >> >rm -f et-h-ss_err.et et-h-ss_err.c et-h-ss_err.h > >> >cp /usr/src/crypto/krb5/src/util/ss/ss_err.et et-h-ss_err.et > >> >compile_et -d /usr/src/crypto/krb5/src/util/et --textdomain mit-krb5 > >> >et-h-ss_err.et > >> >+ /usr/bin/awk -f /usr/src/crypto/krb5/src/util/et/et_h.awk > >> >'outfile=et-h-ss_err.h' et-h-ss_err.et > >> >+ /usr/bin/awk -f /usr/src/crypto/krb5/src/util/et/et_c.awk > >> >'outfile=et-h-ss_err.c' 'textdomain=mit-krb5' 'localedir=' > et-h-ss_err.et > >> >mv et-h-ss_err.h ss_err.h > >> >rm -f et-h-ss_err.et et-h-ss_err.h > >> >./mk_cmds /usr/src/crypto/krb5/src/util/ss/std_rqs.ct > >> >make[4]: exec(./mk_cmds): Permission denied > >> >*** Error code 1 > >> > > >> >Stop. > >> >make[4]: stopped making "all" in /usr/src/krb5/util/ss > >> >*** Error code 1 > >> > > >> >Stop. > >> >make[3]: stopped making "bootstrap-tools" in /usr/src > >> > 10.16 real 8.75 user 1.04 sys > >> >*** Error code 1 > >> > > >> >Stop. > >> >make[2]: stopped making "_bootstrap-tools" in /usr/src > >> >*** Error code 1 > >> > > >> >Stop. > >> >make[1]: stopped making "buildetc" in /usr/src > >> >*** Error code 1 > >> > > >> >Stop. > >> >make: stopped making "buildetc" in /usr/src > >> > > >> >It fails, because it is not allowed to "exec ./mk_cmds", after setting > >> >"chmod 0755" for "mk_cmds"? Within "/usr/src" "mk_cmds" just does not > >> exist > >> >– as long as "find /usr/src -iname '*mk_cmds*' -print" would be able to > >> >find it: > >> >/usr/src # find . -iname '*/mk_cmds/*' > >> >/usr/src # > >> > > >> >"/usr/src" is mounted > >> >zroot/usr/src on /usr/src (zfs, local, noexec, nosuid, nfsv4acls) > >> > > >> >AND what makes it even more mysterious: > >> ># cd /usr/src > >> ># make buildetc > >> > > >> >works without reporting any error. If called from "etcupdate extract" > it > >> >fails. > >> > > >> >It works for FreeBSD: > >> >- stable/13 > >> >- stable/14 > >> > > >> >but not for: > >> >- stable/15 > >> > > >> >Full log is attached. > >> > > >> >On Tue, Nov 4, 2025 at 4:00 PM Sulev-Madis Silber < > >> >freebsd-current-freebsd-org111@ketas.si.pri.ee> wrote: > >> > > >> >> something bad clearly happened in that machine if it's able to > selfhost > >> >> build itself and work, except etcupdate. and that for a long time > >> >> > >> >> i don't see any reason getting pissed about his machine mysteriously > not > >> >> working, despite i haven't broken any machine myself since i > installed > >> >> first fbsd, 4.6 > >> >> > >> >> unsure how to go from here. unsure if src.conf or make.conf matters > here > >> >> > >> >> if this happens in currently actively supported fbsd release, maybe > >> >> etcupdate needs a bugfix to cleanup for edge cases > >> >> > >> >> if it's in unsupported, that should not cause any "pissages" either. > fix > >> >> is somewhere > >> >> > >> >> so, the admin updated files manually because he was not able to get > it > >> >> working? i bet you can recreate it and fix it for future cases > >> >> > >> >> since i haven't ever broken etcupdate, i don't know which data it > uses > >> as > >> >> input. but seems like it reads wrong data out of somewhere. and > entire > >> >> machine works, except this? > >> >> > >> >> i mean if this was bad upgrade leftover, how to fix? i mean doesn't > >> >> etcupdate get rework now, for pkgbase. while we do that, maybe it > could > >> be > >> >> made to handle this. or maybe even given non-destructive nuke mode so > >> >> people can start clean > >> >> > >> >> i don't think it's really entirely user error, considering he didn't > >> break > >> >> the machine > >> >> > >> >> i'm curious too. at worst you could do other install in another > machine > >> >> and compare dirs as something must differ there. then, according to > >> >> findings, fix the old machine. maybe while doing this, some new > bugfix > >> idea > >> >> appears > >> >> > >> >> but hell with getting mad over machine. he's pissed too, everyone's > >> >> pissed, server is not working and no productive work have been done > here > >> >> > >> >> maybe something got lost in translation too > >> >> > >> >> > >> > > >> > >> > > > > -- Thomas [-- Attachment #2 --] <div dir="ltr">Seems like someone "optimized" "./mk_cmds" by changing permissions to "chmod 0755 m/mk_cmds*", then directly calling "./mk_cmds". Within older versions of "etcupdate" it was executed by "/bin/sh ./mk_cmds". "/bin/sh ./mk_cmds" circumvents "noexec". But it makes it possible to compile sources without executing anything within /usr/src directly.<div><br></div><div>But: while /usr/src is mounted</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace">zroot/usr/src on /usr/src (zfs, local, noexec, nosuid, nfsv4acls)</font></div></blockquote><div><br></div><div>/usr/obj is mounted</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace">zroot/usr on /usr (zfs, local, nfsv4acls)</font></div></blockquote><div><br></div><div>And both mounts are what FreeBSD on zfs with GPT (<a href="https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot" target="_blank">https://wiki.freebsd.org/RootOnZFS/GPTZFSBoot</a>) expects (copied right from the handbook pages):</div><div><pre style="border:1pt solid rgb(174,189,204);background-color:rgb(243,245,247);padding:5pt;font-family:courier,monospace;color:rgb(0,0,0);font-size:16px">zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp zfs create -o canmount=off -o mountpoint=/usr zroot/usr zfs create zroot/usr/home zfs create -o exec=off -o setuid=off zroot/usr/src zfs create zroot/usr/obj zfs create -o mountpoint=/usr/ports -o setuid=off zroot/usr/ports zfs create -o exec=off -o setuid=off zroot/usr/ports/distfiles zfs create -o exec=off -o setuid=off zroot/usr/ports/packages zfs create -o canmount=off -o mountpoint=/var zroot/var zfs create -o exec=off -o setuid=off zroot/var/audit zfs create -o exec=off -o setuid=off zroot/var/crash zfs create -o exec=off -o setuid=off zroot/var/log zfs create -o atime=on -o exec=off -o setuid=off zroot/var/mail zfs create -o exec=on -o setuid=off zroot/var/tmp</pre></div><div><br></div><div>Since "etcupdate" worked about four months ago, then it started to stop working, something has changed with how it works. For me it looks like there creating mk_cmds within /usr/src, instead within /usr/obj. This was ok as long as mk_cmds was called with "/bin/sh ./mk_cmds". As soon as this was changed to "chmod 0755 mk_cmds*; ./mk_cmds" it broke.</div><div><br></div><div>I've tested a lot last days. It is not "noexec" set for "/usr/src". It is something changed just before forking "stable/15". Going back to this makes it working again (as it does copying "etcupdate" from "stable/14".</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Nov 9, 2025 at 2:49 AM Sulev-Madis Silber <<a href="mailto:freebsd-current-freebsd-org111@ketas.si.pri.ee" target="_blank">freebsd-current-freebsd-org111@ketas.si.pri.ee</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> <br> On November 8, 2025 10:36:43 PM GMT+02:00, Thomas Schweikle <<a href="mailto:tschweikle@gmail.com" target="_blank">tschweikle@gmail.com</a>> wrote:<br> >On Sat, Nov 8, 2025 at 8:00 PM Sulev-Madis Silber <<br> ><a href="mailto:freebsd-current-freebsd-org111@ketas.si.pri.ee" target="_blank">freebsd-current-freebsd-org111@ketas.si.pri.ee</a>> wrote:<br> ><br> >> first thing that comes to my mind...<br> >><br> >> is there any other fs where noexec is used?<br> >><br> >The other systems are set up the same way, mounting /usr/src<br> >zroot/usr/src on /usr/src (zfs, local, noexec, nosuid, nfsv4acls)<br> ><br> ><br> >> especially on which the etcupdate actually runs on?<br> >><br> >It runs on three other systems:<br> >- stable/13<br> >- stable/14<br> >and a pre stable/15 (before ALPHA was out and the branch done).<br> ><br> >It does not run if executed on a<br> >- stable/15<br> >- current<br> ><br> >in both cases "make buildetc" works as expected if not called from inside<br> >"etcupdate".<br> ><br> ><br> >> i haven't so far studied the whole thing but if you have put noexec<br> >> everywhere, including tmp dirs or whatever etcupdate uses, it could be it!<br> >><br> >noexec does not touch "root", but standard users. Does etcupdate switch to<br> >some other user while executing? Does it do so before branching stable/15?<br> <br> unfortunately noexec don't even let root execute anything directly<br> <br> read works, shell scripts, makefiles<br> <br> if you have other fses in that very same machine mounted noexec then this code will fail. and it indeed does give permission denied<br> <br> i don't know where mk_cmds is. perhaps in obj? i'm currently not debugging this<br> <br> whether it should be able to run noexec, no idea, not everything can be<br> <br> noexec is there by purpose i assume, but maybe it could be temporary disabled<br> <br> btw noexec as a security measure can be also bypassed<br> <br> tell us if disabling noexec on fses helped<br> <br> that's also fun thing, that not many add, or if they do, they know what happens. many building type tasks like to execute something in dirs<br> <br> you didn't tell that you had noexec on and nobody asked either so maybe it's noexec for all that time and you had to do manual work<br> <br> i hope it's fixed now!!!<br> <br> ><br> >And just switching into /usr/src, then executing "make etcupdate" does call<br> >"./mk_cmds" and it does not fail because it is forbidden to run -- it runs.<br> ><br> >just a wild guess. noexec is not always used and it often blows up random<br> >> thing unexpectedly. others have it disabled and therefore don't see it<br> >><br> >><br> >><br> >> On November 8, 2025 8:21:22 PM GMT+02:00, Thomas Schweikle <<br> >> <a href="mailto:tschweikle@gmail.com" target="_blank">tschweikle@gmail.com</a>> wrote:<br> >> >Looking at the log from etcupdate I found it failing with<br> >> ><br> >> >chmod 755 mk_cmds<br> >> >rm -f <a href="http://et-h-ss_err.et" rel="noreferrer" target="_blank">et-h-ss_err.et</a> et-h-ss_err.c et-h-ss_err.h<br> >> >cp /usr/src/crypto/krb5/src/util/ss/<a href="http://ss_err.et" rel="noreferrer" target="_blank">ss_err.et</a> <a href="http://et-h-ss_err.et" rel="noreferrer" target="_blank">et-h-ss_err.et</a><br> >> >compile_et -d /usr/src/crypto/krb5/src/util/et --textdomain mit-krb5<br> >> ><a href="http://et-h-ss_err.et" rel="noreferrer" target="_blank">et-h-ss_err.et</a><br> >> >+ /usr/bin/awk -f /usr/src/crypto/krb5/src/util/et/et_h.awk<br> >> >'outfile=et-h-ss_err.h' <a href="http://et-h-ss_err.et" rel="noreferrer" target="_blank">et-h-ss_err.et</a><br> >> >+ /usr/bin/awk -f /usr/src/crypto/krb5/src/util/et/et_c.awk<br> >> >'outfile=et-h-ss_err.c' 'textdomain=mit-krb5' 'localedir=' <a href="http://et-h-ss_err.et" rel="noreferrer" target="_blank">et-h-ss_err.et</a><br> >> >mv et-h-ss_err.h ss_err.h<br> >> >rm -f <a href="http://et-h-ss_err.et" rel="noreferrer" target="_blank">et-h-ss_err.et</a> et-h-ss_err.h<br> >> >./mk_cmds /usr/src/crypto/krb5/src/util/ss/std_rqs.ct<br> >> >make[4]: exec(./mk_cmds): Permission denied<br> >> >*** Error code 1<br> >> ><br> >> >Stop.<br> >> >make[4]: stopped making "all" in /usr/src/krb5/util/ss<br> >> >*** Error code 1<br> >> ><br> >> >Stop.<br> >> >make[3]: stopped making "bootstrap-tools" in /usr/src<br> >> > 10.16 real 8.75 user 1.04 sys<br> >> >*** Error code 1<br> >> ><br> >> >Stop.<br> >> >make[2]: stopped making "_bootstrap-tools" in /usr/src<br> >> >*** Error code 1<br> >> ><br> >> >Stop.<br> >> >make[1]: stopped making "buildetc" in /usr/src<br> >> >*** Error code 1<br> >> ><br> >> >Stop.<br> >> >make: stopped making "buildetc" in /usr/src<br> >> ><br> >> >It fails, because it is not allowed to "exec ./mk_cmds", after setting<br> >> >"chmod 0755" for "mk_cmds"? Within "/usr/src" "mk_cmds" just does not<br> >> exist<br> >> >– as long as "find /usr/src -iname '*mk_cmds*' -print" would be able to<br> >> >find it:<br> >> >/usr/src # find . -iname '*/mk_cmds/*'<br> >> >/usr/src #<br> >> ><br> >> >"/usr/src" is mounted<br> >> >zroot/usr/src on /usr/src (zfs, local, noexec, nosuid, nfsv4acls)<br> >> ><br> >> >AND what makes it even more mysterious:<br> >> ># cd /usr/src<br> >> ># make buildetc<br> >> ><br> >> >works without reporting any error. If called from "etcupdate extract" it<br> >> >fails.<br> >> ><br> >> >It works for FreeBSD:<br> >> >- stable/13<br> >> >- stable/14<br> >> ><br> >> >but not for:<br> >> >- stable/15<br> >> ><br> >> >Full log is attached.<br> >> ><br> >> >On Tue, Nov 4, 2025 at 4:00 PM Sulev-Madis Silber <<br> >> ><a href="mailto:freebsd-current-freebsd-org111@ketas.si.pri.ee" target="_blank">freebsd-current-freebsd-org111@ketas.si.pri.ee</a>> wrote:<br> >> ><br> >> >> something bad clearly happened in that machine if it's able to selfhost<br> >> >> build itself and work, except etcupdate. and that for a long time<br> >> >><br> >> >> i don't see any reason getting pissed about his machine mysteriously not<br> >> >> working, despite i haven't broken any machine myself since i installed<br> >> >> first fbsd, 4.6<br> >> >><br> >> >> unsure how to go from here. unsure if src.conf or make.conf matters here<br> >> >><br> >> >> if this happens in currently actively supported fbsd release, maybe<br> >> >> etcupdate needs a bugfix to cleanup for edge cases<br> >> >><br> >> >> if it's in unsupported, that should not cause any "pissages" either. fix<br> >> >> is somewhere<br> >> >><br> >> >> so, the admin updated files manually because he was not able to get it<br> >> >> working? i bet you can recreate it and fix it for future cases<br> >> >><br> >> >> since i haven't ever broken etcupdate, i don't know which data it uses<br> >> as<br> >> >> input. but seems like it reads wrong data out of somewhere. and entire<br> >> >> machine works, except this?<br> >> >><br> >> >> i mean if this was bad upgrade leftover, how to fix? i mean doesn't<br> >> >> etcupdate get rework now, for pkgbase. while we do that, maybe it could<br> >> be<br> >> >> made to handle this. or maybe even given non-destructive nuke mode so<br> >> >> people can start clean<br> >> >><br> >> >> i don't think it's really entirely user error, considering he didn't<br> >> break<br> >> >> the machine<br> >> >><br> >> >> i'm curious too. at worst you could do other install in another machine<br> >> >> and compare dirs as something must differ there. then, according to<br> >> >> findings, fix the old machine. maybe while doing this, some new bugfix<br> >> idea<br> >> >> appears<br> >> >><br> >> >> but hell with getting mad over machine. he's pissed too, everyone's<br> >> >> pissed, server is not working and no productive work have been done here<br> >> >><br> >> >> maybe something got lost in translation too<br> >> >><br> >> >><br> >> ><br> >><br> >><br> ><br> <br> </blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr">Thomas</div></div>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2BrGx5e44JhSLTwjLJt=LXT47e-%2B2n=uicdqAbZgx6raXJg=Ow>