Date: Sat, 14 Sep 2024 19:38:43 -0400 From: John Schneider <john.a.schneider@gmail.com> To: Mark Millard <marklmi@yahoo.com> Cc: FreeBSD Mailing List <freebsd-ports@freebsd.org> Subject: Re: Official package builder poudriere.conf update? Message-ID: <CAJdGomXFtic4a6fPoEfgegPn04nibUOxi=e3Ko1Ztgpt8fUG2g@mail.gmail.com> In-Reply-To: <4EB71408-1A4D-4C05-BAED-8756EE6464ED@yahoo.com> References: <8843FFE6-84C9-4DCE-8524-7B912D6C43DC.ref@yahoo.com> <8843FFE6-84C9-4DCE-8524-7B912D6C43DC@yahoo.com> <CAJdGomWgfDrbFV-uyZQzfO2M%2Bp0g%2BSUW%2BkGVUW4oyQYPxxeceg@mail.gmail.com> <4EB71408-1A4D-4C05-BAED-8756EE6464ED@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000c0d8cd06221cd7e9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Excellent. I'm glad poudriere has this change already ( https://github.com/freebsd/poudriere/commit/9ea59e59f12f2b77699de9cafa5ceea= 1f7814eaf#diff-afbea33008c9478b94f3379278925c57204b3283411000094ee48d238829= 3a5f ). I hadn't realized common.sh had been updated and I simply needed to take out the related config from my poudriere.conf I'm using 3.4.1_1. Thanks for the assistance. On Sat, Sep 14, 2024 at 7:05=E2=80=AFPM Mark Millard <marklmi@yahoo.com> wr= ote: > As of 2023-Dec-19 or source when FreeBSD's poudriere on the official > builders > started being based on 3.4.0. So the MAX_FILES default is 8192, not 4096, > as > of that change. Numerous official builds succeed based on the default > value, > including just recently. > > If there is an explicit MAX_FILES_vscode=3D4096 somewhere that would use > poudriere > 3.4.0 or later, the assignment should simply be deleted (instead of > separately raised). > > What version of poudriere are you using? Is it based on 3.4.0 or later? > > Mark > > > On Sep 14, 2024, at 15:45, John Schneider <john.a.schneider@gmail.com> > wrote: > > Hi Mark. Thanks for this very thorough explanation. I'm currently > successful at building electron via poudriere with multiple cores in a > reasonable amount of time, with no special config. The thing that was > biting me was the maximum number of files knob for vscode > (MAX_FILES_vscode) in poudriere.conf. I imagine we're really close to th= e > limit with value of 4096 at present. Perhaps sometimes 4K is enough and > with VSCode source code updates we go back and forth over this limit? > > I can currently reproduce the build error described in bug > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871 a > MAX_FILES_vscode=3D4096 and the build is successful with a higher value. = I > don't know if a value of 8192 is appropriate or not but it definitely fix= es > the package build. > > Assuming we continue to have official stable package builds for electron, > I think the master poudriere config needs to change to have > MAX_FILES_vscode be raised. > > Thanks, > John > > On Fri, Sep 13, 2024 at 9:12=E2=80=AFPM Mark Millard <marklmi@yahoo.com> = wrote: > >> John Schneider <john.a.schneider_at_gmail.com> wrote on >> Date: Fri, 13 Sep 2024 21:58:52 UTC : >> >> > I've noticed the FreeBSD package mirrors (pkg.freebsd.org) haven't >> included >> > packages for editors/vscode. Reference: >> > http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/ >> >> http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/ is >> frozen as things were just before 14.1-RELEASE was made: >> back on 2024-May-19. It is never updated as far as I know. >> So: as-is, including what failed to be built at the time. >> It is what is incuded on the official release DVD's that >> have the built port packages. >> >> http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_0/ is >> similar but goes back to 2023-Oct-15. >> >> The port package repos for 14.x that are updated over time >> are: >> >> http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/quarterly/ >> http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/latest/ >> >> In my testing just now, use of: >> >> url: "pkg+https://pkg.FreeBSD.org/${ABI}/quarterly >> <https://pkg.freebsd.org/$%7BABI%7D/quarterly>", >> >> did not find vscode as stands. Later below I report >> on why/how. But use of: >> >> url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest >> <https://pkg.freebsd.org/$%7BABI%7D/latest>", >> >> did find vscode. >> >> I used a chroot into a stable/14.1 context on a >> machine that booted main [so: 15] (both kernel and >> world). releng/14.* and stable/14 use the same >> pacakge repos via the ABI text also being the same >> in the URL's. . . . >> >> # file /bin/sh >> /bin/sh: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), >> dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.1 >> (1401501), FreeBSD-style, stripped >> >> # pkg search vscode >> vscode-1.92.2 Visual Studio Code - Open Source ("Code - >> OSS") >> >> This was after a round of it synchronizing my context: >> >> # env ABI=3DFreeBSD:14:amd64 IGNORE_OSVERSION=3Dyes pkg search vscode >> pkg: Repository FreeBSD has a wrong packagesite, need to re-create >> database >> vscode-1.92.2 Visual Studio Code - Open Source ("Code - >> OSS") >> >> https://www.freshports.org/editors/vscode/ reports only 3 vscode >> builds: >> >> FreeBSD:13:latest has 1.92.2 >> FreeBSD:14:latest has 1.92.2 >> FreeBSD:15:latest has 1.92.2 >> >> Also, vscode is listed as depending on: devel/electron30 at >> this time. >> >> It is common for all but one devel/electron* to be >> disabled for pkg builds: "blacklisted". >> >> >> https://pkg-status.freebsd.org/beefy20/build.html?mastername=3D140amd64-= quarterly&build=3D59d8804dcdd7 >> >> (started 2024-Sep-05) reports that vscode was skipped because >> electron29-29.4.6 did not build at the time. Note that this >> is a September example of quarterly. >> >> In turn it reports that electron29-29.4.6 was an Ingored Port >> for the Reason: "Blacklisted". >> >> In essence the electron* 's are treated has too many using too >> many resources and time to build them all without other >> negative tradeoffs for the overall context. >> >> > Could this be because the official package vscode build is failing? >> >> I've only shown the one example. No claim that a electron* >> is the only way for vscode builds to fail. >> >> > I was >> > able to overcome a problem with too many file descriptors during the >> build >> > by changing my poudriere.conf file to double MAX_FILES_vscode=3D4096 t= o >> > MAX_FILES_vscode=3D8192 >> >> That will not help with the required electron* not being >> available. >> >> > This was previously addressed in bug 242871 - >> > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871 >> > >> > If there's a better mailing list for this issue, please refer me to it= . >> >> >> I'll note that on the aarch64 (so: Tier 1) builder for >> main (ampere2) the following all fail to build after >> between 4hr and 49hr of attempting to build >> (build/timeout failures): >> >> www/ungoogled-chromium >> www/iridium >> www/chromium >> devel/electron30 >> >> For how things are configured, attempting to build those >> seems to mostly just be wasted resources and time for >> aarch64. Other devel/electron* would likely be the same. >> (Other configurations would have other tradeoffs and >> might not prove sufficient for the overall context.) >> >> I'll note that the next biggest time taker fails for >> other reasons in under 3hr. Also, the largest time >> taker that builds is www/qt6-webengine at between >> 27hr and 28hr currently. The next 2 builds by time >> take between 19hr and 20hr currently >> ( databases/mongodb80 and www/qt5-webengine ). >> >> There is a huge difference for building the various: >> >> www/ungoogled-chromium >> www/iridium >> www/chromium >> devel/electron* >> >> compared to anything else. (I'm not spanning >> prerequisites here: just the direct part of >> the overall build for each.) >> >> I've no clue what time limits would allow those to >> all build on the ampere*, leaving things configured >> the same otherwise. I also do not know what would >> happen if they all started building at about the >> same time, up to 13 builders, given 13 FreeBSD cpus >> in each ampere* . >> >> Anything based on a devel/electron* suffers the >> consequences of the resource/time issues for the >> matching devel/electron* having to be built. >> >> =3D=3D=3D >> Mark Millard >> marklmi at yahoo.com >> >> > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > --000000000000c0d8cd06221cd7e9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Excellent.=C2=A0 I'm glad poudriere has this chan= ge already ( <a href=3D"https://github.com/freebsd/poudriere/commit/9ea59e5= 9f12f2b77699de9cafa5ceea1f7814eaf#diff-afbea33008c9478b94f3379278925c57204b= 3283411000094ee48d2388293a5f">https://github.com/freebsd/poudriere/commit/9= ea59e59f12f2b77699de9cafa5ceea1f7814eaf#diff-afbea33008c9478b94f3379278925c= 57204b3283411000094ee48d2388293a5f</a> ).=C2=A0 I hadn't realized commo= n.sh had been updated and I simply needed to take out the related config fr= om my poudriere.conf=C2=A0 I'm using 3.4.1_1.</div><div><br></div><div>= Thanks for the assistance.<br></div><div><br></div></div><br><div class=3D"= gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Sat, Sep 14, 2024 at = 7:05=E2=80=AFPM Mark Millard <<a href=3D"mailto:marklmi@yahoo.com">markl= mi@yahoo.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" styl= e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin= g-left:1ex"><div>As of 2023-Dec-19 or source when FreeBSD's poudriere o= n the official builders<div>started being based on 3.4.0. So the MAX_FILES = default is 8192, not 4096, as</div><div>of that change. Numerous official b= uilds succeed based on the default value,</div><div>including just recently= .</div><div><div><br></div><div>If there is an explicit=C2=A0<span style=3D= "color:rgb(0,0,0)">MAX_FILES_vscode=3D4096</span><font color=3D"#000000">= =C2=A0somewhere that would=C2=A0use poudriere</font></div><div><font color= =3D"#000000"><span>3.4.0 or later, the assignment should simply be deleted = (instead of separately=C2=A0raised).</span></font></div><div><br></div><div= >What version of poudriere are you using? Is it based on 3.4.0 or later?</d= iv><div><br></div><div>Mark</div><div><br></div><div><div><br><div>On Sep 1= 4, 2024, at 15:45, John Schneider <<a href=3D"mailto:john.a.schneider@gm= ail.com" target=3D"_blank">john.a.schneider@gmail.com</a>> wrote:</div><= br><div><div dir=3D"ltr"><div>Hi Mark.=C2=A0 Thanks for this very thorough = explanation.=C2=A0 I'm currently successful at building electron via po= udriere with multiple cores in a reasonable amount of time, with no special= config.=C2=A0 The thing that was biting me was the maximum number of files= knob for vscode (MAX_FILES_vscode) in poudriere.conf.=C2=A0 I imagine we&#= 39;re really close to the limit with value of 4096 at present.=C2=A0 Perhap= s sometimes 4K is enough and with VSCode source code updates we go back and= forth over this limit?</div><div><br></div><div>I can currently reproduce = the build error described in bug=C2=A0<a href=3D"https://bugs.freebsd.org/b= ugzilla/show_bug.cgi?id=3D242871" target=3D"_blank">https://bugs.freebsd.or= g/bugzilla/show_bug.cgi?id=3D242871</a> a MAX_FILES_vscode=3D4096 and the b= uild is successful with a higher value.=C2=A0 I don't know if a value o= f 8192 is appropriate or not but it definitely fixes the package build.<br>= </div><div><br></div><div>Assuming we continue to have official stable pack= age builds for electron, I think the master poudriere config needs to chang= e=C2=A0 to have MAX_FILES_vscode be raised.</div><div><br></div><div>Thanks= ,</div><div>John<br></div></div><br><div class=3D"gmail_quote"><div dir=3D"= ltr" class=3D"gmail_attr">On Fri, Sep 13, 2024 at 9:12=E2=80=AFPM Mark Mill= ard <<a href=3D"mailto:marklmi@yahoo.com" target=3D"_blank">marklmi@yaho= o.com</a>> wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"ma= rgin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:= 1ex">John Schneider <<a href=3D"http://john.a.schneider_at_gmail.com/" r= el=3D"noreferrer" target=3D"_blank">john.a.schneider_at_gmail.com</a>> w= rote on<br> Date: Fri, 13 Sep 2024 21:58:52 UTC :<br> <br> > I've noticed the FreeBSD package mirrors (<a href=3D"http://pkg.fr= eebsd.org/" rel=3D"noreferrer" target=3D"_blank">pkg.freebsd.org</a>) haven= 't included<br> > packages for editors/vscode. Reference:<br> > <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/" re= l=3D"noreferrer" target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:a= md64/release_1/</a><br> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/" rel=3D"= noreferrer" target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/= release_1/</a> is<br> frozen as things were just before 14.1-RELEASE was made:<br> back on 2024-May-19. It is never updated as far as I know.<br> So: as-is, including what failed to be built at the time.<br> It is what is incuded on the official release DVD's that<br> have the built port packages.<br> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_0/" rel=3D"= noreferrer" target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/= release_0/</a> is<br> similar but goes back to 2023-Oct-15.<br> <br> The port package repos for 14.x that are updated over time<br> are:<br> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/quarterly/" rel=3D"= noreferrer" target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/= quarterly/</a> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/latest/" rel=3D"nor= eferrer" target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/lat= est/</a> <br> <br> In my testing just now, use of:<br> <br> url: "pkg+<a href=3D"https://pkg.freebsd.org/$%7BABI%7D/quarterly" rel= =3D"noreferrer" target=3D"_blank">https://pkg.FreeBSD.org/${ABI}/quarterly<= /a>",<br> <br> did not find vscode as stands. Later below I report<br> on why/how. But use of:<br> <br> url: "pkg+<a href=3D"https://pkg.freebsd.org/$%7BABI%7D/latest" rel=3D= "noreferrer" target=3D"_blank">https://pkg.FreeBSD.org/${ABI}/latest</a>&qu= ot;,<br> <br> did find vscode.<br> <br> I used a chroot into a stable/14.1 context on a<br> machine that booted main [so: 15] (both kernel and<br> world). releng/14.* and stable/14 use the same<br> pacakge repos via=C2=A0 the ABI text also being the same<br> in the URL's. . . .<br> <br> # file /bin/sh<br> /bin/sh: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynami= cally linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.1 (1401501),= FreeBSD-style, stripped<br> <br> # pkg search vscode<br> vscode-1.92.2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Visual Studio Code - Open Source ("Code - OSS")<br> <br> This was after a round of it synchronizing my context:<br> <br> # env ABI=3DFreeBSD:14:amd64 IGNORE_OSVERSION=3Dyes pkg search vscode<br> pkg: Repository FreeBSD has a wrong packagesite, need to re-create database= <br> vscode-1.92.2=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= Visual Studio Code - Open Source ("Code - OSS")<br> <br> <a href=3D"https://www.freshports.org/editors/vscode/" rel=3D"noreferrer" t= arget=3D"_blank">https://www.freshports.org/editors/vscode/</a>; reports onl= y 3 vscode<br> builds:<br> <br> FreeBSD:13:latest has 1.92.2<br> FreeBSD:14:latest has 1.92.2<br> FreeBSD:15:latest has 1.92.2<br> <br> Also, vscode is listed as depending on: devel/electron30 at<br> this time.<br> <br> It is common for all but one devel/electron* to be<br> disabled for pkg builds: "blacklisted".<br> <br> <a href=3D"https://pkg-status.freebsd.org/beefy20/build.html?mastername=3D1= 40amd64-quarterly&build=3D59d8804dcdd7" rel=3D"noreferrer" target=3D"_b= lank">https://pkg-status.freebsd.org/beefy20/build.html?mastername=3D140amd= 64-quarterly&build=3D59d8804dcdd7</a><br> <br> (started 2024-Sep-05) reports that vscode was skipped because<br> electron29-29.4.6 did not build at the time. Note that this<br> is a September example of quarterly.<br> <br> In turn it reports that electron29-29.4.6 was an Ingored Port<br> for the Reason: "Blacklisted".<br> <br> In essence the electron* 's are treated has too many using too<br> many resources and time to build them all without other<br> negative tradeoffs for the overall context.<br> <br> > Could this be because the official package vscode build is failing?<br= > <br> I've only shown the one example. No claim that a electron*<br> is the only way for vscode builds to fail.<br> <br> > I was<br> > able to overcome a problem with too many file descriptors during the b= uild<br> > by changing my poudriere.conf file to double MAX_FILES_vscode=3D4096 t= o<br> > MAX_FILES_vscode=3D8192<br> <br> That will not help with the required electron* not being<br> available.<br> <br> > This was previously addressed in bug 242871 -<br> > <a href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871"= rel=3D"noreferrer" target=3D"_blank">https://bugs.freebsd.org/bugzilla/sho= w_bug.cgi?id=3D242871</a><br> > <br> > If there's a better mailing list for this issue, please refer me t= o it.<br> <br> <br> I'll note that on the aarch64 (so: Tier 1) builder for<br> main (ampere2) the following all fail to build after<br> between 4hr and 49hr of attempting to build<br> (build/timeout failures):<br> <br> www/ungoogled-chromium<br> www/iridium<br> www/chromium<br> devel/electron30<br> <br> For how things are configured, attempting to build those<br> seems to mostly just be wasted resources and time for<br> aarch64. Other devel/electron* would likely be the same.<br> (Other configurations would have other tradeoffs and<br> might not prove sufficient for the overall context.)<br> <br> I'll note that the next biggest time taker fails for<br> other reasons in under 3hr. Also, the largest time<br> taker that builds is www/qt6-webengine at between<br> 27hr and 28hr currently. The next 2 builds by time<br> take between 19hr and 20hr currently<br> ( databases/mongodb80 and www/qt5-webengine ).<br> <br> There is a huge difference for building the various:<br> <br> www/ungoogled-chromium<br> www/iridium<br> www/chromium<br> devel/electron*<br> <br> compared to anything else. (I'm not spanning<br> prerequisites here: just the direct part of<br> the overall build for each.)<br> <br> I've no clue what time limits would allow those to<br> all build on the ampere*, leaving things configured<br> the same otherwise. I also do not know what would<br> happen if they all started building at about the<br> same time, up to 13 builders, given 13 FreeBSD cpus<br> in each ampere* .<br> <br> Anything based on a devel/electron* suffers the<br> consequences of the resource/time issues for the<br> matching devel/electron* having to be built.<br> <br> =3D=3D=3D<br> Mark Millard<br> marklmi at <a href=3D"http://yahoo.com/" rel=3D"noreferrer" target=3D"_blan= k">yahoo.com</a><br> <br> </blockquote></div> </div></div><br><div> <div dir=3D"auto" style=3D"color:rgb(0,0,0);letter-spacing:normal;text-alig= n:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing= :0px;text-decoration:none"><div dir=3D"auto" style=3D"color:rgb(0,0,0);lett= er-spacing:normal;text-align:start;text-indent:0px;text-transform:none;whit= e-space:normal;word-spacing:0px;text-decoration:none"><div dir=3D"auto" sty= le=3D"color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0= px;text-transform:none;white-space:normal;word-spacing:0px"><div style=3D"o= verflow-wrap: break-word;"><div style=3D"color:rgb(0,0,0);font-variant-caps= :normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transfo= rm:none;white-space:normal;word-spacing:0px"><div style=3D"margin:0px;line-= height:normal"><font face=3D"Courier New">=3D=3D=3D</font></div><div style= =3D"margin:0px;line-height:normal"><font face=3D"Courier New">Mark Millard<= /font></div><div style=3D"margin:0px;line-height:normal"><font face=3D"Cour= ier New"><span style=3D"font-size:12px">marklmi at <a href=3D"http://yahoo.= com" target=3D"_blank">yahoo.com</a><br></span></font></div></div></div></d= iv></div></div> </div> <br></div></div></div></blockquote></div> --000000000000c0d8cd06221cd7e9--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJdGomXFtic4a6fPoEfgegPn04nibUOxi=e3Ko1Ztgpt8fUG2g>