Date: Sat, 14 Sep 2024 16:04:56 -0700 From: Mark Millard <marklmi@yahoo.com> To: John Schneider <john.a.schneider@gmail.com> Cc: FreeBSD Mailing List <freebsd-ports@freebsd.org> Subject: Re: Official package builder poudriere.conf update? Message-ID: <4EB71408-1A4D-4C05-BAED-8756EE6464ED@yahoo.com> In-Reply-To: <CAJdGomWgfDrbFV-uyZQzfO2M%2Bp0g%2BSUW%2BkGVUW4oyQYPxxeceg@mail.gmail.com> References: <8843FFE6-84C9-4DCE-8524-7B912D6C43DC.ref@yahoo.com> <8843FFE6-84C9-4DCE-8524-7B912D6C43DC@yahoo.com> <CAJdGomWgfDrbFV-uyZQzfO2M%2Bp0g%2BSUW%2BkGVUW4oyQYPxxeceg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_556B8EC8-243C-4CD1-80C5-3C0D8D7E98B3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 As of 2023-Dec-19 or source when FreeBSD's poudriere on the official = builders started being based on 3.4.0. So the MAX_FILES default is 8192, not = 4096, as of that change. Numerous official builds succeed based on the default = value, including just recently. If there is an explicit MAX_FILES_vscode=3D4096 somewhere that would use = poudriere 3.4.0 or later, the assignment should simply be deleted (instead of = separately raised). What version of poudriere are you using? Is it based on 3.4.0 or later? Mark On Sep 14, 2024, at 15:45, John Schneider <john.a.schneider@gmail.com> = wrote: Hi Mark. Thanks for this very thorough explanation. I'm currently = successful at building electron via poudriere with multiple cores in a = reasonable amount of time, with no special config. The thing that was = biting me was the maximum number of files knob for vscode = (MAX_FILES_vscode) in poudriere.conf. I imagine we're really close to = the limit with value of 4096 at present. Perhaps sometimes 4K is enough = and with VSCode source code updates we go back and forth over this = limit? I can currently reproduce the build error described in bug = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871 a = MAX_FILES_vscode=3D4096 and the build is successful with a higher value. = I don't know if a value of 8192 is appropriate or not but it definitely = fixes the package build. Assuming we continue to have official stable package builds for = electron, I think the master poudriere config needs to change to have = MAX_FILES_vscode be raised. Thanks, John On Fri, Sep 13, 2024 at 9:12=E2=80=AFPM Mark Millard <marklmi@yahoo.com = <mailto:marklmi@yahoo.com>> wrote: > John Schneider <john.a.schneider_at_gmail.com = <http://john.a.schneider_at_gmail.com/>>; wrote on > Date: Fri, 13 Sep 2024 21:58:52 UTC : >=20 > > I've noticed the FreeBSD package mirrors (pkg.freebsd.org = <http://pkg.freebsd.org/>) haven't included > > packages for editors/vscode. Reference: > > http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/ >=20 > http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/ is > frozen as things were just before 14.1-RELEASE was made: > back on 2024-May-19. It is never updated as far as I know. > So: as-is, including what failed to be built at the time. > It is what is incuded on the official release DVD's that > have the built port packages. >=20 > http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_0/ is > similar but goes back to 2023-Oct-15. >=20 > The port package repos for 14.x that are updated over time > are: >=20 > http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/quarterly/=20 > http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/latest/=20 >=20 > In my testing just now, use of: >=20 > url: "pkg+https://pkg.FreeBSD.org/${ABI}/quarterly = <https://pkg.freebsd.org/$%7BABI%7D/quarterly>", >=20 > did not find vscode as stands. Later below I report > on why/how. But use of: >=20 > url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest = <https://pkg.freebsd.org/$%7BABI%7D/latest>", >=20 > did find vscode. >=20 > I used a chroot into a stable/14.1 context on a > machine that booted main [so: 15] (both kernel and > world). releng/14.* and stable/14 use the same > pacakge repos via the ABI text also being the same > in the URL's. . . . >=20 > # file /bin/sh > /bin/sh: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), = dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.1 = (1401501), FreeBSD-style, stripped >=20 > # pkg search vscode > vscode-1.92.2 Visual Studio Code - Open Source ("Code = - OSS") >=20 > This was after a round of it synchronizing my context: >=20 > # env ABI=3DFreeBSD:14:amd64 IGNORE_OSVERSION=3Dyes pkg search vscode > pkg: Repository FreeBSD has a wrong packagesite, need to re-create = database > vscode-1.92.2 Visual Studio Code - Open Source ("Code = - OSS") >=20 > https://www.freshports.org/editors/vscode/ reports only 3 vscode > builds: >=20 > FreeBSD:13:latest has 1.92.2 > FreeBSD:14:latest has 1.92.2 > FreeBSD:15:latest has 1.92.2 >=20 > Also, vscode is listed as depending on: devel/electron30 at > this time. >=20 > It is common for all but one devel/electron* to be > disabled for pkg builds: "blacklisted". >=20 > = https://pkg-status.freebsd.org/beefy20/build.html?mastername=3D140amd64-qu= arterly&build=3D59d8804dcdd7 >=20 > (started 2024-Sep-05) reports that vscode was skipped because > electron29-29.4.6 did not build at the time. Note that this > is a September example of quarterly. >=20 > In turn it reports that electron29-29.4.6 was an Ingored Port > for the Reason: "Blacklisted". >=20 > In essence the electron* 's are treated has too many using too > many resources and time to build them all without other > negative tradeoffs for the overall context. >=20 > > Could this be because the official package vscode build is failing? >=20 > I've only shown the one example. No claim that a electron* > is the only way for vscode builds to fail. >=20 > > I was > > able to overcome a problem with too many file descriptors during the = build > > by changing my poudriere.conf file to double MAX_FILES_vscode=3D4096 = to > > MAX_FILES_vscode=3D8192 >=20 > That will not help with the required electron* not being > available. >=20 > > This was previously addressed in bug 242871 - > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871 > >=20 > > If there's a better mailing list for this issue, please refer me to = it. >=20 >=20 > I'll note that on the aarch64 (so: Tier 1) builder for > main (ampere2) the following all fail to build after > between 4hr and 49hr of attempting to build > (build/timeout failures): >=20 > www/ungoogled-chromium > www/iridium > www/chromium > devel/electron30 >=20 > For how things are configured, attempting to build those > seems to mostly just be wasted resources and time for > aarch64. Other devel/electron* would likely be the same. > (Other configurations would have other tradeoffs and > might not prove sufficient for the overall context.) >=20 > I'll note that the next biggest time taker fails for > other reasons in under 3hr. Also, the largest time > taker that builds is www/qt6-webengine at between > 27hr and 28hr currently. The next 2 builds by time > take between 19hr and 20hr currently > ( databases/mongodb80 and www/qt5-webengine ). >=20 > There is a huge difference for building the various: >=20 > www/ungoogled-chromium > www/iridium > www/chromium > devel/electron* >=20 > compared to anything else. (I'm not spanning > prerequisites here: just the direct part of > the overall build for each.) >=20 > I've no clue what time limits would allow those to > all build on the ampere*, leaving things configured > the same otherwise. I also do not know what would > happen if they all started building at about the > same time, up to 13 builders, given 13 FreeBSD cpus > in each ampere* . >=20 > Anything based on a devel/electron* suffers the > consequences of the resource/time issues for the > matching devel/electron* having to be built. >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com <http://yahoo.com/>; >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com --Apple-Mail=_556B8EC8-243C-4CD1-80C5-3C0D8D7E98B3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;">As of = 2023-Dec-19 or source when FreeBSD's poudriere on the official = builders<div>started being based on 3.4.0. So the MAX_FILES default is = 8192, not 4096, as</div><div>of that change. Numerous official builds = succeed based on the default value,</div><div>including just = recently.</div><div><div><br></div><div>If there is an = explicit <span style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0);">MAX_FILES_vscode=3D4096</span><font color=3D"#000000"> somewhere= that would use poudriere</font></div><div><font = color=3D"#000000"><span style=3D"caret-color: rgb(0, 0, 0);">3.4.0 or = later, the assignment should simply be deleted (instead of = separately raised).</span></font></div><div><br></div><div>What = version of poudriere are you using? Is it based on 3.4.0 or = later?</div><div><br></div><div>Mark</div><div><br></div><div><div><br><di= v>On Sep 14, 2024, at 15:45, John Schneider = <john.a.schneider@gmail.com> wrote:</div><br = class=3D"Apple-interchange-newline"><div><div dir=3D"ltr"><div>Hi = Mark. Thanks for this very thorough explanation. I'm = currently successful at building electron via poudriere with multiple = cores in a reasonable amount of time, with no special config. The = thing that was biting me was the maximum number of files knob for vscode = (MAX_FILES_vscode) in poudriere.conf. I imagine we're really close = to the limit with value of 4096 at present. Perhaps sometimes 4K = is enough and with VSCode source code updates we go back and forth over = this limit?</div><div><br></div><div>I can currently reproduce the build = error described in bug <a = href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871">https:= //bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871</a> a = MAX_FILES_vscode=3D4096 and the build is successful with a higher = value. I don't know if a value of 8192 is appropriate or not but = it definitely fixes the package = build.<br></div><div><br></div><div>Assuming we continue to have = official stable package builds for electron, I think the master = poudriere config needs to change to have MAX_FILES_vscode be = raised.</div><div><br></div><div>Thanks,</div><div>John<br></div></div><br= ><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On = Fri, Sep 13, 2024 at 9:12=E2=80=AFPM Mark Millard <<a = href=3D"mailto:marklmi@yahoo.com">marklmi@yahoo.com</a>> = wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left:1px solid = rgb(204,204,204);padding-left:1ex">John Schneider <<a = href=3D"http://john.a.schneider_at_gmail.com/" rel=3D"noreferrer" = target=3D"_blank">john.a.schneider_at_gmail.com</a>> wrote on<br> Date: Fri, 13 Sep 2024 21:58:52 UTC :<br> <br> > I've noticed the FreeBSD package mirrors (<a = href=3D"http://pkg.freebsd.org/" rel=3D"noreferrer" = target=3D"_blank">pkg.freebsd.org</a>) haven't included<br> > packages for editors/vscode. Reference:<br> > <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/" = rel=3D"noreferrer" = target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/<= /a><br> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/" = rel=3D"noreferrer" = target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_1/<= /a> is<br> frozen as things were just before 14.1-RELEASE was made:<br> back on 2024-May-19. It is never updated as far as I know.<br> So: as-is, including what failed to be built at the time.<br> It is what is incuded on the official release DVD's that<br> have the built port packages.<br> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_0/" = rel=3D"noreferrer" = target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/release_0/<= /a> is<br> similar but goes back to 2023-Oct-15.<br> <br> The port package repos for 14.x that are updated over time<br> are:<br> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/quarterly/" = rel=3D"noreferrer" = target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/quarterly/<= /a> <br> <a href=3D"http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/latest/" = rel=3D"noreferrer" = target=3D"_blank">http://pkg0.nyi.freebsd.org/FreeBSD:14:amd64/latest/</a>= <br> <br> In my testing just now, use of:<br> <br> url: "pkg+<a href=3D"https://pkg.freebsd.org/$%7BABI%7D/quarterly" = rel=3D"noreferrer" = target=3D"_blank">https://pkg.FreeBSD.org/${ABI}/quarterly</a>",<br>; <br> did not find vscode as stands. Later below I report<br> on why/how. But use of:<br> <br> url: "pkg+<a href=3D"https://pkg.freebsd.org/$%7BABI%7D/latest" = rel=3D"noreferrer" = target=3D"_blank">https://pkg.FreeBSD.org/${ABI}/latest</a>",<br>; <br> did find vscode.<br> <br> I used a chroot into a stable/14.1 context on a<br> machine that booted main [so: 15] (both kernel and<br> world). releng/14.* and stable/14 use the same<br> pacakge repos via the ABI text also being the same<br> in the URL's. . . .<br> <br> # file /bin/sh<br> /bin/sh: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), = dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.1 = (1401501), FreeBSD-style, stripped<br> <br> # pkg search vscode<br> vscode-1.92.2 = Visual Studio Code - Open Source ("Code - OSS")<br> <br> This was after a round of it synchronizing my context:<br> <br> # env ABI=3DFreeBSD:14:amd64 IGNORE_OSVERSION=3Dyes pkg search = vscode<br> pkg: Repository FreeBSD has a wrong packagesite, need to re-create = database<br> vscode-1.92.2 = Visual Studio Code - Open Source ("Code - OSS")<br> <br> <a href=3D"https://www.freshports.org/editors/vscode/" rel=3D"noreferrer" = target=3D"_blank">https://www.freshports.org/editors/vscode/</a>; reports = only 3 vscode<br> builds:<br> <br> FreeBSD:13:latest has 1.92.2<br> FreeBSD:14:latest has 1.92.2<br> FreeBSD:15:latest has 1.92.2<br> <br> Also, vscode is listed as depending on: devel/electron30 at<br> this time.<br> <br> It is common for all but one devel/electron* to be<br> disabled for pkg builds: "blacklisted".<br> <br> <a = href=3D"https://pkg-status.freebsd.org/beefy20/build.html?mastername=3D140= amd64-quarterly&build=3D59d8804dcdd7" rel=3D"noreferrer" = target=3D"_blank">https://pkg-status.freebsd.org/beefy20/build.html?master= name=3D140amd64-quarterly&build=3D59d8804dcdd7</a><br> <br> (started 2024-Sep-05) reports that vscode was skipped because<br> electron29-29.4.6 did not build at the time. Note that this<br> is a September example of quarterly.<br> <br> In turn it reports that electron29-29.4.6 was an Ingored Port<br> for the Reason: "Blacklisted".<br> <br> In essence the electron* 's are treated has too many using too<br> many resources and time to build them all without other<br> negative tradeoffs for the overall context.<br> <br> > Could this be because the official package vscode build is = failing?<br> <br> I've only shown the one example. No claim that a electron*<br> is the only way for vscode builds to fail.<br> <br> > I was<br> > able to overcome a problem with too many file descriptors during = the build<br> > by changing my poudriere.conf file to double MAX_FILES_vscode=3D4096 = to<br> > MAX_FILES_vscode=3D8192<br> <br> That will not help with the required electron* not being<br> available.<br> <br> > This was previously addressed in bug 242871 -<br> > <a href=3D"https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D242871= " rel=3D"noreferrer" = target=3D"_blank">https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D2428= 71</a><br> > <br> > If there's a better mailing list for this issue, please refer me to = it.<br> <br> <br> I'll note that on the aarch64 (so: Tier 1) builder for<br> main (ampere2) the following all fail to build after<br> between 4hr and 49hr of attempting to build<br> (build/timeout failures):<br> <br> www/ungoogled-chromium<br> www/iridium<br> www/chromium<br> devel/electron30<br> <br> For how things are configured, attempting to build those<br> seems to mostly just be wasted resources and time for<br> aarch64. Other devel/electron* would likely be the same.<br> (Other configurations would have other tradeoffs and<br> might not prove sufficient for the overall context.)<br> <br> I'll note that the next biggest time taker fails for<br> other reasons in under 3hr. Also, the largest time<br> taker that builds is www/qt6-webengine at between<br> 27hr and 28hr currently. The next 2 builds by time<br> take between 19hr and 20hr currently<br> ( databases/mongodb80 and www/qt5-webengine ).<br> <br> There is a huge difference for building the various:<br> <br> www/ungoogled-chromium<br> www/iridium<br> www/chromium<br> devel/electron*<br> <br> compared to anything else. (I'm not spanning<br> prerequisites here: just the direct part of<br> the overall build for each.)<br> <br> I've no clue what time limits would allow those to<br> all build on the ampere*, leaving things configured<br> the same otherwise. I also do not know what would<br> happen if they all started building at about the<br> same time, up to 13 builders, given 13 FreeBSD cpus<br> in each ampere* .<br> <br> Anything based on a devel/electron* suffers the<br> consequences of the resource/time issues for the<br> matching devel/electron* having to be built.<br> <br> =3D=3D=3D<br> Mark Millard<br> marklmi at <a href=3D"http://yahoo.com/" rel=3D"noreferrer" = target=3D"_blank">yahoo.com</a><br> <br> </blockquote></div> </div></div><br><div> <div dir=3D"auto" style=3D"caret-color: rgb(0, 0, 0); color: rgb(0, 0, = 0); letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: = break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;"><div dir=3D"auto" style=3D"caret-color: rgb(0, 0, = 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;"><div dir=3D"auto" style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;"><div = style=3D"word-wrap: break-word; -webkit-nbsp-mode: space; line-break: = after-white-space;"><div style=3D"color: rgb(0, 0, 0); = font-variant-caps: normal; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div style=3D"orphans:= 2; widows: 2; margin: 0px; line-height: normal;"><font face=3D"Courier = New">=3D=3D=3D</font></div><div style=3D"orphans: 2; widows: 2; margin: = 0px; line-height: normal;"><font face=3D"Courier New">Mark = Millard</font></div><div style=3D"orphans: 2; widows: 2; margin: 0px; = line-height: normal;"><font face=3D"Courier New"><span style=3D"font-size:= 12px;">marklmi at = yahoo.com<br></span></font></div></div></div></div></div></div> </div> <br></div></div></body></html>= --Apple-Mail=_556B8EC8-243C-4CD1-80C5-3C0D8D7E98B3--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EB71408-1A4D-4C05-BAED-8756EE6464ED>