Date: Tue, 11 Jul 2023 21:37:26 -0700 From: Kevin Oberman <rkoberman@gmail.com> To: Mark Millard <marklmi@yahoo.com> Cc: "Hurling, Rainer" <rhurlin@freebsd.org>, FreeBSD Mailing List <freebsd-ports@freebsd.org> Subject: Re: Problem with the package builds Message-ID: <CAN6yY1vA3V8KhCK0a=VdSCJMYOMsH6xa2BFYe3PfF-ZpmjG-=g@mail.gmail.com> In-Reply-To: <3670BA04-D165-44C3-B5DF-5B5AFD4CFD30@yahoo.com> References: <2DCC0132-0307-4A73-A473-B749ABF87367.ref@yahoo.com> <2DCC0132-0307-4A73-A473-B749ABF87367@yahoo.com> <1aaeafcc-9812-64bd-a369-bae7b9fc0e5f@gwdg.de> <422DBC3B-D85F-4AFB-ABDE-842A08482EC8@yahoo.com> <58250d08-ce0a-c1ac-ed15-7d55d517218e@gwdg.de> <2ea168aa-3e0e-ee31-d4a3-82b4ce46b330@gwdg.de> <CAN6yY1sqtLCbqW4%2BxtXfduN7WPPQBnd=FbiknnvDFtGMisveaw@mail.gmail.com> <78c1497b-1ec7-08b6-a407-d671e3c4f9dd@gwdg.de> <ED4C0309-19BB-496D-AA48-462A3303412F@yahoo.com> <CAN6yY1sMceeGHOZHECUHTQKa2UXoXt9x5fdm7A5-b-uyaxCzWQ@mail.gmail.com> <44201182-73C9-46BC-A846-7DC765F5B81F@yahoo.com> <CAN6yY1uhOTMKkfQZ09Esz4VW-jZ-CNuet_CAKTQ4rTO-pnL2ag@mail.gmail.com> <3670BA04-D165-44C3-B5DF-5B5AFD4CFD30@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000c59e94060042c6b2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jul 11, 2023 at 9:15=E2=80=AFPM Mark Millard <marklmi@yahoo.com> wr= ote: > On Jul 11, 2023, at 21:05, Kevin Oberman <rkoberman@gmail.com> wrote: > > > On Tue, Jul 11, 2023 at 1:10=E2=80=AFPM Mark Millard <marklmi@yahoo.com= > wrote: > > On Jul 11, 2023, at 12:57, Kevin Oberman <rkoberman@gmail.com> wrote: > > > > > On Mon, Jul 10, 2023 at 9:42=E2=80=AFPM Mark Millard <marklmi@yahoo.c= om> > wrote: > > > On Jul 10, 2023, at 21:27, Rainer Hurling <rhurlin@gwdg.de> wrote: > > > > > > > As I understand it, the ports-mgmt/pkg of the system running > Poudriere must be updated beforehand? > > > > > > > > At least on my side, this seems to work as expected :) > > > > > > > > > > poudriere builds pkg updates first (if needed) and then uses the pkg = it > > > built for building the later ports into packages. > > > > > > But, after the restarts of main-* builds, the FreeBSD build servers a= re > > > still showing examples were, after an 1hr, some builds are still in > > > build-depends. Also there was an example I saw were after 1.5 hr it w= as > > > still in run-depends. > > > > > > It may be that things are improved but not fully fixed relative to > > > some performance issues. > > > > > > =3D=3D=3D > > > Mark Millard > > > marklmi at yahoo.com > > > A new build started this morning at 1:06 UTC and, with pkg-1.20.2, > it's better, but not much. It's running at 21 packages/hour, a 100% > improvement on the last attempt which appears to have been killed last > night. The logs indicate the installation of dependencies, but I don't se= e > any sign of caching. It's a re-install every time. (I may not understand > how poudriere does things, but I am pretty sure that caching is done.) > > > > Just about "caching" relative to "poudriere bulk" builds . . . > > > > Nope. At the end of a builder run of a port build the context is > destroyed. > > At the start of the builder building its next port the context is > recreated > > from scratch. The only ports installed are exactly the declared > dependencies, > > no more, no less, for the new port to be built. > > > > Caching installed state would imply access to ports from prior build > > activity that do not apply: It would make the build environment pollute= d > with > > irrelevant history. poudriere's purpose is to have a "clean-room" > context for > > each port build. Thus its construction of such a context for each port > build. > > > > Caching vs. not is not the source of the large increase in how long > things > > take to build. > > > > =3D=3D=3D > > Mark Millard > > marklmi at yahoo.com > > OK. I knew it started in a clean jail. I just thought that it might use > some caching technique to speed repeated installs of a single port... not > that I have any idea of how this might be done. > > > > I think I'll monitor the speed of installs. I iish build logs included > timestamps > > FYI: Freshports now shows a pkg 1.20.3 described with: > > pkg*: new regression fixes release > > Changes: > - speed up pkg add again, and greatly reduce its memory footprint > - more compatibility with libfetch (SSL_* variables) > - fixed FETCH_TIMEOUT adaptation to libcurl > > Looks to have been committed about 16 hr ago. > > Last I looked the official package builders had not been stopped > and restarted with a newer ports tree yet. > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > As I expected, each package install takes a very long time. It appears that the average time to install a package with 1.20.2 was running about 45-50 seconds. Hopefully this latest update will fix that. No rush to stop the current build, I guess. The next build won't start until 1:00 UTC tomorrow. It might be that Bapt wants some time to work with it before. killing the current build or just entered a request to the builds manager in some timezone who sees no rush. --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000c59e94060042c6b2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_default" style=3D"fon= t-family:tahoma,sans-serif;font-size:small">On Tue, Jul 11, 2023 at 9:15=E2= =80=AFPM Mark Millard <<a href=3D"mailto:marklmi@yahoo.com">marklmi@yaho= o.com</a>> wrote:</div></div><div class=3D"gmail_quote"><blockquote clas= s=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid r= gb(204,204,204);padding-left:1ex">On Jul 11, 2023, at 21:05, Kevin Oberman = <<a href=3D"mailto:rkoberman@gmail.com" target=3D"_blank">rkoberman@gmai= l.com</a>> wrote:<br> <br> > On Tue, Jul 11, 2023 at 1:10=E2=80=AFPM Mark Millard <<a href=3D"ma= ilto:marklmi@yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>> wrote:<= br> > On Jul 11, 2023, at 12:57, Kevin Oberman <<a href=3D"mailto:rkoberm= an@gmail.com" target=3D"_blank">rkoberman@gmail.com</a>> wrote:<br> > <br> > > On Mon, Jul 10, 2023 at 9:42=E2=80=AFPM Mark Millard <<a href= =3D"mailto:marklmi@yahoo.com" target=3D"_blank">marklmi@yahoo.com</a>> w= rote:<br> > > On Jul 10, 2023, at 21:27, Rainer Hurling <<a href=3D"mailto:r= hurlin@gwdg.de" target=3D"_blank">rhurlin@gwdg.de</a>> wrote:<br> > > <br> > > > As I understand it, the ports-mgmt/pkg of the system running= Poudriere must be updated beforehand?<br> > > > <br> > > > At least on my side, this seems to work as expected :)<br> > > > <br> > > <br> > > poudriere builds pkg updates first (if needed) and then uses the = pkg it<br> > > built for building the later ports into packages.<br> > > <br> > > But, after the restarts of main-* builds, the FreeBSD build serve= rs are<br> > > still showing examples were, after an 1hr, some builds are still = in<br> > > build-depends. Also there was an example I saw were after 1.5 hr = it was<br> > > still in run-depends.<br> > > <br> > > It may be that things are improved but not fully fixed relative t= o<br> > > some performance issues.<br> > > <br> > > =3D=3D=3D<br> > > Mark Millard<br> > > marklmi at <a href=3D"http://yahoo.com" rel=3D"noreferrer" target= =3D"_blank">yahoo.com</a><br> > >=C2=A0 A new build started this morning at 1:06 UTC and, with pkg-= 1.20.2, it's better, but not much. It's running at 21 packages/hour= , a 100% improvement on the last attempt which appears to have been killed = last night. The logs indicate the installation of dependencies, but I don&#= 39;t see any sign of caching. It's a re-install every time. (I may not = understand how poudriere does things, but I am pretty sure that caching is = done.)<br> > <br> > Just about "caching" relative to "poudriere bulk" = builds . . .<br> > <br> > Nope. At the end of a builder run of a port build the context is destr= oyed.<br> > At the start of the builder building its next port the context is recr= eated<br> > from scratch. The only ports installed are exactly the declared depend= encies,<br> > no more, no less, for the new port to be built.<br> > <br> > Caching installed state would imply access to ports from prior build<b= r> > activity that do not apply: It would make the build environment pollut= ed with<br> > irrelevant history. poudriere's purpose is to have a "clean-r= oom" context for<br> > each port build. Thus its construction of such a context for each port= build.<br> > <br> > Caching vs. not is not the source of the large increase in how long th= ings<br> > take to build.<br> > <br> > =3D=3D=3D<br> > Mark Millard<br> > marklmi at <a href=3D"http://yahoo.com" rel=3D"noreferrer" target=3D"_= blank">yahoo.com</a><br> > OK. I knew it started in a clean jail. I just thought that it might us= e some caching technique to speed repeated installs of a single port... not= that I have any idea of how this might be done.<br> > <br> > I think I'll monitor the speed of installs. I iish build logs incl= uded timestamps<br> <br> FYI: Freshports now shows a pkg 1.20.3 described with:<br> <br> pkg*: new regression fixes release<br> <br> Changes:<br> - speed up pkg add again, and greatly reduce its memory footprint<br> - more compatibility with libfetch (SSL_* variables)<br> - fixed FETCH_TIMEOUT adaptation to libcurl<br> <br> Looks to have been committed about 16 hr ago.<br> <br> Last I looked the official package builders had not been stopped<br> and restarted with a newer ports tree yet.<br> <br> =3D=3D=3D<br> Mark Millard<br> marklmi at <a href=3D"http://yahoo.com" rel=3D"noreferrer" target=3D"_blank= ">yahoo.com</a><br> <br> </blockquote></div><div style=3D"font-family:tahoma,sans-serif;font-size:sm= all" class=3D"gmail_default">As I expected, each package install takes a ve= ry long time. It appears that the average time to install a package with 1.= 20.2 was running about 45-50 seconds. Hopefully this latest update will fix= that.</div><div style=3D"font-family:tahoma,sans-serif;font-size:small" cl= ass=3D"gmail_default">No rush to stop the current build, I guess. The next = build won't start until 1:00 UTC tomorrow. It might be that Bapt wants = some time to work with it before. killing the current build or just entered= a request to the builds manager in some timezone who sees no rush.<br></di= v><span class=3D"gmail_signature_prefix">-- </span><br><div dir=3D"ltr" cla= ss=3D"gmail_signature"><div dir=3D"ltr"><div><div dir=3D"ltr"><div><div dir= =3D"ltr"><div><div dir=3D"ltr">Kevin Oberman, Part time kid herder and reti= red Network Engineer<br>E-mail: <a href=3D"mailto:rkoberman@gmail.com" targ= et=3D"_blank">rkoberman@gmail.com</a><br></div><div>PGP Fingerprint: D03FB9= 8AFA78E3B78C1694B318AB39EF1B055683</div></div></div></div></div></div></div= ></div></div> --000000000000c59e94060042c6b2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAN6yY1vA3V8KhCK0a=VdSCJMYOMsH6xa2BFYe3PfF-ZpmjG-=g>