From nobody Wed Jul 12 04:37:26 2023 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4R14jN4cD7z4mnvp for ; Wed, 12 Jul 2023 04:37:44 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-yw1-x1129.google.com (mail-yw1-x1129.google.com [IPv6:2607:f8b0:4864:20::1129]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4R14jN1Zmvz3LkV; Wed, 12 Jul 2023 04:37:44 +0000 (UTC) (envelope-from kob6558@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-yw1-x1129.google.com with SMTP id 00721157ae682-579ed2829a8so68712557b3.1; Tue, 11 Jul 2023 21:37:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689136663; x=1691728663; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=rm70YAiA7B+6NetmnfdpkQvTkq/OM9KidvrlLi9FC60=; b=EuLSQngJs61gmBl9vgCdoD+oqwqoSV27Fy0WM6tfw6jDrMbIVhwrSa4zDIQJAy4rMn iO8gbJKQuJ0mY5yM3d71QUR35oSBKaH359J5sFWTd93yMSBicQsA/TMzQbkbWWV12Vgx Yq2se0HZknJOTQwDXvIIXHnvaFLGlzGAXt1dBPYzz4ee9DX/QIZOsvLnhc1lGEXffogm X1Ti6Qem8L8Xip9/4f+OvWFXVV7Qe+T08iBO3/BjObbJmvgJ6AKY86caIFoNuyUbI2Kw lG5LePjdCV30iRtJAYmbACbj1RDL/u5VbW/V9JQWibdJAdORc5xSdNpCnbuzOJZDrZWu KhNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689136663; x=1691728663; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=rm70YAiA7B+6NetmnfdpkQvTkq/OM9KidvrlLi9FC60=; b=iJF/9aWurJmcJVNm8TIXWjpPmLhX0RvfazKEgVTDx+/KCeuTStc9paWjnpckqPIBSb JQwuYOhGLEAmS3olWOIyaVIcfM1wHLRNYQ/yYLj0BbsaG6Bt3I64lstVT51m3lTjvBIr IOxdg+Qk9FFoqAlpYFqeZo8Niu/fINJy8BvVuwHl0zU5dA4uA2v1MoqYErXsNaRCsEyg peBD+0/p8lzewAfJzRrq9K69zuYo9OuNimMCxHH9mtY++j125Tur9i3bn4eL9eMlociB LF4jCFHS+AJeMlosebo2s8RSB3FDTTXcC7AKFR0aFeoWT1Z8Hek+/KJ7tqcKtSNClBjH zQ9w== X-Gm-Message-State: ABy/qLbWdWi9ApBxt9rX/U85mEYj5FWkX1DObXxYW+g75G0FkDS8VdAH p44L7p9J2tLjxyBVueKzDW5ecVseQ+5qWCYCUOc612Hl X-Google-Smtp-Source: APBJJlH/K2YmAfkYwH4O3ePpQH/jukZIvFYkHh4xrLrURaalN+yukG3cyQcpOxgwmhANZb+mmNcZS85oFvkkbfBqpJE= X-Received: by 2002:a81:4f4c:0:b0:579:e5fb:f6c5 with SMTP id d73-20020a814f4c000000b00579e5fbf6c5mr17676038ywb.2.1689136662944; Tue, 11 Jul 2023 21:37:42 -0700 (PDT) List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 References: <2DCC0132-0307-4A73-A473-B749ABF87367.ref@yahoo.com> <2DCC0132-0307-4A73-A473-B749ABF87367@yahoo.com> <1aaeafcc-9812-64bd-a369-bae7b9fc0e5f@gwdg.de> <422DBC3B-D85F-4AFB-ABDE-842A08482EC8@yahoo.com> <58250d08-ce0a-c1ac-ed15-7d55d517218e@gwdg.de> <2ea168aa-3e0e-ee31-d4a3-82b4ce46b330@gwdg.de> <78c1497b-1ec7-08b6-a407-d671e3c4f9dd@gwdg.de> <44201182-73C9-46BC-A846-7DC765F5B81F@yahoo.com> <3670BA04-D165-44C3-B5DF-5B5AFD4CFD30@yahoo.com> In-Reply-To: <3670BA04-D165-44C3-B5DF-5B5AFD4CFD30@yahoo.com> From: Kevin Oberman Date: Tue, 11 Jul 2023 21:37:26 -0700 Message-ID: Subject: Re: Problem with the package builds To: Mark Millard Cc: "Hurling, Rainer" , FreeBSD Mailing List Content-Type: multipart/alternative; boundary="000000000000c59e94060042c6b2" X-Rspamd-Queue-Id: 4R14jN1Zmvz3LkV X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000c59e94060042c6b2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Jul 11, 2023 at 9:15=E2=80=AFPM Mark Millard wr= ote: > On Jul 11, 2023, at 21:05, Kevin Oberman wrote: > > > On Tue, Jul 11, 2023 at 1:10=E2=80=AFPM Mark Millard wrote: > > On Jul 11, 2023, at 12:57, Kevin Oberman wrote: > > > > > On Mon, Jul 10, 2023 at 9:42=E2=80=AFPM Mark Millard > wrote: > > > On Jul 10, 2023, at 21:27, Rainer Hurling wrote: > > > > > > > As I understand it, the ports-mgmt/pkg of the system running > Poudriere must be updated beforehand? > > > > > > > > At least on my side, this seems to work as expected :) > > > > > > > > > > poudriere builds pkg updates first (if needed) and then uses the pkg = it > > > built for building the later ports into packages. > > > > > > But, after the restarts of main-* builds, the FreeBSD build servers a= re > > > still showing examples were, after an 1hr, some builds are still in > > > build-depends. Also there was an example I saw were after 1.5 hr it w= as > > > still in run-depends. > > > > > > It may be that things are improved but not fully fixed relative to > > > some performance issues. > > > > > > =3D=3D=3D > > > Mark Millard > > > marklmi at yahoo.com > > > A new build started this morning at 1:06 UTC and, with pkg-1.20.2, > it's better, but not much. It's running at 21 packages/hour, a 100% > improvement on the last attempt which appears to have been killed last > night. The logs indicate the installation of dependencies, but I don't se= e > any sign of caching. It's a re-install every time. (I may not understand > how poudriere does things, but I am pretty sure that caching is done.) > > > > Just about "caching" relative to "poudriere bulk" builds . . . > > > > Nope. At the end of a builder run of a port build the context is > destroyed. > > At the start of the builder building its next port the context is > recreated > > from scratch. The only ports installed are exactly the declared > dependencies, > > no more, no less, for the new port to be built. > > > > Caching installed state would imply access to ports from prior build > > activity that do not apply: It would make the build environment pollute= d > with > > irrelevant history. poudriere's purpose is to have a "clean-room" > context for > > each port build. Thus its construction of such a context for each port > build. > > > > Caching vs. not is not the source of the large increase in how long > things > > take to build. > > > > =3D=3D=3D > > Mark Millard > > marklmi at yahoo.com > > OK. I knew it started in a clean jail. I just thought that it might use > some caching technique to speed repeated installs of a single port... not > that I have any idea of how this might be done. > > > > I think I'll monitor the speed of installs. I iish build logs included > timestamps > > FYI: Freshports now shows a pkg 1.20.3 described with: > > pkg*: new regression fixes release > > Changes: > - speed up pkg add again, and greatly reduce its memory footprint > - more compatibility with libfetch (SSL_* variables) > - fixed FETCH_TIMEOUT adaptation to libcurl > > Looks to have been committed about 16 hr ago. > > Last I looked the official package builders had not been stopped > and restarted with a newer ports tree yet. > > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > > As I expected, each package install takes a very long time. It appears that the average time to install a package with 1.20.2 was running about 45-50 seconds. Hopefully this latest update will fix that. No rush to stop the current build, I guess. The next build won't start until 1:00 UTC tomorrow. It might be that Bapt wants some time to work with it before. killing the current build or just entered a request to the builds manager in some timezone who sees no rush. --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000c59e94060042c6b2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Tue, Jul 11, 2023 at 9:15=E2= =80=AFPM Mark Millard <marklmi@yaho= o.com> wrote:
On Jul 11, 2023, at 21:05, Kevin Oberman = <rkoberman@gmai= l.com> wrote:

> On Tue, Jul 11, 2023 at 1:10=E2=80=AFPM Mark Millard <marklmi@yahoo.com> wrote:<= br> > On Jul 11, 2023, at 12:57, Kevin Oberman <rkoberman@gmail.com> wrote:
>
> > On Mon, Jul 10, 2023 at 9:42=E2=80=AFPM Mark Millard <marklmi@yahoo.com> w= rote:
> > On Jul 10, 2023, at 21:27, Rainer Hurling <rhurlin@gwdg.de> wrote:
> >
> > > As I understand it, the ports-mgmt/pkg of the system running= Poudriere must be updated beforehand?
> > >
> > > At least on my side, this seems to work as expected :)
> > >
> >
> > poudriere builds pkg updates first (if needed) and then uses the = pkg it
> > built for building the later ports into packages.
> >
> > But, after the restarts of main-* builds, the FreeBSD build serve= rs are
> > still showing examples were, after an 1hr, some builds are still = in
> > build-depends. Also there was an example I saw were after 1.5 hr = it was
> > still in run-depends.
> >
> > It may be that things are improved but not fully fixed relative t= o
> > some performance issues.
> >
> > =3D=3D=3D
> > Mark Millard
> > marklmi at yahoo.com
> >=C2=A0 A new build started this morning at 1:06 UTC and, with pkg-= 1.20.2, it's better, but not much. It's running at 21 packages/hour= , a 100% improvement on the last attempt which appears to have been killed = last night. The logs indicate the installation of dependencies, but I don&#= 39;t see any sign of caching. It's a re-install every time. (I may not = understand how poudriere does things, but I am pretty sure that caching is = done.)
>
> Just about "caching" relative to "poudriere bulk" = builds . . .
>
> Nope. At the end of a builder run of a port build the context is destr= oyed.
> At the start of the builder building its next port the context is recr= eated
> from scratch. The only ports installed are exactly the declared depend= encies,
> no more, no less, for the new port to be built.
>
> Caching installed state would imply access to ports from prior build > activity that do not apply: It would make the build environment pollut= ed with
> irrelevant history. poudriere's purpose is to have a "clean-r= oom" context for
> each port build. Thus its construction of such a context for each port= build.
>
> Caching vs. not is not the source of the large increase in how long th= ings
> take to build.
>
> =3D=3D=3D
> Mark Millard
> marklmi at yahoo.com
> OK. I knew it started in a clean jail. I just thought that it might us= e some caching technique to speed repeated installs of a single port... not= that I have any idea of how this might be done.
>
> I think I'll monitor the speed of installs. I iish build logs incl= uded timestamps

FYI: Freshports now shows a pkg 1.20.3 described with:

pkg*: new regression fixes release

Changes:
- speed up pkg add again, and greatly reduce its memory footprint
- more compatibility with libfetch (SSL_* variables)
- fixed FETCH_TIMEOUT adaptation to libcurl

Looks to have been committed about 16 hr ago.

Last I looked the official package builders had not been stopped
and restarted with a newer ports tree yet.

=3D=3D=3D
Mark Millard
marklmi at yahoo.com

As I expected, each package install takes a ve= ry long time. It appears that the average time to install a package with 1.= 20.2 was running about 45-50 seconds. Hopefully this latest update will fix= that.
No rush to stop the current build, I guess. The next = build won't start until 1:00 UTC tomorrow. It might be that Bapt wants = some time to work with it before. killing the current build or just entered= a request to the builds manager in some timezone who sees no rush.
--
Kevin Oberman, Part time kid herder and reti= red Network Engineer
E-mail: rkoberman@gmail.com
PGP Fingerprint: D03FB9= 8AFA78E3B78C1694B318AB39EF1B055683
--000000000000c59e94060042c6b2--