From nobody Tue Mar 3 07:03:59 2026 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fQ6Q10W4Vz6Tv9f for ; Tue, 03 Mar 2026 07:10:17 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-ot1-x32a.google.com (mail-ot1-x32a.google.com [IPv6:2607:f8b0:4864:20::32a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fQ6Pz46gGz4LFs for ; Tue, 03 Mar 2026 07:10:15 +0000 (UTC) (envelope-from kob6558@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=HvIvx5kf; dmarc=pass (policy=none) header.from=gmail.com; arc=pass ("google.com:s=arc-20240605:i=1"); spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2607:f8b0:4864:20::32a as permitted sender) smtp.mailfrom=kob6558@gmail.com Received: by mail-ot1-x32a.google.com with SMTP id 46e09a7af769-7d556c1a79eso7299320a34.3 for ; Mon, 02 Mar 2026 23:10:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1772521813; cv=none; d=google.com; s=arc-20240605; b=eRSQgdMHQq8zOYbLET41nM3uo8HWkMu54DtNYSqCFi2BTAhQRuyHM/3OkT5IlIl8I6 Im3NSeYJn03KnfaPz/gCNLR/JoeUrF01eFIHnTjmjr19JVCbnbGZQ410Sjfj2UbaU/y4 hs5a59VJgq4TPn4gnvrnCyUJZJzlY0hn/ldEZHtZuaeNMLOS4dWZh108/5+waZixTIfd BLWBm6/9XpXK5/1wm/8yvjo2oZWPTlYYHv29fiE6AEi8a5bM5wr+cv6tUWV2hc8nbrUO FOomBzso/0Bnk2uVnbxSh/ovaF5qrbw0339uvkD4YN7lu3z7M/jDlBH9JcntNDIUq0Ot CD1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=dDOvY8LZYcKVrci4BtNcICDWuEE5S7hIQ5RcgkYEYkM=; fh=i3ClYvk79168TQw8GxlFgwVjvJiEd73Nj7a7Gf4aJ/Y=; b=hgEdm/hPOxaG39nNcrMu5jMGUjh67511UNPBbgQ8RqUmqkP41nX5pmDmpjxKrx1z7L UBC7uBLTENzKZEAep+15mAW4/jp5mHSLx+/PFA54E8Y6xcnvW24YxIuzIjN2w1Sqdf8V pO2k6eKPYSqUmKKsf4UaEHQ6Sr3TJVDqUWHcnLmOwGUjFQb1+6h8t2B60FTUZ74rzIPg UzdeIY/xlBeZmwoGwi+ZngnS6YFOF2TKcHGtGD04A/ZiXAOu6If2Vz7sjows4/ruZHnX iikUus8oQYqBrzxurGYl407u/n5tWUsfumJgnUZapZWrRMZrhCloRhtXKBnUGEHk5Y/9 lnwA==; darn=freebsd.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772521813; x=1773126613; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=dDOvY8LZYcKVrci4BtNcICDWuEE5S7hIQ5RcgkYEYkM=; b=HvIvx5kfraagXBJPA/WAuRGq/tow8sJZlGgfGLZVnRKGg0EiyjvMiate5X9mR3gf5o sFPyZ/PiHiHpUDx8dOa+UVmi1jl8fQhGbpaAtrIPaUPl6aPyg978cmxNlH7QoKE6Niha jhGqSzVe9gltjDo4/31+4J+5mFFYKp9sYH0sZVS1pICqC7veME5TqFaoQ9/+4Xv9wrKk YtNzrGycZTue4+H+RJoi/wmn16VkGX8KIYIDXVtFUHAoNWRhG9MvWYxxPZGOA7T7xAAp tFgLHuWcCX49DeC+8Dv0nMYzWcpuJx/+7xGE6RnX5DvuCbNi0Qg8IwHPnGdXrFucboBb 1UTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772521813; x=1773126613; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=dDOvY8LZYcKVrci4BtNcICDWuEE5S7hIQ5RcgkYEYkM=; b=xCzOImNz8DXHcfQZVLrL52+dml5WzpyOvowBsUIu63Ayel9cXVD3zbZwjYZtPfFIjp 3dds/jS6gU3i/OM6CpNs+KwEK90XKJC2oj+CjU5nU4bTCYg2p3ZYwMsOR3QGJF2lPIMY WcGXduYHWqylr3ToHPk6ps22EFOdjBNuW/FyhlCUtZkE9vGTwhtPK1cSPz0TSVkMh+yo 1x2Vei0+58/qeMF3KWcMtBfSjuYHeu57tUMphuZqQjJDSHuTOzk6yAY816TweeIJzkwm w6pdEdGi9y0IcShhbmrEUQISIhR8aaVIFbTzApa4jvWTekmR0TggcLXajXsu2Q2GBbqk EEXA== X-Forwarded-Encrypted: i=1; AJvYcCXdDW8istV96FyF0e5jrp56xHv/s8kCnCY0Yy1NPHx09aRaYPFcun6biRXi/+vw66VfGJ843wgiO6Rw7VvQvQ==@freebsd.org X-Gm-Message-State: AOJu0Yzqn9g26rwmjIJhhPcOIs7tuc9//yTXaX87u7jdL3HMcoYPHET8 gsm14ZxJVeE6FgazDvbC0FQmWlKQ+2lRYVQLDVJv9SgpDVhIN8IFxjY0uKYw1KOcfSuKFYqlL3m l7lVJE/21RHCbuhp3srPnE6hp+asKYWT3Rg== X-Gm-Gg: ATEYQzzYbZnAzsZxqaYOT/G0HdAe+Gbb6x7DqoNirUQjTlMT8QMLq/VA00qYCtvHabG csku1uidWIt59qFPod0TYFE07IZISRj78g2OZYajTKCIHWD7Sl9SO7F2dCXrh/wV+ZbTyGhLh1M ahBaj5qZaoeNG6POqFvh4Yct/QKiTrMX22xebNAStHaRQg35tCpRKAcpUfM3EDpomh6CIt2FsmS 1/5nsdHVlq5fLrrLH+ITg3aRA2EIWThU1lIi1CUfqGNrNNfsnL7teF5H2S6IBJVC2UIQO7Vxz8I v13dZFS023QHvfOZ8mqmCazT/4s7GADUJpJ8L9Y2rHBXzXIP+p+T/OaFXu0B2Uwqs2zemQpEzRh dYKw= X-Received: by 2002:a53:e8cd:0:b0:64c:b56c:1acf with SMTP id 956f58d0204a3-64cc228918bmr10963513d50.54.1772521456125; Mon, 02 Mar 2026 23:04:16 -0800 (PST) List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-stable@freebsd.org Sender: owner-freebsd-stable@FreeBSD.org MIME-Version: 1.0 References: <1895ba62-b069-4e28-a910-6c666703ee8e@gmail.com> <6d613720-fa00-4686-be70-1767dbdc1fc2@yahoo.com> In-Reply-To: <6d613720-fa00-4686-be70-1767dbdc1fc2@yahoo.com> From: Kevin Oberman Date: Mon, 2 Mar 2026 23:03:59 -0800 X-Gm-Features: AaiRm52WuA7tDsyEUsf1HmsWVPGKiY5Smq768AzSu04Y-bydfmGHnrH6aXI41pI Message-ID: Subject: Re: 504 gateway time-outs To: Mark Millard Cc: Graham Perrin , freebsd-current@freebsd.org, freebsd-stable@freebsd.org Content-Type: multipart/alternative; boundary="000000000000bfce64064c194f08" X-Spamd-Result: default: False [1.31 / 15.00]; R_SUSPICIOUS_URL(5.00)[pkg-]; ARC_ALLOW(-1.00)[google.com:s=arc-20240605:i=1]; URI_COUNT_ODD(1.00)[41]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; GREYLIST(0.00)[pass,body]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MLMMJ_DEST(0.00)[freebsd-stable@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::32a:from] X-Rspamd-Queue-Id: 4fQ6Pz46gGz4LFs X-Spamd-Bar: + --000000000000bfce64064c194f08 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Feb 28, 2026 at 8:55=E2=80=AFPM Mark Millard wr= ote: > On 2/28/26 17:57, Kevin Oberman wrote: > > On Sat, Feb 28, 2026 at 5:48=E2=80=AFPM Kevin Oberman > > wrote: > > > > On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perrin > > > wrote: > > > > On 28/02/2026 10:32, Mark Millard wrote: > > > =E2=80=A6 > > > > > > The following got "504 Gateway Time-out" when I tried them: > > > > > > > mastername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b= 8d > > > mastername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b= 8d>> > > > > > > > mastername=3D150amd64-default&build=3Ddf4f957ea181 > status.freebsd.org/beefy23/build.html?mastername=3D150amd64- > > default&build=3Ddf4f957ea181>> > > > > > > Both somewhat slow to load, however they do load for me. > > > > Thanks > > > > > > beefy22 is showing hte same issues. Something is tying up the > > builders with builds hanging in "build-depends" and "lib-depends" > > for very long periods of time, as long as over 4 hours) with high > > load averages. This has been going on for months and seems to be > > getting worse. Chromium builds which used to take around 20 hours > > are now running around 40 and I really don't think that this is jus= t > > code bloat. At times, about half of the active builds are in some > > state other than "build"; mostly one of the depends states which > > should never take long as all dependencies are built before a build > > is started. > > > > After some time passes (often hours) things clear up. Dependencies > > are suddenly loaded in a few minutes or less. Note that things do > > continue to complete, but the 10 minute averages are in single > > digits and frequently go to 0. During this time, my connection to > > beefy22 will timeout and sometimes I can't establish a connection. > > > > I don't have any special access to the machines... just via pkg- > > status, so I'm fairly limited in any analysis. > > > > > > Just before I started my last message, all was well, but I last saw > > were a dozen builds in one of the "depends" states and "impulse" at 74 > > when I lost my connection. I can reconnect to the beefy22 status page, > > but it has not updated for several minutes. > > [Do not expect that I edited my all notes in presentation-text-order.] > > Via : > > beefy22: 143amd64-default 17:21:38 > > Accurate would be over 27 hrs (see below). > > Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space. > > < > https://pkg-status.freebsd.org/beefy22/build.html?mastername=3D143amd64-d= efault&build=3Ddf4f957ea181 > > > : > > Load Averages Swapinfo Elapsed > (100%) 63.94 69.19 70.56 0.24% 27:37:33 > > > -- > > Kevin Oberman, Part time kid herder and retired Network Engineer > > E-mail: rkoberman@gmail.com > > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 > > > I see such on beefy24 (main-amd64-default) currently but also got lucky > for an detailed page display. Notably: > > < > https://pkg-status.freebsd.org/beefy24/build.html?mastername=3Dmain-amd64= -default&build=3Dpdf4f957ea181_s178d0b5b8d > > > > shows: > > Load Averages Swapinfo Elapsed > (107%) 102.71 106.93 107.91 55.15% 26:35:05 > > but shows > 23:10:02 for Elapsed. > > (Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The > poudriere configuration is allowing 45 builders at once. Each builder is > likely to be configured to allow, say, up to 3 make processes, > MAKE_JOBS_NUMBER=3D3 . It is difficult to look at logs now to check on th= e > figure. I've no information about the amount of RAM or the size of the > SWAP space for beefy24 or beefy23.) > > The longest-still-active port-package builders elapsed-time-so-far > figures shown were: > > qt6-webengine-6.10.2 23:33:08 > chromium-145.0.7632.109 20:43:03 > electron39-39.7.0 20:41:59 > apache-openoffice-devel-4.2.1768900765_1,4 11:20:05 > > Those are likely the major users of tmpfs space that competes for > RAM+SWAP for USE_TMPFS=3Dall (or other large USE_TMPFS settings). > > I'd guess that the paging is thrashing for the above but have no way of > checking. It could be that some use of TMPFS_BLACKLIST to avoid the use > of TMPFS for the port-packages that have huge TMPFS involved could help > (without otherwise disabling USE_TMPFS=3Dall to still speed up most > builders). > > (I will note that TMPFS_BLACKLIST entries do not necessarily end up with > no tmpfs use, just less. But that less can help avoid paging that ends > up thrashing.) > > Note during my editing/research and other activities, the web page had > not updated at all after its initial display. It is now significantly > later than when I started editing this note. > > Later: Hmm. > > https://pkg-status.freebsd.org/builds?type=3Dpackage&all=3D1 > > 05:48:27 beefy23 (150amd64-default) > > but . . . > > < > https://pkg-status.freebsd.org/beefy23/build.html?mastername=3D150amd64-d= efault&build=3Ddf4f957ea181 > > > > Load Averages Swapinfo Elapsed > (112%) 107.81 100.28 97.16 0.04% 27:42:2 > > > -- > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > beefy22 started a full ports build Saturday at 00:10 UTC. Most of last Saturday and early Sunday had 'impulse' of no more than double digits and over a 20 hour period the number of completed builds was just over 1000. Chromium and electron were killed when they reached 48 hours. The build now exceeds 72 hours. Last month's fill build was about 64.5 hours. It really looks like things are getting worse. I suspect some resource is reaching exhaustion. One suggestion I saw that looks possible is tmp which I believe is a tempfs, but I really can't tell too much with what is visible on pkg-status. On to wild speculation. Maybe reduce the number of concurrent builds. If a small amount of DRAM could help, that would be great, but DRAM is getting very expensive and reducing the number of builds might speed things up at no cost. --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000bfce64064c194f08 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sat, Feb 28, 2026 at 8:55=E2= =80=AFPM Mark Millard <marklmi@yaho= o.com> wrote:
On 2/28/26 17:57, K= evin Oberman wrote:
> On Sat, Feb 28, 2026 at 5:48=E2=80=AFPM Kevin Oberman <rkoberman@gmail.com
> <mailto:rk= oberman@gmail.com>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perr= in
>=C2=A0 =C2=A0 =C2=A0<grahamperrin@gmail.com <mailto:grahamperrin@gmail.com>> wrot= e:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0On 28/02/2026 10:32, Mark Millard wro= te:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> =E2=80=A6
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> The following got "504 Gate= way Time-out" when I tried them:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> <http= s://pkg-status.freebsd.org/beefy24/build.html?
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3Dmain-amd64-default&b= uild=3Dpdf4f957ea181_s178d0b5b8d
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<https://p= kg-status.freebsd.org/beefy24/build.html?
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3Dmain-amd64-default&b= uild=3Dpdf4f957ea181_s178d0b5b8d>>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> <http= s://pkg-status.freebsd.org/beefy23/build.html?
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3D150amd64-default&bui= ld=3Ddf4f957ea181 <https://pkg-
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status.freebsd.org/beefy23/build.html?mastername=3D150amd64-
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0default&build=3Ddf4f957ea181>&= gt;
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Both somewhat slow to load, however t= hey do load for me.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Thanks
>
>=C2=A0 =C2=A0 =C2=A0=C2=A0
>=C2=A0 =C2=A0 =C2=A0beefy22 is showing hte same issues. Something is ty= ing up the
>=C2=A0 =C2=A0 =C2=A0builders with builds hanging in "build-depends= " and "lib-depends"
>=C2=A0 =C2=A0 =C2=A0for very long periods of time, as long as over 4 ho= urs) with high
>=C2=A0 =C2=A0 =C2=A0load averages. This has been going on for months an= d seems to be
>=C2=A0 =C2=A0 =C2=A0getting worse. Chromium builds which used to take a= round 20 hours
>=C2=A0 =C2=A0 =C2=A0are now running around 40 and I really don't th= ink=C2=A0that this is just
>=C2=A0 =C2=A0 =C2=A0code bloat. At times, about half of the active buil= ds=C2=A0are in some
>=C2=A0 =C2=A0 =C2=A0state=C2=A0other than "build"; mostly one= of the depends states which
>=C2=A0 =C2=A0 =C2=A0should never take long as all dependencies are buil= t before a build
>=C2=A0 =C2=A0 =C2=A0is started.
>
>=C2=A0 =C2=A0 =C2=A0After some time passes (often hours) things clear u= p. Dependencies
>=C2=A0 =C2=A0 =C2=A0are suddenly loaded in a few minutes or less. Note = that things do
>=C2=A0 =C2=A0 =C2=A0continue to complete, but the 10 minute averages ar= e in single
>=C2=A0 =C2=A0 =C2=A0digits and frequently go to 0. During this time, my= connection to
>=C2=A0 =C2=A0 =C2=A0beefy22 will timeout and sometimes I can't esta= blish a connection.
>
>=C2=A0 =C2=A0 =C2=A0I don't have any special access to the machines= ... just via pkg-
>=C2=A0 =C2=A0 =C2=A0status, so I'm fairly limited in any analysis.<= br> >
>
> Just before I started my last message,=C2=A0 all was well, but I last= =C2=A0saw
> were a dozen builds in one of the=C2=A0"depends" states and = "impulse" at 74
> when I lost my connection. I can reconnect to the beefy22 status page,=
> but it has not updated for several=C2=A0minutes.

[Do not expect that I edited my all notes in presentation-text-order.]

Via <https://pkg-status.freebsd.o= rg/builds?type=3Dpackage&all=3D1> :

beefy22: 143amd64-default 17:21:38

Accurate would be over 27 hrs (see below).

Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space.

<https://pkg-status.freebsd.org/beefy22/build.html?mastername=3D143a= md64-default&build=3Ddf4f957ea181>
:

=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed
(100%) 63.94 69.19 70.56=C2=A0 =C2=A0 =C2=A00.24%=C2=A0 =C2=A0 =C2=A027:37:= 33

> --
> Kevin Oberman, Part time kid herder and retired Network Engineer
> E-mail: rkobe= rman@gmail.com <mailto:rkoberman@gmail.com>
> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683


I see such on beefy24 (main-amd64-default) currently but also got lucky
for an detailed page display. Notably:

<https://pkg-status.freebsd.org/beefy24/build.html?ma= stername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b8d>=

shows:

=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed
(107%) 102.71 106.93 107.91=C2=A0 55.15%=C2=A0 =C2=A0 26:35:05

but <https://pkg-status.freebsd.o= rg/builds?type=3Dpackage&all=3D1> shows
23:10:02 for Elapsed.

(Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The
poudriere configuration is allowing 45 builders at once. Each builder is likely to be configured to allow, say, up to 3 make processes,
MAKE_JOBS_NUMBER=3D3 . It is difficult to look at logs now to check on the<= br> figure. I've no information about the amount of RAM or the size of the<= br> SWAP space for beefy24 or beefy23.)

The longest-still-active port-package builders elapsed-time-so-far
figures shown were:

qt6-webengine-6.10.2=C2=A0 =C2=A0 23:33:08
chromium-145.0.7632.109 20:43:03
electron39-39.7.0=C2=A0 =C2=A0 =C2=A0 =C2=A020:41:59
apache-openoffice-devel-4.2.1768900765_1,4 11:20:05

Those are likely the major users of tmpfs space that competes for
RAM+SWAP for USE_TMPFS=3Dall (or other large USE_TMPFS settings).

I'd guess that the paging is thrashing for the above but have no way of=
checking. It could be that some use of TMPFS_BLACKLIST to avoid the use
of TMPFS for the port-packages that have huge TMPFS involved could help
(without otherwise disabling USE_TMPFS=3Dall to still speed up most builder= s).

(I will note that TMPFS_BLACKLIST entries do not necessarily end up with no tmpfs use, just less. But that less can help avoid paging that ends
up thrashing.)

Note during my editing/research and other activities, the web page had
not updated at all after its initial display. It is now significantly
later than when I started editing this note.

Later: Hmm.

https://pkg-status.freebsd.org/build= s?type=3Dpackage&all=3D1

05:48:27 beefy23 (150amd64-default)

but . . .

<https://pkg-status.freebsd.org/beefy23/build.html?mastername=3D150a= md64-default&build=3Ddf4f957ea181>

=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed
(112%) 107.81 100.28 97.16=C2=A0 =C2=A00.04%=C2=A0 =C2=A0 =C2=A027:42:2


--
=3D=3D=3D
Mark Millard
marklmi at yahoo.com

beefy22 started a= full ports build Saturday at 00:10 UTC. Most of last Saturday and early Su= nday had 'impulse' of no more than double digits and=C2=A0
over a 20 hour period the number of completed builds was just over 1= 000. Chromium and electron=C2=A0were killed when they reached 48 hours. The= build now exceeds=C2=A072 hours. Last month's fill build was about 64.= 5 hours.

It really looks like things are= getting worse. I suspect some resource is reaching exhaustion. One suggest= ion I saw that looks possible is tmp which I believe is a tempfs, but I rea= lly can't tell too much with what is visible on pkg-status.=C2=A0
=

On to wild speculation. Maybe reduce the numb= er of concurrent builds. If a small amount of DRAM could help, that would b= e great, but DRAM is getting very expensive and reducing the=C2=A0number of= builds might speed things up at no cost.=C2=A0
--
<= div dir=3D"ltr">
Kevin Oberman, Part time kid herder and retired Network EngineerE-mail: rkoberma= n@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB= 39EF1B055683
--000000000000bfce64064c194f08--