From nobody Tue Mar 3 07:03:59 2026 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fQ6S63jXrz6Tv9K for ; Tue, 03 Mar 2026 07:12:06 +0000 (UTC) (envelope-from kob6558@gmail.com) Received: from mail-oi1-x22a.google.com (mail-oi1-x22a.google.com [IPv6:2607:f8b0:4864:20::22a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "WR4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4fQ6S55yL2z4Mnw for ; Tue, 03 Mar 2026 07:12:05 +0000 (UTC) (envelope-from kob6558@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=gJ7R+00P; dmarc=pass (policy=none) header.from=gmail.com; arc=pass ("google.com:s=arc-20240605:i=1"); spf=pass (mx1.freebsd.org: domain of kob6558@gmail.com designates 2607:f8b0:4864:20::22a as permitted sender) smtp.mailfrom=kob6558@gmail.com Received: by mail-oi1-x22a.google.com with SMTP id 5614622812f47-45f053b7b90so3598113b6e.0 for ; Mon, 02 Mar 2026 23:12:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1772521924; cv=none; d=google.com; s=arc-20240605; b=IIlTwNBYA2q/+Vp5aKvz83AC04BcD3TKS77Is/WNQMHlp9r5baDQnN7Sy3YXKy4rIy CoVoAoL73PX0LD0hi4OWdYedOUoeHfLFo/8VzOTWwxFTPC6Yz6X9qEE+npY6Bqbv8G2H V+p1D84LkGs1kl+AEH8trlb6WkauXYkCumFYOVvcF+jTpdE/E0UnUHcJG/kWPk0AYl1Q E9P4oyAmWaN4axP/sXsaGDIzebxY8qDjsbOMY+anAhJxS/ABBgpYD4m/I3t6oTig6Jhi Vg8mNkzT8QGtW06h/7gu2UiTuMMgkw0rJPrU5lVvFAfljzmCiureIISmQ7xTYuoZPK5/ 9/Kg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=dDOvY8LZYcKVrci4BtNcICDWuEE5S7hIQ5RcgkYEYkM=; fh=QAuinG8DdctZZdUFwzeZu8OkrS+/xhA6LwD85eCrHKk=; b=RLLfDHvuW8mOla+sD23QXaAgK3oi2UxlNN0rLjAxrkt1BMzBnheWa3bZ46XCKJwYLB K/xq8p1UGXXQDm1utpPNq5AcvRPogMJ+P9k67rZJAc9oZXOaKNWIHQnIl8+Jbf4h3z3o PfuwY3PyIsy43JOQn81Y/XQYGE3o/bSjKk99ql9PChMKQLfR4dhPWmq2WQVN1jsIcdkR FKSWU9fVutzEGHUx41y+zhRFPUuD+iaXStO7VFuN69gf7Rdf8/v3UqxEk5A3/Xb1bxa8 m8P7+RwePC/rTZUj46Pxj3v7zQQUVLXZFQTkQ8x0eLj/Nj6P3F+n2vksunCrHcZBLPCL tHnw==; darn=freebsd.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1772521924; x=1773126724; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=dDOvY8LZYcKVrci4BtNcICDWuEE5S7hIQ5RcgkYEYkM=; b=gJ7R+00PU43f2dWEzHA1JOLNzgv2BF7KnjzXc1zOL9FDhYkTLWh7xv5h6O0zEIgNKu N3AJLfKhVKHJrLquR/WlUjIkAyNgfiR/4C3uhA34S9d+YrIVwh2grlPUPeCqc2adEVgF 5WOZqU/IQIse0XmANrm/5gMqEzpuxDxPI0shnDYObdLmwyHBHfpm0tbpgshrfJtXl7gh 450liF4lg7/VfJsgP4FjTaXVzRVudkJDrTu2reEHQSqpsd3USGsW+B1oXj3/NSb43rt7 rRWOKzqSEZh0K1U0s3BKk6fb7h5/CVlA10drp9r2srsvxPJObnVIaX2ZS4o5J5GyUF+A /kkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772521924; x=1773126724; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=dDOvY8LZYcKVrci4BtNcICDWuEE5S7hIQ5RcgkYEYkM=; b=rKnaGrQFEXbd2HYvVwk2j1TsiBccLBRPQbIIiVSyyrb+3Ot4sJjeuvCNUPR91hqdrw 5F6j7MLT2cE9VPCmcdnZJ2QQxgf/uNuP2mPuVw8lp9JHfUxLDy3SgimVg9unSziQPQ/m KF6z7N+3RkFkGZIfarei0XVcsX08Wal0vFXu5avvF1roKCE0SrXaCiw5MBwspNRAj3HG ZJpGG1rVAMLl/22S9wrpje4s0XVm/G2GnQP0BuPtycLZ4+jJS3bLipKl3V9WClO8DSjY GNSj8R285ii2e59elZauqzrbLu4/A2x+Apat+2BRUlSBaLKCf/FnFVbgSk6KQltkLFEo U0CA== X-Forwarded-Encrypted: i=1; AJvYcCVxPfS3m/7dw8ynGJKlFNzFjh6OjHcRK12+DxZMW4egL12cDtNFuhpodFISsOr9uytCJC5coPqJNxR9THlLqTg=@freebsd.org X-Gm-Message-State: AOJu0YzXpSFdT8VFIUYolnOp85QsVNqhr7qH78vWthRdiZfUegHRuNet /x7Sc4nUNdbFyji98VghwVPo3e1AEC/Ko/NwAkziGG65ucpVrMlGSKeZcTYx+zB2aD0xGYfGa8P y9lLr8mjvQUiVnqNiCqn0XXGFVWveN8GFvg== X-Gm-Gg: ATEYQzyE+zL1uPuLJ6bpjxFQB2umV8GRnx9o5rLOGXyXgON9Sw3tuE8ff8FTDMmZD+N Cz/MEo9fadkGHLVUcr/KL/ONv1dNUulBqLA4nkG273FvfH8vEaIo0hjVRTiMXyjACV+FCn/0qHJ MoYcpTjgnRJqxcbg3qj4dlw3QUFYDcD+4d11RIQfnJY8j6MgExLHl8CGcoeirCSIG80CuCgq32B Ncdo8dV/UxNevq4JEoO7bR4TBL9DR4+xKuQYkre7SCb99RODpTXSWt/OzrBzj13UaLZYGKncXER +eTYVAAebHs5rfevyfNSZajK+dpzysvCW9f3yy0Pn/UwV++LysGdNmcUdaGvn9Yu7Gnc X-Received: by 2002:a53:e8cd:0:b0:64c:b56c:1acf with SMTP id 956f58d0204a3-64cc228918bmr10963513d50.54.1772521456125; Mon, 02 Mar 2026 23:04:16 -0800 (PST) List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 References: <1895ba62-b069-4e28-a910-6c666703ee8e@gmail.com> <6d613720-fa00-4686-be70-1767dbdc1fc2@yahoo.com> In-Reply-To: <6d613720-fa00-4686-be70-1767dbdc1fc2@yahoo.com> From: Kevin Oberman Date: Mon, 2 Mar 2026 23:03:59 -0800 X-Gm-Features: AaiRm52WuA7tDsyEUsf1HmsWVPGKiY5Smq768AzSu04Y-bydfmGHnrH6aXI41pI Message-ID: Subject: Re: 504 gateway time-outs To: Mark Millard Cc: Graham Perrin , freebsd-current@freebsd.org, freebsd-stable@freebsd.org Content-Type: multipart/alternative; boundary="000000000000bfce64064c194f08" X-Spamd-Result: default: False [1.31 / 15.00]; R_SUSPICIOUS_URL(5.00)[pkg-]; ARC_ALLOW(-1.00)[google.com:s=arc-20240605:i=1]; URI_COUNT_ODD(1.00)[41]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.988]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FORGED_SENDER(0.30)[rkoberman@gmail.com,kob6558@gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; GREYLIST(0.00)[pass,body]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[gmail.com:+]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_NEQ_ENVFROM(0.00)[rkoberman@gmail.com,kob6558@gmail.com]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MISSING_XM_UA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::22a:from] X-Rspamd-Queue-Id: 4fQ6S55yL2z4Mnw X-Spamd-Bar: + --000000000000bfce64064c194f08 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sat, Feb 28, 2026 at 8:55=E2=80=AFPM Mark Millard wr= ote: > On 2/28/26 17:57, Kevin Oberman wrote: > > On Sat, Feb 28, 2026 at 5:48=E2=80=AFPM Kevin Oberman > > wrote: > > > > On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perrin > > > wrote: > > > > On 28/02/2026 10:32, Mark Millard wrote: > > > =E2=80=A6 > > > > > > The following got "504 Gateway Time-out" when I tried them: > > > > > > > mastername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b= 8d > > > mastername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b= 8d>> > > > > > > > mastername=3D150amd64-default&build=3Ddf4f957ea181 > status.freebsd.org/beefy23/build.html?mastername=3D150amd64- > > default&build=3Ddf4f957ea181>> > > > > > > Both somewhat slow to load, however they do load for me. > > > > Thanks > > > > > > beefy22 is showing hte same issues. Something is tying up the > > builders with builds hanging in "build-depends" and "lib-depends" > > for very long periods of time, as long as over 4 hours) with high > > load averages. This has been going on for months and seems to be > > getting worse. Chromium builds which used to take around 20 hours > > are now running around 40 and I really don't think that this is jus= t > > code bloat. At times, about half of the active builds are in some > > state other than "build"; mostly one of the depends states which > > should never take long as all dependencies are built before a build > > is started. > > > > After some time passes (often hours) things clear up. Dependencies > > are suddenly loaded in a few minutes or less. Note that things do > > continue to complete, but the 10 minute averages are in single > > digits and frequently go to 0. During this time, my connection to > > beefy22 will timeout and sometimes I can't establish a connection. > > > > I don't have any special access to the machines... just via pkg- > > status, so I'm fairly limited in any analysis. > > > > > > Just before I started my last message, all was well, but I last saw > > were a dozen builds in one of the "depends" states and "impulse" at 74 > > when I lost my connection. I can reconnect to the beefy22 status page, > > but it has not updated for several minutes. > > [Do not expect that I edited my all notes in presentation-text-order.] > > Via : > > beefy22: 143amd64-default 17:21:38 > > Accurate would be over 27 hrs (see below). > > Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space. > > < > https://pkg-status.freebsd.org/beefy22/build.html?mastername=3D143amd64-d= efault&build=3Ddf4f957ea181 > > > : > > Load Averages Swapinfo Elapsed > (100%) 63.94 69.19 70.56 0.24% 27:37:33 > > > -- > > Kevin Oberman, Part time kid herder and retired Network Engineer > > E-mail: rkoberman@gmail.com > > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 > > > I see such on beefy24 (main-amd64-default) currently but also got lucky > for an detailed page display. Notably: > > < > https://pkg-status.freebsd.org/beefy24/build.html?mastername=3Dmain-amd64= -default&build=3Dpdf4f957ea181_s178d0b5b8d > > > > shows: > > Load Averages Swapinfo Elapsed > (107%) 102.71 106.93 107.91 55.15% 26:35:05 > > but shows > 23:10:02 for Elapsed. > > (Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The > poudriere configuration is allowing 45 builders at once. Each builder is > likely to be configured to allow, say, up to 3 make processes, > MAKE_JOBS_NUMBER=3D3 . It is difficult to look at logs now to check on th= e > figure. I've no information about the amount of RAM or the size of the > SWAP space for beefy24 or beefy23.) > > The longest-still-active port-package builders elapsed-time-so-far > figures shown were: > > qt6-webengine-6.10.2 23:33:08 > chromium-145.0.7632.109 20:43:03 > electron39-39.7.0 20:41:59 > apache-openoffice-devel-4.2.1768900765_1,4 11:20:05 > > Those are likely the major users of tmpfs space that competes for > RAM+SWAP for USE_TMPFS=3Dall (or other large USE_TMPFS settings). > > I'd guess that the paging is thrashing for the above but have no way of > checking. It could be that some use of TMPFS_BLACKLIST to avoid the use > of TMPFS for the port-packages that have huge TMPFS involved could help > (without otherwise disabling USE_TMPFS=3Dall to still speed up most > builders). > > (I will note that TMPFS_BLACKLIST entries do not necessarily end up with > no tmpfs use, just less. But that less can help avoid paging that ends > up thrashing.) > > Note during my editing/research and other activities, the web page had > not updated at all after its initial display. It is now significantly > later than when I started editing this note. > > Later: Hmm. > > https://pkg-status.freebsd.org/builds?type=3Dpackage&all=3D1 > > 05:48:27 beefy23 (150amd64-default) > > but . . . > > < > https://pkg-status.freebsd.org/beefy23/build.html?mastername=3D150amd64-d= efault&build=3Ddf4f957ea181 > > > > Load Averages Swapinfo Elapsed > (112%) 107.81 100.28 97.16 0.04% 27:42:2 > > > -- > =3D=3D=3D > Mark Millard > marklmi at yahoo.com > beefy22 started a full ports build Saturday at 00:10 UTC. Most of last Saturday and early Sunday had 'impulse' of no more than double digits and over a 20 hour period the number of completed builds was just over 1000. Chromium and electron were killed when they reached 48 hours. The build now exceeds 72 hours. Last month's fill build was about 64.5 hours. It really looks like things are getting worse. I suspect some resource is reaching exhaustion. One suggestion I saw that looks possible is tmp which I believe is a tempfs, but I really can't tell too much with what is visible on pkg-status. On to wild speculation. Maybe reduce the number of concurrent builds. If a small amount of DRAM could help, that would be great, but DRAM is getting very expensive and reducing the number of builds might speed things up at no cost. --=20 Kevin Oberman, Part time kid herder and retired Network Engineer E-mail: rkoberman@gmail.com PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 --000000000000bfce64064c194f08 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sat, Feb 28, 2026 at 8:55=E2= =80=AFPM Mark Millard <marklmi@yaho= o.com> wrote:
On 2/28/26 17:57, K= evin Oberman wrote:
> On Sat, Feb 28, 2026 at 5:48=E2=80=AFPM Kevin Oberman <rkoberman@gmail.com
> <mailto:rk= oberman@gmail.com>> wrote:
>
>=C2=A0 =C2=A0 =C2=A0On Sat, Feb 28, 2026 at 3:33=E2=80=AFAM Graham Perr= in
>=C2=A0 =C2=A0 =C2=A0<grahamperrin@gmail.com <mailto:grahamperrin@gmail.com>> wrot= e:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0On 28/02/2026 10:32, Mark Millard wro= te:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> =E2=80=A6
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> The following got "504 Gate= way Time-out" when I tried them:
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> <http= s://pkg-status.freebsd.org/beefy24/build.html?
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3Dmain-amd64-default&b= uild=3Dpdf4f957ea181_s178d0b5b8d
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<https://p= kg-status.freebsd.org/beefy24/build.html?
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3Dmain-amd64-default&b= uild=3Dpdf4f957ea181_s178d0b5b8d>>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> <http= s://pkg-status.freebsd.org/beefy23/build.html?
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0mastername=3D150amd64-default&bui= ld=3Ddf4f957ea181 <https://pkg-
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0status.freebsd.org/beefy23/build.html?mastername=3D150amd64-
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0default&build=3Ddf4f957ea181>&= gt;
>
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Both somewhat slow to load, however t= hey do load for me.
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Thanks
>
>=C2=A0 =C2=A0 =C2=A0=C2=A0
>=C2=A0 =C2=A0 =C2=A0beefy22 is showing hte same issues. Something is ty= ing up the
>=C2=A0 =C2=A0 =C2=A0builders with builds hanging in "build-depends= " and "lib-depends"
>=C2=A0 =C2=A0 =C2=A0for very long periods of time, as long as over 4 ho= urs) with high
>=C2=A0 =C2=A0 =C2=A0load averages. This has been going on for months an= d seems to be
>=C2=A0 =C2=A0 =C2=A0getting worse. Chromium builds which used to take a= round 20 hours
>=C2=A0 =C2=A0 =C2=A0are now running around 40 and I really don't th= ink=C2=A0that this is just
>=C2=A0 =C2=A0 =C2=A0code bloat. At times, about half of the active buil= ds=C2=A0are in some
>=C2=A0 =C2=A0 =C2=A0state=C2=A0other than "build"; mostly one= of the depends states which
>=C2=A0 =C2=A0 =C2=A0should never take long as all dependencies are buil= t before a build
>=C2=A0 =C2=A0 =C2=A0is started.
>
>=C2=A0 =C2=A0 =C2=A0After some time passes (often hours) things clear u= p. Dependencies
>=C2=A0 =C2=A0 =C2=A0are suddenly loaded in a few minutes or less. Note = that things do
>=C2=A0 =C2=A0 =C2=A0continue to complete, but the 10 minute averages ar= e in single
>=C2=A0 =C2=A0 =C2=A0digits and frequently go to 0. During this time, my= connection to
>=C2=A0 =C2=A0 =C2=A0beefy22 will timeout and sometimes I can't esta= blish a connection.
>
>=C2=A0 =C2=A0 =C2=A0I don't have any special access to the machines= ... just via pkg-
>=C2=A0 =C2=A0 =C2=A0status, so I'm fairly limited in any analysis.<= br> >
>
> Just before I started my last message,=C2=A0 all was well, but I last= =C2=A0saw
> were a dozen builds in one of the=C2=A0"depends" states and = "impulse" at 74
> when I lost my connection. I can reconnect to the beefy22 status page,=
> but it has not updated for several=C2=A0minutes.

[Do not expect that I edited my all notes in presentation-text-order.]

Via <https://pkg-status.freebsd.o= rg/builds?type=3Dpackage&all=3D1> :

beefy22: 143amd64-default 17:21:38

Accurate would be over 27 hrs (see below).

Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space.

<https://pkg-status.freebsd.org/beefy22/build.html?mastername=3D143a= md64-default&build=3Ddf4f957ea181>
:

=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed
(100%) 63.94 69.19 70.56=C2=A0 =C2=A0 =C2=A00.24%=C2=A0 =C2=A0 =C2=A027:37:= 33

> --
> Kevin Oberman, Part time kid herder and retired Network Engineer
> E-mail: rkobe= rman@gmail.com <mailto:rkoberman@gmail.com>
> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683


I see such on beefy24 (main-amd64-default) currently but also got lucky
for an detailed page display. Notably:

<https://pkg-status.freebsd.org/beefy24/build.html?ma= stername=3Dmain-amd64-default&build=3Dpdf4f957ea181_s178d0b5b8d>=

shows:

=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed
(107%) 102.71 106.93 107.91=C2=A0 55.15%=C2=A0 =C2=A0 26:35:05

but <https://pkg-status.freebsd.o= rg/builds?type=3Dpackage&all=3D1> shows
23:10:02 for Elapsed.

(Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The
poudriere configuration is allowing 45 builders at once. Each builder is likely to be configured to allow, say, up to 3 make processes,
MAKE_JOBS_NUMBER=3D3 . It is difficult to look at logs now to check on the<= br> figure. I've no information about the amount of RAM or the size of the<= br> SWAP space for beefy24 or beefy23.)

The longest-still-active port-package builders elapsed-time-so-far
figures shown were:

qt6-webengine-6.10.2=C2=A0 =C2=A0 23:33:08
chromium-145.0.7632.109 20:43:03
electron39-39.7.0=C2=A0 =C2=A0 =C2=A0 =C2=A020:41:59
apache-openoffice-devel-4.2.1768900765_1,4 11:20:05

Those are likely the major users of tmpfs space that competes for
RAM+SWAP for USE_TMPFS=3Dall (or other large USE_TMPFS settings).

I'd guess that the paging is thrashing for the above but have no way of=
checking. It could be that some use of TMPFS_BLACKLIST to avoid the use
of TMPFS for the port-packages that have huge TMPFS involved could help
(without otherwise disabling USE_TMPFS=3Dall to still speed up most builder= s).

(I will note that TMPFS_BLACKLIST entries do not necessarily end up with no tmpfs use, just less. But that less can help avoid paging that ends
up thrashing.)

Note during my editing/research and other activities, the web page had
not updated at all after its initial display. It is now significantly
later than when I started editing this note.

Later: Hmm.

https://pkg-status.freebsd.org/build= s?type=3Dpackage&all=3D1

05:48:27 beefy23 (150amd64-default)

but . . .

<https://pkg-status.freebsd.org/beefy23/build.html?mastername=3D150a= md64-default&build=3Ddf4f957ea181>

=C2=A0 =C2=A0 =C2=A0 =C2=A0Load Averages=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0S= wapinfo=C2=A0 Elapsed
(112%) 107.81 100.28 97.16=C2=A0 =C2=A00.04%=C2=A0 =C2=A0 =C2=A027:42:2


--
=3D=3D=3D
Mark Millard
marklmi at yahoo.com

beefy22 started a= full ports build Saturday at 00:10 UTC. Most of last Saturday and early Su= nday had 'impulse' of no more than double digits and=C2=A0
over a 20 hour period the number of completed builds was just over 1= 000. Chromium and electron=C2=A0were killed when they reached 48 hours. The= build now exceeds=C2=A072 hours. Last month's fill build was about 64.= 5 hours.

It really looks like things are= getting worse. I suspect some resource is reaching exhaustion. One suggest= ion I saw that looks possible is tmp which I believe is a tempfs, but I rea= lly can't tell too much with what is visible on pkg-status.=C2=A0
=

On to wild speculation. Maybe reduce the numb= er of concurrent builds. If a small amount of DRAM could help, that would b= e great, but DRAM is getting very expensive and reducing the=C2=A0number of= builds might speed things up at no cost.=C2=A0
--
<= div dir=3D"ltr">
Kevin Oberman, Part time kid herder and retired Network EngineerE-mail: rkoberma= n@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB= 39EF1B055683
--000000000000bfce64064c194f08--