Date: Sat, 28 Feb 2026 20:55:37 -0800 From: Mark Millard <marklmi@yahoo.com> To: Kevin Oberman <rkoberman@gmail.com>, Graham Perrin <grahamperrin@gmail.com> Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: 504 gateway time-outs Message-ID: <6d613720-fa00-4686-be70-1767dbdc1fc2@yahoo.com> In-Reply-To: <CAN6yY1uzkQTtEaaL0BpB9kvh7FZSbi-jT3jYDxy%2BB_O9=2COmw@mail.gmail.com> References: <d5fd8b56-35cd-4700-8e9b-7767cbb088a6@yahoo.com> <1895ba62-b069-4e28-a910-6c666703ee8e@gmail.com> <CAN6yY1vN8T=6SsNNtr%2B-NO7n5yCJGTPYz=HjVnwPOjM9827x8Q@mail.gmail.com> <CAN6yY1uzkQTtEaaL0BpB9kvh7FZSbi-jT3jYDxy%2BB_O9=2COmw@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
On 2/28/26 17:57, Kevin Oberman wrote: > On Sat, Feb 28, 2026 at 5:48 PM Kevin Oberman <rkoberman@gmail.com > <mailto:rkoberman@gmail.com>> wrote: > > On Sat, Feb 28, 2026 at 3:33 AM Graham Perrin > <grahamperrin@gmail.com <mailto:grahamperrin@gmail.com>> wrote: > > On 28/02/2026 10:32, Mark Millard wrote: > > … > > > > The following got "504 Gateway Time-out" when I tried them: > > > > <https://pkg-status.freebsd.org/beefy24/build.html? > mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d > <https://pkg-status.freebsd.org/beefy24/build.html? > mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d>> > > > > <https://pkg-status.freebsd.org/beefy23/build.html? > mastername=150amd64-default&build=df4f957ea181 <https://pkg- > status.freebsd.org/beefy23/build.html?mastername=150amd64- > default&build=df4f957ea181>> > > > Both somewhat slow to load, however they do load for me. > > Thanks > > > beefy22 is showing hte same issues. Something is tying up the > builders with builds hanging in "build-depends" and "lib-depends" > for very long periods of time, as long as over 4 hours) with high > load averages. This has been going on for months and seems to be > getting worse. Chromium builds which used to take around 20 hours > are now running around 40 and I really don't think that this is just > code bloat. At times, about half of the active builds are in some > state other than "build"; mostly one of the depends states which > should never take long as all dependencies are built before a build > is started. > > After some time passes (often hours) things clear up. Dependencies > are suddenly loaded in a few minutes or less. Note that things do > continue to complete, but the 10 minute averages are in single > digits and frequently go to 0. During this time, my connection to > beefy22 will timeout and sometimes I can't establish a connection. > > I don't have any special access to the machines... just via pkg- > status, so I'm fairly limited in any analysis. > > > Just before I started my last message, all was well, but I last saw > were a dozen builds in one of the "depends" states and "impulse" at 74 > when I lost my connection. I can reconnect to the beefy22 status page, > but it has not updated for several minutes. [Do not expect that I edited my all notes in presentation-text-order.] Via <https://pkg-status.freebsd.org/builds?type=package&all=1>; : beefy22: 143amd64-default 17:21:38 Accurate would be over 27 hrs (see below). Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space. <https://pkg-status.freebsd.org/beefy22/build.html?mastername=143amd64-default&build=df4f957ea181>; : Load Averages Swapinfo Elapsed (100%) 63.94 69.19 70.56 0.24% 27:37:33 > -- > Kevin Oberman, Part time kid herder and retired Network Engineer > E-mail: rkoberman@gmail.com <mailto:rkoberman@gmail.com> > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 I see such on beefy24 (main-amd64-default) currently but also got lucky for an detailed page display. Notably: <https://pkg-status.freebsd.org/beefy24/build.html?mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d>; shows: Load Averages Swapinfo Elapsed (107%) 102.71 106.93 107.91 55.15% 26:35:05 but <https://pkg-status.freebsd.org/builds?type=package&all=1>; shows 23:10:02 for Elapsed. (Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The poudriere configuration is allowing 45 builders at once. Each builder is likely to be configured to allow, say, up to 3 make processes, MAKE_JOBS_NUMBER=3 . It is difficult to look at logs now to check on the figure. I've no information about the amount of RAM or the size of the SWAP space for beefy24 or beefy23.) The longest-still-active port-package builders elapsed-time-so-far figures shown were: qt6-webengine-6.10.2 23:33:08 chromium-145.0.7632.109 20:43:03 electron39-39.7.0 20:41:59 apache-openoffice-devel-4.2.1768900765_1,4 11:20:05 Those are likely the major users of tmpfs space that competes for RAM+SWAP for USE_TMPFS=all (or other large USE_TMPFS settings). I'd guess that the paging is thrashing for the above but have no way of checking. It could be that some use of TMPFS_BLACKLIST to avoid the use of TMPFS for the port-packages that have huge TMPFS involved could help (without otherwise disabling USE_TMPFS=all to still speed up most builders). (I will note that TMPFS_BLACKLIST entries do not necessarily end up with no tmpfs use, just less. But that less can help avoid paging that ends up thrashing.) Note during my editing/research and other activities, the web page had not updated at all after its initial display. It is now significantly later than when I started editing this note. Later: Hmm. https://pkg-status.freebsd.org/builds?type=package&all=1 05:48:27 beefy23 (150amd64-default) but . . . <https://pkg-status.freebsd.org/beefy23/build.html?mastername=150amd64-default&build=df4f957ea181>; Load Averages Swapinfo Elapsed (112%) 107.81 100.28 97.16 0.04% 27:42:2 -- === Mark Millard marklmi at yahoo.comhome | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6d613720-fa00-4686-be70-1767dbdc1fc2>