From nobody Sun Mar 1 04:55:37 2026 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4fNqWr5f3Mz6T2ZF for ; Sun, 01 Mar 2026 04:55:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic316-8.consmr.mail.gq1.yahoo.com (sonic316-8.consmr.mail.gq1.yahoo.com [98.137.69.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4fNqWr22v8z3F7r for ; Sun, 01 Mar 2026 04:55:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1772340944; bh=35OfwaBHNNfZRKPyeAvTJhTgu0pECyHJ99dejJy285E=; h=Date:Subject:To:Cc:References:From:In-Reply-To:From:Subject:Reply-To; b=DG3YT6Ygh/v/yQa4ee2zOtBQMvizuMBKTnqg0Zh1wwSxxKzPCiD06OEAePLtApIR+Hm0Hkw9PiM1SRV2gAblalpeRPMpX0x+dCQJIPvQBSNp1Rykmez7w0rMQrAdcaIZzPsRyaTA80dEWAj+JYvaQ3Ldg3NHW1tkY3yR1YtkGYxXpJNE4gv1V0R2s/bEIXC0T5QEPLX8mF1DJu5pazq6wpyOUrU5Qow9Tn6gWrJ2j73PDGTQ2tTujCeoyiXyjtMOviWgQciKk8hOqEwUTkVcI05prXBewKvc9P0IhEeO/7oCDl0LbZGWRb3mGkqlal/h98ZneGTqW0RvV4FAeYARLA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1772340944; bh=5BpL5iqIt89Np6Hzth2t+eymc0ug7qK/7LVtpqext4m=; h=X-Sonic-MF:Date:Subject:To:From:From:Subject; b=ZICFyCCaSVFwSD4W1NL1HAkpM1h92PSrOE/E2QMP3VbussT7aMeh/ibihOGfyATe/A8zyexBUdX6iOG7w/mfi2CpXo0fr1uXMoRZrtFXlLVApJEbLllvNxHclhaYsfnm5uDbaERlLEaoG7QSVd1DtDpRjz0f+QnvxI+ueo3b3VBX0j4uLKWlJhdaqyEISn6tCuzTxYa4OiT+viG5YIX2m1e+CoNkiuuuS75QrFX4+WFNQtmGbTGWyNhG/8Szc20avsAHpI7+K4dK8cSmaPTJxf6V145lIYKGV9A833+W5m62Hefb5QwmWfemEV7bDW82YDXoSfYHObLjsuVJOaMkqQ== X-YMail-OSG: iICHPKIVM1npCCvflVyrtRaPy4UqPpvsGPv18.QvoZiITpunPCf4GW4vYy2uy8c gme3j9seBd8Xy2zk94VUwRpow7sjImp9mzL9XSb7_zMNV2RBQu62BMnKqhE1igKiKenjKXEpyb7R v3dBBCl89przR.hkdaIZIaGfjx0LTz_pv5L4LW91pYq9bn4lifSKCSb_aqpeIxiMB9gDWRMp8Y4J XWA2g2PhrClumArWQGM5miIQZbBwSVMGOurczjR.rfekeXQrGZPhSdw8xmN9AZN94DI6nW9r7DaX 2iMWEm1LVpH5FHGDPFwHnvN1KCGF_z7e1q_ueU3fqKMM59TOlxxOVbJBSRfKWIcghHHG3VPIKvG9 pyB6PCJQxEKrKTj3pxjV1_ceP1FVG4a1KhcuYzt0paitsGmYBPX2rlD2UouBl6IE81YeuQr6EmsN bTm2qhdBa77EfbD7Ws1smpDA3oepiC9UkISIaGboG1FcMUgaOm_j6CrFji_HFWB7Fvirp1CFu45I BELEwmZQB6db9jcdVPoGGsjVao38h1.fimoy9yiPcTLrRwOzDG6ZxSno5QnlQuELSGcbEytGl44. DRnbf0Ri06McSjjq45FBU5H0MKF_Svu1K8j5NM6aNHx_Eon2pffAOq_6W2Kdfz0lSemla4jCway0 dpz0Q.NDu_T5yYxE9G3j1LqBrjBwimgJ5bz9m1LFajJV7cSVbQ30xokv7eD0KLoq6bULbdAJex91 1iOxu4aUnCpoHJGsUxZcy3Hda.0rTjFT8sdP1RrlDtB79ItyET22.geWorn2J7R388g9iYtRMaMk Kj97xidzfLOZnaUqoG0cI7QN1FVcKzfxPc6FY.j5uDA7wLw7oDy4gYlGMKPdZS44UPMpZkS38MCf 62u01K9if.7ZtWz.16tGu_a2HnC07Li3lKWY.1lpYJ1RaUdlUCo8sQcnzS1WzZCDqk1VS0BCm5P9 zbHLQbVRfbc.jz2E2Cx89gk4H.cYYVWicIzVD5f.0X0Z7z38QfnN5plu661SnL2nemEbs9jvCGqA stC0iyCvaZLTNKLqTctznOB0xiBXrviRW60_kpbA757kFiAaemwWUL6ITKoM8cYrwjgG9uP9CSTJ HV1qilze6POsM8kl.0iMQTiz4C0xKD0iwqYks6hTMAP1F9DGKuhlbWwnAP9wmuEOweD4luO4Y7XM 0KASfbhVq_b5YUPe13ZZDL1p3z9k7OEeOPyW901nSC8GVpjC9j6CWmOHlHdNb4VxjPdrmeeD3kwl VA4sFOzDmmhGXj6pmh9rQMHSbq_qbJB8_8NKo4mepth8TkxnJPQ7WEGZtSPkypmlwWBqnqoj8sNN IGC201kkY.19QumCGqig3Dul2D0Cn.noxeDqqxE41IGklYNZJYAKfJdMR6L1eHwn4ydbzI6_imHV U.aILVgmhwe09305EpaKUHHAO6VkvJzHOh59eQ8SycFqQjlhjJPXT.lrRSoRVX2pdStDeEkT.Lvx .SK9JCQwyut1Mv_ouGU8YXctxjNR3xZz13EWPiAZyrUx8hzOsIL3vi7smDFDRTaIWT5Z7FYG7Ulq erQKOZgJ6L7_4uyawVYtNqJvUzR0YrhzNvN0CAp7YLmG6slGudFbGWrj73zxsiE4EtUtUKT0S6Zh fNH7tlgyS.bQD3PUQ4chFS8VvvcTe9a2B_6xLnsAdm_y7LxqM1kTyEZsy.a7NKgpZOC8cBgluwl8 yVOy4lR8krCU8gx43c_whQ2229imiGTjxpqRV0nWAGDLbuzoUNoDEycZLIN3lsfJ3pfY7JfhaD8R 3euoiKhYFyxVDJmUOd_JBKnteIgmXZmikGzkhDb2AffG5HiBnGyh5tKbpoph9UODIGAGC3p1s7KI j0.Zeg1r8BcV9lgMfs1e_SnTbSSnXRcT.zMTYlhVifwxe0EQ00fqkF6IIKk4A89mOgNzn1M.V2cK a.oy498R1bS7KPuZIG2I.VOXVIMsRoXKzl4hs8FuJ4KG5.kiKYopvdKyvybg9IYY_XdvsQcAnzLH 2n1gYho7_HjWqrXrr7ekQTSEZbNil6P5K_WFHuZrCR9yyNmhIAdVqbvpeug9_N.vFkIOuqvywHPq edd8tkrEUjoOfJDgRmaqxBJMD7PTH5U9GnCSc_Mg7hyA_uqn4Q7s7lKMnOUKu70rghfnAl6fkRru jmhACWpMNAm26AyOPDKAiC95M2.RBzU6ACDjHhQYaiEKm93CuSQYthrzlRm0SlCgrmHvaQgdm5sY tDegsu5C6EfsGz.QWhAGBI4oj1zpw41L52_6jB3SHkQAIIMzSCH3vuHqzoPepOEbnULW4p0xvhJO _kQfOW_6pht7tqEWalu3EGh6x_xKsj6eCLbhguB8mr0Kd1f.WMx0J X-Sonic-MF: X-Sonic-ID: 2358846a-ae70-4ecd-bb2a-38d28f105d5c Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Sun, 1 Mar 2026 04:55:44 +0000 Received: by hermes--production-gq1-6dfcf9f8b-gxljn (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID c217506b6260cb9ab7e9519a48a26134; Sun, 01 Mar 2026 04:55:39 +0000 (UTC) Message-ID: <6d613720-fa00-4686-be70-1767dbdc1fc2@yahoo.com> Date: Sat, 28 Feb 2026 20:55:37 -0800 List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: 504 gateway time-outs To: Kevin Oberman , Graham Perrin Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org References: <1895ba62-b069-4e28-a910-6c666703ee8e@gmail.com> Content-Language: en-US From: Mark Millard In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Mailer: WebService/1.1.25198 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4fNqWr22v8z3F7r X-Spamd-Bar: ---- On 2/28/26 17:57, Kevin Oberman wrote: > On Sat, Feb 28, 2026 at 5:48 PM Kevin Oberman > wrote: > > On Sat, Feb 28, 2026 at 3:33 AM Graham Perrin > > wrote: > > On 28/02/2026 10:32, Mark Millard wrote: > > … > > > > The following got "504 Gateway Time-out" when I tried them: > > > > mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d > mastername=main-amd64-default&build=pdf4f957ea181_s178d0b5b8d>> > > > > mastername=150amd64-default&build=df4f957ea181 status.freebsd.org/beefy23/build.html?mastername=150amd64- > default&build=df4f957ea181>> > > > Both somewhat slow to load, however they do load for me. > > Thanks > >   > beefy22 is showing hte same issues. Something is tying up the > builders with builds hanging in "build-depends" and "lib-depends" > for very long periods of time, as long as over 4 hours) with high > load averages. This has been going on for months and seems to be > getting worse. Chromium builds which used to take around 20 hours > are now running around 40 and I really don't think that this is just > code bloat. At times, about half of the active builds are in some > state other than "build"; mostly one of the depends states which > should never take long as all dependencies are built before a build > is started. > > After some time passes (often hours) things clear up. Dependencies > are suddenly loaded in a few minutes or less. Note that things do > continue to complete, but the 10 minute averages are in single > digits and frequently go to 0. During this time, my connection to > beefy22 will timeout and sometimes I can't establish a connection. > > I don't have any special access to the machines... just via pkg- > status, so I'm fairly limited in any analysis. > > > Just before I started my last message,  all was well, but I last saw > were a dozen builds in one of the "depends" states and "impulse" at 74 > when I lost my connection. I can reconnect to the beefy22 status page, > but it has not updated for several minutes. [Do not expect that I edited my all notes in presentation-text-order.] Via : beefy22: 143amd64-default 17:21:38 Accurate would be over 27 hrs (see below). Note: beefy22 has 64 FreeBSD cpus. I do not know the RAM or SWAP space. : Load Averages Swapinfo Elapsed (100%) 63.94 69.19 70.56 0.24% 27:37:33 > -- > Kevin Oberman, Part time kid herder and retired Network Engineer > E-mail: rkoberman@gmail.com > PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683 I see such on beefy24 (main-amd64-default) currently but also got lucky for an detailed page display. Notably: shows: Load Averages Swapinfo Elapsed (107%) 102.71 106.93 107.91 55.15% 26:35:05 but shows 23:10:02 for Elapsed. (Note: There are 96 FreeBSD cpus in beefy24 and in beefy23. The poudriere configuration is allowing 45 builders at once. Each builder is likely to be configured to allow, say, up to 3 make processes, MAKE_JOBS_NUMBER=3 . It is difficult to look at logs now to check on the figure. I've no information about the amount of RAM or the size of the SWAP space for beefy24 or beefy23.) The longest-still-active port-package builders elapsed-time-so-far figures shown were: qt6-webengine-6.10.2 23:33:08 chromium-145.0.7632.109 20:43:03 electron39-39.7.0 20:41:59 apache-openoffice-devel-4.2.1768900765_1,4 11:20:05 Those are likely the major users of tmpfs space that competes for RAM+SWAP for USE_TMPFS=all (or other large USE_TMPFS settings). I'd guess that the paging is thrashing for the above but have no way of checking. It could be that some use of TMPFS_BLACKLIST to avoid the use of TMPFS for the port-packages that have huge TMPFS involved could help (without otherwise disabling USE_TMPFS=all to still speed up most builders). (I will note that TMPFS_BLACKLIST entries do not necessarily end up with no tmpfs use, just less. But that less can help avoid paging that ends up thrashing.) Note during my editing/research and other activities, the web page had not updated at all after its initial display. It is now significantly later than when I started editing this note. Later: Hmm. https://pkg-status.freebsd.org/builds?type=package&all=1 05:48:27 beefy23 (150amd64-default) but . . . Load Averages Swapinfo Elapsed (112%) 107.81 100.28 97.16 0.04% 27:42:2 -- === Mark Millard marklmi at yahoo.com