From nobody Mon Aug 15 01:40:49 2022 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4M5cS93jDlz4ZZ0x for ; Mon, 15 Aug 2022 01:40:57 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4M5cS85HqNz4Mg3 for ; Mon, 15 Aug 2022 01:40:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1660527654; bh=8RYXUDxnsRbdA4ibnV20jdpHvf+9xh/RMEnjO7+E2lU=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=kqD7oymrGml+dW3yPV/etKvTFSHjlEtMCkirKI5WcWmT8mwBPXPpgvvDwpAU0xZY3UHnca++81RMfrEqUAscMYuZLBMWehSMUd/4gaQXVvDhswTQojddPiWjIjK1jAP1lcvWRxvHLstaXRiRnAbbZttJwnOuvQQUY089ypqEx75ekIWAthcFuonDn0J/66qJGfrOJ5RNw9Mj+LVvS6U7S+CP/67cGdaxCrTPOBR5sljYIMO1/6wEd78ohBVH1W8JVCEdiACU8P+Sm0gff395uB2KM+dC9sjRL1o60sC9pzmmJEGqdugbtnOX81LZydtV2dA+wAnW+NI1PQ5rrjRQag== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1660527654; bh=LMoqXnrTbPDMGugW8OwC06UEOs704vJZHc3TTeh/vfx=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=HwTonmcylkCFKZYB4hJHqS2bVTKGWD9pokGta+Cgy89DsgPqgOQq6CzGbQ9+pB7CzIM26eTL19/BwbbY6KJ0Xd+OjysLl8zSaeCrozhQ6GpbmjEnDXLJEZP0C+IY2PXqN8aUzHblw0xdl5v5Je/wQf2lLEYXTi64Uxs5rVALAL0jRabDwo5hakGizqoH/opLuqFO4LpIiTegpD1zw8duMmFcbny5sAOAHymrDrsbVnHrXBI1t9LrkomqzZT/VGEXqj6FwZGUCQiElEX/E/EGcseDxaLWmcN2waK61/aMdahQeycF4T9ultO5gnlSVcmXycpA8M3y3+fnKNKTIAtZ7w== X-YMail-OSG: ArB2J20VM1kJHTL8FRFE4HJotZszq42NTmNdDGy8ybW12nUXj4QbCW7_JbhkCBq 73J1Ry_HZL6MerpGkoPsBRc82Rxhe7brLmVDdtznVjxSfKA.npplCjiqhrvwjQ1gxzfJs0fX6UE. BwM.R.eKy6L8uIt.K18cBaCURF1gIy.6ncR9PawRayTYaSYTy4TjbE..KxQvUQXTU1EIazdQbASv flGNTxUtAFLdu6uhWxmjM6MdbOAdvSZBYLVyGqB74BOTVR2IcUByLgRpKYjkteGPMleEJ0kT3afE SafNaMYqsGlmL0VbKcHI7s4LTJ0W.HxFRzsa31yHrC_edONleeiEY479L5lu0N1T_2jf.x1eQmUv jNwPMToy6JnG_z5Yqr2LNaWquBfjL561SuSP3Z0HHAHGZH_1nQuW0FXGsL_zJm43uRBRH5snZISb KSH7GBHHvjfxMH8vy4X04YRgTdF0vyDGILxAEIKKuuNhNH4a9qCshlu1VI5XR7fywLEi1kskLV3g yfzl9jAoYRdUg7sHmo82sH7bXBnpPotj8dbGThcf1zDkRDXWGKaT8KN14ID5USIhYD8AUtASdPCt smEJzb.NGBehzvTf7bbyBS4sRIG1DTefHFxIgIuCpknSIG7toQbjRtMC8G8IuyUaZlLdm4bOGei2 66EbUKUByP5WbTKNDWcb3K_oMkzoigFw1ebLWq8RlLj.e.tbBJQrZiqFISWvQUVmzpJYdYs1PJUQ EcWxOvXnemAT2y6txO_cyEqZUZH5hkO_Aavg0nNgxRBiLLbnkq1FGZnIUjm93I3y.DuTKUg6Pvra NkQT8DfizBWXcLJK9SAlCakomeJ07CwpGMRWsm5W_gTh4RdC6HmrGvzs1_juSPc6rVFGP.qtBj9Z 4UQc9yGgOkRTwFUQr5SIqHi.XjD0kh3Rj0TeAXPMTS8KYpST24imo1IDWmuCc_FvF.G4hg00A6O0 TMCfhLEJwqqQeiRYSgjS0PFfyHTyOaNwf.KQbimGwy0K.ZpWaK4JR.7kWwornjkKD.SgLZA7gV7R 61EoJIeNOscmmwPeQ1bolRlXD16bx7D6Nc.aSbdyw5fnPguWl2zRwzQxcbk9WKsBuh1xxsam2uQt eTbLGBb380Ux1pd6WveNC16rnhebvem0JSS4DIwQ3DyBnMiVOeP5c49gMCYZ0NueMas7lD0NGsqB 7a7Nsurqu8d0chuTswkdQ61_ZXrpZNdxIvQ8O94QepM3U6olHauD8TU7u4Xgzy5pbZjCuZHGD5GI E70LakV2hbefzQQ5aTids4OV7qblICYMerED0nKwpQCp5eK2VOsXVK2i.r7CR.0IQ27sk9tIyfES mVjSlUZWhVyLAD_z0rqHn8Jt2G_.RjJLkDIZijaYdUlCEp2TocyCMsCZK8sJZGTOOYWHZZdhuSNs D.xSsFucblxenYEes8CTwIHvUhHZxfp1qFAqSKz69g_clDG_6GL0IWNedkQjtOiEsr2Elrcyq7W. vuNhW.r_nqvKOWAi3CCoIsZlcn9qLqwUYb_qZ1sHO8sLuwKli43vm2S9NcvXypoqLk5aKO0hsbZw mVBs0bGJdnuP_R2b2L.JPdxgp.IWqLQt_YIl7CTy2F_jPUJ4Dmgf0Om9BovkhZML7xS4T2T38giX K9QzKRayFR9yONKQn7vluu4dp8eQqlWvHzrgR3BC1qDaJpW.zp7r0ejIukSjDt.KsJRK2Og05skd BAttb5xkKpgnA44c8.0Kxx4RUYrTpp6w0CQx4fc3z1IO7JHQ.ObiQ2dfSvvvxxg5GZg4drmDDy0d xFx4vrnh57qLlRym5IVwTVpTFx3uuN8hSv5XzHz.eWiGs267E9KbVZ1RL8AZfe3jAfgOLhO9Xg_J otlf.bBOiI_gIl_2JGN0bQIrZXq3rp71m1zaEDVhZfT5OP3beqWzo1mCYeYfc7pKgc_qbnrn1lN7 Hy1dhZR8EBobJg9CEr4QEt.M4sHkvBPZvzOiQ8hKrm_v3pW93aefUNZrRPkRaDPlU_qCl1JymT0g iNfD9lc9Z2wp.mXxdAWc.nstGPDlnfUOYM2HFS8FT3MlGLjTsgnLilJDh8JVH4qO.SC5hT7IMytM YcOLmFvmkZRBP4WKd.7Whyg.eJKlCilkqZVPb.bYjh013dO9WmdFrEuiHbNrHVnm.NP5VDx065b_ h8_ZSnFNPnBOf6I06Uur8MOeldVtpk_BN0.TfGfrl8VT7xTbAAMAtPhLwJnp0KfmXCUYKjvUcx0X 4_CSNuOFZ6ZKsnts_xCx1ZgsPLvy32rvuMRkItFLBivhSitoCI4CWnPPiV5auQ7mFl35fZ646yx3 1vzd7 X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Mon, 15 Aug 2022 01:40:54 +0000 Received: by hermes--production-ne1-6649c47445-8p8nm (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 8a4544467dd12ca0f4c57a8b9c6ea2f3; Mon, 15 Aug 2022 01:40:50 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Resolved: devel/llvm13 build: "ninja: build stopped: subcommand failed" From: Mark Millard In-Reply-To: <7CDC63F3-8B68-420E-8012-B1692667E293@yahoo.com> Date: Sun, 14 Aug 2022 18:40:49 -0700 Cc: FreeBSD Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <1D4C14BD-8955-4B86-9C99-3E58D7603122.ref@yahoo.com> <1D4C14BD-8955-4B86-9C99-3E58D7603122@yahoo.com> <7CDC63F3-8B68-420E-8012-B1692667E293@yahoo.com> To: Nuno Teixeira X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4M5cS85HqNz4Mg3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=kqD7oymr; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.40 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; NEURAL_HAM_MEDIUM(-0.90)[-0.904]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.147:from]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-ports@freebsd.org]; TO_DN_ALL(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.66.147:from] X-ThisMailContainsUnwantedMimeParts: N On 2022-Aug-14, at 09:35, Mark Millard wrote: > On 2022-Aug-14, at 07:50, Nuno Teixeira wrote: >=20 > . . . >> have some time now and it's caused by a build peak of memory that = affects people with less than 32/64GB mem and to solve building it must = be build using one builder with one core thats takes about 7 hours on my = machine or with 6c+6t on 12.3 i386 that takes about 45min (123i386 is = the only jail that I can use all cores). >=20 > Last I tried I built all the various devel/llvm* on a 8 GiByte > RPi4B, 4 builders active and ALLOW_MAKE_JOBS=3Dyes in use. > 4 FreeBSD cpus. So the load average would have been around 16+ > much of the time during devel/llvm13 's builder activity. > USE_TMPFS=3Ddata in use. >=20 > Similarly for a 16 GiByte machine --but it is also an aarch64 > context, also 4 FreebSD cpus. >=20 > But I use in /boot/loader.conf: >=20 > # > # Delay when persistent low free RAM leads to > # Out Of Memory killing of processes: > vm.pageout_oom_seq=3D120 >=20 > This has been historically important to avoiding the likes of > "was killed: failed to reclaim memory" and related notices on > various armv7 and aarch64 small board computers used to > buildworld buildkernel and build ports, using all the cores. >=20 > The only amd64 system that I've access to has 32 FreeBSD cpus > and 128 GiBytes of RAM. Not a good basis for a comparison test > with your context. I've no i386 access at all. >=20 >> llvm 12 build without problems >=20 > Hmm. I'll try building devel/llvm13 on aarch64 with periodic > sampling of the memory use to see maximum observed figures > for SWAP and for various categories of RAM, as well as the > largest observed load averages. >=20 > ZFS context use. I could try UFS as well. >=20 > Swap: 30720Mi Total on the 8GiByte RPi4B. > So about 38 GiBytes RAM+SWAP available. > We should see how much SWAP is used. >=20 > Before starting poudriere, shortly after a reboot: >=20 > 19296Ki MaxObs(Act+Lndry+SwapUsed) > (No SWAP in use at the time.) >=20 > # poudriere bulk -jmain-CA72-bulk_a -w devel/llvm13 >=20 > for the from scratch build: reports: >=20 > [00:00:34] Building 91 packages using up to 4 builders >=20 > The ports tree is about a month back: >=20 > # ~/fbsd-based-on-what-commit.sh -C /usr/ports/ > branch: main > merge-base: 872199326a916efbb4bf13c97bc1af910ba1482e > merge-base: CommitDate: 2022-07-14 01:26:04 +0000 > 872199326a91 (HEAD -> main, freebsd/main, freebsd/HEAD) = devel/ruby-build: Update to 20220713 > n589512 (--first-parent --count for merge-base) >=20 > But, if I gather right, the problem you see goes back > before that. >=20 > I can not tell how 4 FreeBSD cpus compares to the > count that the Lenovo Legion 5 gets. >=20 > I'll report on its maximum observed figures once the > build stops. It will be a while before the RPi4B > gets that far. >=20 > The ports built prior to devel/llvm13's builder starting > will lead to load averages over 4 from up to 4 > builders, each potentially using up to around 4 > processes. I'll see about starting a separate tracking > once devel/llvm13 's builder has started if I happen > to observe it at the right time frame for doing such. >=20 > . . . I actually have tried a few builds on different machines. The 8GiByte RPi4B takes a long time and is currently omitted from this report. 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): but using MAKE_JOBS_NUMBER=3D4 (with both FLANG and MLIR) On amd64 I started a build with FLANG and MLIR enabled, using MAKE_JOBS_NUMBER=3D4 in devel/llvm13/Makefile to limit the build to 4 FreeBSD cpus. It is a ZFS context. Given the 128 GiBytes of RAM, there will not be much for effects of memory-pressure. But will record the=20 MaxObs(Act+Lndry+SwapUsed) and the like. ---Begin OPTIONS List--- =3D=3D=3D> The following configuration options are available for = llvm13-13.0.1_3: BE_AMDGPU=3Don: AMD GPU backend (required by mesa) BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) CLANG=3Don: Build clang COMPILER_RT=3Don: Sanitizer libraries DOCS=3Don: Build and/or install documentation EXTRAS=3Don: Extra clang tools FLANG=3Don: Flang FORTRAN compiler GOLD=3Don: Build the LLVM Gold plugin for LTO LIT=3Don: Install lit and FileCheck test tools LLD=3Don: Install lld, the LLVM linker LLDB=3Don: Install lldb, the LLVM debugger MLIR=3Don: Multi-Level Intermediate Representation OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library PYCLANG=3Don: Install python bindings to libclang =3D=3D=3D=3D> Options available for the single BACKENDS: you have to = select exactly one of them BE_FREEBSD=3Doff: Backends for FreeBSD architectures BE_NATIVE=3Doff: Backend(s) for this architecture (X86) BE_STANDARD=3Don: All non-experimental backends =3D=3D=3D> Use 'make config' to modify these settings ---End OPTIONS List--- [02:23:55] [01] [02:04:29] Finished devel/llvm13 | llvm13-13.0.1_3: = Success For just the devel/llvm13 builder activity, no parallel builds and excluding the prerequisites being built: load averages: . . . MaxObs: 6.76, 4.75, 4.38 6812Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed. Note: MAKE_JOBS_NUMBER does not constrain any lld=20 procoess from using all available FreeBSD cpus (via threading) --and multiple lld's can be active at the same time. So this looks to fit in a 16 GiByte RAM context just fine, no SWAP needed. I'll try MAKE_JOBS_NUMBER=3D12 instead and rerun on the same machine. 128 GiByte amd64 ThreadRipper 1950X (16 cores, so 32 FreeBSD cpus): but using MAKE_JOBS_NUMBER=3D12 (with both FLANG and MLIR) ---Begin OPTIONS List--- =3D=3D=3D> The following configuration options are available for = llvm13-13.0.1_3: BE_AMDGPU=3Don: AMD GPU backend (required by mesa) BE_WASM=3Don: WebAssembly backend (required by firefox via wasi) CLANG=3Don: Build clang COMPILER_RT=3Don: Sanitizer libraries DOCS=3Don: Build and/or install documentation EXTRAS=3Don: Extra clang tools FLANG=3Don: Flang FORTRAN compiler GOLD=3Don: Build the LLVM Gold plugin for LTO LIT=3Don: Install lit and FileCheck test tools LLD=3Don: Install lld, the LLVM linker LLDB=3Don: Install lldb, the LLVM debugger MLIR=3Don: Multi-Level Intermediate Representation OPENMP=3Don: Install libomp, the LLVM OpenMP runtime library PYCLANG=3Don: Install python bindings to libclang =3D=3D=3D=3D> Options available for the single BACKENDS: you have to = select exactly one of them BE_FREEBSD=3Doff: Backends for FreeBSD architectures BE_NATIVE=3Doff: Backend(s) for this architecture (X86) BE_STANDARD=3Don: All non-experimental backends =3D=3D=3D> Use 'make config' to modify these settings ---End OPTIONS List--- [00:55:37] [01] [00:55:30] Finished devel/llvm13 | llvm13-13.0.1_3: = Success load averages: . . . MaxObs: 12.45, 12.20, 11.52 13074Mi MaxObs(Act+Lndry+SwapUsed) but no use of SWAP observed. Note: MAKE_JOBS_NUMBER does not constrain any lld=20 procoess from using all available FreeBSD cpus (via threading) --and multiple lld's can be active at the same time. (16+4)*1024 Mi - 13074 Mi =3D=3D 7406 Mi for other RAM+SWAP use. (Crude estimates relative to your context.) That would seem to be plenty. Conclusion: It is far from clear what all was contributing to your (16+4)*1024 MiBytes proving to be insufficient. Unintentional tmpfs use, such as a typo in USE_TMPFS in /usr/local/etc/poudriere.conf ? I really have no clue: the example is arbitrary. Other notes: # uname -apKU FreeBSD amd64_ZFS 14.0-CURRENT FreeBSD 14.0-CURRENT #50 = main-n256584-5bc926af9fd1-dirty: Wed Jul 6 17:44:43 PDT 2022 = root@amd64_ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1400063 1400063 Note that the above is without WITNESS and without INVARIANTS and the = like. The only thing commited to main's contrib/llvm-project after that was 9ef1127008 : QUOTE Apply tentative llvm fix for avoiding fma on PowerPC SPE Merge llvm review D77558, by Justin Hibbits: PowerPC: Don't hoist float multiply + add to fused operation on SPE SPE doesn't have a fmadd instruction, so don't bother hoisting a multiply and add sequence to this, as it'd become just a library call. Hoisting happens too late for the CTR usability test to veto using the = CTR in a loop, and results in an assert "Invalid PPC CTR loop!". END QUOTE =3D=3D=3D Mark Millard marklmi at yahoo.com