From nobody Wed Feb 15 21:02:26 2023 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PH9Wq31cRz3rLQl for ; Wed, 15 Feb 2023 21:02:47 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4PH9Wp0jyNz3JGw for ; Wed, 15 Feb 2023 21:02:46 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=JM5lqJ6t; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676494964; bh=4PKRouqqcV0qMEyDRT5D0ZaVtWwwxRLh9hxrcf9eiIo=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=JM5lqJ6tc4dYzSiGy0SXFStIaA3+snWDb3WWmPgPStYqfoLFnkbosgmEDmvn/CTfd6EB3pOWhXtBHbv1ooChE2TpyTlQOZNqba1PcqC1HY3bxPxO99vOOUd4AWbO283iurqPTtO73DjvobB/8VNi+Tczk3PxOZ9ZKqhpEfyXy675Qvc3HNocNr3Wa9HTDa9G1xs0rfI/vpWjNCV1ay2iRFhmsVW22rs564jPbkGYq/V1u2bRH8tZTDbWWMPUCln2a4LVHczrsxF/aiitJN2nOqE4LHmKRHjEGNs7aO5jFYEfrSMOTnVuMT6cyPNzz7g0N4F0OE+x5BUjcLICboa5wQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1676494964; bh=c3CcoMl40ojJJEdEKgxuTsxDmOt+RArBfDWqeYV0knX=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=UOXxU9s973whWRD2hIhwU4qF1adR2PuU5cbCbzAxe26UK+jbnjBct4sQFY61rHpEAVNzIm8GYmLpvPWkK7oiVVvHruKpWIbg+QjAZcpkr3Z3Q4vEr+C2yYeGfRo2AFyQvV53eO5Mj7VEok3paKOPeiK+H6aN6gxrcwogLd8Bf3IP479Qc0LVYa+4osJuqp0vstiV1xymuNVNmZI1xXnxtsrAMRNsfpBgzx7q0mB9eMfAERgabfIVohxR2EkDtcx7jjjlNOi7U1DZ/1sVl0htfJ3jRv2OTLNGQtSt16+/vExSv+02s7+Ux6JmoTAr0atwALKUyJPX2CXlMSUtl7qLdA== X-YMail-OSG: r9mQeZ4VM1nyRbVzWc7rxhlFAINvtoZeU1S6AtVtgS.4R51sOKmhM65P5Uu80RG C0FZycdDszsj3rV4DYMMf46nbU_ykDa3G2n459ZcmOo_7WTPTvuXBEolWRc0BabKiCOi_wVmhv1j _sdVcQ4Enr_Y5OYDCkhIR_fGNred.YhthcnhZm09UIKeOiXPeoBsct_3WcaPyfn2qjsAzusp9OHI 35_2jxSNBZgKh1lTbLp7vBt5t2H1Efcg9gln1.S0lPsBUNb._3zoxXhWgdNXCtfbT.KCjeEincup otF3yJn4Z7D7dDFUhMB_AkVwmCfQKglxnf8A_15VRhivXCLWDovTOgSLmTEOH0Ndg39bbA15eLJr w053ZdJb8PlgL9TcXpiTpTJmCmvnyZF8_fprv4dCAehGcDD_vWl3BCIMIW90VwE7prhpG.vHF0RZ hX9Wy9OY9_Est9eR_IOCZRzNzMHnwnVLAqYRPccdsk.mzW.QxTkHNpRP8Z0WXnNcF5kVyoqWtDV6 g3mu_DBKcok1fZb6lLqfON12Ozqt6aCgQR7VaaStXbtWLbrRBymDmC6ZdUTunyg8IJaJRG22Fy2P q8I_KTqYg1eqjUXt_pqHoGKA.v_Mw_i_ywlsxUYBC_UL38FqrpQUCmR19_7WbLZhBmhxJSX4NK4B QppddMRtFHZkiATuESJJDKlLNSstS5ni3hpx4MyeKZLaItrV8aKkyOb.Ia2fl7TIFTeRjafFnRM5 wHiyxRivWoC8lncIoYxC51JPtKPRi1ClERoi2ncfYX_sLpG8YPPvj0n3FeEC73IBLTl9.vpd0ydk zmlFmrAEf38LUkUfcUjxAJW9Lqga5ZQtrV8JT6GSG.SNFXqiWW8Zk.KBzspQ0aRTp2mekM84L.St B02jq4QrWvOwYgbXljkukKhF0Tlub_m5n5C.JaU_pnFEElrCv2PG2CRNsiYmVIjuzGkuprkz8LuC r.e8j2DN4IennlcvZWxJ8SqQclcsJXIbAZflmD.bBRuq9n.0Osmrl.u9ugJ5uViWSybV.eH03TB4 Cdc3zyXuqFTAta105i.dpGKwuyQ2h1d1TmMCYJcGYEmuqC4HqGlJ_Sx8p.edBEnFP6EJlXxy0HsI tu6a5m1j.cGDHBAKJl5pTfIgCDHCoDjm1JDNCuYV2Q34cGNxS921Af8kh9GcovmjiA5kXCtyUQQ0 U04Mc2qkA0WF0Q_g4jQrKB5JJhXqSwwPgfXivLXinGJRrluePWlZTp1XHRlgUeSp33bAGDySntzO Md9qZF.x2lHbweKtc4Xl3seSMu2r2IIHPV1yID6Vl.gCSg_jT.QgBmmpAsaaEWzoy_f1_p.elkcV ZGeZeVPWXXKWR9Q6SvQYZTWG_RrB_Qix_7LD_Vropp1bsy3Nx8ECXE3HmsgKIVagbD2zpPTBBaAo cDYIK51nvdVLF.1tFCL2yWYoRargEAHGkGu7o4vHV.L6EPjnZuOIp8upENz_VnIWH.XEm3gdTBMi 0b3euRGlWHvRn0z6yqZt2oN8AW25AQ.MTkjPTbcoWQyAHkdK2GLecLgLyt3FtXZyDcKnVcIdUYdV 1Rv.hVBU0zFn7jCFp61sU_BNGcyWN_Ml7ndbpBURzaETHHBNmQx93SAU.J9XJXW081M2t62vzGd9 KJ7o3ZuOgrkVdqEt9EcQ_1ggkIjoG_P8o.gEHfSO4.qjOndNmlcxjYMgqSONtM_pl8a555WbrgP7 c7NNyHnvVvVTFJXK6wLwmYPqCeI4cj3LtP6tdmNU8QhOKdJhZyoO5QoU9tiVecPUmaFch9phZzOM QH5KGkka3zz93aX9RX_vw03d4IuhjV_G_ZRIkesOsGde1HwyAMAQubwIZeeCNJTEK0GP80YwAw0j 0_oNf5EG.akW1sxyoHfRIIn_e_ahYtuYP2a4Jc08wLkwNW7v1TlUlFNpoF8HKsJ4jNC7uya15mqF IuTNhnj_wL8VxExBdssXoQrt6QPt2wd2BIx6HY_eUppjCnl4nZ_GnT_GXpppjszOZiNh7IFY814z SGgR3F9n5.pO_rXxEc89ahmKipOKfiqt5BtHQqr2IGDm5fvPvGyO.6uj3E4vrQRntxZ7sSWc3cwa ULp0pPH.g0yaXfqxPAmmNesAKSuMfLXPtViz7cJkmXffhgAjjAUgK2Z23LKFWAgVnp9o.sqRCE9d zA.RrkwKAbBZbPoyf0NehOHv.sMEpNyFZELxM0fF0hoOP350aojA.l36ArzG8HWhrFyJZHJLR63q f7DX5Mr3wCR9Va_mUYyYB4E6.TKFm3UczqpGBL_jKbNPtrPqRWOc_EXSgqGbuAcCBbMeX2qQgpNq rVjI- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Wed, 15 Feb 2023 21:02:44 +0000 Received: by hermes--production-ne1-746bc6c6c4-z5xkm (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID eabec3a2465f2e19298ab703ec6f7b27; Wed, 15 Feb 2023 21:02:39 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.300.101.1.3\)) Subject: RE: How poudriere's PACKAGE_FETCH_WHITELIST should work? Message-Id: <9B296C55-6F06-4E10-9056-ECAD05630920@yahoo.com> Date: Wed, 15 Feb 2023 13:02:26 -0800 To: 000.fbsd@quip.cz, FreeBSD Mailing List X-Mailer: Apple Mail (2.3731.300.101.1.3) References: <9B296C55-6F06-4E10-9056-ECAD05630920.ref@yahoo.com> X-Spamd-Result: default: False [-2.00 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; SUBJECT_ENDS_SPACES(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.84:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MLMMJ_DEST(0.00)[freebsd-ports@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; FROM_EQ_ENVFROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.84:from] X-Rspamd-Queue-Id: 4PH9Wp0jyNz3JGw X-Spamd-Bar: - X-ThisMailContainsUnwantedMimeParts: N Miroslav Lachman <000.fbsd_at_quip.cz> wrote on Date: Wed, 15 Feb 2023 19:50:59 UTC : > Poudriere and fetching build dependecies - I would like to know how it=20= > is supposed to work? >=20 > I use poudriere a bunch of years, with ports overlays etc. As I need = to=20 > rebuild packages for our machines many times a month I am tired of=20 > building huge and slow packages like llvm, gcc, rust (as they often = eat=20 > all memory+swap then build is killed by OOM). You may or may not have run into the following sort of thing. Using rust as an example: it uses 10 GiByte+ of file system space, as I have things set up, 17 GiByte+. So, if this ends u pin tmpfs space, RAM+SWAP has to cover that large file system space. Only 2 simple USE_TMPFS=3D settings avoid this: USE_TMPFS=3Ddata USE_TMPFS=3Dno There is also the following pair that can otherwise be used to avoid such for just specific ports when using other USE_TMPFS=3D settings. For example: TMPFS_BLACKLIST=3D"rust" TMPFS_BLACKLIST_TMPDIR=3D${BASEFS}/data/cache/tmp (Of course, the file system for TMPFS_BLACKLIST_TMPDIR needs to have sufficient space available for whatever combination of blacklisted ports might be building at the same time.) (Note: my familiarity is with poudriere-devel .) (RAM+SWAP use for processes memory for building the likes of rust are smaller than such tmpfs use but are still notable.) (There are other relevant system settings involved for OOM should the above sort of thing prove insufficient. But I'll stop with the above for now.) > I tied to setup PACKAGE_FETCH with the two following variables in=20 > poudriere.conf > PACKAGE_FETCH_BRANCH=3Dquarterly > PACKAGE_FETCH_WHITELIST=3D"gcc* rust llvm* gcc90 gcc10 gcc11 gcc12=20 > gcc13 gcc14 llvm10 llvm11 llvm12 llvm13 llvm14 llvm15 lua54" >=20 > When I started usual "poudriere bulk" only gcc12 were fetched, llvm10=20= > and rust built from sources: >=20 > [00:01:02] Package fetch: Will fetch 1 packages from remote or local=20= > pkg cache > The following packages will be fetched: > New packages to be FETCHED: > gcc12: 12.2.0_5 (81 MiB: 100.00% of the 81 MiB to download) > Number of packages to be fetched: 1 >=20 > I thought maybe I have some different options selected for llvm and = rust=20 > than the default on official FreeBSD packages, I double checked, = removed=20 > stored options and started another poudriere bulk with different = package=20 > set (llvm10 and rust will be needed for the set). >=20 > This time the rust package was downloaded, but llvm10 built from = source=20 > again: >=20 > [00:00:22] Package fetch: Will fetch 1 packages from remote or local=20= > pkg cache > The following packages will be fetched: > New packages to be FETCHED: > rust: 1.66.0 (112 MiB: 100.00% of the 112 MiB to download) > Number of packages to be fetched: 1 > The process will require 112 MiB more space. > 112 MiB to be downloaded. > [xxxxx] Fetching rust-1.66.0.pkg: 100% 112 MiB 39.2MB/s 00:03 >=20 > But the mystery is that "poudriere bulk" failed on building rust even = if=20 > it should be used from fetched package: A possibility for the type of issue: Using 1.66 vs. 1.67 as an example, there was a time frame when the most recent package available to download was 1.66 based but the port had been updated to 1.67 . The package for 1.67 showed up later after the FreeBSD build-server poudriere bulk activity and the distribution to the download server that you happen to (potentially) use. So it might have downloaded 1.66 but discovered that the ports tree involved was at 1.67 instead. Unless port updates were delayed until the package is also available so they can be published as a unit, there is no way to avoid such mismatches. Attempts to build before the new version of the package is available leads to doing the build locally in order to match the ports tree involved. This is more of a problem for ports updated frequently. There is also that the packages for distribution are updated after the FreeBSD build server's poudriere bulk that involved the package: the packages are updated more like "as a unit" relative to the overall bulk build but distribution is spread over time and space. > [04:11:44] Failed ports: lang/rust:build > [04:11:44] Skipped ports: devel/cargo-c graphics/libimagequant=20 > graphics/py-pillow@py37 >=20 > I checked the logs and the rust build process was killed by OOM = killer.=20 > (but why poudriere was building it if it was already fetched?) >=20 > I started poudriere bulk again, rust was fetch again a this time it = was=20 > really used to build other packages, no rebuild of rust from sources = needed. In my illustration above, this could have been that the package for 1.67 had become available and was downloaded. > I am really confused why Poudriere fetches only 1 package at a time if=20= > it should fetch gcc12, rust and llvm10? My notes are not directly about the above issue. > Why Poudriere tried to build rust if it fetches it as pkg? I've tried to indicate a possibility for some examples of this. > And why it does not fetch llvm10 even if it is available and we do not=20= > have options stored for devel_llvm10? Using a more overall example context: Using poudriere, it rebuilds a port that are dependent on ports that have new versions, even if the port in question does not have a new version number. After rebuilding, the lack of a new version leads to package-update activity not installing an update. So: built but not put to use. This gets to be sort of like the earlier 1.66 vs. 1.67 illustration: The downloadable package might have been based on an earlier version of some other port(s) and, until rebuilt and distributed, is somewhat mismatched with some port(s) it was dependent on. This could lead to a rebuild, even if the rebuild might end up not being used because of a lack of version number change. =3D=3D=3D Mark Millard marklmi at yahoo.com