From nobody Thu Jul 8 02:11:59 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5744911F3959 for ; Thu, 8 Jul 2021 02:12:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-20.consmr.mail.gq1.yahoo.com (sonic302-20.consmr.mail.gq1.yahoo.com [98.137.68.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GL0D810kYz4k84 for ; Thu, 8 Jul 2021 02:12:07 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625710326; bh=sGhS34aYH4fUWsaFak5WObxueJakWrxFKBguweOkVq8=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=Rx07qxyxK8k/lXcAii+i3VqxYzcjnb5G50Kk+76LhBpwYOjoAm4Xo74T1Iy9BKCk7993Hic/uv2UcGRfswyzpFfx1CIJxFgktAye/Lm/GNic1yZNIScUqSyfwRZCPM9/hqBYFG8RZYw2UZcp9wESRBaBNUpFlvKKVoH0K8fYUVpFvbomU4BtjiAdgjkY9TSYTEpAtS2ov5535hfT8XZqmpzr+yUktsxzynQ+QKTpchCiy97jqCOrh09hosgREbeBfvosOOn0o0DlSWhYuFc64wpgPUAhtsIdJ+4WOpRJ7+L5YK1tuQuogF76XGdMWJe/VwsylGUklbePtWWJHX8BwQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625710326; bh=3xeMZEcGKpWElgMQ7XkH2U0VKoF3di41c2WFEtuOEl4=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=C/VdoZmR830wIITY24SZc5d9lJhlAMOE3NYxiNQziUteJidR45IoXQGlGgNVPw4pEfRR4kw3Y8NqJtS5uoWPJ3WfUQNYpmBK99r7TR9ikO/nmsfP56WWGblY6EWU1a88FaWpmx3SRgR6X2Wa3gYUB94GpArk5e7CCARIrPWLC+X7Ran/ZbIiJ9qLx7dlcWifoNQkdWsPwphHYAB+3Kp7GBfxYlUWOOZjitK4pdHPfl/7CRv0b/W7rFXg/5pzHu3vdBQ28G7/TAnspE3/SrhdCo6NX1s36nA1Vjr6xa4ixmms1pK/oZTJ3EbBcwaXh8e+eraUSeU87TnCAt4FJmbIbQ== X-YMail-OSG: 9LFZM.oVM1kmFRRlTMKY3CI4nOzGiS9lnwNOob1reoSbsxS83GRlArNNvy_JbOB TC1p82uUJ_60B.Dhn5RTqqNOQhc5w3E7CEJA8bry3hq6d52qeUXUOYu_xhBlyWsqHLlthc8xyIVN ucMSbCl4MJrGehDvUlCazbhctHesLzhq.erfzlkrO9DovmgcXyAVNtRF2ufXrHx7P_0SV_bKHKie S.3eoIN_emgFguu_IPw2XeHQ3RprOgAGFs9H4UyQwLjiiafnvav.3BBdSIlIIDjLCHAUm_7wyvNc AHaX4.aoFrMZajVRKCLBEiFLEwZ7MalOmEzIs7xS7VGIRzVzivyuMWkMl68LVDd13_6RRgazTkEf lnWf9_dEcexzgp.UyrNhboq.Yb7aYyZVffTZVzIW4ACNlFXg25duJcgP0LukRwHcFR8WSS0FRYq8 2oUQWR4xG8e35uyUNeO9usT8RukHYO0mH2AZRJXuQtXxFYQ4YuEQagFMm8nAqIcDtXgCXVEFFSmM KyLtSJlzK3qkH7sethZ5rHD1IyhWA4yD_Vd8Zi2DLyy8Lzbx6cW9SUqHR.Ssg37JGbB4gZbELvfa JLyVJtfzttwCYCaPzL7MgI49tOkX0nd9D4h.RSn9kE9gj2mnz84nEIshsHQgZtupd29YZXrifTvH UrPWLjfZ_AXNKFf4UqAFkEFuX2nyrMC_9.LE8STgreWnkkLaCtYq8onIg5EmAi.MwwoSsn5WfQLe hfP_NaLtHscPc7D4zv1wbUFC.9VbgXuGmxG9RkWk043kXmXJRIgvzPpSWyffB5yq71JW4t6eGEcx eEHVOO7r2U5VOffDQ_E_dyi4PRcJlwUrNdzL6_HJdkE9ULPkMhjh03DKPSaeqn0YjeBycDs.RL4S H3fZKGATJbAD79ADQnqqz67b7FslOiAiZddv8EJM12ABbW8Mr2.xVitmKmkqubjLlF8S2zwELrY7 Ml8A2EoRfN0R_9totxENYDReMnS.VO9CEsz4avTi3qtMZa6_xSnMwPdN1UojxunRSsOoRjpiQNyh ruWedr6yAytIN6QDWm65mrwrkcSFIrxRmIUNl0YtkPcwaJX2uxGq4bf8jRRpL5Ty3s46flHjzeCC SOdPhUoFMC3KpwzeMrnte86Kr7OKYBk4_CKblLIVmIw_Rtf2K29LrIuDt_0YOWQH0_LUPcde9Gho OXezO4LfkjlvZzCGi1Ru6jIDezec3DS3vUm02PCplB33TzjTv9GDHvOQ.WgwnYXam2esTtJ0HgbY F3l1w_f0sOM9Yp6_NDkxWlTRbDj9v26Bwvpf77lVc2xbjgPUujvB0o1gdMVtIwIhqA4yC.h37WVT 4BGz26fhKH1XCLulq0kZQdxmkUwx6FWdwRMB5hrIBGtM0vGlcl5aCbUpDlSqAnOPa9SwwrGgubAv QqwyFHCipHkTf7.lmsAR0hqez2fiesQ7VNNjgPynva69ihRc7h8v4PRaGZeqLE3LRGREj.MVT617 RLzuUK0_VV7eK4Op8DSSDb2TyylWEmhNZAIrHEuEOAnpiVIb374Pwy8E1JGwdKPq0LHa9S2wvf1N tJb9wZNaqs2VO2OjCeTIjncXz_PuHAf7MqrBW_OYxQoezNPLfIrxhcSWKqkdzaJfqCKMvyv2QQE2 UMTZhKXaqrXU2C9ZA6fuWtjqr4jDo6YDoNpK5CBDndEbO_YtZ28C0ctzvv0ayNBGutgrULitK68Y 90zgGqW21jSw1.rKETG6ja166Pbm6s2mOVe5yL4Wu8e9UkXXv1rXb6eELidX91ATBqBLcggDApak 7d.PTp_B5f9jn7IUnbxV7vwVESmFyN9sDy5Os5Ztx0IHCISERYscunZHHya68TdI4XqTkqVD0dIh 2cxLtiKIPlHM5cvyw6fv563qiW5Ih90XED1g8V14mveyyQUa8NHCfblHzkLDKbDk8qbWsD1gbSKr OKWbPcZ6FPPWedLaG0yVO_ctIPuAjLIan0FFBoLvG_dHA5lnnM2cdYicJLQeXhhsBjl01Ww_DvPk nNNRzdCn4s7GueEqbIDtyNxV4dZao4mY14xl64jRZFDSt6hhk4C3.rRyzfPCVNxdlG2RJY52HnnK 9yuO4mLbHLGmiQXHp0WwlEW.l1YkMbLZTErUvP.dSFpAut7txj.1rw3NNJtYJM3u0PMxQkSjOiEm lfahl5AMIoi8fczLOCApbqGWvgcTX40hZODchtjQNcDciA1GW5St2YkM4ZH31uvEPc3rpcjlbC_e huJn0KfJNt9_xxBHnsK5.gH_KphCBTVPxUKEGSqR1KVqdbKX6Tkv5SbDSfWVVathxSnhBLO4h.fJ wWYoItHHMYAXzndQy39Nc4Itoy9d5GdKAyA6M2t3IjmAAXQgPW52M6wqbBS7drjRif5r_66OQQn6 ro20kHZ9msH8uWc_eCnbDdGegE0ZLhS1mk8Ocv7ab_nMkmEbWyDq5nGg_jIP1sJNnyg5GhKBIZCv 99nRzHj89lrkFOIrQuQvUcyVuRh1vI9nOp7X5YfVgj_rorVbbv4GhpfHUKJQIMVgV2IeBhDHtfaW O_3TYctM- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Thu, 8 Jul 2021 02:12:06 +0000 Received: by kubenode534.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID a09f798e6d0029076ecb1fe2ce24428d; Thu, 08 Jul 2021 02:12:00 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: Too many pythons in poudriere In-Reply-To: <044A7E63-2734-41F4-A1A2-AE5096C6A62C@yahoo.com> Date: Wed, 7 Jul 2021 19:11:59 -0700 Cc: freebsd-ports@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <044A7E63-2734-41F4-A1A2-AE5096C6A62C@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GL0D810kYz4k84 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Rx07qxyx; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.48 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.98)[-0.980]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.68.146:from]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.68.146:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.146:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.146:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-ports] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-7, at 19:04, Mark Millard wrote: > From: bob prohaska wrote on > Date: Wed, 7 Jul 2021 14:53:20 -0700 : >=20 >> In trying to compile www/chromium under poudriere on a Pi3 there >> comes a point when five python2.7 sessions totaling more than 2 GB >> in size are running at once. I should have orignially noted right here that such is because of the use of: ALLOW_MAKE_JOBS=3Dyes You need to avoid that for the www/chromium build. >> Obviously, swap is swamped and the Pi3 >> is running at a crawl. It hasn't givem up, however. >>=20 >> Poudriere was started with -J 2 and make is limited to 2 jobs.=20 >> Is there an analogous limit to how many pythons are loosed at >> once? It looks like there's only one builder, so it isn't >> obvious that -J 1 would help; I'll try it if this job stops >> prematurely. >=20 > It will not help. There were no competing build jobs. >=20 >> Progress, such as it is, can be seen at >>=20 >> = http://www.zefox.org/~bob/poudriere/data/logs/bulk/main-default/2021-07-05= _14h06m26s/build.html >=20 > By the time I looked it had run out of swap space: >=20 > Swapinfo 100.00% >=20 > and had stopped for build/timeout after 50:33:26 . >=20 > For reference, including "swap_pager: out of swap space" > and "swp_pager_getswapspace(1): failed": >=20 > QUOTE >=20 > Wed Jul 7 15:20:34 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1831744 11456 99% > /dev/mmcsd0s2b 1843200 1832328 10872 99% > Total 3686400 3664072 22328 99% > . . . > Wed Jul 7 15:20:46 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1838356 4844 100% > /dev/mmcsd0s2b 1843200 1838928 4272 100% > Total 3686400 3677284 9116 100% > . . . > Wed Jul 7 15:20:56 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1841260 1940 100% > /dev/mmcsd0s2b 1843200 1841836 1364 100% > Total 3686400 3683096 3304 100% > . . . > Wed Jul 7 15:21:08 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843000 200 100% > /dev/mmcsd0s2b 1843200 1843124 76 100% > Total 3686400 3686124 276 100% > . . . > Jul 7 15:20:58 www kernel: swap_pager: out of swap space > . . . > Wed Jul 7 15:21:20 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843128 72 100% > /dev/mmcsd0s2b 1843200 1843140 60 100% > Total 3686400 3686268 132 100% > . . . > Jul 7 15:20:58 www kernel: swap_pager: out of swap space > . . . > Wed Jul 7 15:21:30 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843160 40 100% > /dev/mmcsd0s2b 1843200 1843116 84 100% > Total 3686400 3686276 124 100% > . . . > Jul 7 15:20:58 www kernel: swap_pager: out of swap space > . . . > Wed Jul 7 15:21:45 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843192 8 100% > /dev/mmcsd0s2b 1843200 1843192 8 100% > Total 3686400 3686384 16 100% > Jul 7 15:20:58 www kernel: swap_pager: out of swap space > Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed > . . . > Wed Jul 7 15:22:05 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843192 8 100% > /dev/mmcsd0s2b 1843200 1843192 8 100% > Total 3686400 3686384 16 100% > Jul 7 15:20:58 www kernel: swap_pager: out of swap space > Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed > . . . > Wed Jul 7 15:48:46 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843192 8 100% > /dev/mmcsd0s2b 1843200 1843192 8 100% > Total 3686400 3686384 16 100% > Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed > Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed > . . . > Wed Jul 7 15:57:01 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843192 8 100% > /dev/mmcsd0s2b 1843200 1843192 8 100% > Total 3686400 3686384 16 100% > Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed > Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed > . . . > Wed Jul 7 15:57:21 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843192 8 100% > /dev/mmcsd0s2b 1843200 1843192 8 100% > Total 3686400 3686384 16 100% > Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed > Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed > . . . > Wed Jul 7 16:31:52 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1843192 8 100% > /dev/mmcsd0s2b 1843200 1843192 8 100% > Total 3686400 3686384 16 100% > Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed > Jul 7 16:13:16 www kernel: swp_pager_getswapspace(3): failed > . . . > Wed Jul 7 17:47:11 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 32696 1810504 2% > /dev/mmcsd0s2b 1843200 33572 1809628 2% > Total 3686400 66268 3620132 2% >=20 > END QUOTE >=20 > It looks like for the configuration as it is, the > bulk build needs to build such that the load > average stays near 1 or less, avoiding near 2 or > more: no use of ALLOW_MAKE_JOBS=3Dyes during the bulk > build is one way to do that. >=20 > In http://www.zefox.org/~bob/poudriere.conf (modified > for illustration): >=20 > # By default MAKE_JOBS is disabled to allow only one process per cpu > # Use the following to allow it anyway > #ALLOW_MAKE_JOBS=3Dyes > # List of packages that will always be allowed to use MAKE_JOBS > # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports > # which holdup the rest of the queue to build more quickly. > #ALLOW_MAKE_JOBS_PACKAGES=3D"pkg ccache py*" >=20 > I'll also note that: >=20 > http://www.zefox.org/~bob/poudriere.d/make.conf >=20 > should not ever have the "ALLOW_MAKE_JOBS=3Dyes" that > is does: it is the wrong symbol for that kind of > context. poudriere converts ALLOW_MAKE_JOBS to something > else used in make. >=20 > QUOTE (not modified for illustration) > ALLOW_MAKE_JOBS=3Dyes > MAKE_JOBS_NUMBER=3D2 > #.if ${.CURDIR:M*www/chromium} > #MAKE_JOBS_NUMBER_LIMIT=3D2 > #.endif > #.if ${.CURDIR:M*databases/sqlite3} > #MAKE_JOBS_NUMBER_LIMIT=3D2 > #.endif > #.if ${.CURDIR:M*www/firefox} > #MAKE_JOBS_NUMBER_LIMIT=3D2 > #.endif > END QUOTE >=20 >=20 >>=20 >> The last time www/chromium was compiled (using make) on this machine=20= >> I don't remember seeing such a python jam. If it happened at all the=20= >> Pi3 worked through it faster than presently.=20 >=20 > Which was the old make: -j1 vs. -j2 vs. -j3 vs. -j4 ? >=20 > The ALLOW_MAKE_JOBS=3Dyes use is like -j4 in your 4 core > context. Lack of ALLOW_MAKE_JOBS=3Dyes is like -j1 . >=20 > The -jN is for the number of make processes allowed to be > active per builder, including when there is only one builder. >=20 > The ALLOW_MAKE_JOBS=3Dyes meant that there was likely a massive > amount of paging activity that was taking up much of the > time. That would still have been true with a much lager > swap space: it is a type of context where Lack of > ALLOW_MAKE_JOBS=3Dyes may well take notably less time to > finish. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)