From nobody Thu Jul 8 02:27:54 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2EF3511F6760 for ; Thu, 8 Jul 2021 02:28:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-21.consmr.mail.gq1.yahoo.com (sonic313-21.consmr.mail.gq1.yahoo.com [98.137.65.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GL0ZX0Tk9z4n6l for ; Thu, 8 Jul 2021 02:28:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625711282; bh=ITeRNE/Xp19hxwo5zMztE8XUnSh/YFMRrLPeQtdDHdo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=PyaDnmVB2++EBTp8tUI8aOG13URPso1+dBQq+1VDv6trwN0m2ciHO8vEjVj/bGRVAdPkHWooYu/U/V67Y+3bQH84sPTTD5yDXoHyuDM2gUxpTClqYV1ZenC7NUXTlEuWJQvcptuTOytm8vUDW34CHhcLSy0XIuJOly1GoU8TGO5dl9wxcUx+ByZHuLWog2jteII5RAKG/I17rTfejkT6V/zUK8XnYvyN2xOsDK3vFOSdqUTei/yQncmYBB3hcagQisV//dJkuHv3Lz9qZgzHcXTWAY3pMTNUIbGJa3zz/nx6MZrCyo7YxUrOOfMTwRFIxbAIhSohSmI+HELpD3gBQQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625711282; bh=niV8lUpndxPSB0u/HsI5D0Cj6TAQLONRxUyO7jch9pZ=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=OhtGAkusDUBk99ZqC+3hN7mqkmYTFps7PMqt4JG3O0zPtWNHMVG6DYOUAd2HLM0gnQqx6+GhSMHbGqjRH2X58d4aOefVwJzWox3d+aG24ckTthoGzjoxvJNRo+ZUGV2o71zVSiB+4RGap67uTca1WZsadc7Zq1uRDrpXoF8fGD8lSqZg5KYBTJ2Y/Fsj9vJ7li8xb8YRmneELSGir0hS0ASK6ofwnevRLHe+OzLgdzz/0dZIS68Nc8DdjZREXz3E54T0+JRntluPYoyCpCrva7kNGMsQRpUHEpmIwCwE1cm/Pb67aCF4x8oxxqbzhR4itxprSBQ9PGnCPLQuZNMUeg== X-YMail-OSG: 7UIGmwwVM1mZl8S3kKkRXjSNT1Yz0j9khMUarGKvSpYTJA1M_QiARjsTh7pk6o_ HkP1uEvvCDjjvkHpWhcVN2hn8fHSADzM7xQrECNisiRKKHjoiotF55W49PLp2nKnOvuIOY84nSoW D1Efb7PjOeyVgei_pYdE3l28Wh5wZtMsQbtV6zqIJhfstS_wTaRpkyjk9ijkFVqJhSOdIKfK.WRh YI_F4S6byJZGClpDYTtNf.bAVlZwwsHCaHqSWBD27qkTD04NYGE81sK_XzLv__zo5K5C3BrVeW6e jPiOdeTFjjd.o4vribEteRnABW7TMYpmQhl6j0c2cQYjZXUkSAzNp3BapqmTMWEx.aNBqjfvLzgB KCXbW4OlJbIFxEw06VeyV_wHfUzYQEESNdI58b4t3_eFI4i1q5Xjud1KVQeoqmLkHKYX6LWpI6ca py4jhg3EawR68bZx4LL1cqFMGn2.aKgRASU6z48GfAkz0vLIXxfHRYL15_iXLhmrgf.dffZ6qMPp 02A5dfYQuq38LwFXdNicn0__2DJ5Ir13_ciIBYwDWrTcrR1JFxh0pgH7XMJ_n_7UIVHF7ACxZFqD XcbDmiYb9aVmhTQUEwXg057IGq6BCk0HuVjNyKvERWbye5a2mWRIsbMmjgKMyCq0w2OMtP8Qkc_. RlOQTBpCKMeTPQRoKeNTCNaCFEzGwx0ziA7Ikc2WWkH7SIfKh2NP_inDi13gylK_grd77Qmzmsd4 yQ6AiHEknM3tkxURIYfNbT9F.9PDonUXpkeG0.TUSww1lA6wW1Pi27rh1Pd5MB_Th7sJF8YaFRRH .wl6NukxVhFgh6S2SbGDkQt..ZJEicmXaaBh08MBRW4QbWdUJ1TsD17awk9HBYkgTbn1kLcDuZ2u r7rP.GoWFuwpK_Yvnmp3Vg6cq5M0xLx1Ezh6urxGjjficGuUKkkDVV_p0X2unlX3C8RJYURbFxEY Jjp8gOHO7irgsxI7ayiTcmj2oVhXQNL8rZpzzOXIVjlxCZcu1gSfuPTg8NRtirEHYMjoBWT4327G 7P.DMVb7fLUa0BCi86nw5Lg.mtP90VtrHPM.ukQ.LSh7tSPq9ZbnDTacV3suhP2qhs8IjVDBgXmG GH1oi9tS7WqwHAPBJzA6nyrAMOgFKHW0P3GJSd6k_IJKBNxgADFx86lqV0SluQEmla3aeiUC1Zjj _R1CI_fTV6IzBqsOvA3NvE2ANzeFgdGXboTraKWqxMCPmcwamkg9_b9SHOatxj9vnJVjwkJ7xcm2 JJLwzbKiAXtmhidaBAZNapqXe9ZFrcFEfGnHjaDO5HCMQYae4EwOMR4tdRh2gkPtvHPZhwqfkCV3 sdYWE3UaLfyiwV3_cNbhG.5SUMAgwkOUXHlMyXzUeeujCN1A117Oj9QtEybnrvcedu8enYJXEBYv xW8jt7meeQHtjDJjElFa250sGsc3wom4h3lxbYIZFBXYEaEiKQn0rg.bdi_tJN9GUrHq45ZmlLGi GNu3SZgRfNSoz6Jyr4XXoaPOnJMrHVfVrOXVQUpKe1Zg76UMZT2JQNa.TOMtMINvkYNzOvqHT7o2 Jc4kl9l7WWIuJuC4XKTKuV1RVo_qj90gktMqzomHKkghyV7K7GAvHN9tjIJGvHfoSXUTi0qBGkGK 9tizY6vlsR3MSGhHc6GQ73tcQtbor.u4BdvUArbnK2EMu.W3pNW4l6seY4J7AQwJMYVWKlpzOnQp ZiX4bx5wMteiXkyF9_7omFs5guEGXFl43DvYMdkw5BlqDjE5BNjfCVHfwblvmzOU7NiE.91qkpgn 4rtmNxhymXyYItTxOoNN3znWO7N308Dl7ZqhryaF3C.jiWa1boqG5YsrHGixK86Lqekov_2R7FLB 6w5p3eNgKcgBJxuT_9PfgTZLIL40o2LrXPtZoqxQk5v9gvdJsgxkk3eO7jsLrutZkT1ZsJiF4Y7L 76hq6u8V4XL2ak.UWlMY2GMp6bPXBcrwCPg5PLqq.94quBnxnI7eg7TcEPL4yCdWQltpdyRiMbDe Ya_DE8AR8N7n61NrbCsMf1WmtCW.tPaWrF_oglkJk3N5Jd9V3ypExUgqEND0GJdO7OUizQkxEaP9 YeuoxjGKDuVRMfZOjegXOokPjIcqI.RSTRsqO24VxEOtPztN0h6.YYjzZnWJO5SmripCXzfmlHgl NqoRfm7Voi3Ee6lZIbJbXKP0FMZ8PM9k0tuPZSRDUF9ap0MfsEZC5rLjkk5rqsFwcZsKdx5jvPA1 ZeqRfIcml3VuZcEPz8DyxY1mouT2Wv84OIm4o7zr4NmM8tyJhv_Bau_6cuj55Jv53sSRAAUiGy9c 6Y4rIOo8EeWT2vh1jreHN3VfheFiXXPBre3j85igqG.EW_dDAsKfaZSebRtiInRia6Q4mSG_Chln Bzfdg0cYZu9mjrRHwKFiak_YKLU8g_FjauC6ePpzgM3TDI9Bxg6CFR6IKL2TSZ8HJrpdiQC3S8D0 jtJFJm7t0bAmvAGbYSwd9I1MqaHA0s9_1TviXMpVBMHW2xDns2Vw2OIMuemRjsJHkKlKEP07_tkq tnCyWegBSo68- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Thu, 8 Jul 2021 02:28:02 +0000 Received: by kubenode523.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 6b36816130b852653b4c65610c84d375; Thu, 08 Jul 2021 02:27:56 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: Too many pythons in poudriere In-Reply-To: Date: Wed, 7 Jul 2021 19:27:54 -0700 Cc: freebsd-ports@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <044A7E63-2734-41F4-A1A2-AE5096C6A62C@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GL0ZX0Tk9z4n6l X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=PyaDnmVB; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.65.84:from]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.65.84:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.84:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.84:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-ports] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-7, at 19:11, Mark Millard wrote: > On 2021-Jul-7, at 19:04, Mark Millard wrote: >=20 >> From: bob prohaska wrote on >> Date: Wed, 7 Jul 2021 14:53:20 -0700 : >>=20 >>> In trying to compile www/chromium under poudriere on a Pi3 there >>> comes a point when five python2.7 sessions totaling more than 2 GB >>> in size are running at once. >=20 > I should have orignially noted right here that such is > because of the use of: ALLOW_MAKE_JOBS=3Dyes >=20 > You need to avoid that for the www/chromium build. >=20 >>> Obviously, swap is swamped and the Pi3 >>> is running at a crawl. It hasn't givem up, however. >>>=20 >>> Poudriere was started with -J 2 and make is limited to 2 jobs.=20 >>> Is there an analogous limit to how many pythons are loosed at >>> once? It looks like there's only one builder, so it isn't >>> obvious that -J 1 would help; I'll try it if this job stops >>> prematurely. >>=20 >> It will not help. There were no competing build jobs. >>=20 >>> Progress, such as it is, can be seen at >>>=20 >>> = http://www.zefox.org/~bob/poudriere/data/logs/bulk/main-default/2021-07-05= _14h06m26s/build.html >>=20 >> By the time I looked it had run out of swap space: >>=20 >> Swapinfo 100.00% >>=20 >> and had stopped for build/timeout after 50:33:26 . >>=20 >> For reference, including "swap_pager: out of swap space" >> and "swp_pager_getswapspace(1): failed": >>=20 >> QUOTE >>=20 >> Wed Jul 7 15:20:34 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1831744 11456 99% >> /dev/mmcsd0s2b 1843200 1832328 10872 99% >> Total 3686400 3664072 22328 99% >> . . . >> Wed Jul 7 15:20:46 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1838356 4844 100% >> /dev/mmcsd0s2b 1843200 1838928 4272 100% >> Total 3686400 3677284 9116 100% >> . . . >> Wed Jul 7 15:20:56 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1841260 1940 100% >> /dev/mmcsd0s2b 1843200 1841836 1364 100% >> Total 3686400 3683096 3304 100% >> . . . >> Wed Jul 7 15:21:08 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843000 200 100% >> /dev/mmcsd0s2b 1843200 1843124 76 100% >> Total 3686400 3686124 276 100% >> . . . >> Jul 7 15:20:58 www kernel: swap_pager: out of swap space >> . . . >> Wed Jul 7 15:21:20 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843128 72 100% >> /dev/mmcsd0s2b 1843200 1843140 60 100% >> Total 3686400 3686268 132 100% >> . . . >> Jul 7 15:20:58 www kernel: swap_pager: out of swap space >> . . . >> Wed Jul 7 15:21:30 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843160 40 100% >> /dev/mmcsd0s2b 1843200 1843116 84 100% >> Total 3686400 3686276 124 100% >> . . . >> Jul 7 15:20:58 www kernel: swap_pager: out of swap space >> . . . >> Wed Jul 7 15:21:45 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843192 8 100% >> /dev/mmcsd0s2b 1843200 1843192 8 100% >> Total 3686400 3686384 16 100% >> Jul 7 15:20:58 www kernel: swap_pager: out of swap space >> Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed >> . . . >> Wed Jul 7 15:22:05 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843192 8 100% >> /dev/mmcsd0s2b 1843200 1843192 8 100% >> Total 3686400 3686384 16 100% >> Jul 7 15:20:58 www kernel: swap_pager: out of swap space >> Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed >> . . . >> Wed Jul 7 15:48:46 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843192 8 100% >> /dev/mmcsd0s2b 1843200 1843192 8 100% >> Total 3686400 3686384 16 100% >> Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed >> Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed >> . . . >> Wed Jul 7 15:57:01 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843192 8 100% >> /dev/mmcsd0s2b 1843200 1843192 8 100% >> Total 3686400 3686384 16 100% >> Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed >> Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed >> . . . >> Wed Jul 7 15:57:21 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843192 8 100% >> /dev/mmcsd0s2b 1843200 1843192 8 100% >> Total 3686400 3686384 16 100% >> Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed >> Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed >> . . . >> Wed Jul 7 16:31:52 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 1843192 8 100% >> /dev/mmcsd0s2b 1843200 1843192 8 100% >> Total 3686400 3686384 16 100% >> Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed >> Jul 7 16:13:16 www kernel: swp_pager_getswapspace(3): failed >> . . . >> Wed Jul 7 17:47:11 PDT 2021 >> Device 1K-blocks Used Avail Capacity >> /dev/da0s2b 1843200 32696 1810504 2% >> /dev/mmcsd0s2b 1843200 33572 1809628 2% >> Total 3686400 66268 3620132 2% >>=20 >> END QUOTE >>=20 >> It looks like for the configuration as it is, the >> bulk build needs to build such that the load >> average stays near 1 or less, avoiding near 2 or >> more: no use of ALLOW_MAKE_JOBS=3Dyes during the bulk >> build is one way to do that. >>=20 >> In http://www.zefox.org/~bob/poudriere.conf (modified >> for illustration): >>=20 >> # By default MAKE_JOBS is disabled to allow only one process per cpu >> # Use the following to allow it anyway >> #ALLOW_MAKE_JOBS=3Dyes >> # List of packages that will always be allowed to use MAKE_JOBS >> # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports >> # which holdup the rest of the queue to build more quickly. >> #ALLOW_MAKE_JOBS_PACKAGES=3D"pkg ccache py*" >>=20 >> I'll also note that: >>=20 >> http://www.zefox.org/~bob/poudriere.d/make.conf >>=20 >> should not ever have the "ALLOW_MAKE_JOBS=3Dyes" that >> is does: it is the wrong symbol for that kind of >> context. poudriere converts ALLOW_MAKE_JOBS to something >> else used in make. >>=20 >> QUOTE (not modified for illustration) >> ALLOW_MAKE_JOBS=3Dyes >> MAKE_JOBS_NUMBER=3D2 >> #.if ${.CURDIR:M*www/chromium} >> #MAKE_JOBS_NUMBER_LIMIT=3D2 >> #.endif >> #.if ${.CURDIR:M*databases/sqlite3} >> #MAKE_JOBS_NUMBER_LIMIT=3D2 >> #.endif >> #.if ${.CURDIR:M*www/firefox} >> #MAKE_JOBS_NUMBER_LIMIT=3D2 >> #.endif >> END QUOTE >=20 Looks like you could use: MAKE_JOBS_NUMBER=3D2 .if ${.CURDIR:M*www/chromium} MAKE_JOBS_UNSAFE=3D .endif to have www/chromium build as, essentially, -j1 . One thing that MAKE_JOBS_UNSAFE=3D leads to is: MAKE_JOBS_NUMBER=3D1 (Or you could put the MAKE_JOBS_UNSAFE=3D in the www/chromium makefile .) >>=20 >>>=20 >>> The last time www/chromium was compiled (using make) on this machine=20= >>> I don't remember seeing such a python jam. If it happened at all the=20= >>> Pi3 worked through it faster than presently.=20 >>=20 >> Which was the old make: -j1 vs. -j2 vs. -j3 vs. -j4 ? >>=20 >> The ALLOW_MAKE_JOBS=3Dyes use is like -j4 in your 4 core >> context. Lack of ALLOW_MAKE_JOBS=3Dyes is like -j1 . >>=20 >> The -jN is for the number of make processes allowed to be >> active per builder, including when there is only one builder. >>=20 >> The ALLOW_MAKE_JOBS=3Dyes meant that there was likely a massive >> amount of paging activity that was taking up much of the >> time. That would still have been true with a much lager >> swap space: it is a type of context where Lack of >> ALLOW_MAKE_JOBS=3Dyes may well take notably less time to >> finish. >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)