From nobody Thu Jul 8 02:04:25 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9823711F2856 for ; Thu, 8 Jul 2021 02:04:33 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.gq1.yahoo.com (sonic314-21.consmr.mail.gq1.yahoo.com [98.137.69.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GL03N30Vlz4j8C for ; Thu, 8 Jul 2021 02:04:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625709869; bh=ESxIk9LA3/TkulmtR+k6OhYt3qH9p0mCBx5iAmB7FRE=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=aQ8GbkW2rharZJSJcuCSl9fFZ1l3morJkORzQ/MnjnkIJNg2iKBthPrNy3tWwfIO0rw2qH/jjQ8X0O+iS1czR71sN3ChQPztf5enth8wF/O922t2optVvDpIoReJ4EBrGE+xmLdaVvd6RZWvS9QgvIm2bhfS2snwxdfCffEY9hMthn4QfnQvIexe4uwpuBNun0uEZ2GoV7JO1wwij3A4Zhg9jZz4sgKKX2fvwkzEG0sfdcDEBo3PvoSXk+JGrGf//1ESkr+aO68b0islBLo3jGehw3tfo41socrxxiimxespzHiZNXyb6RKlX/2LI97SguTqgxGd7eMIqqxTVa3zdw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625709869; bh=4i5M887ku+QcKPbm0KyAPQhYHa7iXjP/MvpXtQ/+Vh/=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=j0Qps7eXheGYLUplJEkGsKfQkLma6HqBI9gtf1QEi+juTvuspFYCFIsJSrnprR+/ndppA61uCTo4VEv93imGDPuq0qnjOAPvq7zHhD4sQr44gSMg+sKaFMo6r2ZHrys2k/wxu77XoOwtSdzDlG9PDGDqAm/yv7LthmLtN/jjG7yE6/m0i61SEDBoI2QccQFxLUr8VIqQlhuWWLyOKLjjHlrIbaJjYgP5GG1637MLhQoy5XtR7iM8gJttsvp6qkgrSfwVAKaAvV9dXice9aBLV4t9SuOGPba9RsiDheO2InxK1siv9iXCtv4EOrGu2z4MW5hSaHtx9OTHJv0jzuvbRg== X-YMail-OSG: 5Z3mhuEVM1mUYo14Wc7GGNmdyKMBR.79aXPSWLGT.wZk8_jn56xJn_upjfiCgTz pcooyeotCLMihb22tupHehT8kZQVTu85SecAZmyoH3teD7kTgy2mr1_m67SWOynV.lWoLZ7Ey.c9 YYD7MVw0n1xWPoRivOXMV_D0YXHeXZLvUDQ.fm9E6kLQYeQk3HiEg4XbNrdr8B5wCDB8gos.11.O EaIP8bC.vAVYReElXoDK73HUP56_lb4W4N4na7uT8O8hOlQze_mz4XECTJHN83MtgM8x1Q10nCTk AyCAdzdU.trI_ERL.qTIayS03qkjzGNSi.R_CIyJ0tO8PTnuBmnHC9kLAYCBuf_OLqGPNz7X8WdW Inm3l57S7Xbfeer5D0eEQ50kbVbhvcJ8h34DE69mbYEqeXVJ4GEVYgVJCXp34NEfco9NyG_iKZ5X mZiMMWGptykaN_Al4hR.eRwavNA7GeLja1IS0ngtw4LtZQkZpcn_eifi3MrrZKFuVIzi2y9Un_iF mAWoVPfn1GoHXUSTIGarXZWkKGq16zii_HyF8Tvu.lEqrmoHXC9jbOEdp9rzNCg8AqB52kW0nhs9 zlpKr2d.ctBmgwdMcL8PtXIBXijER4wxGJ0tpkcePlabKCLOyJQQU1iS4ABL4BRIGawYH5VpRmXI qpiFZbdPsZwQKXE83IdtQucpPA3XVadoRhcVDi3BpOzeNxLyVOCRtI2.TNljfnbcSMARL5g_VjHP DcP0s2dlwAYLI5AFYhKAYBJjA5M6U1uekHTArsqovt9jIKlTTacTKpKGqgsu9bM_UeeE.KK837L7 cfDzzW5s9_pJQlEOgPS7Y1Zd8vQZeuI2L3ulSz3yHjNPTPdB6FJ_tYdTUfpmG2e.4xoXuK1GR1mR LDAQ4q9Njc7x00SCnDgukk8DLGhmw.vJUZJhFOpotXiknO3vMGtzG_OqxsoTGRK1My71VUjA8xDL Y_7goGzK.UmGf74I0WvH8OBMpTKII_sX2s.cPcJYCgD9xrfVId1N_v6q6agjfuzXYQDeYL6s_qas 63ociwksLZLbM5SWcqmrQT.WYJv1N4NfGHxu9lApKjVFFIbcuVVfhY9WvRgNt6Z4KwlgVV9g07u0 i0v1q48Usu5nnJDgyeNQsgX361RNvKci5dSSjtUJpUhksappTNVTw_UCqGa9fRX1fji.sZJhZ.OE UcE0xgHB_mb0KDiJ1XVNAi0netsdYG_L.aMbVdvq671CINjs0PX9P2cLK9Ebgip2az0LPOjrn49K mx3LqC0_kiMwlLwxHRHEb9yPFdc7zaPfQVZIA1MvMKyZZjdq4uEuPOzpFZjaVQKcPAa6QmVtSown capnjcHdrlzJT54H7R_2e269asTu3x9j_xG8RJXZyCTC4iUez8HCPk_BpR5Wny96r6wYJ7ANQSXx FZ_8NYSWrxWXlAvyBlIu2E.cZVH5HCgcTzuBLd0dP4R0KeFxW76wGHtrgRO7rm.NR2QWsjTTOBlj Q7AWtIPYe92Oz6vW9uvPzCmz8k8AzTrPYWcHH3eOpG99oeYxGmn0myvoy4MnVr0jqJ62UPxf6uOO F8mEGisDfK4GxO8Pi7i.aVLnSTY6B2b5lVwEb1q9prEWs.15Prz7z65nedcL833s2wKZxLeIf1bV 0U8xqi12mvtGK4176CfoZbS3wIbiGOvmJUFvEIm_8zVuQ4L4tHOloEW9CdO2ThgVJZNZy3.PoSpw WkYTBgoMbr32i8F02EW6y31EPLURLofuRCYD9AFXImQJe5YBentOjNmQld6ddb0e5YZjttMSOjze Wq5QAuKu0d0Nhq2i3aNA7F8k1yRsl7aW8VecIqu8fBdVKiF5MNNFY2t9mhwyiU6fZC6ufOUq6ubr 5d6CIL9FwC3iHTJhay8E2ZAZ1R4Jm2nwQnjtqxSfmjW8Y9WCmrFMJZzI8r8eKe1wdIAUGtt0.N45 dPNmtP_lKXZzrzQqxZAUqmj4ojy2JcPz8uXAARz2SJgOJbokTxY7ENDuHf0OrkCcruXGPdO8qPtw 6hX5K94wYfjW2UzzYgEyGQDOAWqQHKqCZdNU1cgNnGzluhdQpjuwS28Fbu_eXdjgW4iBlEBKnI7a YA.Gm2OSbED167n12WAwTp0RnMbBdLFrEMjl3duFVCilQKqqWjfYRHtjtb1obr7cH6cs88dL9Kgr GR729yvQWuCTarFJUxnQsI.ok.01EYvMW6OFhfG7kvOUrxz2Df81ZH6dlctXuqnwyvcEwVduXC59 gk1oEm6ANUVRmhIU3ieMs_lEIRhC.BmlLjaPVl2nBnUCtX_foUpEvykU83dDvcxZ8R0L9nYXyaTa gmNBy43jhP7hHHbr0cDyQoXOK9zSwat8OlTLpnhR60fdBzh_lazpSMGUGfAqcZ0jiwQuxbHEG0Kz aZma5ToVM2e9KY_vgzMU.v0pkKhfEGmypj4aTVcw015pzZnBw47_278NhDbooEjr6FKnE3Zm8gSG TzZuaem8SM7.FssO3zgIDMVYfglvij9w- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Thu, 8 Jul 2021 02:04:29 +0000 Received: by kubenode542.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 63ffb5a6afe926f50819420cfab9f40c; Thu, 08 Jul 2021 02:04:27 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: Too many pythons in poudriere Message-Id: <044A7E63-2734-41F4-A1A2-AE5096C6A62C@yahoo.com> Date: Wed, 7 Jul 2021 19:04:25 -0700 Cc: freebsd-ports@freebsd.org To: fbsd@www.zefox.net X-Mailer: Apple Mail (2.3654.100.0.2.22) References: <044A7E63-2734-41F4-A1A2-AE5096C6A62C.ref@yahoo.com> X-Rspamd-Queue-Id: 4GL03N30Vlz4j8C X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=aQ8GbkW2; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.84 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.48 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; TO_DN_NONE(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.98)[-0.982]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.69.84:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.69.84:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.84:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.84:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-ports] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N From: bob prohaska wrote on Date: Wed, 7 Jul 2021 14:53:20 -0700 : > In trying to compile www/chromium under poudriere on a Pi3 there > comes a point when five python2.7 sessions totaling more than 2 GB > in size are running at once. Obviously, swap is swamped and the Pi3 > is running at a crawl. It hasn't givem up, however. >=20 > Poudriere was started with -J 2 and make is limited to 2 jobs.=20 > Is there an analogous limit to how many pythons are loosed at > once? It looks like there's only one builder, so it isn't > obvious that -J 1 would help; I'll try it if this job stops > prematurely. It will not help. There were no competing build jobs. > Progress, such as it is, can be seen at >=20 > = http://www.zefox.org/~bob/poudriere/data/logs/bulk/main-default/2021-07-05= _14h06m26s/build.html By the time I looked it had run out of swap space: Swapinfo 100.00% and had stopped for build/timeout after 50:33:26 . For reference, including "swap_pager: out of swap space" and "swp_pager_getswapspace(1): failed": QUOTE Wed Jul 7 15:20:34 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1831744 11456 99% /dev/mmcsd0s2b 1843200 1832328 10872 99% Total 3686400 3664072 22328 99% . . . Wed Jul 7 15:20:46 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1838356 4844 100% /dev/mmcsd0s2b 1843200 1838928 4272 100% Total 3686400 3677284 9116 100% . . . Wed Jul 7 15:20:56 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1841260 1940 100% /dev/mmcsd0s2b 1843200 1841836 1364 100% Total 3686400 3683096 3304 100% . . . Wed Jul 7 15:21:08 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843000 200 100% /dev/mmcsd0s2b 1843200 1843124 76 100% Total 3686400 3686124 276 100% . . . Jul 7 15:20:58 www kernel: swap_pager: out of swap space . . . Wed Jul 7 15:21:20 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843128 72 100% /dev/mmcsd0s2b 1843200 1843140 60 100% Total 3686400 3686268 132 100% . . . Jul 7 15:20:58 www kernel: swap_pager: out of swap space . . . Wed Jul 7 15:21:30 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843160 40 100% /dev/mmcsd0s2b 1843200 1843116 84 100% Total 3686400 3686276 124 100% . . . Jul 7 15:20:58 www kernel: swap_pager: out of swap space . . . Wed Jul 7 15:21:45 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843192 8 100% /dev/mmcsd0s2b 1843200 1843192 8 100% Total 3686400 3686384 16 100% Jul 7 15:20:58 www kernel: swap_pager: out of swap space Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed . . . Wed Jul 7 15:22:05 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843192 8 100% /dev/mmcsd0s2b 1843200 1843192 8 100% Total 3686400 3686384 16 100% Jul 7 15:20:58 www kernel: swap_pager: out of swap space Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed . . . Wed Jul 7 15:48:46 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843192 8 100% /dev/mmcsd0s2b 1843200 1843192 8 100% Total 3686400 3686384 16 100% Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed . . . Wed Jul 7 15:57:01 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843192 8 100% /dev/mmcsd0s2b 1843200 1843192 8 100% Total 3686400 3686384 16 100% Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed . . . Wed Jul 7 15:57:21 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843192 8 100% /dev/mmcsd0s2b 1843200 1843192 8 100% Total 3686400 3686384 16 100% Jul 7 15:21:33 www kernel: swp_pager_getswapspace(3): failed Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed . . . Wed Jul 7 16:31:52 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1843192 8 100% /dev/mmcsd0s2b 1843200 1843192 8 100% Total 3686400 3686384 16 100% Jul 7 15:48:44 www kernel: swp_pager_getswapspace(1): failed Jul 7 16:13:16 www kernel: swp_pager_getswapspace(3): failed . . . Wed Jul 7 17:47:11 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 32696 1810504 2% /dev/mmcsd0s2b 1843200 33572 1809628 2% Total 3686400 66268 3620132 2% END QUOTE It looks like for the configuration as it is, the bulk build needs to build such that the load average stays near 1 or less, avoiding near 2 or more: no use of ALLOW_MAKE_JOBS=3Dyes during the bulk build is one way to do that. In http://www.zefox.org/~bob/poudriere.conf (modified for illustration): # By default MAKE_JOBS is disabled to allow only one process per cpu # Use the following to allow it anyway #ALLOW_MAKE_JOBS=3Dyes # List of packages that will always be allowed to use MAKE_JOBS # regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports # which holdup the rest of the queue to build more quickly. #ALLOW_MAKE_JOBS_PACKAGES=3D"pkg ccache py*" I'll also note that: http://www.zefox.org/~bob/poudriere.d/make.conf should not ever have the "ALLOW_MAKE_JOBS=3Dyes" that is does: it is the wrong symbol for that kind of context. poudriere converts ALLOW_MAKE_JOBS to something else used in make. QUOTE (not modified for illustration) ALLOW_MAKE_JOBS=3Dyes MAKE_JOBS_NUMBER=3D2 #.if ${.CURDIR:M*www/chromium} #MAKE_JOBS_NUMBER_LIMIT=3D2 #.endif #.if ${.CURDIR:M*databases/sqlite3} #MAKE_JOBS_NUMBER_LIMIT=3D2 #.endif #.if ${.CURDIR:M*www/firefox} #MAKE_JOBS_NUMBER_LIMIT=3D2 #.endif END QUOTE >=20 > The last time www/chromium was compiled (using make) on this machine=20= > I don't remember seeing such a python jam. If it happened at all the=20= > Pi3 worked through it faster than presently.=20 Which was the old make: -j1 vs. -j2 vs. -j3 vs. -j4 ? The ALLOW_MAKE_JOBS=3Dyes use is like -j4 in your 4 core context. Lack of ALLOW_MAKE_JOBS=3Dyes is like -j1 . The -jN is for the number of make processes allowed to be active per builder, including when there is only one builder. The ALLOW_MAKE_JOBS=3Dyes meant that there was likely a massive amount of paging activity that was taking up much of the time. That would still have been true with a much lager swap space: it is a type of context where Lack of ALLOW_MAKE_JOBS=3Dyes may well take notably less time to finish. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)