From nobody Tue Aug 24 08:58:13 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7DA79178E566 for ; Tue, 24 Aug 2021 08:58:26 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-8.consmr.mail.gq1.yahoo.com (sonic315-8.consmr.mail.gq1.yahoo.com [98.137.65.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gv31F1nPGz4VYf for ; Tue, 24 Aug 2021 08:58:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1629795497; bh=wjUKVpP/UulxRMvm8VidgKjx4wa5hWKsZ0GGyhGcTos=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=P4WutpNGSFVQRi288I3sUBEdChbIIYIc8s7wxjU6GpBC7AHHGu0uFrmUi/jZ5L5rg5BmLC5BnP/Bh8cm1MnUM65CPVVCBE/oZfbrKaewUouBQOCNwsRl2mXbY4Thl2TLvWtANtbExxv/R+D3Ro7joqUQ3IqOazLE75vi6fXTLjT2Lw+Ckme9FSpirQW9OWzaLMvSKjVY2ZTN6OvbZylf18xw4tWedHY2Dxx0rKLAPhCwC/CxDN1ZXAs0kMAKTyIC1L1uD7Qj0ZBYGtROOEFecKSy/QHbLsKdPrlDGlSpsLNIAg5ryppX3wiOcfU/bGuWQOBawn3ZMCpZ0PJTCa4T0w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1629795497; bh=5CBC1OV1ownmXEAlOxJW3GVvwpdidAHHJrikTjwoaUL=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Hqf4F0mDUjIANXraXnXFdRv2Rg2XcA4ZDkQGxinm8JWkCwXu7VadoOifudYAKoPYvtswhKvGX9DwP66nnLuxD+2pQtDFKjtxnsHbg7Bq0M55Pkz8i4nHzwldeWkTJortsE3CrQ/kJ40eE+jP2f6eBxiknPQUQ6KKgeA/yqJjY0xeDb7yfyCYSCWGZg1aCGiOk12LoQPiIRxl196Iaw9BWl6QQClVg3Ej4HO4M8J3qmZ+gogol+1HJmLB0HyR3RjGU95sptHImGAihyeDDNsN+MdFGO57CwaI8Mp1iPdTQaIzIyxnMhI9amHQX2qpjnNbpPZ1w1dDSABE3KK0O8NSXQ== X-YMail-OSG: 9AXOGAAVM1k1RyeYOZ5zWM3fmJWux9gUXGgqmY_OYVD7PpCcfsGJ5RlUC083TWG rvjC7EXEcIzCp2sKAW93l6wg0hY94mfqjf9HG1nRT7IjrO1GE77Opm2DlD4ZiP62Y1o41eHoq2md KSNv0yaA7z7cX9Riv4Llf5JE08dz5jodauqmoLyeyOMkQGlzyt_oE7DJ0vTdxrXFBeR0Mwx7BvdD DwQXT.s3paEl46k5VupwPzMCLHVLglCgOjLvRaqgTXKYR8U6Hgrc.5NtnI4lUbguSv5GNV7IZy.V 8N5ied1yei0VaprrObcp8MtYPbwPDiWGHMHDpQUI2L3FJcrZCjnX0RKK3rTZJfQTwlz1ajFK9ZqZ t5SJrTaes4VyQUcpM5sAHJp5bEzuFgNaH9vgQ5rRGDgXik0OGe0q2uBdTJBgv7dCvz6_3ea9Fmo2 l_wVAjxJ9vvJOQyjFq1vVMZmBitHPkNiGxUNcmoO.RHBubxAU0VZXmt6oxiTA.bOooEyNCuUxKxW A442sgPwcWI3cGD_AEDX4hoPbZSj6Ako37EqlngxvckoXsDbAhD02b8siXj1P9vBVblds4npjcS5 M77NisQMV7_X3Xfv4NTKtXn0kYZLZtDZns0cAdDXgqGQRUAPo774Fr4ac7a.BDFquRhdIHE.8yP6 TLCvwv2dnSO1ouwWDNstB5..5uEeU4qoYBNmxDoa..ge4fKICLOk.Y5IYN8uk3UtTiRlnO2tvk_5 8LWyBjOaQONNCoLweafALJpTCHaghVRESJMtUDIppdZNATUsB16FHjZ0UkzidHR_oAwfAIITWcLU Kd_AxuKYhptwdvgh5JBf6JIEmd7Njie3yWT4kzI3utjuKylPay45jHjGo244lD.gZi4Pz7aZBw6c Kai11pSAOX7qKzlsiS2c9n4xE9LmZZm4i8kh_5ESSMzS9WwkYdJm1cZOI7O0FjCU.nZXIti2HNRb BxTh3b2aqbfvuQsm7jScffOdfp3KxSTEy7CrcacDVJ0rnfba4D4HMjW9rzw1rjqYWXW71IHqddkF qnveUkx3Mbv17raf8PtvYUseBhqGrevLtK4sk6G.3W.OPaMn2OMXkD23WtvJwBY5.wjfidpERfA4 KS0BE0GSnL3.GKWMmAUGRcoDjy_NtrpynkfslrC_xqQYlQ9viTJ47UnzQsHG93.CZI6JLd_GotSU jltjkbiZTMk4.89vOBk_c3S0DBLLK_OVQlYfG.b.CmFiYLm9GtesF.NNomxeKU_U56F1XXkPe_D4 KSVGr3BV5IWaIK4mNtCbTOTxACDXp9AFApFZM4wUfVGABcTq_c.0JKbh0xXlc_dfn_5P2hL_7e7p C9A31Jdg_ZKyp3deTc19MsDkTF4z_WWyjvhzl8..DtFDY7BhqL6KXPHyHmRcIznftDsmxBkAVTBr iht_CkCa.mwnrj2n8is8xHtQ7Rz8o0Sr4Fz.a2.uFgNiJyXxKSFCCtgamGcOQZKxQ7o9TtNTwB9Y tylbUaOiFNWLspJ5D0U9y1vS416o9A94oImRu2xG3wSoxRP9blCQjInAJvJUW_G.kjf6_2V55JTh 5V_FwSNTIO7fkOL9j2dHZjPhFrZY.pm83PceJTHWDxrARG018Eo7jkoYAdyKqnBc0adatAe7ESbn 9TAulrR9M3nqMva72pNbsPKCPBhBVpas64MwFSmuJSlxcFjTB7edoaj6JSKF4Ey1K0sSxYXeOHJ4 iElXcd3PADZrRlnPxcV1.RAAp1bULocCss8jKfNsrPO6FgMkfSYdCYj24J.H43z5oaov4KLbUOJu O8.NYU8PLxw_SFfiK6xCdrUFGQwiVyk6lj3WC3CZPW_Cra6HpLFwoDlYo.Fx9GZlGURHrOLA17Lk _WSj.H2vOnGAZRhxMZF9h2i0EoY5cG0VX1kH6dU5EjdsDFb975FfQB8vgUHqjlvjdysrtoOSgp.r k3jiKeCwsP11XpHVOeO26eHlHqMDLpx8QhdQqNBxRE20P0PMb0X86Gy08Xvj_xBKimFEssdToaAG uaKkH_cBfp2jed_euBz4yNVVpnPf59VAWa5HACX4ngeeq2Cx0e5AB1fdw9EBKco2yqtdIWdKAUZ. aY8a_NpDOmnrxMPF6JiUrXFUX2Dsk6k3F1i9gD9HSz5BAZr7yhB.XAl2a4v9dJMCGxZJSpEq_ial H7IhiMgTTmeZOhDLLRAbOQ4gz.9ggz3687zvKKBWMa9GkMqNSGIIDCNo9qO4AlUc4IjYFGrvbmUw jdKuuFtlplCHesu6gDYSAGeSztzbMtnw1QUBlmvMK825m7URjbIlb6msAl_bIpvk6VCE9xYDO8z2 IC.14o6kiTQm6JOZC9XICHiaAGZ9_d3hNmL0aKcga6rCuSi6K03wv.p8z81rPMQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Tue, 24 Aug 2021 08:58:17 +0000 Received: by kubenode510.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 3229aba3aec797e8fcbb95baa1299f52; Tue, 24 Aug 2021 08:58:14 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Controlling python when building www/chromium Message-Id: <32251809-3419-4DBA-A623-B8B4B2294440@yahoo.com> Date: Tue, 24 Aug 2021 01:58:13 -0700 To: freebsd-ports@freebsd.org, bob prohaska X-Mailer: Apple Mail (2.3654.120.0.1.13) References: <32251809-3419-4DBA-A623-B8B4B2294440.ref@yahoo.com> X-Rspamd-Queue-Id: 4Gv31F1nPGz4VYf X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=P4WutpNG; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.32:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.32:from] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N bob prohaska wrote on Date: Mon, 23 Aug 2021 08:27:33 -0700 : > When trying to build www/chromium on a 1 GB Pi3 the system > gets bogged down by five instances of python2.7 running > simultaneously. This happens using both poudriere and make. >=20 > It wasn't a problem a year ago, so presumably something has > changed in chromium's internal build machinery. I've searched >=20 > https://www.chromium.org/developers/how-tos/get-the-code > and >=20 > https://groups.google.com/a/chromium.org/g/chromium-dev >=20 > but couldn't find any references to python when compiling, only > when running the browser. =20 >=20 > Is there some way to control how many pythons are loosed at one time? > Most likely two could be accomodated, possibly three. On an 8 GB > Pi4 the five pythons coexist happily, so the behavior is probably > not considered a bug. =20 Bob did not show the context. Below I show an example from his public poudriere logs, a copy from an off-list mail, for reference: QUOTE When I looked recently, the peak swap usage reported was: Fri Aug 6 00:39:58 PDT 2021 Device 1K-blocks Used Avail Capacity /dev/da0s2b 1843200 1617020 226180 88% /dev/mmcsd0s2b 1843200 1615244 227956 88% Total 3686400 3232264 454136 88% and was for (showing the one after that total): `-- /bin/sh ./buildscript.chromium `-- /usr/local/libexec/poudriere/sh -e = /usr/local/share/poudriere/bulk.sh -j main www/chromium |-- /usr/local/libexec/poudriere/sh -e = /usr/local/share/poudriere/bulk.sh -j main www/chromium |-- /usr/local/libexec/poudriere/sh -e = /usr/local/share/poudriere/bulk.sh -j main www/chromium `-- sh: poudriere[main-default][01]: build_pkg = (chromium-91.0.4472.114_1) (sh) |-- sh: poudriere[main-default][01]: build_pkg = (chromium-91.0.4472.114_1) (sh) | `-- /usr/bin/make -C /usr/ports/www/chromium build | `-- (sh) | `-- ninja -j1 -C out/Release chromedriver -v chrome | `-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . | |-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . | |-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . | |-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . | `-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . `-- timestamp END QUOTE This was the most extreme swap/paging space usage from somewhat analogous use of a generate_bindings. The swap/paging space usage makes trying multiple builders impractical: it actually does run out of swap/paging space. (There are limits to how big of a swap avoids potential mistuning for a given size RAM. swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much larger than on a RPi3B, without getting notices suggesting a mistuned environment.) (It is not necessarily Python 2.7. The build, overall, only uses 2.7 sometimes in some places.) = https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_par= ty/blink/renderer/bindings/scripts/generate_bindings.py shows: bind_gen.init(web_idl_database_path=3Doptions.web_idl_database, root_src_dir=3Doptions.root_src_dir, root_gen_dir=3Doptions.root_gen_dir, component_reldirs=3Dcomponent_reldirs, enable_style_format=3Doptions.format_generated_files) task_queue =3D = bind_gen.TaskQueue(single_process=3Doptions.single_process) for task in options.tasks: dispatch_table[task](task_queue) which I would guess is the code initiating the parallel python processes above. Looking at the history, the first use of .TaskQueue here seems to have been at: chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / = . / third_party / blink / renderer / bindings / scripts / = generate_bindings.py commit 3b8f5a3b2903f2aa50efb971f430ddce57a17d16 [log] [tgz] author Yuki Shiino Thu Jun 04 05:01:43 2020 committer Commit Bot Thu Jun 04 = 05:01:43 2020 tree 46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57 parent 54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame] (I did not look for other contexts with .TaskQueue uage additions.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)