From nobody Tue Aug 31 21:05:31 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7040A179D8C6 for ; Tue, 31 Aug 2021 21:05:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-55.consmr.mail.gq1.yahoo.com (sonic307-55.consmr.mail.gq1.yahoo.com [98.137.64.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gzfq92YNvz3j8V for ; Tue, 31 Aug 2021 21:05:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1630443934; bh=VT06UHMKUELykU2trEbuOW/CUcvdIdr3MrAz62+Sdvk=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=fdt3QRqGElDrfUbgHMJLAzv8Y7oc0Qawhvh05GYNdpA4qOp2XjozA7Yh9qmqXIBzzJfdHTGU14b8TmlBLEwRt/SMW2IVA2N/JQRSW0iEPlmlGXugfdQsD4j4spfbBQEiX1W8gaF09s8zB9GkXd+gOeTxjv0b8GIptpOTkIzROFs8WY2w2xTsxzxnO4yr4Z3f2v8qtEu3l0qithCftl4zMRkGR0ARxJITGwwxRWUiNln751RtcjP1oSl905ng7sc6YYH3GK1mtvEk7709ur6ohNj4jmsjZ9Cgl1jWHd3s/4cKGIThQhUjEoS+7j6yrfXYiN2VAdJ6mVAvoMBxVaZAUg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1630443934; bh=9DGlG7Ae8A8CPgrvK17/TMwMEX7FHhedxEtwJKB7h4o=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=cErcQS5OCbV2yCefvZqztjQSUB8wKvzcCaZro4UvGuZrsNfr6mJlogKljS8WpRNud0g/V+cVlmpnp35WJJrUtLVFQpcmpP8g1BRk+bdqsg4hXlvGczCWESQGoSnNyt8ukbzLvQtAp4+rjuUl2UBQ2qqtB5gChsyBx/TXH/mpsuTM5fZrwwjp/yEwpIPrbe1x7qdLFYSJX6s+/o8GtR4ZIeTKdgoXkvcUVwbReYWimMlQqzWbb8j+a+3HTmFLRXyWMkE7ZNPTDSOMPNkx6ASdRESnzeNLLCThykn8MSCqkBrV23kv2kuEPLCpu3am6g1AwlHAQGTgqlsJDNwUws+4NA== X-YMail-OSG: dpTV..gVM1lmVApJZKDtnYP.KQps0guBhO0NROqvPnsOKZLuJkohidQW2Zq6_Mb ujf4QyYA911s9IMQzJWBzy4Fl8Qg.b.Jifts5GR5QZPFi_ZMpRHhyQgPcLTBj6ELFS0UzJw.Yk61 5S_xlWmozwZcHkrvFC91Q0cIMSRJupKaEElaoSuD6dairkphLvQdgSgIKU76eFJBe1NJrENVMF7t 2YJr1lOh_JMmp1W9GOOB8e9SUvTXlR_Hxii10Lp3aCvP5UFUeHeA7ss4VSzwg17DG1TxkNIYNFNT .FvsHgf70oi1ES1j1NvByCcZnClo7uYJ9JUkPfHL0O35esAOm2CsQrO7A4hhAa_kP3ws6rVVYPW4 QjA.m.dJtIUaondhRYwhDPXgBZa05CZ9NC6Txg9iMI8hlePfiW6DZUfuFlxDDLBI59_ZZUl.E7nF 1rJi0kESMnyXeNSINemHtQDKuUhTq_TRhpCH6vrhK_j.iTWHxo5Tw3k3I4u_k4mn2wmf93HgQH4S R5_dwhbbwWECWF8QC1vPQavp6.i70AaldN2.HZk5FyidsOsakGiyLsmKXRrD5w7zzi50fd2At0AA gtkuAk3qNmKMFLHOBs7G2J5kEV21W2tHNRsQEe9NVnoTJFxuUi_yumUiU7.Yb5.hvc4ufR1z.moa 69LWlA.tGKSyAciDmZ2A_sGaYFcCQT6uILYOfBgYaBwBQphYCXuBwdAHUjHkVhfWkOOvHrvmQfyC iWFkYk_fWTDzcuDPAwKjpHRZkICBvg4fMaS0jtygm1rfF1L4nZ1cMJS4NeXgqFkRFtVfycSQ5dFE 9AsWpwPBZ.lJzYJgkZpje0PmTv6lrgfxSxlrjQIKNEnBPI_9owZ2D5S_2h4FzvWh82Jz58xdzTWd resh4HhJQDnRxz_s0qWXt4nc1NFiPIdPfFHwGfvIU9pOdNwQKswsGsqHmnuHDTK2YN8R0_2y6unj I0wHnXdyd62o93SWkkxBnb.Bb_Yw6EftRvcZlB_C4ZBX4DQXPbNa5lEZRNTToO33Yyj7ol9cwtzh DH4keI0ksPE.LV0S.29aU1sw4xCHgdHtZF_iuDmTEUEaYvsKMD24MS.3mJMW916fYeSqY8Svt8PM EigXkCRruVAB87pZzeMFdw3c7aPHhZC_3WXdCj54QoJU_yt.9iLvS8EoHuDi.jyBsyHgYVv9rQbz e4MuESrGmegJWhLmx1scFz.8s4yuscR1SGtdU.8bIqxwnLGcbd_Egz4ilgm1Jk00RkrOkGzR5aa0 yobqL.zSw7_Ym5udTkAcMNz.2KpxZ2t8NSqbAFcVZ6V7wl04kq6jr0DikDdnBVcKtp.nkQpMEHHB bUj09QllyilA_Q0rJaeDNOdinxNz1d4mVH_31ZW7rLs5GGUfSIJ19fbtDFPKsB8z3FAiNUWGf4aA umWRBzkmT5aPdZTxh1u6zh_54.fF0t_eFxlZ0KR9jByo.LxCfXZZcjHi5bFUszmukMiVinnpHss5 VhijXmi0U09KSCC7ITLrbipYtQyX95Gsu6cDuZ2YUxzahEyCjBjbi389qPzE1ksGM7uUJDM0CbR8 ZGQG2U9Bgs6TLd5B.oIZpE9aRuL.DsPQAZkdLnwskVPKo7JGHjV_2uiqy6Emikv6vlexW23nLDGU 3j4x3dbLIo5spKnOvW5e2NvAZmuNJPTlGvOxwXV4e0cI63pQ2KT_Mapt6iwme6faAISYj7x8S9sb 9J7J80bynN5uBXY.P6a.9acRO9FXbeZ36r8lyhzwodmgsrAVtYAAW7vlhZ3Rt1J0BNNEbpyRsous szY10m7Iy04JQsapg4lCt5.6fSh3sTsiyAF4GnkeuGT6gvV8__XnKJ0xUFcgA.2J4lGbTDfZm4UG B9gsovD9YxVZf5TJfgMwhwv610V4oCaBBj7pZkr5iYOadA9xjynVlr4XidSq7Cw56ul6GAh9Vn1o Q9rtcCWl37ascB3AvYKZVsAxxzf4_7_0SURXC3GPj_ZlPHy4k5_6pmH9uXdUgNjFqmDqzNYhY34v pjSKVl896zzwdcZcJr88JnxdMEbYOkM4samRTG6jiapGhLkpgNf.DkyHpLBvEyOA9Y81osiuhHO7 .p8GZtTaxRpcNiRG2j.ViUeEVBcOUD.3qLuAR6JmbG3S5GfYHVAmpzgfp_DxF8SfAskWVSXKalYc sOF5.X3hcwl1LOlYWRD2LrOYhjW.2yzUpu7GmEXazqIUQZOyPKQGfX29chVXpMNREl25WlP5y.nN 5w0g1Fimd2VYCMKV0y3NSUYk9SbZg4FGybHAvWzMKT2cmbzJEPjlMMnsNnhq8L5tYt9aqB0fH9rU fhD8tVzAsrU4MmXZkKe51kki5pMPUZ3FioDq4VxOCYPVx7fmn6JxBdG6zB8UuwA-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Tue, 31 Aug 2021 21:05:34 +0000 Received: by kubenode522.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 1aa4ea9121993521e384974a97fc71cf; Tue, 31 Aug 2021 21:05:32 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Controlling python when building www/chromium Message-Id: <20499B7C-C20C-4571-B069-F59E974E5637@yahoo.com> Date: Tue, 31 Aug 2021 14:05:31 -0700 To: bob prohaska , freebsd-ports@freebsd.org X-Mailer: Apple Mail (2.3654.120.0.1.13) References: <20499B7C-C20C-4571-B069-F59E974E5637.ref@yahoo.com> X-Rspamd-Queue-Id: 4Gzfq92YNvz3j8V X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=fdt3QRqG; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.31:from]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.31:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N From: bob prohaska wrote on Date: Tue, 24 Aug 2021 08:35:54 -0700 : > On Tue, Aug 24, 2021 at 11:09:54AM +0200, Ronald Klop wrote: > >=20 > > Van: Mark Millard via freebsd-ports > > Datum: dinsdag, 24 augustus 2021 10:58 > > Aan: freebsd-ports_at_freebsd.org, bob prohaska = > > Onderwerp: Re: Controlling python when building www/chromium > > >=20 > > >=20 > > > bob prohaska wrote on > > > Date: Mon, 23 Aug 2021 08:27:33 -0700 : > > >=20 > > > > When trying to build www/chromium on a 1 GB Pi3 the system > > > > gets bogged down by five instances of python2.7 running > > > > simultaneously. This happens using both poudriere and make. > > > > > > > > It wasn't a problem a year ago, so presumably something has > > > > changed in chromium's internal build machinery. I've searched > > > > > > > >=20 > https://www.chromium.org/developers/how-tos/get-the-code >=20 > > > > and > > > > > > > >=20 > https://groups.google.com/a/chromium.org/g/chromium-dev >=20 > > > > > > > > but couldn't find any references to python when compiling, only > > > > when running the browser. > > > > > Is there some way to control how many pythons are loosed at one = time? > > > > Most likely two could be accomodated, possibly three. On an 8 GB > > > > Pi4 the five pythons coexist happily, so the behavior is = probably > > > > not considered a bug. > > >=20 > > >=20 > > > Bob did not show the context. Below I show an example from his > > > public poudriere logs, a copy from an off-list mail, for > > > reference: > > >=20 > > > QUOTE > > > When I looked recently, the peak swap usage reported was: > > >=20 > > > Fri Aug 6 00:39:58 PDT 2021 > > > Device 1K-blocks Used Avail Capacity > > > /dev/da0s2b 1843200 1617020 226180 88% > > > /dev/mmcsd0s2b 1843200 1615244 227956 88% > > > Total 3686400 3232264 454136 88% > > >=20 > > > and was for (showing the one after that total): > > >=20 > > > `-- /bin/sh ./buildscript.chromium > > > `-- /usr/local/libexec/poudriere/sh -e = /usr/local/share/poudriere/bulk.sh -j main www/chromium > > > |-- /usr/local/libexec/poudriere/sh -e = /usr/local/share/poudriere/bulk.sh -j main www/chromium > > > |-- /usr/local/libexec/poudriere/sh -e = /usr/local/share/poudriere/bulk.sh -j main www/chromium > > > `-- sh: poudriere[main-default][01]: build_pkg = (chromium-91.0.4472.114_1) (sh) > > > |-- sh: poudriere[main-default][01]: build_pkg = (chromium-91.0.4472.114_1) (sh) > > > | `-- /usr/bin/make -C /usr/ports/www/chromium build > > > | `-- (sh) > > > | `-- ninja -j1 -C out/Release chromedriver -v chrome > > > | `-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | |-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | |-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | |-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > | `-- python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > > > `-- timestamp > > > END QUOTE > > >=20 > > > This was the most extreme swap/paging space usage from somewhat > > > analogous use of a generate_bindings. The swap/paging space > > > usage makes trying multiple builders impractical: it actually > > > does run out of swap/paging space. (There are limits to how > > > big of a swap avoids potential mistuning for a given size RAM. > > > swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much > > > larger than on a RPi3B, without getting notices suggesting > > > a mistuned environment.) > > >=20 > > > (It is not necessarily Python 2.7. The build, overall, only uses > > > 2.7 sometimes in some places.) > > >=20 > > >=20 > = https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_par= ty/blink/renderer/bindings/scripts/generate_bindings.py > shows: > > >=20 > > >=20 > > > bind_gen.init(web_idl_database_path=3Doptions.web_idl_database, > > > root_src_dir=3Doptions.root_src_dir, > > > root_gen_dir=3Doptions.root_gen_dir, > > > component_reldirs=3Dcomponent_reldirs, > > > = enable_style_format=3Doptions.format_generated_files) > > > task_queue =3D = bind_gen.TaskQueue(single_process=3Doptions.single_process) > > > for task in options.tasks: > > > dispatch_table[task](task_queue) > > >=20 > > > which I would guess is the code initiating the parallel python > > > processes above. > > >=20 > > > Looking at the history, the first use of .TaskQueue here seems to = have > > > been at: > > >=20 > > > chromium / chromium / src / = 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / = renderer / bindings / scripts / generate_bindings.py > > > commit 3b8f5a3b2903f2aa50efb971f430ddce57a17d16 [log] [tgz] > > > author Yuki Shiino Thu Jun 04 = 05:01:43 2020 > > > committer Commit Bot Thu Jun 04 = 05:01:43 2020 > > > tree 46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57 > > > parent 54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame] > > >=20 > > >=20 > > > (I did not look for other contexts with .TaskQueue uage = additions.) > > >=20 > > >=20 > > >=20 > > > =3D=3D=3D > > > Mark Millard > > > marklmi at yahoo.com > > > ( dsl-only.net went > > > away in early 2018-Mar) > > >=20 > > >=20 > > >=20 > > >=20 > >=20 > >=20 > > Hi, > >=20 > > Without looking into the code, I would guess that it should be = possible to set "options.single_process" (if -j1 is set). > > As the RPI3 has 4 cores it apparently auto-scales to the amount of = CPUs. > > But I don't have the time to look into the ninja build file. > >=20 >=20 > After the latest poudriere failure I again tried using make in > www/chromium. It also got bogged down with five pythons and > eventually stopped. At that point I simply restarted the make > session without cleaning and left the system to run overnight. >=20 > As of this morning there are two c++ instances, the system is > only ~45% idle and it seems to be making good progress:[ 18% = 5544/29872] >=20 > The two c++ instances are consistent with /etc/make.conf . >=20 > Does this behavior offer any hints into when and how the surplus > of pythons gets hatched? I've already shown part of the code change that runs the parallel python commands instead of one at a time --and when upstream changed that. See above. Once FreeBSD's ports updated to such a version, the port started doing the parallel subprocesses (plus the parent process). Ronald Klop already reported on what option exists to avoid such. I have looked at the code for bind_gen.TaskQueue and confirmed the option is for such. (But it is viewed as a debugging aid in how it is described in the code.) The default in the code is based on the number freebsd cpus for the number of subprocesses used (unless the option is used). (Such is in code I've not shown.) > The build session is visible via >=20 > http://www.zefox.org/~bob/usr.ports/www/chromium/ No, when I looked at the time, the only log file was from the 2nd run after the problem occurred and so only had information from you started tat 2nd make run. Note: At the time a fair amount of old material was still exposed from prior poudriere experiments and such. If they are still exposed, you might want to remove them. > The actvivity logging script is still running from the poudriere > session and the output is visible at >=20 > http://www.zefox.org/~bob/swaplogs/20210816.log >=20 > however the file is now somewhat unwieldy in size. Note: it might help to stop this and start another one with a new file once and a while (daily?). Such would make getting information of interst more reasonable. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)