Date: Tue, 24 Aug 2021 11:09:54 +0200 (CEST) From: Ronald Klop <ronald-lists@klop.ws> To: freebsd-ports@freebsd.org Subject: Re: Controlling python when building www/chromium Message-ID: <1626116829.117.1629796194106@localhost> In-Reply-To: <32251809-3419-4DBA-A623-B8B4B2294440@yahoo.com> References: <32251809-3419-4DBA-A623-B8B4B2294440.ref@yahoo.com> <32251809-3419-4DBA-A623-B8B4B2294440@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_116_1458766480.1629796194044 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Van: Mark Millard via freebsd-ports <freebsd-ports@freebsd.org> Datum: dinsdag, 24 augustus 2021 10:58 Aan: freebsd-ports@freebsd.org, bob prohaska <fbsd@www.zefox.net> Onderwerp: Re: Controlling python when building www/chromium > > > bob prohaska <fbsd_at_www.zefox.net> wrote on > Date: Mon, 23 Aug 2021 08:27:33 -0700 : > > > When trying to build www/chromium on a 1 GB Pi3 the system > > gets bogged down by five instances of python2.7 running > > simultaneously. This happens using both poudriere and make. > > > > It wasn't a problem a year ago, so presumably something has > > changed in chromium's internal build machinery. I've searched > > > > https://www.chromium.org/developers/how-tos/get-the-code > > and > > > > https://groups.google.com/a/chromium.org/g/chromium-dev > > > > but couldn't find any references to python when compiling, only > > when running the browser. > > > > Is there some way to control how many pythons are loosed at one time? > > Most likely two could be accomodated, possibly three. On an 8 GB > > Pi4 the five pythons coexist happily, so the behavior is probably > > not considered a bug. > > > Bob did not show the context. Below I show an example from his > public poudriere logs, a copy from an off-list mail, for > reference: > > QUOTE > When I looked recently, the peak swap usage reported was: > > Fri Aug 6 00:39:58 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1617020 226180 88% > /dev/mmcsd0s2b 1843200 1615244 227956 88% > Total 3686400 3232264 454136 88% > > and was for (showing the one after that total): > > `-- /bin/sh ./buildscript.chromium > `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > `-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > |-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > | `-- /usr/bin/make -C /usr/ports/www/chromium build > | `-- (sh) > | `-- ninja -j1 -C out/Release chromedriver -v chrome > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > `-- timestamp > END QUOTE > > This was the most extreme swap/paging space usage from somewhat > analogous use of a generate_bindings. The swap/paging space > usage makes trying multiple builders impractical: it actually > does run out of swap/paging space. (There are limits to how > big of a swap avoids potential mistuning for a given size RAM. > swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much > larger than on a RPi3B, without getting notices suggesting > a mistuned environment.) > > (It is not necessarily Python 2.7. The build, overall, only uses > 2.7 sometimes in some places.) > > https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/bindings/scripts/generate_bindings.py shows: > > > bind_gen.init(web_idl_database_path=options.web_idl_database, > root_src_dir=options.root_src_dir, > root_gen_dir=options.root_gen_dir, > component_reldirs=component_reldirs, > enable_style_format=options.format_generated_files) > task_queue = bind_gen.TaskQueue(single_process=options.single_process) > for task in options.tasks: > dispatch_table[task](task_queue) > > which I would guess is the code initiating the parallel python > processes above. > > Looking at the history, the first use of .TaskQueue here seems to have > been at: > > chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / renderer / bindings / scripts / generate_bindings.py > commit 3b8f5a3b2903f2aa50efb971f430ddce57a17d16 [log] [tgz] > author Yuki Shiino <yukishiino@chromium.org> Thu Jun 04 05:01:43 2020 > committer Commit Bot <commit-bot@chromium.org> Thu Jun 04 05:01:43 2020 > tree 46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57 > parent 54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame] > > > (I did not look for other contexts with .TaskQueue uage additions.) > > > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > > > > Hi, Without looking into the code, I would guess that it should be possible to set "options.single_process" (if -j1 is set). As the RPI3 has 4 cores it apparently auto-scales to the amount of CPUs. But I don't have the time to look into the ninja build file. Regards, Ronald. ------=_Part_116_1458766480.1629796194044--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1626116829.117.1629796194106>