From nobody Tue Aug 24 09:09:54 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id AB0D417948BC for ; Tue, 24 Aug 2021 09:10:02 +0000 (UTC) (envelope-from SRS0=4vZC=NP=klop.ws=ronald-lists@realworks.nl) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.24]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Gv3Gd2fG2z4ZmZ for ; Tue, 24 Aug 2021 09:10:01 +0000 (UTC) (envelope-from SRS0=4vZC=NP=klop.ws=ronald-lists@realworks.nl) Date: Tue, 24 Aug 2021 11:09:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=klop.ws; s=rw2; t=1629796194; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wbATcs3kpEgzWymN5rAPolntnSI2apvPtk3d3vfw8TA=; b=HWDWh6bMupvDlJZxLh8Bss0CuY+uYgce/t8T9N4QQey2mJZlxujBD1nET3TWIR5LcjW8P/ FrO5/yeD94Bf36M9tJMzYX/VmFVBrVdkhReQ7M5ixlkbWCHFehXTkP24FF+p4wYPB2wbsG iwwCsYE3SFUiwS1Q1qBt4FdDrKG6cEGV3L4arYIzr1uwGuL3kboWOgjmOJsgUI0N+HueCs hzzSFjZsnudl1e8enqaoFHUIYRDTDGMtEM34rFYqRyZJxo7Y/AIpG0twoOIxSsO4l3C2j7 +U+t1lYfOfoKJMyClCvzY5k8ziTB7f/cxnvkeUxf18PtAS+r7V7mS/z9cwRvFg== From: Ronald Klop To: freebsd-ports@freebsd.org Message-ID: <1626116829.117.1629796194106@localhost> In-Reply-To: <32251809-3419-4DBA-A623-B8B4B2294440@yahoo.com> References: <32251809-3419-4DBA-A623-B8B4B2294440.ref@yahoo.com> <32251809-3419-4DBA-A623-B8B4B2294440@yahoo.com> Subject: Re: Controlling python when building www/chromium List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_116_1458766480.1629796194044" X-Mailer: Realworks (572.25.5dfdc28) Importance: Normal X-Priority: 3 (Normal) X-Rspamd-Queue-Id: 4Gv3Gd2fG2z4ZmZ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=klop.ws header.s=rw2 header.b=HWDWh6bM; dmarc=pass (policy=none) header.from=klop.ws; spf=pass (mx1.freebsd.org: domain of "SRS0=4vZC=NP=klop.ws=ronald-lists@realworks.nl" designates 194.109.157.24 as permitted sender) smtp.mailfrom="SRS0=4vZC=NP=klop.ws=ronald-lists@realworks.nl" X-Spamd-Result: default: False [-3.20 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[klop.ws:s=rw2]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:194.109.157.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DKIM_TRACE(0.00)[klop.ws:+]; DMARC_POLICY_ALLOW(-0.50)[klop.ws,none]; HAS_X_PRIO_THREE(0.00)[3]; RCVD_IN_DNSWL_NONE(0.00)[194.109.157.24:from]; FORGED_SENDER(0.30)[ronald-lists@klop.ws,SRS0=4vZC=NP=klop.ws=ronald-lists@realworks.nl]; RCVD_COUNT_ZERO(0.00)[0]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.109.157.24:from]; MIME_TRACE(0.00)[0:+,1:+,2:~]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:3265, ipnet:194.109.0.0/16, country:NL]; FROM_NEQ_ENVFROM(0.00)[ronald-lists@klop.ws,SRS0=4vZC=NP=klop.ws=ronald-lists@realworks.nl] X-ThisMailContainsUnwantedMimeParts: Y ------=_Part_116_1458766480.1629796194044 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Van: Mark Millard via freebsd-ports Datum: dinsdag, 24 augustus 2021 10:58 Aan: freebsd-ports@freebsd.org, bob prohaska Onderwerp: Re: Controlling python when building www/chromium > > > bob prohaska wrote on > Date: Mon, 23 Aug 2021 08:27:33 -0700 : > > > When trying to build www/chromium on a 1 GB Pi3 the system > > gets bogged down by five instances of python2.7 running > > simultaneously. This happens using both poudriere and make. > > > > It wasn't a problem a year ago, so presumably something has > > changed in chromium's internal build machinery. I've searched > > > > https://www.chromium.org/developers/how-tos/get-the-code > > and > > > > https://groups.google.com/a/chromium.org/g/chromium-dev > > > > but couldn't find any references to python when compiling, only > > when running the browser. > > > > Is there some way to control how many pythons are loosed at one time? > > Most likely two could be accomodated, possibly three. On an 8 GB > > Pi4 the five pythons coexist happily, so the behavior is probably > > not considered a bug. > > > Bob did not show the context. Below I show an example from his > public poudriere logs, a copy from an off-list mail, for > reference: > > QUOTE > When I looked recently, the peak swap usage reported was: > > Fri Aug 6 00:39:58 PDT 2021 > Device 1K-blocks Used Avail Capacity > /dev/da0s2b 1843200 1617020 226180 88% > /dev/mmcsd0s2b 1843200 1615244 227956 88% > Total 3686400 3232264 454136 88% > > and was for (showing the one after that total): > > `-- /bin/sh ./buildscript.chromium > `-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > |-- /usr/local/libexec/poudriere/sh -e /usr/local/share/poudriere/bulk.sh -j main www/chromium > `-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > |-- sh: poudriere[main-default][01]: build_pkg (chromium-91.0.4472.114_1) (sh) > | `-- /usr/bin/make -C /usr/ports/www/chromium build > | `-- (sh) > | `-- ninja -j1 -C out/Release chromedriver -v chrome > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | |-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > | `-- python ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py --web_idl_database gen/third_party/blink/renderer/bindings/web_idl_database.pickle . . . > `-- timestamp > END QUOTE > > This was the most extreme swap/paging space usage from somewhat > analogous use of a generate_bindings. The swap/paging space > usage makes trying multiple builders impractical: it actually > does run out of swap/paging space. (There are limits to how > big of a swap avoids potential mistuning for a given size RAM. > swap/paging+RAM can be larger on a 8 GiByte RPi4B can be much > larger than on a RPi3B, without getting notices suggesting > a mistuned environment.) > > (It is not necessarily Python 2.7. The build, overall, only uses > 2.7 sometimes in some places.) > > https://chromium.googlesource.com/chromium/src/+/refs/heads/main/third_party/blink/renderer/bindings/scripts/generate_bindings.py shows: > > > bind_gen.init(web_idl_database_path=options.web_idl_database, > root_src_dir=options.root_src_dir, > root_gen_dir=options.root_gen_dir, > component_reldirs=component_reldirs, > enable_style_format=options.format_generated_files) > task_queue = bind_gen.TaskQueue(single_process=options.single_process) > for task in options.tasks: > dispatch_table[task](task_queue) > > which I would guess is the code initiating the parallel python > processes above. > > Looking at the history, the first use of .TaskQueue here seems to have > been at: > > chromium / chromium / src / 3b8f5a3b2903f2aa50efb971f430ddce57a17d16^! / . / third_party / blink / renderer / bindings / scripts / generate_bindings.py > commit 3b8f5a3b2903f2aa50efb971f430ddce57a17d16 [log] [tgz] > author Yuki Shiino Thu Jun 04 05:01:43 2020 > committer Commit Bot Thu Jun 04 05:01:43 2020 > tree 46e7ea9138d3452f1b864cbe1d9c3d1adaa62a57 > parent 54c74ea75c7984bbfa11ed6232821af1943ef922 [diff] [blame] > > > (I did not look for other contexts with .TaskQueue uage additions.) > > > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > > > > Hi, Without looking into the code, I would guess that it should be possible to set "options.single_process" (if -j1 is set). As the RPI3 has 4 cores it apparently auto-scales to the amount of CPUs. But I don't have the time to look into the ninja build file. Regards, Ronald. ------=_Part_116_1458766480.1629796194044--