From nobody Thu Jul 8 21:11:43 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id E1CBF1224B0C for ; Thu, 8 Jul 2021 21:11:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-25.consmr.mail.gq1.yahoo.com (sonic303-25.consmr.mail.gq1.yahoo.com [98.137.64.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GLTWG4YxCz4v3l for ; Thu, 8 Jul 2021 21:11:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625778712; bh=HXJfEonMycDgNGThSLFSX01kdYU4cCVKpW0FzYWAcAo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=toBvKtHllhy94+2yX45GnY/kXiaBd3DpLN5rj5hgleAVgLi/THpjMoObjHTR3wH2rAe1UvstHWiZkrvstV//h9/vra7XqTAK7D1T8a7rHoJHkK4IGlyNpYY8xH3auR8XhUpOR7IHOdvJGppTBINUIP7+p+ZOJ+O/5jrdQNkUm8Sttf/iUUDGCIT1eKNuyOROSvghZnbJ5bcZLA0YgmI9ySpm3fqrscxn+jpS76p6IVsnAlVqsj+MPro4r2IKpO4+4hDsnzEk0pQPUB27t4FWz97iJ8r6k5NTGgn26U1NA0CSEUI8TGs5TtqxjNkKYepNIbjUkenElyJh7/eiSHgdKQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625778712; bh=Uk0UbZOBLTqZljtPGZFIC0vv6PnOLEaV7K6IvQwq+FV=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=DKbE4lkRnF0l3MnezX7upNdn+cXgH0eptWLc71exNvA2rl1DRyzQE6VhrzULio4evuSMGoFct+y/kxeWvh6XTPDA5ulzwcNZwaL+TEbGJlCx1aDYDBOQnKeyLiZ2vULCwsd2Ksn6xQGiV8KUmw/a+DZWZugHeyZY/Bn4PsVf/I+1oPz0rEr9lvz3legk55vW2kkXgoMzEdgTjSToi0fO8ky5iFBTFJ93zadOO2A/GmlxA7NgwMGNXA8KbD4j5FP5PwaFwxZw0LJooNdBx6WPkucSRvh0Qq8DN2Y7umwSsz4EhBT2QPMsmRl9X8pA62NgMJUe4yHQTwKQw24atue+lw== X-YMail-OSG: k2AmuoQVM1l86SMX38lzPRbxC8ihzaNkUwI1NFEKu26ZHFUuPzm8RHvvOIGLy2p diHvBqmA.jL51rHdE7FQlcJL6F8s.pLhDJdZUFE9dCYumpKA4ThxK01R220Til3em122HmtQA1eK CdvDHcaS5eEOh_wbS2Z0iaOX68UIuxvJ37sBgz9ZvyelUPrdnrPZPkI1tc28w49iILaEW259DxFi 3vctWmg0OP5PjOe2ELJxYLubtdD7xQ8Mg.8FiB4kBYzVPzaxSBsEpAwQ2LTW1wbhcpqxvsAkJm4g MrclPMT9k2Fx5gCvPUhdYjsoRNnr9HT3ukXYUe1f8awhAfNsQfQvKdfrZwqHpmNMqkQ7TlPbBgyP _1_ikdy7ziNNVy5E_351CTZmJJlexHqyuNRuHuY0ZKAWMWHXecGdgA3rSBWYDFl.g95XuHlnQ4kY ldLXCo2dZU8d2Dr0W72Q5b4_T39ifVst6ZFLMLyJDRILl_4jFfn86SbCYYUP3yUdFrVeIRM2vzyt 47V48K3pXQRZ0o_z92cjoM5uw0oQYyQX8l1vbW2aDUXPJX90hNrpMXttel1dfrMTOMt6YEtyn31g 4nlpFt_cHoOz5PQlpo9AMfQ6QV3FYAc8jbnQZTxsaLMRiGREHd8lj7ObFG3uskNUfZcj7xz6d5un TCQ9fAiKVdpZYuCKa.4mCPChOtxzjBLntAgXo6WmRtY3Kj4IYy44cMTyG3Wh78QJo4Im8GCN9MuK HgNHMsG1SeuZm5deIJYjObBzOTEu.jjI_AGM9JUYuqcCqnUz1zfw0lz9w3kz9uCOiLLtXuTFqdR9 wkAZE8WMzkocirBrZ296ht9cEhNK7hHYll_NuhMnViycQ8IFAKxV0kAvd6VwM7wqEefiKUGDVhpq R4.9jqLHgUfGkqnYWF.YXT234VFBTvzmf7R.KlGcM3V4.87AJdLONtZzJ9VaN4XmWa71l0vLHitr Nj8bC6QTHfEftjnu2nC80WueSFoIxwh22nC6hRHMjP2ZWJjpfylIOVLBgVe1XttrllkhejT.qyUk IPsSSYWKdmrowYha4I5muntsUzBntLYw0XkFYE4R1cjkdRFtilKYjsFrwiC_CyCo8nOfHfdBAA.M XMdilmIgUmd9bi70ONoYHZSzBPQaNyiBONaDu_H7RZMocXuyr7b1naPkkks_y9FBjn6hwmR0tnb3 PgglgUWo9mrQYg7gZtoKYm2g6NhmBZzOowqpi9UHgtBonZjpdBBur_pCeC5kBNlXsp8LC1wlSOhN rsOzzbyrECC72FkvQVCnr3P1NuhOklbg00v4rO.TxgnGaL_KTieTiVaIGXnHiv3JcDaF4TxEcr21 C8lLGR7deUuC2ygEML4mer_Lfmo5LNOkuFfwAqa0hgBBMATXWa7k6EMVUf1Tk3wiL4vIqqQMWMAg XifQwdAM_PVAzvAYQp0qF9eyLrIQUEEcBi0HOrRtEvWK9VCsLqCRnj5vyX0H.XyxFUseO5bU_cWa C0cnsMojhf316DOPRbbg51jpyHvEjUM6vsLbh1OBqOUZ6jKGehMuFmdypXnzXX2L5VK8mLC0_U10 aix89Ks0lFl0okM0HUgnQy1QDr_Gy0dnAMvpQXyA_g4FbkcCGBV1hxk1UptyDJjn.CFKldbYBc0f KK7ezLIoc0e8H08.HhX4BTdMeQQFVPAdaM2_f13ytcHW78476jNPSNnT3ZAvOQDYTWqu3f_M.EE0 4PgnKUUoDi1DsH4koGA6W.xqiS94haJihc4nPIYcxF5AhayimJab8sHbyq5QumOGTRTZs7Y_HHYM Q7wagu1pZfCEUKsAee9Fe6HBVRwfLmYtiafu__EKSu.qaE.tTZTvekVj62haKx4k.oVeL6u8ArUQ Jf2njl0.lje6GsWurvCYdSLAUlj.wudqlzo0mxPwc7phXfg03ck5BVAXJbAXrQyPlu3a.5zakxau Iyye5AgJTec_YoVAq0WINRfW3b8jdI3mlOVfBxtcVruxlKRO4Njba83u_vvGoZBaxsxewNIJaOu6 D1CO9mmvEXpGyMEMN7.ofZzhYGWxJp9S5qah0n4vKHGWRfMgZ_FmHNCOJfqWmmvxiAxkGe_s.xV3 lnx6LODHeMH9I926ZBEGNcBv8ZUL.oqh5YgPYOop1ODXPeUtCzM8G4sjaC7.DH0iiRRvEKrr6t6B ySaAUm6nKNKIyOFP9pLIkh4.bKuRm9CMeKWT46Oi4K8VPFhSQmeDgUrNV_SaLjXUczfdz.lT0PIL puTndOlozeaOqalmCjwHRbjHXpPy6K5gG7v41AL4STEsD4z.8sbg_556FLQ9sAx27cQXHPjYHTXi YpJSq.aEoPCll8B1gvBOUdvaHKw_s5lZwpf3oINWwJ8UyK7cUiBuddyoJppc1uNnpT8NqdnaOuOX wkOGNlQnmOaJBysHwXN_WQ6hN3tmkPPM9EjY5IWeNqKQweZ57JKAf5wivQwGpQSqd3rYzxfn_cHk Eruv.43Sr712a8ntwMJqtrKnTrzjPMytecA2rfrTRm4sXl5kCISfqL9clPraCdNmDqieBjSgrNAT S33V1UBf8sQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.gq1.yahoo.com with HTTP; Thu, 8 Jul 2021 21:11:52 +0000 Received: by kubenode520.mail-prod1.omega.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 6fc7813bae8c6bec61a763c36b9f014d; Thu, 08 Jul 2021 21:11:46 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: Too many pythons in poudriere In-Reply-To: <20210708175436.GA70414@www.zefox.net> Date: Thu, 8 Jul 2021 14:11:43 -0700 Cc: freebsd-ports@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <24795168-CEBF-4049-A4BD-D529F70E7F8D@yahoo.com> References: <044A7E63-2734-41F4-A1A2-AE5096C6A62C@yahoo.com> <20210708154558.GB60914@www.zefox.net> <20210708175436.GA70414@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GLTWG4YxCz4v3l X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-8, at 10:54, bob prohaska wrote: > On Thu, Jul 08, 2021 at 09:41:13AM -0700, Mark Millard wrote: >>=20 >>=20 >> On 2021-Jul-8, at 08:45, bob prohaska wrote: >>=20 >>> Even with -J1 and no ALLOW_MAKE_JOBS I'm still >>> seeing five pythons occupying at least 3 GB on >>> the loose. >>=20 >> Actually I just looked and saw: >>=20 >> Swapinfo 7.36% >>=20 >> (Unlike the 83% or so I saw somewhat around 3 hours ago.) >>=20 >> Load Averages (220%) 2.20 2.18 1.76 >>=20 >> Elapsed 12:54:56 >>=20 >> I do not see a swaplog in http://www.zefox.org/~bob/swaplogs/ >> to look at. So I can not see how much the peak swap space >> usage was so far (approximately). >=20 > Started a new swaplog: > http://www.zefox.org/~bob/swaplogs/202107182930.log >=20 > It came within a whisker of running out of swap, then abruptly > the python threads vanished and the build seems to be proceeding. Did you update any thing, such as /usr/ports/ , between the 50+ hr run and the new one? The new one spent over 6 hours at: [05:50:59] [ 8% 3914/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = gen/third_party/blink/renderer/modules/event_interface_modules_names.json5= --output_dir gen/third_party/blink/renderer/modules [12:22:32] [ 8% 3915/47953] python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle = --root_src_dir ../../ --root_gen_dir gen --output_core_reldir = third_party/blink/renderer/bindings/core/v8/ --output_modules_reldir = third_party/blink/renderer/bindings/modules/v8/ enumeration = callback_function callback_interface interface namespace typedef union [12:22:36] [ 8% 3916/47953] touch = obj/third_party/blink/renderer/bindings/generate_bindings_all.stamp [12:22:39] [ 8% 3917/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_modules_names.stamp [12:22:42] [ 8% 3918/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = ../../third_party/blink/renderer/modules/event_target_modules_names.json5 = --output_dir gen/third_party/blink/renderer/modules [12:22:42] [ 8% 3919/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_target_modules_names= .stamp The 50+ hour one did not: [03:56:05] [ 8% 3848/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = gen/third_party/blink/renderer/modules/event_interface_modules_names.json5= --output_dir gen/third_party/blink/renderer/modules [03:56:05] [ 8% 3849/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_modules_names.stamp [03:56:06] [ 8% 3850/47953] python = ../../third_party/blink/renderer/bindings/scripts/collect_idl_files.py = --idl_list_file = __third_party_blink_renderer_bindings_web_idl_in_core_for_testing___build_= toolchain_linux_clang_arm64__rule.rsp --component core --output = gen/third_party/blink/renderer/bindings/web_idl_in_core_for_testing.pickle= --for_testing [03:56:06] [ 8% 3851/47953] touch = obj/third_party/blink/renderer/bindings/web_idl_in_core_for_testing.stamp [03:56:06] [ 8% 3852/47953] touch = obj/third_party/blink/renderer/bindings/scripts/cached_jinja_templates.sta= mp [03:56:06] [ 8% 3853/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = ../../third_party/blink/renderer/modules/event_target_modules_names.json5 = --output_dir gen/third_party/blink/renderer/modules [03:56:09] [ 8% 3854/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools = ../../third_party/blink/renderer/build/scripts/core/style/make_computed_st= yle_initial_values.py = ../../third_party/blink/renderer/core/css/css_properties.json5 = ../../third_party/blink/renderer/core/css/computed_style_field_aliases.jso= n5 = ../../third_party/blink/renderer/platform/runtime_enabled_features.json5 = ../../third_party/blink/renderer/core/style/computed_style_extra_fields.js= on5 --output_dir gen/third_party/blink/renderer/core/style --gperf gperf [03:56:09] [ 8% 3855/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_target_modules_names= .stamp The build step numbers are different for the same command: 3914/47953 vs. 3848/47953 (But I do not know if the build technique tries to keep the partial ordering for build steps stable across build attempts from the similar starting conditions.) It almost seems like like a system level check in what process(s) have large amounts swap space in use would be appropriate when this sort of thing happens, not that I've done such before. My understanding is that top's per-process reporting of swap space usage is problematical when the display is set to display such information. > I'm curious if this was blind luck, or some adaptive behavior > by poudrirere. Poudriere does not control chrome's internel build steps. The notation like [ 8% 3849/47953] is not from poudriere. > One other oddity: occasionally one see in top a > PID using more than 100% WCPU. Is one thread occupying two cores? A process can have multiple threads instead of just one but each thread runs on at most one cpu (core here) at a time. Top can do either of: A) show a count of threads in each process shown vs. B) show a line for each thread instead of for each process Top can also display the unique thread number instead of the process number (shared by all threads in the process). (But not both at the same time.) Showing thread numbers is probably only commonly selected when (B) is in use. In (B) mode, if the process number is being shown, then there will be one example of the number per thread shown that is from the process. But I'll note that, if I remember right, python is a single threaded language. It would probably take use of language bindings to other langauges for multiple threads to be involved (indirectly) in a python process. >>=20 >>> I'm fairly sure this didn't happen >>> when using make by itself (IIRC it was -j2). It did not happen at that point in the 50+ hr run either. >>> I also got rid of the mistaken directive in >>> poudriere.d/make.conf. >>=20 >> When I look at http://www.zefox.org/~bob/poudriere.d/make.conf >> now I see: >>=20 >> ALLOW_MAKE_JOBS=3Dyes >> #MAKE_JOBS_NUMBER=3D2 >> #.if ${.CURDIR:M*www/chromium} >> #MAKE_JOBS_NUMBER_LIMIT=3D2 >> #.endif >> #.if ${.CURDIR:M*databases/sqlite3} >> #MAKE_JOBS_NUMBER_LIMIT=3D2 >> #.endif >> #.if ${.CURDIR:M*www/firefox} >> #MAKE_JOBS_NUMBER_LIMIT=3D2 >> #.endif >>=20 >> which does not match your wording. >>=20 >=20 > Thank you for catching my error. _now_ it's fixed. >=20 > [snip] >> To see what is getting CPU time that leads to >> the load averages being around 2 might take >> using something like top sorted by cpu time >> and watching for a while. >>=20 >>> There is a=20 >>> #MAX_MEMORY=3D8 >>> in poudriere.conf, presumably GB. >>=20 >> Documented as GiB: >>=20 >> # How much memory to limit jail processes to for *each builder* >> # in GiB (default: none) >> #MAX_MEMORY=3D8 >>=20 >> Per builder, not per-make-process. >> Within a builder each make-process shares >> that size space with the others. >>=20 >>> That >>> looks like a good knob to play with. Would >>> setting it to something like 3 or 4 help? >>=20 >> If the memory use exceeds what you set, the builder >> process is likely killed.=20 > [snip]=20 >=20 > I was hopeful it might inhibit starting new PIDs > when memory/swap is below some threshold. Guess not.=20 poudriere does not control the internals of the builder's attempted operations beyond what the port supports. (And there are not port-specific interfaces for such control. poudriere treats ports the same way in general.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)