From nobody Fri Jul 9 19:46:47 2021 X-Original-To: freebsd-ports@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4D8BE1270C87 for ; Fri, 9 Jul 2021 19:46:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-20.consmr.mail.gq1.yahoo.com (sonic314-20.consmr.mail.gq1.yahoo.com [98.137.69.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4GM3Zk6Hd9z4vtN for ; Fri, 9 Jul 2021 19:46:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625860013; bh=2ZA0TNzEnX7EDNGVRK7UWqvdSjgP/pqcEScHmatCCH4=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=WsG5+zp/F8ZvEpqSJEu32TUUJb/BmGv8xpHNVJL8LGSW2llbf+gRifiItAsLtOgvCu5gFfQd6X2Zg4aD2paXlUMxB/PQ7P6z88xJzExpAS0jZSbuazVFxTMx1ybtZDjBo/augcPlmDxRqLyrTD2aizAiM1jcj8dqZq3MTJe3Xq+ccIVYUblTfsOMtGXAeIbjXWboIOBH5sIeEBF2hoXdPe63En4pqxQFiP+tEnXA5FfGCIp9kHU7G5r3ICd6BmdagLX8ol+eZB2c2j0SEPYhbSzZllvDZscRzv7rxwSgj/TSc9fXjJ6J+dlvArjYefGXPGFv5Qzttg9x6/zGBiuQjw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1625860013; bh=u2BQUwTzbKPUyiRAXnxJjfRYjzEU0sdf1Z6P9QeHlmS=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=r24LrjP7yN5twRms1QHcaStmw6LU6Dk0muAkhBySyEmq7pxryu+wmGfYbU0nLEHWXoTqml7UI3kEDBF6AJJP93igHa4PAtkIkDGFv7Se+ZqEzW20HOuKSL16uJgC1FoNF2UwHZc62jTFVepB93wi5GqI16UqeJaoBIBuFwh/8CfiWLs22g7plcSBoEJOP3bWbvGXIyJJV5RPvtnk+2gkkN/1/5GGRYhNLPlKcgZcePBlM9yWVo+plchmScwPr7A1tdObiBIKHqy3uL3JYc0eIAKL4yLrsjokGrGH/eNmqfCfH8G2PDdFoNbgU+d/Gb0xXHXrHgyOZ01ObAEXHivVwQ== X-YMail-OSG: dW0eOJYVM1mQMuUAld9VEYtR7MnYqxWlE3cGDoF1aV4ZxgiGiDv3mmWWgM_GlM_ VTPrzPdoovofzqs.EOdXbE6.bFg7B3eRBryLLnPH4tjOqBIwrtpu2BXdjs4GJmM9GiCi_oFh6EXD f_3WPMg9r2t5qZRWi5Jz_sjzGmAH.yT1DrO5buSRr9j73JqFLkeXPG6gRmmpNqW7k2K9ijlFEM9P rEHOEXLSICqSwgMiIv.4KdY2t0JNIDtGTpyfbGGa3xy3l5lcpzVBFfD.MvSZXTswiSds4Aa7kYTm XWds9dJoqtgVPHjd1mgEEN84PoZnraIUzOSBLvnptEI0GSDIUgVzMljGK5eAV77vnuohB3e0wNgX 3UvVO7HKVaVXrfcxIYNfQjMUyoy8S9vxX6OJ.CrdYA4xRU_hPBt_mFOW1iF8GiDbQIQM_Sda5cw9 2RSOrSNlAF_LEem.yRUj3Xj1Lsu0xrLA3xUfLYKJ2hayTBUjdMhbmHS28n476Vh7hwsjoDKzRsuH mNTgSKrxkfJ4O4IXWJwM9R0cPyqWbnpIvSO0ffX2D95D0AWEfL4jF_tLSCMfuqpvZJj84zFJUg15 ItF7c.fln.4Xa2kuwUxsmzGvhU6Fl61YAE3bjCBx19X2K3wFLEpI3J_T2NWI8iV_OHHP.JYucZTB bdsUkBcyCIqNqdJa9VS51upvRXrytzkchfnTTnN3yPoRQDJ5vpa_7ocaSk264Tje5U3ZPfebxx1n QXJK4pwGxchIdL0RY0vFkTiamnxJNMf43vsiH6MTJWt1IEh5qc4fyE0lOsvHUywcK_eWXljyHt5w mVhcbZACQK.fQlCaduXzpSOgm4q4BVRVWP8iXPURuIh3TOfvtxZY3EiDnnPoCRlT0apY8.QjyZgT xmTXeySnRPy08amBsur0rhl23XudYsnp6aZ4FBeRMtsNYG5beutuYubvHsdNGMbYovEwHrbKquh9 WibZIK7fSq543vwFU_f2Coi0ImUr_BcIRwmjjmSZkdUkplKgFgpw1mamzC.SIENjvZQF257oi6JG WKNlUjlRJXIql6_L6Bin7_ahnaBlQlpXIvrLimzOmWBs_Bfj6GmGD9822RfI4gjnKdwFE1pKlr.N c0h.n5qE_8WvwGInTmn.0F2cnS3kwU_tkUDRvGU8Z.6IlqWTotyVJk9hTPN7cMs2ZncvmYR7sbRX aDskSzw8eTsi4hVPPL.KxNT6.Ulk.sfwKGlZxELt18PvY7M1i5rNB0KXZmA9_5mzqruYyK3vKin0 nEtQVuLfJHq0uAd8GmvlwS7bNqOtIpa4JnG3pKSWkAcW60LvdgU289nO3bCCvsjUjfX7cFjnFUVO K5ZpwdDulQz7kl9RPrWskEQYO3f2euOd70cGIXXaPL1q2sG1W4wtgkGTsvPKNw3u65_CHdWRsKxF RHie1LEkMuFMGK9D2teecKMh.XnryI1f9eHvbOJrLc0iex3.Q5Wq0_EZ5FGJK.US87xVXlrE6MIp BjRelY4S0qTMUwPFpHom4zF9AqLJbKR98TEAz3Dh2Rsbs0fVCWXHZNn2OCHb4SD_PiuPLyO5XpKq .FIuEUDTEYXCowBRCkhfWpovzfo0wFj60E_pgRuwjh_clnmrKu5dNhE1wSKzG4k4nKP9jAIQK9JH rO4YLWqMsiSAlqMOnzNKvGkqqjt2R.IhWnXuNyAvPmzRJWpriowIaSOEBDQKtCBAJxmmNYlOlxKG 3fT66H65M7N.rNMsH6gL2RybFQo979pJBliQDagKeAoF.BHN1KDjze3kHdhwVhzNEV.zzkcIY41g LtBsX3YTKzNxl2Wj9vnXUnqEy4s9AmEOkSYa6A1fC8km4UMPhkakp64rSJYkY.RxNFU0VGMjZSDT p9Bk.zTIMo2zXExmXbb3EEAhGF3tRUkgW.sptY034DoejFwH3_Qq1WlCBodDZvaQ9Uck4Hq3H0kX dgDjJ0FwC1LFq0tAcIVpUCg24SIk0PdrmFbSrfkmQuN.ISCIGQrdDs2JP6SaE4DjRsX7uzzlBA2P e7jtd.5PYs5N9VPEd7BlrVV6Pj5akwyR3Ilg79mfvmEfeQSnvFMZ.NEHZSxMcxEWKdjVgSOy_3Kc zxC24my5XJfQiuK7Gox.7bwEicJZBwp2Ra9FIoKyMdFpu2H1PSF9D4dG_RSXNxlJPvRlb0qFDHOx 4TApKK.G2iDtHU4e0XRdUcuqV9FPs_dCLp9Yg1ArL.XvX7y_nAQvZh6qrbVYXIf5KugAHnXsPQ43 Em_iCAaSuXsytfYcpVvW_p4VVPcYAzLfKTj6ttbNzBs40F02Fxv6Lw6YDJAVngX95WMXhdel2aWf cl3q6IkhBcVlInOMMmoekaPZSV.y4L4s31QY.OJTadxyx8T5IJ5A3wxuOKXIqSjtWVlUS__q4ci2 xBKy6WDoCnE830.WVT_nNmbtHrEEoq13I._RLxRLXBHo4M.NLcIpoLhCEH.c0U2x_XQC_KRY_Vhe DxG7X4i00oHBSOEG2Cv4An2L2.fUmX4v.lw-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Fri, 9 Jul 2021 19:46:53 +0000 Received: by kubenode524.mail-prod1.omega.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID fca4721c91b8223b7b9cb2e93ef89c55; Fri, 09 Jul 2021 19:46:48 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Porting software to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-ports List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports@freebsd.org X-BeenThere: freebsd-ports@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.100.0.2.22\)) Subject: Re: Too many pythons in poudriere In-Reply-To: <24795168-CEBF-4049-A4BD-D529F70E7F8D@yahoo.com> Date: Fri, 9 Jul 2021 12:46:47 -0700 Cc: freebsd-ports@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <6D9BAA86-CB85-42E9-B4FC-6272141CFFDB@yahoo.com> References: <044A7E63-2734-41F4-A1A2-AE5096C6A62C@yahoo.com> <20210708154558.GB60914@www.zefox.net> <20210708175436.GA70414@www.zefox.net> <24795168-CEBF-4049-A4BD-D529F70E7F8D@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.100.0.2.22) X-Rspamd-Queue-Id: 4GM3Zk6Hd9z4vtN X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=WsG5+zp/; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.46 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.69.83:from]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.96)[-0.957]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.69.83:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.83:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.83:from]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-ports] Reply-To: marklmi@yahoo.com From: Mark Millard via freebsd-ports X-Original-From: Mark Millard X-ThisMailContainsUnwantedMimeParts: N On 2021-Jul-8, at 14:11, Mark Millard wrote: > On 2021-Jul-8, at 10:54, bob prohaska wrote: >=20 >> On Thu, Jul 08, 2021 at 09:41:13AM -0700, Mark Millard wrote: >>>=20 >>>=20 >>> On 2021-Jul-8, at 08:45, bob prohaska wrote: >>>=20 >>>> Even with -J1 and no ALLOW_MAKE_JOBS I'm still >>>> seeing five pythons occupying at least 3 GB on >>>> the loose. >>>=20 >>> Actually I just looked and saw: >>>=20 >>> Swapinfo 7.36% >>>=20 >>> (Unlike the 83% or so I saw somewhat around 3 hours ago.) >>>=20 >>> Load Averages (220%) 2.20 2.18 1.76 >>>=20 >>> Elapsed 12:54:56 >>>=20 >>> I do not see a swaplog in http://www.zefox.org/~bob/swaplogs/ >>> to look at. So I can not see how much the peak swap space >>> usage was so far (approximately). >>=20 >> Started a new swaplog: >> http://www.zefox.org/~bob/swaplogs/202107182930.log >>=20 >> It came within a whisker of running out of swap, then abruptly >> the python threads vanished and the build seems to be proceeding. >=20 > Did you update any thing, such as /usr/ports/ , between the 50+ hr > run and the new one? The new one spent over 6 hours at: >=20 > [05:50:59] [ 8% 3914/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = gen/third_party/blink/renderer/modules/event_interface_modules_names.json5= --output_dir gen/third_party/blink/renderer/modules > [12:22:32] [ 8% 3915/47953] python = ../../third_party/blink/renderer/bindings/scripts/generate_bindings.py = --web_idl_database = gen/third_party/blink/renderer/bindings/web_idl_database.pickle = --root_src_dir ../../ --root_gen_dir gen --output_core_reldir = third_party/blink/renderer/bindings/core/v8/ --output_modules_reldir = third_party/blink/renderer/bindings/modules/v8/ enumeration = callback_function callback_interface interface namespace typedef union > [12:22:36] [ 8% 3916/47953] touch = obj/third_party/blink/renderer/bindings/generate_bindings_all.stamp > [12:22:39] [ 8% 3917/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_modules_names.stamp > [12:22:42] [ 8% 3918/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = ../../third_party/blink/renderer/modules/event_target_modules_names.json5 = --output_dir gen/third_party/blink/renderer/modules > [12:22:42] [ 8% 3919/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_target_modules_names= .stamp >=20 >=20 > The 50+ hour one did not: >=20 > [03:56:05] [ 8% 3848/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = gen/third_party/blink/renderer/modules/event_interface_modules_names.json5= --output_dir gen/third_party/blink/renderer/modules > [03:56:05] [ 8% 3849/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_modules_names.stamp > [03:56:06] [ 8% 3850/47953] python = ../../third_party/blink/renderer/bindings/scripts/collect_idl_files.py = --idl_list_file = __third_party_blink_renderer_bindings_web_idl_in_core_for_testing___build_= toolchain_linux_clang_arm64__rule.rsp --component core --output = gen/third_party/blink/renderer/bindings/web_idl_in_core_for_testing.pickle= --for_testing > [03:56:06] [ 8% 3851/47953] touch = obj/third_party/blink/renderer/bindings/web_idl_in_core_for_testing.stamp > [03:56:06] [ 8% 3852/47953] touch = obj/third_party/blink/renderer/bindings/scripts/cached_jinja_templates.sta= mp > [03:56:06] [ 8% 3853/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools ../../third_party/blink/renderer/build/scripts/make_names.py = ../../third_party/blink/renderer/modules/event_target_modules_names.json5 = --output_dir gen/third_party/blink/renderer/modules > [03:56:09] [ 8% 3854/47953] python = ../../third_party/blink/renderer/build/scripts/run_with_pythonpath.py -I = ../../third_party/blink/renderer/build/scripts -I ../../third_party -I = ../../tools = ../../third_party/blink/renderer/build/scripts/core/style/make_computed_st= yle_initial_values.py = ../../third_party/blink/renderer/core/css/css_properties.json5 = ../../third_party/blink/renderer/core/css/computed_style_field_aliases.jso= n5 = ../../third_party/blink/renderer/platform/runtime_enabled_features.json5 = ../../third_party/blink/renderer/core/style/computed_style_extra_fields.js= on5 --output_dir gen/third_party/blink/renderer/core/style --gperf gperf > [03:56:09] [ 8% 3855/47953] touch = obj/third_party/blink/renderer/bindings/modules/event_target_modules_names= .stamp >=20 >=20 > The build step numbers are different for the same > command: >=20 > 3914/47953 > vs. > 3848/47953 >=20 > (But I do not know if the build technique tries to > keep the partial ordering for build steps stable > across build attempts from the similar starting > conditions.) >=20 > It almost seems like like a system level check in what > process(s) have large amounts swap space in use would > be appropriate when this sort of thing happens, not > that I've done such before. My understanding is that > top's per-process reporting of swap space usage is > problematical when the display is set to display such > information. Going in different direction, when you updated: /usr/local/poudriere/poudriere-system to have Jail OSVERSION: 1400024 (and probably matching the host OS main-n247590-5dd84e315a9), matching /usr/src/ as well, did you rebuild all your ports under the coherent combination? Or are you still using port builds in poudriere that were built with the jail OSVERSION being older than the /usr/src/ the builds were based on? Just like devel/llvm10 had a command that only-sometimes had problems when built via the incoherent combination, other ports could have odd behavior --but only sometimes. My guess is that you have not done such because it would have taken far more time than I've noticed in your activities. I recommended before doing such a rebuild of the ports and then a reinstall of them. Given the odd behaviors being observed that do not repeat each time tried, I still make the recommendation. >> I'm curious if this was blind luck, or some adaptive behavior >> by poudrirere. >=20 > Poudriere does not control chrome's internel build steps. > The notation like [ 8% 3849/47953] is not from poudriere. >=20 >> One other oddity: occasionally one see in top a >> PID using more than 100% WCPU. Is one thread occupying two cores? >=20 > A process can have multiple threads instead of just one > but each thread runs on at most one cpu (core here) at a > time. >=20 > Top can do either of: >=20 > A) show a count of threads in each process shown > vs. > B) show a line for each thread instead of for each > process >=20 > Top can also display the unique thread number instead > of the process number (shared by all threads in the > process). (But not both at the same time.) Showing > thread numbers is probably only commonly selected when > (B) is in use. >=20 > In (B) mode, if the process number is being shown, > then there will be one example of the number per > thread shown that is from the process. >=20 > But I'll note that, if I remember right, python is > a single threaded language. It would probably take > use of language bindings to other langauges for > multiple threads to be involved (indirectly) in a > python process. >=20 >>>=20 >>>> I'm fairly sure this didn't happen >>>> when using make by itself (IIRC it was -j2). >=20 > It did not happen at that point in the 50+ hr run > either. >=20 >>>> I also got rid of the mistaken directive in >>>> poudriere.d/make.conf. >>>=20 >>> When I look at http://www.zefox.org/~bob/poudriere.d/make.conf >>> now I see: >>>=20 >>> ALLOW_MAKE_JOBS=3Dyes >>> #MAKE_JOBS_NUMBER=3D2 >>> #.if ${.CURDIR:M*www/chromium} >>> #MAKE_JOBS_NUMBER_LIMIT=3D2 >>> #.endif >>> #.if ${.CURDIR:M*databases/sqlite3} >>> #MAKE_JOBS_NUMBER_LIMIT=3D2 >>> #.endif >>> #.if ${.CURDIR:M*www/firefox} >>> #MAKE_JOBS_NUMBER_LIMIT=3D2 >>> #.endif >>>=20 >>> which does not match your wording. >>>=20 >>=20 >> Thank you for catching my error. _now_ it's fixed. >>=20 >> [snip] >>> To see what is getting CPU time that leads to >>> the load averages being around 2 might take >>> using something like top sorted by cpu time >>> and watching for a while. >>>=20 >>>> There is a=20 >>>> #MAX_MEMORY=3D8 >>>> in poudriere.conf, presumably GB. >>>=20 >>> Documented as GiB: >>>=20 >>> # How much memory to limit jail processes to for *each builder* >>> # in GiB (default: none) >>> #MAX_MEMORY=3D8 >>>=20 >>> Per builder, not per-make-process. >>> Within a builder each make-process shares >>> that size space with the others. >>>=20 >>>> That >>>> looks like a good knob to play with. Would >>>> setting it to something like 3 or 4 help? >>>=20 >>> If the memory use exceeds what you set, the builder >>> process is likely killed.=20 >> [snip]=20 >>=20 >> I was hopeful it might inhibit starting new PIDs >> when memory/swap is below some threshold. Guess not.=20 >=20 > poudriere does not control the internals of the > builder's attempted operations beyond what the > port supports. (And there are not port-specific > interfaces for such control. poudriere treats > ports the same way in general.) >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)