From nobody Sun Jun 5 19:42:23 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3229B1BF7C1B for ; Sun, 5 Jun 2022 19:42:38 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-20.consmr.mail.gq1.yahoo.com (sonic317-20.consmr.mail.gq1.yahoo.com [98.137.66.146]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LGRq10KxJz3NK3 for ; Sun, 5 Jun 2022 19:42:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654458148; bh=y26jBWeKE5yPp+M0qovyqLfny16Z7cDlKDRfQyybqPc=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=dhw0T0CxDult/14eOgf3AqpVFkHBQSHyCx4Z9TDQpXxsOZSyE6SJtXJTK8QYCfn6UvcIlnUX3itSZfNL685MNBOyz8X8H3AVyDRouyA0eJ/jv5P4t7YN8tlzidUyasSeI6i9+Y/k8rx5y9XMEbNCZn5BIdlta3QeGXq4eR7rfMXw+1n8+ztSEWxHLgBNHX7PN3XRmTj5TnRrsUSnwfnvte65z+C9Z3UXGaDlrGjn5U/Cy2he5h/SlGxPwggez75ZyyzHPr08j4dVdHGsikvaVfBy6EJRdIKVtdGYy2MZJCrtFWrJmciDsGOUOO8Sg2WnFmSLjhlmZ1Zav+V1LnlKWA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654458148; bh=CCuUfJ5qsIUv2vD6pZCbneM8Tx9dp+R40gFDWIOJeb1=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=WdIRPREWHJ0jzW2jmnXRdSF48M7WseOtWucSSxSiysf2sE4H2aaQctuqaalATsZL48+B1pCfcy/da3y+9+Bq0rF+G0CgEs2uvBSc8xWYXW2D4GBr9p7bDaKCuZkUqBVesvzQ3/AxpUhq6uZwuD8YEkQix1nFCTNky/pihDmIyIvTQxhsU9Y/abZGjc0clQBMwLNuqiRAXaTr/hR8lelBJG+FwAA3IAV0sq6jL/K4PtsreAFwv9UCcGgHjkgQoBHE/ohTnfXBvKULJ65ocCTmKF8f9Qst8AOUVaQOtHc8ipCjOsPyaL8MBg7KT0B2FlEAuixyXRx/PnWKSPKuuJbNiw== X-YMail-OSG: Ddl4KAUVM1nWsZdkULpKAC2fHFxT.EXrxoQil3Sg84e5pSPcVt9kOvjwxlYTgLk vivvxKaM4VCLHbZWKGsrCmf0z7cr72MDhywjHPzUxrfO0mW3YTBQohwjwoaO8RrTAyNhNlnC2CTU 07Q.EE0xzU6HPMSe0dWAPw6MxbjFuB9hO.ew29iKUWCaQKrm8iAlKqWXc5jevfxzIt80UPstTwwx _EpN8oU87VAELtfuAssNmZfOYvR2DFmrfXseInL6g35muYjlWr1SLl7Sj.Qxp2rig5CE4FBzySsy AoVrMNCZNKotejVxSGUxYtlYnLuOmsKmKC7l59yVxaETYVfcgnD4cg2N3MEKfJewHphyMwDlIIq9 ._oCz7enpob03XXd30PQPLvZTFj6QrTjdLLMcs7ksGD6Oxy24tCvmyTjaxfx446ZjqjQYusGL_Gm xlmKduQHqcDtrM_XaZz1RwJUYCmXS747MjPW_YVyug3GQUE_bZVb5qc8lRwqGIp2rtvx0SZBfUDU GzDjVYQpfXmR0bgAwJEgWMm_.wNajAXkez5HjxZGnF0TwC5vpGVt64WGGpp.AdoKGjom7K4OkEx2 jdazEaWxG9lkYJV_x2h9QOZB9pdw2z9nW.5nuUVQPkMqWTbhQgV.UDYW9F77Z5bKoYkTl1C3Fhxa VHhC9yLBrMI.34IWh0V.r4JSkwU0WqN5Up6xCyzASTnix9ac7K01iZ844Cn2BtLsPEaBvt_LaKtf P63T65L1KtgxUMtP5ya_c3DtJ5JzPphJKx8NGpvhNtiNuEiIpFy_3U3UTU3gWqGi7y9HVeCnO8FJ NoaXU1SLZkxuoAsNJR5F1maf209Qd2_EXHQs7prYiUH8JswfngWclw4ubeRD9lt9Wcs34iDWXbxG joFtn0KeyjQ9qybockmUfcUCPApNRuVdwLBmYsm0dOFxQKAOHfJbGJfqGnqEX6dxCARHFyFx4OmL ctUkcAIkKofCa_ds0UzZGTmAX0wqE0GP0I09ql5tT8.Zcm4_K7Dy5ASFKy2gpFi1AyqfkmX_qQ2I bcC8hq7tsyWPVBRTZp8YCu2SHq4Z9c3sYNsPymW1y72iw8nK3rnIrnzmfKUyvQjTPYNoHRKVVdjx EZAkYhuPsC_4idGZz88YFUispm1.Xda0s9OPPogUXo9Egd4Duvw55j1DlMZgVLf_Hk0.jev9IogR VXSIrXqr2K8TueIu0qaMd3ceynOp.7IksgD3fD1O5GqPIKEqdVWBGHMnfEUTDIT4gsvLi3BUObYd pqmT.iUx70GvaNcd_gzpge_FkgBR0A6S81BADKkYHPUMWFq0r9HATFnF7OdLp7eOhZtsfdwj3T3F ObraGHQTUAKqr5jMPof6tjQnBA.uhHkzJIis5_4c3CX7ORcKHysRdLOBfvc9DIzbcMC59DkD8dW8 ZeSse06U_UGeVSoN.LhIJybp8cH7UAVFUfKJgneywpA7fpgi5_.aGkRFwPHjayXjXRFjt5hRJOO6 2Ju8PLiAYkM36HNQxhwpvSt2X.cI5E2huF6o3TsvZpQfGS3Ck20zthvO0xr103bzCd5s8oJW08iL qov3n5B7AeSoz5ThpXx.XqEhnkS3FUS5Q_xNMkZB05V1JEsfFnL9p9ku4CLJj6tlzzLjIRgBXcNh Y3t.se48u9RrrUR0fNGZrv4F71n8OErLZcxyWnUzHJyT27a0HSGXuqqzpQVaIx38hF1NP4dJGl6F E02E_eF4Dj6qjwumC8NKyu3dqBMUIPv66aQmC.eCoD.P.BnYjtJp2Vw7n6ON1k16gwFFQKz0UqeP AqkurjzX1yH4yZVbHH4kZ7GNBuAc0gzXIyKOW6acsI6nsI176bc9dRzco7OZGd5_GQJQ5EoRKDFV ykusUExNA0EueAjlM1mUHckwU6HyAwTq4OQDLjyc48x82Ys.MvyCwdE_RVpiDSHzVZ.ByVBwiTvC ClXamP.Su.lVdgRk35p1ft3yKmNmYkBucsWi5Q39c7rz_2oppAj8W6n9N_y26KNMMJppy6aJNJPL otgYQ8Mk6WQ_oFH20dMdphXCLF6GwG5Puqrfx6XywO9MiVRjUiuGtCYel0m1ZnbJ5.Qr68mJZmzS VAiMBIIW9VsRC304jpXmst5yQyxqbTjtgrkoHNHTbcKXUP423hBAkyrZD.DGYQ9Fx2CfZ9ONz4V9 6Dyd4GP1HC2wspctHIctHWwL8aviQZ5OedxYgN4nngyNBVWTDXf8oO5Z5Fyqj5_PUl4ASZSqR0CF Vxc53oqBkOBTlD2YK.LqcsFolpJcnt_OtNDwLkm2l63xpYl_iTMxv63JmRfDpHmVJu1TH0eOZXw- - X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 Jun 2022 19:42:28 +0000 Received: by hermes--canary-production-ne1-799d7bd497-7lvgk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 84b87229faa40357f4ac8221cea7e301; Sun, 05 Jun 2022 19:42:25 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) Message-Id: Date: Sun, 5 Jun 2022 12:42:23 -0700 To: FreeBSD Hackers X-Mailer: Apple Mail (2.3654.120.0.1.13) References: X-Rspamd-Queue-Id: 4LGRq10KxJz3NK3 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=dhw0T0Cx; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.66.146 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.33 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.84)[-0.845]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.983]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[98.137.66.146:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N I have a poudriere bulk -a -c going on a 8 Gibyte aarch64 system. top has been showing an occasionally increasing swap usage but never any sizable decreases. Over 5800 ports have built so far. The context is UFS only. The system is running a non-debug build of main. Part of the context is ( in /etc/sysctl.conf ): vm.swap_enabled=0 vm.swap_idle_enabled=0 Also ( in /usr/local/etc/poudriere.conf ): USE_TMPFS="data" poudriere's TMPFS reports normally total under 128 KiBytes across the 4 builders. For reference, example figures . . . A top variant shows: Swap: 30720Mi Total, 306816Ki Used vmstat -s shows: 78152 swap pager pages paged out Note: (78152*4096)/1024 == 312608Ki So nearly all of the "swap pager pages paged out" pages are still sitting out in the used swap/paging space. Thus, the usage is not held by user processes or is held via very long running processes or is not directly tied to user processes --or some mix. The variant of top reports never having observed more than: 6658Mi MaxObs(Act+Wir+Lndry). ("MaxObs" is short for "Maximum Observed".) Such high usage is for a bounded time, long past at this point. (Until some combination of port builds ends up active that uses such.) So I'm curious: What can I learn about the data that is staying paged out (and is gradually growing)? How can I learn it? Other notes: The poudriere jail being built is: # poudriere jail -jmain-CA7-bulk_a -i Jail name: main-CA7-bulk_a Jail version: 14.0-CURRENT Jail arch: arm.armv7 Jail method: null Jail mount: /usr/obj/DESTDIRs/main-CA7-poud-bulk_a Jail fs: Jail updated: 2022-05-23 02:21:24 Jail pkgbase: disabled (Just in case the armv7 jail usage or the null method or such is important to the issue.) === Mark Millard marklmi at yahoo.com