From nobody Sat Jun 11 04:32:10 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2E581842012 for ; Sat, 11 Jun 2022 04:32:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4LKlKw1yqPz4vtk for ; Sat, 11 Jun 2022 04:32:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654921933; bh=qc+dZL8iIWIuAUpe5BUAcPef9mOFb81gpbx/+vWxyzo=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=kmR1vXfv9R9zRJgAonZtpyBw2XWzdufFcsrdUyJ+Fhl/be+jxhW+cd18PjgQu2r/v8Y+662VTlM4omlaGzQ15yW73WKqcIVGit4PtIJB7ZkrYdhaBx6vWaJn04yA62rZtnGbOmN0ky5MAAIRRbnqgZAq7dLJU03QgDWnJpqUZ0+0PD/pgmJX3R62uSO2hE/qZqSVVCyhbBx2i0pjhev6qj1sVOJz7fmhp5J+/tTteyI2p3uzPnsfiCVHuHnGURhPakOfkYkHdhhQdb4FRJZPqrNK4xJOFhdDjefbYMj5Qg+pSaWP73Y2BtuxKt7nZtRBk0KYMVtOMjXxrbBoA64TJQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1654921933; bh=oZQIH27p6bg/smX1Mys6nk5VHgRpLZ7OZvM77KyVCA1=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=HbzDwUkug5eio1h9KM52oCTxJP7lu5lZzxWeKGuZkUI6pZLiKC9tiW5z44Y9YOwmCJ/lMXFk6bRgh//ZbuvNIMDuZW23fyOB7I49RY1l8qkKLs76o3ShJ+2jWJiS3CAkSQp6+u/8fP2ApIh1jVPZ3IgV9EJMr+ufXmxOCcQEWi02ZJDuZLiis38919CseKn3IEUXlOz+3Zz6zncPquN/Gr7FY0YM+uqKx3FTzctGPSnFAimkKmO3O9JejT7rFPdPUtAYxP+8/n8J3eVw+xwIdxz7hT6EL/q1DoD0Wmteu85HQzb08h7hRgEhBnHeXVY4Ibuz68EvTVMbiL/29UP68g== X-YMail-OSG: G6T8pQEVM1m_ueolD9e74CmOp_uKYKWvjhIuWsAxV0PF3Ih1pwwYQHwDJJFomIq fg8bkbSgKAXau0WDBrrQyB.QP6N3UgaVZaQb82DBRbpOay2chD_sFkPFa3rwEIQLKF..cwsulYo7 yGZsPtTmI4KfUXduJqNltuUq23VSoNfWELSPJXeBFGG5Y5z6.drju5sZjwBW74NYAbYXF1c2JsF7 K8YLYp5cTd8EdQVFNSZ85Gt4KHZJug5VUioG3.9s5b3VlaUsb9ckbh9cr_oLRwEjWbbvgpUNnJu9 h4FNOhMqZnA32KeFGgLVRsz6i4iFjgkUF6kRc6ad8t8zA95Uj6S1c3ClTU_rm1W1krL0ikLloaig CrnmF2.NKYMJNs8jGd8Bl2mxzYZ9EQ09FZ01jRPIJk8i9lS6HWdGoAagIMhaNP6DwHbMYlFlWIaJ .oToTTuoglvm6l1JZI.9dM8b_vgjj8lOdyi.wtWMmW7840scMr.s21M9SM67SOTlYQORDJ.J8NHa hudoxeu4_GJxdQQ1VVTx3qjmkZlfngyqrk4z2pHjiCuog7B5Czyf8fuqFom91PH91keUSHI0OKlh gXK6V5XEb3NQxo.TZRJS61Cq_Ybz7iTIJ26aBU5UJ914eMA16sjstXxyCAfjB8CfbsIQHZzHtGXK Ciptfy3S0g5bbudzbj89UCDA2nFENJDVqLNHviYpQ5OWnoQc366FumYet5e0FoVKpO1d4R1waPs. .YDvDBv7_moPK65NxHclQhF3RSyZriRS1VOzu_ttDmSgKQAc4QtJyNyWu_LdqJ5JiTeupiXjsX44 UXPPgo.f.8MARCyGRcyF4Kk1yJCZ1xGSy_pU9Njd38UhP3eGwE1BJydlWZCBhtRIlC6qLf7uO.gY 8Kn2wqo4kBpECFE9CSvTO3qGM.nCdTBSkC3HX0EUBeoOcYpYKsuUHf_lwGw7qlQpQVSH2NsXEsPc _TEThhXoZZHyeLozib4Oo_iTARR2da.DUUzdnY4YCfZ99SQRLZgk6GAyDxVsjYUieZYbTBiUqKXc aR_y.70bZ0LZI0.Gkjp0GBE2zMSYpGcgqbVJJCrKBBrvQmDDzQuJMadiiRwdiXXc84JVDz9LX9if v5HtpTZgcJ51VVrOeb1iEHAOQWfm3a31odn5VIFG5BgRrJapj7eXUS1W05STgr8NISC9vPLGZgOS vzvCJze7V_ZQgq0nmN9tY4Y5T8Bx6CzTXRuiBNkNQhNAITQWRg7ywh7_YAugDVrxpIaVnWIKn3.6 32gDJW_8wqLYXdmdhA9zP3pEvSYxTjolW6I3OX8B5ZCFhgi1yrJ_k7OyPxliVAtUNFvoFN08m5Yf F1FboL_Xd740FZG3MuUYfIUnGNqzXUWS.qtr3CAUBV71noccQo0OwL26RY8qokTS_7B68rKahGJO WEptmOVDDFhv5hldhiHbppUi557mu6DghY4Un8IXl5OjLCt68iLugk5OaqmyslfRZXNlGEgq1gKS ZgAhI65Ok1QYiy8uWHU8Y0zzYGO51HvHRoCNts16WqITay47MyL1pzGiVneerXRXOr0mZOHZD8sv FdPDYOP7Kv9m4dzEWWAQeY6Nkxm0LSBze_NeMAsVt42dsuicsObCOaU2fZMKSfiV4Hl.ZVNQA8V4 qGGcRp6FoxpJdQWsoqsUd5fxysdIDQ_nRN67goZRxGnsEBx.IguEE_da3.IU5wSddm5eNe1faC9x Yrlmzan1.85GxjAZTcA5wzPBEQY4JEkk7sZQ9GppgO9ZkjuQwMLlgpTP7j2GmMfiVsyRBTFpaNSN I0..qFwtRR_8wXK0B8uWomhAig547.ciVBSNk6lXk321DSv6j9SOrXX3y5P9s0PncUdFbSoBemgu dWA6Ta40cgtJaaUQLBeyVvUMthJEbBUXumgZ0Puw4hF21JXpshVZCLX_qtvwArer1UOMFOuPG8Ic Qgi3Ct7o0Jh4DYZ_soQ1qYCSBLFt.ldYUCc6yXnwt9iiM3ZScKFKHnIn7akfzZH3p25FJnTTPIjx .89IR7q_HlvpKtN2rhcm9jGQg2TN6Wj8TmKXP.iqek2pTC7yOfssdqgNQcMXck0hJReOu0XJJk_Q 4AWrksDyk73ES5uPLucQsELbqmGrU8Ls8GAPigjR1EDiAvrQJ7lGp7ZUdo9aLJ6WqIbiEWuc5qna kJL3A9XOh4qLkXqIEzCPIwxfIAh7w8C3f9z3uSs6orL5z4.Ud_gcGlSvjNN9DAk6OkGOlmLmdQ3i N0DpvY6Pivgv_zAXk4LHrj1rh_n6hqWkeEuwZoXTRxN9hewOoEdlQ74IxDWuRwuvHjCIvdKrZ61f Ebu_D61LBU8Gq X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Sat, 11 Jun 2022 04:32:13 +0000 Received: by hermes--canary-production-gq1-54945cc758-fkbwk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID db8054295ef0d028d7d9735dcdfb8b81; Sat, 11 Jun 2022 04:32:11 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) From: Mark Millard In-Reply-To: <573B8B0C-5209-459D-98AD-EE92DDA4DF83@yahoo.com> Date: Fri, 10 Jun 2022 21:32:10 -0700 Cc: Daniel Ebdrup Jensen , David Cross , Robert Clausecker Content-Transfer-Encoding: quoted-printable Message-Id: <6EA27152-3355-4356-B246-A083F31452F2@yahoo.com> References: <573B8B0C-5209-459D-98AD-EE92DDA4DF83@yahoo.com> To: FreeBSD Hackers X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4LKlKw1yqPz4vtk X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=kmR1vXfv; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.148 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; RCPT_COUNT_THREE(0.00)[4]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.148:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N On 2022-Jun-10, at 21:27, Mark Millard wrote: > On 2022-Jun-5, at 15:04, Mark Millard wrote: >=20 >> On 2022-Jun-5, at 12:42, Mark Millard wrote: >>=20 >>> I have a poudriere bulk -a -c going on a 8 Gibyte >>> aarch64 system. top has been showing an occasionally >>> increasing swap usage but never any sizable decreases. >>> Over 5800 ports have built so far. The context is UFS >>> only. The system is running a non-debug build of main. >>>=20 >>> Part of the context is ( in /etc/sysctl.conf ): >>>=20 >>> vm.swap_enabled=3D0 >>> vm.swap_idle_enabled=3D0 >>>=20 >>> Also ( in /usr/local/etc/poudriere.conf ): >>>=20 >>> USE_TMPFS=3D"data" >>>=20 >>> poudriere's TMPFS reports normally total under 128 >>> KiBytes across the 4 builders. >>>=20 >>> For reference, example figures . . . >>>=20 >>> A top variant shows: >>>=20 >>> Swap: 30720Mi Total, 306816Ki Used >>>=20 >>> vmstat -s shows: >>>=20 >>> 78152 swap pager pages paged out >>>=20 >>> Note: (78152*4096)/1024 =3D=3D 312608Ki >>>=20 >>> So nearly all of the "swap pager pages paged out" >>> pages are still sitting out in the used swap/paging >>> space. Thus, the usage is not held by user processes >>> or is held via very long running processes or is >>> not directly tied to user processes --or some mix. >>>=20 >>> The variant of top reports never having observed >>> more than: 6658Mi MaxObs(Act+Wir+Lndry). >>> ("MaxObs" is short for "Maximum Observed".) >>> Such high usage is for a bounded time, long past >>> at this point. (Until some combination of port >>> builds ends up active that uses such.) >>>=20 >>> So I'm curious: >>>=20 >>> What can I learn about the data that is staying >>> paged out (and is gradually growing)? How can I >>> learn it? >>>=20 >>>=20 >>> Other notes: >>>=20 >>> The poudriere jail being built is: >>>=20 >>> # poudriere jail -jmain-CA7-bulk_a -i >>> Jail name: main-CA7-bulk_a >>> Jail version: 14.0-CURRENT >>> Jail arch: arm.armv7 >>> Jail method: null >>> Jail mount: /usr/obj/DESTDIRs/main-CA7-poud-bulk_a >>> Jail fs: =20 >>> Jail updated: 2022-05-23 02:21:24 >>> Jail pkgbase: disabled >>>=20 >>> (Just in case the armv7 jail usage or the null method >>> or such is important to the issue.) >>=20 >> Hmm. systat -swap reports a toal for the Devices/Paths Used >> that is somewhat less than the total for what reports for the >> Pid . . . Total figures (not the Pid Swap figures!): >>=20 >> # systat -swap >> /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 = /10 >> Load Average |||||||| =20 >>=20 >> Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ = 90\ 100| >> gpt/CA72USBswp14 14G 150M >> gpt/CA72USBswp16 16G 150M >> Total 30G 300M >>=20 >> Pid Username Command Swap/Total Per-Process Per-System >> 1453 root nfsd 1M / 15M 9% 0% >> 1451 root mountd 1M / 15M 7% 0% >> 1481 root sshd 912K / 20M 4% 0% >> 1406 root ntpd 740K / 27M 2% 0% >> 1513 root login 724K / 14M 5% 0% >> 1514 root sh 656K / 13M 4% 0% >> 342 _dhcp dhclient 516K / 13M 3% 0% >> 1363 root rpcbind 448K / 13M 3% 0% >> 1454 root nfsd 400K / 12M 3% 0% >> 341 root dhclient 380K / 13M 2% 0% >> 1341 root syslogd 324K / 12M 2% 0% >> 1505 root getty 292K / 12M 2% 0% >> 1510 root getty 292K / 12M 2% 0% >> 1511 root getty 292K / 12M 2% 0% >> 1512 root getty 292K / 12M 2% 0% >> 1509 root getty 292K / 12M 2% 0% >> 1508 root getty 292K / 12M 2% 0% >> 1507 root getty 292K / 12M 2% 0% >> 1506 root getty 288K / 12M 2% 0% >> 1135 root devd 272K / 11M 2% 0% >> 338 root dhclient 264K / 13M 2% 0% >> 1 root init 244K / 11M 2% 0% >> 1486 root cron 188K / 13M 1% 0% >>=20 >> I'm, Still looking for a clear indication of what >> most of the 300 MiBytes or so of swap/paging space >> is in use for. >=20 > I finally gave up and checked if a swapoff would > actually bring in all the pages from swap space > that were needed (if any) and then un-configure > the swap space. It did. (The bulk -a was still > ongoing. It was not doing memory-hog builder > activity at the time.) >=20 > So such an activity may be a workaround for long > running things like bulk -a to avoid a swap space > accumulation that seems to be happening. >=20 > I do not know how much was brought in to RAM vs. > simply deallocated from swap space (pages not > changed and still in RAM). If I do such a test > again, it would be good to figure out how to > monitor what the swapoff does for bringing in > pages vs. just discarding them --if possible. >=20 > After a while 12136Ki Used showed up after the > swapon that reconfigured the swap space, which is > about the size of the increments that I'd observed > for its sustained increases. >=20 An interesting point for "systat -swap" now: /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 = /10 Load Average ||||||||||||||||||||||||||||| Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ = 90\ 100| gpt/CA72USBswp14 14G 6108K gpt/CA72USBswp16 16G 6028K Total 30G 12M Pid Username Command Swap/Total Per-Process Per-System No process is listed as using swap but the 12M shows as used! That should be a hint, not that it is directly useful for me figuring out what the usage is from/for. =3D=3D=3D Mark Millard marklmi at yahoo.com