From nobody Sun Jun 5 22:45:43 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 81F131BD0CBA for ; Sun, 5 Jun 2022 22:46:11 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Received: from mail.dcrosstech.com (rrcs-24-97-5-250.nys.biz.rr.com [24.97.5.250]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.dcrosstech.com", Issuer "DCrossTech.com LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LGWtk4tQMz4dBZ for ; Sun, 5 Jun 2022 22:46:05 +0000 (UTC) (envelope-from david@crossfamilyweb.com) X-Virus-Scanned: amavisd-new at dcrosstech.com Received: from smtpclient.apple (234.sub-174-215-219.myvzw.com [174.215.219.234]) (authenticated bits=0) by mail.dcrosstech.com (8.15.2/8.15.2) with ESMTPSA id 255Mjnm0074326 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Sun, 5 Jun 2022 22:45:50 GMT (envelope-from david@crossfamilyweb.com) X-Authentication-Warning: mail.priv.dcrosstech.com: Host 234.sub-174-215-219.myvzw.com [174.215.219.234] claimed to be smtpclient.apple Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: David Cross List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (1.0) Subject: Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) Date: Sun, 5 Jun 2022 18:45:43 -0400 Message-Id: <27DD20B8-E19B-444B-BAE7-A09CD5390D2F@crossfamilyweb.com> References: Cc: FreeBSD Hackers In-Reply-To: To: Mark Millard X-Mailer: iPhone Mail (19F77) X-Rspamd-Queue-Id: 4LGWtk4tQMz4dBZ X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of david@crossfamilyweb.com designates 24.97.5.250 as permitted sender) smtp.mailfrom=david@crossfamilyweb.com X-Spamd-Result: default: False [-2.35 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+mx]; HAS_XAW(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-0.55)[-0.552]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_TO(0.00)[yahoo.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11351, ipnet:24.97.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[174.215.219.234:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FREEFALL_USER(0.00)[david]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[crossfamilyweb.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; BLOCKLISTDE_FAIL(0.00)[24.97.5.250:server fail,174.215.219.234:server fail]; MLMMJ_DEST(0.00)[freebsd-hackers]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N > On Jun 5, 2022, at 6:06 PM, Mark Millard wrote: >=20 > =EF=BB=BFOn 2022-Jun-5, at 12:42, Mark Millard wrote: >=20 >> I have a poudriere bulk -a -c going on a 8 Gibyte >> aarch64 system. top has been showing an occasionally >> increasing swap usage but never any sizable decreases. >> Over 5800 ports have built so far. The context is UFS >> only. The system is running a non-debug build of main. >>=20 >> Part of the context is ( in /etc/sysctl.conf ): >>=20 >> vm.swap_enabled=3D0 >> vm.swap_idle_enabled=3D0 >>=20 >> Also ( in /usr/local/etc/poudriere.conf ): >>=20 >> USE_TMPFS=3D"data" >>=20 >> poudriere's TMPFS reports normally total under 128 >> KiBytes across the 4 builders. >>=20 >> For reference, example figures . . . >>=20 >> A top variant shows: >>=20 >> Swap: 30720Mi Total, 306816Ki Used >>=20 >> vmstat -s shows: >>=20 >> 78152 swap pager pages paged out >>=20 >> Note: (78152*4096)/1024 =3D=3D 312608Ki >>=20 >> So nearly all of the "swap pager pages paged out" >> pages are still sitting out in the used swap/paging >> space. Thus, the usage is not held by user processes >> or is held via very long running processes or is >> not directly tied to user processes --or some mix. >>=20 >> The variant of top reports never having observed >> more than: 6658Mi MaxObs(Act+Wir+Lndry). >> ("MaxObs" is short for "Maximum Observed".) >> Such high usage is for a bounded time, long past >> at this point. (Until some combination of port >> builds ends up active that uses such.) >>=20 >> So I'm curious: >>=20 >> What can I learn about the data that is staying >> paged out (and is gradually growing)? How can I >> learn it? >>=20 >>=20 >> Other notes: >>=20 >> The poudriere jail being built is: >>=20 >> # poudriere jail -jmain-CA7-bulk_a -i >> Jail name: main-CA7-bulk_a >> Jail version: 14.0-CURRENT >> Jail arch: arm.armv7 >> Jail method: null >> Jail mount: /usr/obj/DESTDIRs/main-CA7-poud-bulk_a >> Jail fs: =20 >> Jail updated: 2022-05-23 02:21:24 >> Jail pkgbase: disabled >>=20 >> (Just in case the armv7 jail usage or the null method >> or such is important to the issue.) >=20 > Hmm. systat -swap reports a toal for the Devices/Paths Used > that is somewhat less than the total for what reports for the > Pid . . . Total figures (not the Pid Swap figures!): >=20 > # systat -swap > /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 /10 > Load Average |||||||| =20 >=20 > Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ 90\= 100| > gpt/CA72USBswp14 14G 150M > gpt/CA72USBswp16 16G 150M > Total 30G 300M >=20 > Pid Username Command Swap/Total Per-Process Per-System > 1453 root nfsd 1M / 15M 9% 0% > 1451 root mountd 1M / 15M 7% 0% > 1481 root sshd 912K / 20M 4% 0% > 1406 root ntpd 740K / 27M 2% 0% > 1513 root login 724K / 14M 5% 0% > 1514 root sh 656K / 13M 4% 0% > 342 _dhcp dhclient 516K / 13M 3% 0% > 1363 root rpcbind 448K / 13M 3% 0% > 1454 root nfsd 400K / 12M 3% 0% > 341 root dhclient 380K / 13M 2% 0% > 1341 root syslogd 324K / 12M 2% 0% > 1505 root getty 292K / 12M 2% 0% > 1510 root getty 292K / 12M 2% 0% > 1511 root getty 292K / 12M 2% 0% > 1512 root getty 292K / 12M 2% 0% > 1509 root getty 292K / 12M 2% 0% > 1508 root getty 292K / 12M 2% 0% > 1507 root getty 292K / 12M 2% 0% > 1506 root getty 288K / 12M 2% 0% > 1135 root devd 272K / 11M 2% 0% > 338 root dhclient 264K / 13M 2% 0% > 1 root init 244K / 11M 2% 0% > 1486 root cron 188K / 13M 1% 0% >=20 > I'm, Still looking for a clear indication of what > most of the 300 MiBytes or so of swap/paging space > is in use for. >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com My guess is that this is swapped out buffer cache (files).=20=