Date: Fri, 10 Jun 2022 21:27:06 -0700 From: Mark Millard <marklmi@yahoo.com> To: FreeBSD Hackers <freebsd-hackers@freebsd.org> Cc: Daniel Ebdrup Jensen <debdrup@freebsd.org>, David Cross <david@crossfamilyweb.com>, Robert Clausecker <fuz@fuz.su> Subject: Re: What can I learn about data that is staying paged out? (There is a more specific poudriere bulk related context given.) Message-ID: <573B8B0C-5209-459D-98AD-EE92DDA4DF83@yahoo.com> In-Reply-To: <A9C9AC24-62EC-43C1-B713-F2012CD1FD5B@yahoo.com> References: <C337A09C-D546-46FC-A166-DBB3237D1AFC@yahoo.com> <A9C9AC24-62EC-43C1-B713-F2012CD1FD5B@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2022-Jun-5, at 15:04, Mark Millard <marklmi@yahoo.com> wrote:
> On 2022-Jun-5, at 12:42, Mark Millard <marklmi@yahoo.com> wrote:
>=20
>> I have a poudriere bulk -a -c going on a 8 Gibyte
>> aarch64 system. top has been showing an occasionally
>> increasing swap usage but never any sizable decreases.
>> Over 5800 ports have built so far. The context is UFS
>> only. The system is running a non-debug build of main.
>>=20
>> Part of the context is ( in /etc/sysctl.conf ):
>>=20
>> vm.swap_enabled=3D0
>> vm.swap_idle_enabled=3D0
>>=20
>> Also ( in /usr/local/etc/poudriere.conf ):
>>=20
>> USE_TMPFS=3D"data"
>>=20
>> poudriere's TMPFS reports normally total under 128
>> KiBytes across the 4 builders.
>>=20
>> For reference, example figures . . .
>>=20
>> A top variant shows:
>>=20
>> Swap: 30720Mi Total, 306816Ki Used
>>=20
>> vmstat -s shows:
>>=20
>> 78152 swap pager pages paged out
>>=20
>> Note: (78152*4096)/1024 =3D=3D 312608Ki
>>=20
>> So nearly all of the "swap pager pages paged out"
>> pages are still sitting out in the used swap/paging
>> space. Thus, the usage is not held by user processes
>> or is held via very long running processes or is
>> not directly tied to user processes --or some mix.
>>=20
>> The variant of top reports never having observed
>> more than: 6658Mi MaxObs(Act+Wir+Lndry).
>> ("MaxObs" is short for "Maximum Observed".)
>> Such high usage is for a bounded time, long past
>> at this point. (Until some combination of port
>> builds ends up active that uses such.)
>>=20
>> So I'm curious:
>>=20
>> What can I learn about the data that is staying
>> paged out (and is gradually growing)? How can I
>> learn it?
>>=20
>>=20
>> Other notes:
>>=20
>> The poudriere jail being built is:
>>=20
>> # poudriere jail -jmain-CA7-bulk_a -i
>> Jail name: main-CA7-bulk_a
>> Jail version: 14.0-CURRENT
>> Jail arch: arm.armv7
>> Jail method: null
>> Jail mount: /usr/obj/DESTDIRs/main-CA7-poud-bulk_a
>> Jail fs: =20
>> Jail updated: 2022-05-23 02:21:24
>> Jail pkgbase: disabled
>>=20
>> (Just in case the armv7 jail usage or the null method
>> or such is important to the issue.)
>=20
> Hmm. systat -swap reports a toal for the Devices/Paths Used
> that is somewhat less than the total for what reports for the
> Pid . . . Total figures (not the Pid Swap figures!):
>=20
> # systat -swap
> /0 /1 /2 /3 /4 /5 /6 /7 /8 /9 =
/10
> Load Average |||||||| =20
>=20
> Device/Path Size Used |0% /10 /20 /30 /40 / 60\ 70\ 80\ =
90\ 100|
> gpt/CA72USBswp14 14G 150M
> gpt/CA72USBswp16 16G 150M
> Total 30G 300M
>=20
> Pid Username Command Swap/Total Per-Process Per-System
> 1453 root nfsd 1M / 15M 9% 0%
> 1451 root mountd 1M / 15M 7% 0%
> 1481 root sshd 912K / 20M 4% 0%
> 1406 root ntpd 740K / 27M 2% 0%
> 1513 root login 724K / 14M 5% 0%
> 1514 root sh 656K / 13M 4% 0%
> 342 _dhcp dhclient 516K / 13M 3% 0%
> 1363 root rpcbind 448K / 13M 3% 0%
> 1454 root nfsd 400K / 12M 3% 0%
> 341 root dhclient 380K / 13M 2% 0%
> 1341 root syslogd 324K / 12M 2% 0%
> 1505 root getty 292K / 12M 2% 0%
> 1510 root getty 292K / 12M 2% 0%
> 1511 root getty 292K / 12M 2% 0%
> 1512 root getty 292K / 12M 2% 0%
> 1509 root getty 292K / 12M 2% 0%
> 1508 root getty 292K / 12M 2% 0%
> 1507 root getty 292K / 12M 2% 0%
> 1506 root getty 288K / 12M 2% 0%
> 1135 root devd 272K / 11M 2% 0%
> 338 root dhclient 264K / 13M 2% 0%
> 1 root init 244K / 11M 2% 0%
> 1486 root cron 188K / 13M 1% 0%
>=20
> I'm, Still looking for a clear indication of what
> most of the 300 MiBytes or so of swap/paging space
> is in use for.
I finally gave up and checked if a swapoff would
actually bring in all the pages from swap space
that were needed (if any) and then un-configure
the swap space. It did. (The bulk -a was still
ongoing. It was not doing memory-hog builder
activity at the time.)
So such an activity may be a workaround for long
running things like bulk -a to avoid a swap space
accumulation that seems to be happening.
I do not know how much was brought in to RAM vs.
simply deallocated from swap space (pages not
changed and still in RAM). If I do such a test
again, it would be good to figure out how to
monitor what the swapoff does for bringing in
pages vs. just discarding them --if possible.
After a while 12136Ki Used showed up after the
swapon that reconfigured the swap space, which is
about the size of the increments that I'd observed
for its sustained increases.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?573B8B0C-5209-459D-98AD-EE92DDA4DF83>