Date: Sat, 13 Jan 2024 20:49:58 -0800 From: Mark Millard <marklmi@yahoo.com> To: lexi@le-fay.org, Current FreeBSD <freebsd-current@freebsd.org> Subject: RE: poudriere: swap_pager: out of swap space Message-ID: <60C36CF0-43AC-4ACA-B6A5-6997F4425EC5@yahoo.com> References: <60C36CF0-43AC-4ACA-B6A5-6997F4425EC5.ref@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Lexi Winter <lexi_at_le-fay.org> wrote on
Date: Thu, 11 Jan 2024 02:21:19 UTC :
> i'm having a recurring problem with poudriere that i hope someone =
might
> have an idea about.
>=20
> i'm building packages with poudriere on a system with 32GB memory, =
with
> tmpfs and md disabled in poudriere (so it's using ZFS only) and with =
the
> ZFS ARC limited to 8GB.
>=20
> running poudriere produces many kernel log messages like this:
>=20
> Jan 10 21:40:00 ilythia kernel: swap_pager: out of swap space
> Jan 10 21:40:00 ilythia kernel: swp_pager_getswapspace(2): failed
> Jan 10 22:41:55 ilythia kernel: swap_pager: out of swap space
> Jan 10 22:41:55 ilythia kernel: swp_pager_getswapspace(21): failed
> Jan 10 23:48:03 ilythia kernel: swap_pager: out of swap space
> Jan 10 23:48:03 ilythia kernel: swp_pager_getswapspace(8): failed
> Jan 11 00:05:00 ilythia kernel: swp_pager_getswapspace(1): failed
> Jan 11 00:21:45 ilythia kernel: swp_pager_getswapspace(10): failed
>=20
> this is despite the system having a large amount of "Inact" memory
> according to top(1):
>=20
> Mem: 3828M Active, 15G Inact, 2921M Laundry, 9263M Wired, 1559M Buf, =
892M Free
> ARC: 3113M Total, 994M MFU, 884M MRU, 39M Anon, 49M Header, 1139M =
Other
> 1296M Compressed, 4130M Uncompressed, 3.19:1 Ratio
> Swap: 2048M Total, 2048M Used, 8192B Free, 99% Inuse
>=20
> from what i can tell, these swap errors don't cause any issues with =
the
> poudriere build, but they do seem to hinder interactive usage by =
causing
> long hangs.
>=20
> does anyone have some idea what's going on here? i don't really
> understand why the system has used 100% of available swap space when =
it
> has plenty of Inact memory it could free to fulfill requirements.
You seem to be under the impression that "Inact" means "page is not
dirty" and so can be freed without being written out to the swap
space.
Inact pages can be dirty and such pages can not be freed without
being written out to the swap space first. If the swap space
ends up filled, dirty pages that are not in active use stay or
propogate into the Inact or Laundry states, accumulating there
(for later potential use).
("Laundry" just means that those pages have been fully prepared
to be written out to swap space. Dirty Inact pages have more work
to be done before they can be written out --if there is room.)
It appears that you had about 15 GiBytes of dirty pages that
were not in active use but that would not fit in the 8192 Bytes
of free Swap space. So they could not be freed without losing
the "dirt" for which there was no other copy of around that
could later be restored when needed.
You do not report what sort of load average triple the context
had, the number of parallel builders that were active, if
you were using ALLOW_MAKE_JOBS, or if you had defined
MAKE_JOBS_NUMBER or the like to limit the parallelism.
I've certainly seen such memory use figures for allowing lots
of parallel builders when ALLOW_MAKE_JOBS was also enabled
and MAKE_JOBS_NUMBER (or the like) was not used to limit the
number of make jobs each parallel builder is allowed to
have going in parallel. There are also some port builds that
create more parallel activity below each make-job, outside
make's control. I've seen an 8 hardware thread system with
the maximum observed for each of the 3 load average time
frames being the likes of: 43.21, 40.44, 33.70 (from
different times during the bulk run, not simulateously).
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?60C36CF0-43AC-4ACA-B6A5-6997F4425EC5>