Date: Mon, 16 May 2022 12:07:18 -0700 From: Mark Millard <marklmi@yahoo.com> To: Michael Wayne <freebsd07@wayne47.com> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: Can not build kernel on 1GB VM Message-ID: <934C3159-4B1A-4A2F-9C21-93DC7CC90A72@yahoo.com> In-Reply-To: <20220516143753.GY72471@post.wayne47.com> References: <27171A11-13B1-48A8-AF46-605091E1093F.ref@yahoo.com> <27171A11-13B1-48A8-AF46-605091E1093F@yahoo.com> <20220516143753.GY72471@post.wayne47.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2022-May-16, at 07:37, Michael Wayne <freebsd07@wayne47.com> wrote:
> More info. I am running UFS so the ZFS should not be an issue
>=20
> % pstat -s
> Device 1K-blocks Used Avail Capacity
> /dev/md99 1048576 0 1048576 0%
That may well explain some (or all?) of what is going on:
file-system/vnode backed swap spaces are subject to deadlocks.
On 2017-Feb-13, at 7:20 PM, Konstantin Belousov <kostikbel at gmail.com> =
wrote
on the freebsd-arm list:
QUOTE
. . .
swapfile write requires the write request to come through the filesystem
write path, which might require the filesystem to allocate more memory
and read some data. E.g. it is known that any ZFS write request
allocates memory, and that write request on large UFS file might require
allocating and reading an indirect block buffer to find the block number
of the written block, if the indirect block was not yet read.
As result, swapfile swapping is more prone to the trivial and =
unavoidable
deadlocks where the pagedaemon thread, which produces free memory, needs
more free memory to make a progress. Swap write on the raw partition =
over
simple partitioning scheme directly over HBA are usually safe, while =
e.g.
zfs over geli over umass is the worst construction.
END QUOTE
I only use swap partitions for swap space --in order to avoid
the deadlocks in parts of the kernel operation that I ran into
otherwise.
> % df
> Filesystem 1K-blocks Used Avail Capacity Mounted on
> /dev/ufs/rootfs 25388500 17630840 5726580 75% /
> devfs 1 1 0 100% /dev
>=20
> On Sat, May 14, 2022 at 08:02:29AM -0700, Mark Millard wrote:
>> The way to know if out of swap might actually be involved
>> in the context are some other messages that do not of
>> themselves announce kills:
>>=20
>> swap_pager: out of swap space
>> swp_pager_getswapspace(. . .): failed
>>=20
>> If you are getting either of those 2, then you are actually
>> running out of swap space. Otherwise you are not.
>=20
> I am not getting either of those. Watching swap as ctfmerge runs up=20
> confirms that I am likely not running out of swap.
>=20
> Note also that I continue to run a kernel built with the suggested =
patch to=20
> /sys/vm/vm_pageout.c
(The oldest code I cover for extra messaging
relative to OOM kills and and the like is releng/13.0
code these days. As I remember, the code has been
moved around between files some since 12.x . So the
below may read odd for your context.)
Which suggested patch(s)? Any patches for . . .
sys/vm/swap_pager.c ?
swp_pager_meta_build: swap blk uma zone exhausted
swp_pager_meta_build: swap pctrie uma zone exhausted
sys/vm/vm_fault.c ?
vm_fault_allocate_oom: proc %d (%s) failed to alloc page on fault, =
starting OOM
sys/vm/vm_page.c ?
thread %d waiting for memory
For sys/vm/vm_pageout.c I have messages:
vm_pageout_mightbe_oom: kill context: v_free_count: %u,
v_inactive_count: %u, v_laundry_count: %u, v_active_count: %u
(The 2 forming one line of output.)
failed to reclaim memory
a thread waited too long to allocate a page
(The 2 above are now the standard messages in releng/13.1 ,
stable/13 , and main .)
swblk or swpctrie zone exhausted
(In releng/13.1 , stable/13 , and main this is
still shown as the mismoner "out of swap space".)
>> If you are not the real reason is one of 4:
>=20
> Likely the swap device isn't responding within a "reasonable" time. =
(from another mail in this chain)
>=20
>> FYI:
>>=20
>> kernel: swap_pager: indefinite wait buffer: bufobj: . . ., blkno: . . =
., size: . . .
>>=20
>> is for a swap read taking over 20 seconds (including
>> time when queued but waiting in the queue to start
>> the transfer).
>=20
> I got this message on 12.1. I am not getting it on 12.3
>=20
>=20
>> /boot/loader.conf can use the likes of:
>>=20
>> vm.pageout_oom_seq=3D120
>> vm.pfault_oom_attempts=3D-1
>> #vm.pfault_oom_attempts=3D 3
>> #vm.pfault_oom_wait=3D 10
>>=20
>> I do not know if you have tried any of these.
>=20
> Quoting one of my old emails:
> On 12.1 I used sysctl to set
>> vm.pageout_oom_seq=3D120
>> vm.pfault_oom_attempts=3D-1
>>=20
>> There was no improvement. I still see processes getting killed due
>> to no swap space despite only 7-8 MB being reported used. It sorta
>> feels like it's not really able to use swap at all.
>>=20
>> Note that everything worked fine on 11.x, this is a new issue on 12.
>=20
> I have not tried this on 12.3, nor have I tried the other two. Will =
do.
>=20
Can you switch to using a swap partition instead of
file-system/vnode backed swap space?
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?934C3159-4B1A-4A2F-9C21-93DC7CC90A72>