Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 5 Nov 2018 15:04:38 -0800
From:      Mark Millard <marklmi@yahoo.com>
To:        Rozhuk Ivan <rozhuk.im@gmail.com>
Cc:        Konstantin Belousov <kostikbel@gmail.com>, Robert <robert.ayrapetyan@gmail.com>, FreeBSD <freebsd-hackers@freebsd.org>, Mark Johnston <markj@freebsd.org>
Subject:   Re: Sudden grow of memory in "Laundry" state
Message-ID:  <F68000D0-6037-48C0-85BE-D6BFF1934765@yahoo.com>
In-Reply-To: <20181106012107.2898f093@gmail.com>
References:  <ce38cbfa-e1c5-776e-ef2e-2b867c9a520f@gmail.com> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <e705099c-ea42-4985-1012-50e9fa11addd@gmail.com> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <c25e19a4-d3ef-e419-06f8-8a86082dbf31@gmail.com> <E4B508E7-04CC-41BD-934B-19EE69E85800@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> <20181027043819.GX5335@kib.kiev.ua> <20181106012107.2898f093@gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On 2018-Nov-5, at 14:21, Rozhuk Ivan <rozhuk.im at gmail.com> wrote:

> On Sat, 27 Oct 2018 07:38:19 +0300
> Konstantin Belousov <kostikbel at gmail.com> wrote:
>=20
>>> 1. When process allocates memory (mmap), "Active" memory increases,=20=

>>> "Free" memory decreases (that's expected). =20
>> No, mmap(2) call only causes some small amount of the kernel memory
>> allocation, to track the mmaped region.
>>=20
>> Pages are allocated on the first touch, lets ignore the prefaulting
>> mechanism which is not too relevant for the discussion.  Mapped page
>> can be either active or inactive, this is determined by the usage
>> history.
>>=20
>=20
>=20
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D195882
> This is still a proublem.
>=20
> I have 12.0b3 in vbox.
> 16 Gb hdd, 4 Gb RAM.
> Free 6+Gb on hdd.
> Build app to write 6 Gb.
>=20
> run, it was killed=20
> Nov  5 21:05:09 firewall kernel: pid 96603 (testvm), uid 0, was =
killed: out of swap space
> Nov  5 21:05:15 firewall kernel: Nov  5 21:05:09 firewall kernel: pid =
96603 (testvm), uid 0, was killed: out of swap space

Unfortunately, the wording of this message is a misnomer for what
drives the kills: it is actually driven by being unable to gain more
free memory but FreeBSD will not swap-out processes that stay runnable
(or are running), only ones that are waiting. Even a single process
that stays runnable and keeps lots of RAM in the active category can
lead to kills, even if swap is unused or little used. So the
kill-behavior is very workload dependent. Having more swap need not
avoid such kills because there may not be sufficient other
swap-eligible processes that can be used to gain sufficient free RAM
to avoid the kills.

Real "out of swap" conditions tend to also have messages
similar to:

Aug  5 17:54:01 sentinel kernel: swap_pager_getswapspace(32): failed

If you are not seeing such messages, then it is likely that the
mount of swap space that is still free is not the actual issue
driving the kill(s). Instead it is tied to a sustained period
of low free RAM based on processes that stay running/runnable.

(The system simply leaves the dirty pages in memory when a
swap_pager_getswapspace failed message is produced. Of itself,
it does not cause a kill.)

> Mem: 50M Active, 3331M Inact, 192M Laundry, 362M Wired, 208M Buf, 212M =
Free
> Swap: 16G Total, 151M Used, 16G Free

Without extra evidence, do not beleive the "out of swap space"
part of "killed: out of swap space".

But it turns out there is a tunable setting to control how many
tries at freeing memory before kills happen: so, indirectly,
how "long" before the kills will start under sustained low
free RAM conditions.

The default vm.pageout_oom_seq=3D12 can be increased
to increase how long a low-free-RAM condition is tolerated.
I assign vm.pageout_oom_seq in /etc/sysctl.conf --but that may
not be the best for your context.

vm.pageout_oom_seq=3D120 has proved useful. In some extreme
situations (buildworld buildkernel in a low RAM, slow
context, including long I/O latencies) vm.pageout_oom_seq=3D1024
or more has been used to avoid kills when there was plenty
of swap space but low free memory (via processes that
stay runnable and keep large amounts of RAM active relative
to what the system has).

So you may want to try assigning vm.pageout_oom_seq based on
workload characteristics as part of your experiments. You
may find that the kills can be avoided.


> Filesystem                                         Size    Used   =
Avail Capacity iused ifree %iused  Mounted on
> /dev/gptid/6f42a7d9-45b9-11e8-8fd8-0800278fa2e3     17G     14G    =
2.4G    85%    768k  1.4M   35%   /
>=20
> 1. Program does not do it job / killed, but system have more than =
enough swap.
> 2. mem does not free after all disk io done.
> It free only after file deleted or readed.
>=20
> Also there is more strange behaviour if swap file disabled.

The book The Design And Implementation Of The FreeBSD Operating
System Second Edition has material describing that runnable
processes are not swapped out: the policy is deliberate.



=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F68000D0-6037-48C0-85BE-D6BFF1934765>