Date: Mon, 5 Nov 2018 15:04:38 -0800 From: Mark Millard <marklmi@yahoo.com> To: Rozhuk Ivan <rozhuk.im@gmail.com> Cc: Konstantin Belousov <kostikbel@gmail.com>, Robert <robert.ayrapetyan@gmail.com>, FreeBSD <freebsd-hackers@freebsd.org>, Mark Johnston <markj@freebsd.org> Subject: Re: Sudden grow of memory in "Laundry" state Message-ID: <F68000D0-6037-48C0-85BE-D6BFF1934765@yahoo.com> In-Reply-To: <20181106012107.2898f093@gmail.com> References: <ce38cbfa-e1c5-776e-ef2e-2b867c9a520f@gmail.com> <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <e705099c-ea42-4985-1012-50e9fa11addd@gmail.com> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <c25e19a4-d3ef-e419-06f8-8a86082dbf31@gmail.com> <E4B508E7-04CC-41BD-934B-19EE69E85800@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> <20181027043819.GX5335@kib.kiev.ua> <20181106012107.2898f093@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2018-Nov-5, at 14:21, Rozhuk Ivan <rozhuk.im at gmail.com> wrote: > On Sat, 27 Oct 2018 07:38:19 +0300 > Konstantin Belousov <kostikbel at gmail.com> wrote: >=20 >>> 1. When process allocates memory (mmap), "Active" memory increases,=20= >>> "Free" memory decreases (that's expected). =20 >> No, mmap(2) call only causes some small amount of the kernel memory >> allocation, to track the mmaped region. >>=20 >> Pages are allocated on the first touch, lets ignore the prefaulting >> mechanism which is not too relevant for the discussion. Mapped page >> can be either active or inactive, this is determined by the usage >> history. >>=20 >=20 >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D195882 > This is still a proublem. >=20 > I have 12.0b3 in vbox. > 16 Gb hdd, 4 Gb RAM. > Free 6+Gb on hdd. > Build app to write 6 Gb. >=20 > run, it was killed=20 > Nov 5 21:05:09 firewall kernel: pid 96603 (testvm), uid 0, was = killed: out of swap space > Nov 5 21:05:15 firewall kernel: Nov 5 21:05:09 firewall kernel: pid = 96603 (testvm), uid 0, was killed: out of swap space Unfortunately, the wording of this message is a misnomer for what drives the kills: it is actually driven by being unable to gain more free memory but FreeBSD will not swap-out processes that stay runnable (or are running), only ones that are waiting. Even a single process that stays runnable and keeps lots of RAM in the active category can lead to kills, even if swap is unused or little used. So the kill-behavior is very workload dependent. Having more swap need not avoid such kills because there may not be sufficient other swap-eligible processes that can be used to gain sufficient free RAM to avoid the kills. Real "out of swap" conditions tend to also have messages similar to: Aug 5 17:54:01 sentinel kernel: swap_pager_getswapspace(32): failed If you are not seeing such messages, then it is likely that the mount of swap space that is still free is not the actual issue driving the kill(s). Instead it is tied to a sustained period of low free RAM based on processes that stay running/runnable. (The system simply leaves the dirty pages in memory when a swap_pager_getswapspace failed message is produced. Of itself, it does not cause a kill.) > Mem: 50M Active, 3331M Inact, 192M Laundry, 362M Wired, 208M Buf, 212M = Free > Swap: 16G Total, 151M Used, 16G Free Without extra evidence, do not beleive the "out of swap space" part of "killed: out of swap space". But it turns out there is a tunable setting to control how many tries at freeing memory before kills happen: so, indirectly, how "long" before the kills will start under sustained low free RAM conditions. The default vm.pageout_oom_seq=3D12 can be increased to increase how long a low-free-RAM condition is tolerated. I assign vm.pageout_oom_seq in /etc/sysctl.conf --but that may not be the best for your context. vm.pageout_oom_seq=3D120 has proved useful. In some extreme situations (buildworld buildkernel in a low RAM, slow context, including long I/O latencies) vm.pageout_oom_seq=3D1024 or more has been used to avoid kills when there was plenty of swap space but low free memory (via processes that stay runnable and keep large amounts of RAM active relative to what the system has). So you may want to try assigning vm.pageout_oom_seq based on workload characteristics as part of your experiments. You may find that the kills can be avoided. > Filesystem Size Used = Avail Capacity iused ifree %iused Mounted on > /dev/gptid/6f42a7d9-45b9-11e8-8fd8-0800278fa2e3 17G 14G = 2.4G 85% 768k 1.4M 35% / >=20 > 1. Program does not do it job / killed, but system have more than = enough swap. > 2. mem does not free after all disk io done. > It free only after file deleted or readed. >=20 > Also there is more strange behaviour if swap file disabled. The book The Design And Implementation Of The FreeBSD Operating System Second Edition has material describing that runnable processes are not swapped out: the policy is deliberate. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?F68000D0-6037-48C0-85BE-D6BFF1934765>