From owner-freebsd-hackers@freebsd.org Mon Nov 5 23:25:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6AC0110F009 for ; Mon, 5 Nov 2018 23:25:03 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.gq1.yahoo.com (sonic305-21.consmr.mail.gq1.yahoo.com [98.137.64.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CCC337A2DE for ; Mon, 5 Nov 2018 23:25:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: OdahRkgVM1n6HodPqKIkrwgDe35JM3uHVDXB0W65Es7Cpdhbz2pWChIexu3feeH Pw9f9ocfehEnRvsSXnn2uHKarzFRr0Auh5I.q5XJOgy2TPMCO.FqAu3N8ippIalblvtq2RsVAKfJ iTjrIEo5mt1UNEdBQpClFe_LH0GXuhx1SgQsjbNMJQkILKRxUH_JV7RPf_KQ0EhB0P6OomlADeJM eXCCp9IHamLCtUHAmH8q0jSZmbu60ezzseMuWYdrcXxgUcsHrqoZC249RE2F6RoZSXC5BuV8kNf7 ZokUBcJ.rftV3fCt38kQ97UvtqgOegsbjI2O7XaSnAfWMoyQolVGAhPKCOpxFkerJj_.FmqMy9bM d3ryiNAl3UemZtrzrwxz1SG_WSJAZGNJGRJEmTfLypy9UG5J4qRip.e04DU5mCv1C1GB1KB8tzT6 SvjgUVrZTOI6wbx2YNTMn5sdFPxLnKMesGzb6.3eO.s4jNR9cTeuOHCa1JhUr8NLyg1rQq02otPP SAHMZ8EBMcTiMPYv5KS08SRhd28DLnDYZ0OMz1zQcFdzA5_L6TqQ_E.3bwM3VfoWzllzGBkh4WJV vWRcnML2yknuLfRJ.UbzhshaeotUPMJzDB8rQk3IxlxM0ldoIvabdvFclwPBuumTFYgiGhFqWkfJ _WqrVV8p2tOoTtqGJ2t8bv6f0YuGJXvS0WlbkDRBUAs23xQJVE8Q4_FnnvD1vomRz_pk.icXds_u zM8yzLWx0hp_Q.bv4s3ZCa9PFIVknpUIZUhD1SDFBip1Y8OoR6FNV11AKr4PzwCcDIOZFKBwXYWg _5SZ3vShKyZkePQnBxEMapyKwjBtYBcA9NBBVakIzEl32uQ67V.3hUfVyNycrvcyZAKz9YNZqUFv 34sa9ub_i7vm9vj7xE4R9pqc_gCaO8vHPf1TYr6Aju9TT.CQuvDCkS3wGha0sLh6IHuTPNP4L.jS eJDxXMoirMHQ6V7ay5YTYW4zq4kT8ythQtutjxHwL7agHGLxn5lqg7oZMEgP1tHZMOdFeDhH5s28 hnHcR1Lyo_KtL.p2YpkrT.kU7ANQ1zFmZg54VQevZ1XlH7Jq0NLhZr_gdGa4nYWbA3N8r5WnP Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Mon, 5 Nov 2018 23:24:56 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp432.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID fd2c6f61bddc51ab850dd24b2590a1ff; Mon, 05 Nov 2018 23:04:38 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.1 \(3445.101.1\)) Subject: Re: Sudden grow of memory in "Laundry" state From: Mark Millard In-Reply-To: <20181106012107.2898f093@gmail.com> Date: Mon, 5 Nov 2018 15:04:38 -0800 Cc: Konstantin Belousov , Robert , FreeBSD , Mark Johnston Content-Transfer-Encoding: quoted-printable Message-Id: References: <20180911150849.GD92634@raichu> <104be96a-c16b-7e7c-7d0d-00338ab5a106@gmail.com> <20180928152550.GA3609@raichu> <20181024211237.302b72d9@gmail.com> <981C887D-78EB-46D2-AEE5-877E269AF066@yahoo.com> <42f6544f-830c-18c5-e1a8-0acc4c3f09cc@gmail.com> <20181027043819.GX5335@kib.kiev.ua> <20181106012107.2898f093@gmail.com> To: Rozhuk Ivan X-Mailer: Apple Mail (2.3445.101.1) X-Rspamd-Queue-Id: CCC337A2DE X-Spamd-Result: default: False [-3.22 / 200.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.94)[-0.943,0]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.13)[ipnet: 98.137.64.0/21(0.39), asn: 36647(0.31), country: US(-0.06)]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.950,0]; R_DKIM_ALLOW(-0.20)[yahoo.com]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.94)[-0.940,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[84.64.137.98.list.dnswl.org : 127.0.5.0]; FREEMAIL_CC(0.00)[gmail.com] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Nov 2018 23:25:04 -0000 On 2018-Nov-5, at 14:21, Rozhuk Ivan wrote: > On Sat, 27 Oct 2018 07:38:19 +0300 > Konstantin Belousov wrote: >=20 >>> 1. When process allocates memory (mmap), "Active" memory increases,=20= >>> "Free" memory decreases (that's expected). =20 >> No, mmap(2) call only causes some small amount of the kernel memory >> allocation, to track the mmaped region. >>=20 >> Pages are allocated on the first touch, lets ignore the prefaulting >> mechanism which is not too relevant for the discussion. Mapped page >> can be either active or inactive, this is determined by the usage >> history. >>=20 >=20 >=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D195882 > This is still a proublem. >=20 > I have 12.0b3 in vbox. > 16 Gb hdd, 4 Gb RAM. > Free 6+Gb on hdd. > Build app to write 6 Gb. >=20 > run, it was killed=20 > Nov 5 21:05:09 firewall kernel: pid 96603 (testvm), uid 0, was = killed: out of swap space > Nov 5 21:05:15 firewall kernel: Nov 5 21:05:09 firewall kernel: pid = 96603 (testvm), uid 0, was killed: out of swap space Unfortunately, the wording of this message is a misnomer for what drives the kills: it is actually driven by being unable to gain more free memory but FreeBSD will not swap-out processes that stay runnable (or are running), only ones that are waiting. Even a single process that stays runnable and keeps lots of RAM in the active category can lead to kills, even if swap is unused or little used. So the kill-behavior is very workload dependent. Having more swap need not avoid such kills because there may not be sufficient other swap-eligible processes that can be used to gain sufficient free RAM to avoid the kills. Real "out of swap" conditions tend to also have messages similar to: Aug 5 17:54:01 sentinel kernel: swap_pager_getswapspace(32): failed If you are not seeing such messages, then it is likely that the mount of swap space that is still free is not the actual issue driving the kill(s). Instead it is tied to a sustained period of low free RAM based on processes that stay running/runnable. (The system simply leaves the dirty pages in memory when a swap_pager_getswapspace failed message is produced. Of itself, it does not cause a kill.) > Mem: 50M Active, 3331M Inact, 192M Laundry, 362M Wired, 208M Buf, 212M = Free > Swap: 16G Total, 151M Used, 16G Free Without extra evidence, do not beleive the "out of swap space" part of "killed: out of swap space". But it turns out there is a tunable setting to control how many tries at freeing memory before kills happen: so, indirectly, how "long" before the kills will start under sustained low free RAM conditions. The default vm.pageout_oom_seq=3D12 can be increased to increase how long a low-free-RAM condition is tolerated. I assign vm.pageout_oom_seq in /etc/sysctl.conf --but that may not be the best for your context. vm.pageout_oom_seq=3D120 has proved useful. In some extreme situations (buildworld buildkernel in a low RAM, slow context, including long I/O latencies) vm.pageout_oom_seq=3D1024 or more has been used to avoid kills when there was plenty of swap space but low free memory (via processes that stay runnable and keep large amounts of RAM active relative to what the system has). So you may want to try assigning vm.pageout_oom_seq based on workload characteristics as part of your experiments. You may find that the kills can be avoided. > Filesystem Size Used = Avail Capacity iused ifree %iused Mounted on > /dev/gptid/6f42a7d9-45b9-11e8-8fd8-0800278fa2e3 17G 14G = 2.4G 85% 768k 1.4M 35% / >=20 > 1. Program does not do it job / killed, but system have more than = enough swap. > 2. mem does not free after all disk io done. > It free only after file deleted or readed. >=20 > Also there is more strange behaviour if swap file disabled. The book The Design And Implementation Of The FreeBSD Operating System Second Edition has material describing that runnable processes are not swapped out: the policy is deliberate. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)