From owner-freebsd-hackers@freebsd.org Sun Jan 5 00:06:04 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 54C541E6F33 for ; Sun, 5 Jan 2020 00:06:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-21.consmr.mail.gq1.yahoo.com (sonic305-21.consmr.mail.gq1.yahoo.com [98.137.64.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47qzSW2Rr1z4nxv for ; Sun, 5 Jan 2020 00:06:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 1YByXBsVM1l7iIRDXLdXXMYuGN74KNl3wgeC5CE13UHsSYOfhsoVi1_IeIkshVR PAkAdqqhj3zOpdyzVBe9ZFOgrQj_cLKCMuZAtkaBfdy.yOOA4bUGk2SqqJEbrRVP0r3fwhDg.MpG tTqzoCuQFHc3y1L.ZFFy.MgAM.cxjKDWPbrvP0RelfbiDb_QM4Lexp4WjdGqZ_Xbvkh4Whk_JOsT WmwPc9rGI8dkP1vpyHmD3_en.CFu1957zHuv1j93P4gZvmztCLgCWfajIAr1BiOROzOxi0I8w032 wEi42PTFetoA2vH9N85jGoLeELLEWpOpRIY5OTezMkl407z4gWR9QwVsSS4A8pw3xHJGJ_A38CN6 NxpmWDfN4bo3E6ehZVga45LwGx0zXPTQ4liz_Q7zau949Ap7Kh3LpTO0Gcj7IWBuJ5T2Ts9mf4Qo DeeD1Drb04hFzabtYSGbwN2gbud5ciY3XBYrKVLipda.tECBZXHN8TeJ8FHkbaOMC48k_gVTohqd mMaLZ_PRfhdxC.d73mdx_TcwUSdlAIMg_Bz_dlRR44Z7N3e5ebjUADb9feEbnccI8aXND8FE5kAL 2juDNAwDDC2359H0AIlllq43lEWo1f_oHU4E_K_rv4UY3V8ITm33bvmd3eL4pYTV865bsoW5bcP5 G7.Vt8iunTyUsdjzg9pN0AFK8VrkDDtNcsONb.0WIMc_kTtWojBX_5xEJgeE_lVMo6DXABhn6h1T nKzcSl6naz_yaUucn7kmS_vDptFSeITm6J56G.DSwS1eFFqrO8JHrm3TEunX.W7XH5E8s20mN25W RMgZ8USwhFH1QC0INc29u_Svb6qYBmv5HPoHphLFX9WpFHPKXyCxaxCREOyTUFThXJKgsNFbsaFW 6_gXDRI3dlGhVCddHSKA2C9XfgAgkhX2x7v0dbEfVJI5fuFHRnj.VoC9t7rVRLgJqHFPh076y6gV 9ebiIgJDb4Xv30r6bYOkMZuJYCcNGw8xafPR9xMMhCM0wygqOnYjVm1LX_g1bI.h9D9Iq24F015A oYPjxEmtydZJO4Gcqx8932EY756YjV859Of__DVQJ1vcETH0PvaBv8bYODBK4o2aQHp8YyXVGgk5 GcV.u6lx6XM7NFMh3DYLixtJtuM09dgBK9YFgR_i2gi1YCxhRhfktFi1s.2L_PWq8cywjrCvEygy XUcXI_cIdB_rKEEpwRB8xezxO1kCGZPV.JpmE5PwRHlKAbnCnFbw3Tgenb6JCk1_rrQR1lOjDcxw 3MxcBPN8ZJTqERsipfJzBIqD8Y.mC5VudVm_usxOXYibZ7HWp_j1.1O65R.j_9BxLZPQQq5GHudK ND1p5j3jN0sef.nAn47BEUTgAHAog8_U8Dpr6zTju8ZRSeO3XLAJLDWslQaebnf_2k8ncxF9Uhlv z88Vaet4yakD_sNjQdd_EeEVCPSFWZw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 Jan 2020 00:06:01 +0000 Received: by smtp429.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 8ff9280250688b8909cf2047ad72c312; Sun, 05 Jan 2020 00:06:00 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: processes are killed because of out of swap space Message-Id: Date: Sat, 4 Jan 2020 16:05:59 -0800 To: Wojciech Puchar , freebsd-hackers@freebsd.org X-Mailer: Apple Mail (2.3608.40.2.2.4) References: X-Rspamd-Queue-Id: 47qzSW2Rr1z4nxv X-Spamd-Bar: / X-Spamd-Result: default: False [-0.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.10)[-0.105,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-0.30)[-0.300,0]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[84.64.137.98.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (5.69), ipnet: 98.137.64.0/21(0.88), asn: 36647(0.70), country: US(-0.05)]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jan 2020 00:06:04 -0000 Wojciech Puchar wojtek at puchar.net wrote on Sat Jan 4 22:35:35 UTC 2020 : > when i try to use more virtual memory (tested by putting files to tmpfs > /tmp). > > > like that > > pid 16977 (bhyve), jid 0, uid 0, was killed: out of swap space Unfortunately, the wording of this type of message is a misnomer for what typically drives the kills: it is actually driven by being unable to gain more free memory when it is below threshold but FreeBSD will not swap-out processes that stay runnable (or are running), only ones that are waiting. Even a single process that stays runnable and keeps lots of RAM in the active category can lead to kills when swap is unused or little used. So the kill-behavior is very workload dependent. Real "out of swap" conditions (tend to?) also have messages similar to: Aug 5 17:54:01 sentinel kernel: swap_pager_getswapspace(32): failed If you are not seeing such swap_pager_getswapspace messages, then it is likely that the mount of swap space still available is not the actual thing driving the kills. Another thing that can lead to kills is paging I/O that is slow. > the problem is that it's less than 10GB swap used while i have 120GB > available. That fits with the above comments. > before processed begin to be killed system stalls for a while. The below notes may or may not prove useful for your context. For delaying how long free RAM staying low is tolerated, one can increase vm.pageout_oom_seq from 12 to larger. The management of slow paging I've less experience with but do have some notes about below. Examples follow that I use in contexts with sufficient RAM that I do not have to worry about out of swap/page space. These I've set in /etc/sysctl.conf . (Of coruse, I'm not trying to deliberately run out of RAM.) # # Delay when persisstent low free RAM leads to # Out Of Memory killing of processes: vm.pageout_oom_seq=120 (I'll note that figures like 1024 or 1200 or even more are possible. This is controlling how many tries at regaining sufficient free RAM that that level would be tolerated long-term. After that it starts Out Of Memory kills to get some free RAM.) # # For plunty of swap/paging space (will not # run out), avoid pageout delays leading to # Out Of Memory killing of processes: vm.pfault_oom_attempts=-1 (Note: In my context "plunty" really means sufficient RAM that paging is rare. But others have reported on using the -1 in contexts where paging was heavy at times and OOM kills had been happening that were eliminated by the assignment.) I've no experience with the below alternative to that -1 use: # # For possibly insufficient swap/paging space # (might run out), increase the pageout delay # that leads to Out Of Memory killing of # processes: #vm.pfault_oom_attempts= ??? #vm.pfault_oom_wait= ??? # (The multiplication is the total but there # are other potential tradoffs in the factors # multiplied, even for nearly the same total.) I'm not claiming that these 3 vm.???_oom_??? figures are always sufficient. Nor am I claiming that tunables are always available that would be sufficient. Nor that it is easy to find the ones that do exist that might help for specific OOM kill issues. I have seen reports of OOM kills for other reasons when both vm.pageout_oom_seq and vm.pfault_oom_attempts=-1 were in use. As I understand, FreeBSD did not report what kibnd of condition lead to the decision to do an OOM kill. (I do not remember the vm.pageout_oom_seq figures from those reports but no figure is designed to make the delay unbounded. There may be large enough figures to effectively be bounded beyond any reasonable time to wait for an oom.) === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)