From owner-freebsd-hackers@freebsd.org Sun Jan 5 19:58:54 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 345331E45C3 for ; Sun, 5 Jan 2020 19:58:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic302-22.consmr.mail.gq1.yahoo.com (sonic302-22.consmr.mail.gq1.yahoo.com [98.137.68.148]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47rTws04Jsz3CS7 for ; Sun, 5 Jan 2020 19:58:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: WsCHTkcVM1nqt_noZD34Qm0Az_kJ2jN2uc0b8xHX_8ivZzYXIHIWjm5eeCxXuHe HMHWDL083ayBA5FiDvjV26i3gJAYKhLs0dRHKWJoh6nnOkvYXnjYxQ0DnuMlD2YnwpPxRyVvV6Mp rXnnfzCeObz4gEjcAjDVPynJLplu_zdRbjMa_oY.1kNF956e_r2SuSjgmBF2kpDKwX2obqxRlwhh Ch22LZAiNWMFJou6uezNkR9cui0fIXtnR.i4kZrAVc2.Ly5romDqaJN_Ox1DZf409Pz_UpPElO6G h7sSxBLzCPizD_hQv2_vqek8xeZbO28ZiBlHi6WRD6lCLtaaXw1bxL5O_54gqO3m.4sLACPeH7OR SN0uE1e1YCtXXzMDJA3wPdStbvOXFeQ17dcquFT3ioVkRJY2DtioMlqjHx2mPFrRuIW_IWRfrKQ_ fQU4PML_FIMjH.8PA1Hdq0.aUyWljfpMLDzq6skKNBiZ6x_l.46CXSqesn9Sv4FQ82cSWs.9dj3O RFfSHPS4XWdgXKSWEgFJkBPTeI7EJr.aVh2Wzh2yLJDSr06kVSPhsxhPeCM_n2laZURMzdGa4zDW vp1dehPDKSL5ca7y5YAaEDyyepb3Y7TZc2phnQe_dLTffMrlJi2G.fwxQM6HysEbJGGaT4WQxNW. KYJgE3J3ZXZOJjbZzEmj5cecYmdr9d00J9FFTI_NqJ5KgbEOEWiFngQqeWb2YE0KE_Mn1SPRGmP_ pOKHOPg6ihqYX7tUPLvibOpYrGvBfygODJ1_sWHUrGlxCvAjnFpfTzch64aeyf_lz.nDI2J_6dts v.mUjYaw5bvd99syfSQPcLcIJLkWBQzGFfQdOZ_jRvU5QtJ5FSFZzmOQl10jkfNAiKDDNVJEYGYC a7HXDeCzQ9OFsvUigb8qBtcnmPbpO2A4sNtE9pgHoZTwC9zeE6FVlvTy6DgftVEqLw5nqhxNtW4x cyPwTFiQxRi_Z0aeJez1KEMRCe5wjGuXSYZp.qEWcTd3lCgsLgVR8GZaSlhp4EJ0e9JCOvqz2zyK yHRc7qDiDtJStKyY7VbjxozwHDcKrhEQbMjBP7ozZjpWajKYbDbXYTXsgbRq8_MOUioD4JAlZqUW xv4848Ls2JNHkaU8EK1peRdRpPi6X47.RUvO_t.x8F1S5y9GVddDlXXzlnMr6yJgU1EDMGtDaED4 okQis2wQ2PItm7GyUEgnNpUk3SOeClyV2coAfqFpn5Ih3ElqHynrW_GYM00n0DKCb_Iqg1iLI5fc f589YzC9LOhkfNB9XKaUeW2RApzxH9X3xHF_GqCazcVKplvV1Y_vYM2r7YoObzuNWO9KgX3FC67O rDWekyNSRekgOIHv1hz2xi5AUjKtSdhtQE8Y4AWgdfDMCQpqxG965TyW2h873cdq4aKUa18SJ6Gc 2cdFLereGrVh_HYGNoQdmV6p4QXup0z8- Received: from sonic.gate.mail.ne1.yahoo.com by sonic302.consmr.mail.gq1.yahoo.com with HTTP; Sun, 5 Jan 2020 19:58:50 +0000 Received: by smtp418.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 6d11935457f00778a5eb62a91e79b963; Sun, 05 Jan 2020 19:58:50 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.40.2.2.4\)) Subject: Re: processes are killed because of out of swap space From: Mark Millard In-Reply-To: Date: Sun, 5 Jan 2020 11:58:48 -0800 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <9C6B115C-90CC-449D-A29D-429BD97F37C4@yahoo.com> References: To: Wojciech Puchar X-Mailer: Apple Mail (2.3608.40.2.2.4) X-Rspamd-Queue-Id: 47rTws04Jsz3CS7 X-Spamd-Bar: / X-Spamd-Result: default: False [-0.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.28)[-0.278,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_GOOD(-0.10)[text/plain]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-0.20)[-0.205,0]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[148.68.137.98.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.00)[ip: (5.97), ipnet: 98.137.64.0/21(0.88), asn: 36647(0.70), country: US(-0.05)]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Jan 2020 19:58:54 -0000 On 2020-Jan-5, at 05:08, Wojciech Puchar wrote: >=20 >>> If you are not seeing such swap_pager_getswapspace messages, then >>> it is likely that the mount of swap space still available is not the >>> actual thing driving the kills. >>> Another thing that can lead to kills is paging I/O that is >>> slow. >>=20 >> paging device is nvd so it's fast. And system isn't even paging = heavily. but is doing geom_raid5 rebuild right now+copying lots of files = to this raid (new RAID5 just created). >>=20 >>=20 >> but still - there is A LOT of memory to be reclaimed. inactive is = many gigabytes on my server. >>=20 >>> # Delay when persisstent low free RAM leads to >>> # Out Of Memory killing of processes: >>> vm.pageout_oom_seq=3D120 >> set to 300. A good question is if changing this figure significantly changes how long before the OOM kills significantly. If it does not take significantly different time to start OOM kills, then vm.pageout_oom_seq criteria is not what is leading to the kills. As far as I know, that is the only way to test for if vm.pageout_oom_seq criteria are the criteria leading to the kills vs. it being something else. (Short of source code changes, anyway.) >>> some free RAM.) >>> # >>> # For plunty of swap/paging space (will not >>> # run out), avoid pageout delays leading to >>> # Out Of Memory killing of processes: >>> vm.pfault_oom_attempts=3D-1 >>=20 >> i don't have such sysctl. is it in FreeBSD 12? >> i have 11.3 vm.pfault_oom_attempts is in head (13). I've not checked on any potential the MFC status but forgot to mention that. Sorry for the confusion. > after changes - no effect If going from vm.pageout_oom_seq=3D12 to vm.pageout_oom_seq=3D300 did not change the time frame for the OOM kills starting to happen, then vm.pageout_oom_seq criteria are not what is driving the kills. Unfortunately, I'm not aware of another (non-source-code) ways to discover what crieria are leading to the OOM kills. Changing source code would require first analyzing it --or someone already familiar would need to provide the source code changes. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)