From owner-freebsd-arm@freebsd.org Sun Aug 12 23:23:52 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D2F6C105972D for ; Sun, 12 Aug 2018 23:23:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic303-4.consmr.mail.bf2.yahoo.com (sonic303-4.consmr.mail.bf2.yahoo.com [74.6.131.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A19D7F542 for ; Sun, 12 Aug 2018 23:23:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: aFij67IVM1noXKJV_XrywKJxBwMuRR.eCixeNkaUp91DGaO1_ZZLfZUqbQIjwwm rD4VoUja2qMUHBI3HXYtbIMEc5FSBFV5KGaiSsOIHcub4MUqv7zEJuE.t0WAhnsiBJxIbIL_Fuy6 YECIO91MabTxUQnEmHuO7.JJymBzj_3N_H7tIOMkGK7Aj9Lp0GFK1lwFOn_538wban8kevoOZL3V gyDY4aGmaEkB3MCCbwjij58tmwbMJOCaxf215u26i4l5LAbuQZVcCgCsppFhuYOSHWcW.7OzdC1u JVH9ZQtyah_7cBTg99LZFWDThIpV5avRoeRMa3sN.spGQwTtZYoNwQPqkRmf39U.wKSaRhx6oq1o 2GTy85zP1mnHDrzFEV6.CDFGRaxapBYea4cJBBHaZiKIBo6bzjEYcG9wzQ7y2Bsz6ycCIpgBG7o5 fCMveap6jmhb.za1SCXEeskrGP240U4YdIYEWVAj9haQcwNxY6pZ0jBxO50CuUJxvxc9GISZ3RNA wdpfJDL6Rk9EB2EHIrBC2EkZavYM7W9hLfW.7n_cBjcmp0U5RrK2iybrQpa_ZcyqKMroHXPG9N_u Q0_3_TDRLSxRQW.vbl_fVasKZTiIsiE400WmV5KJNS5chWYm_dD75NUqOwQfmgzIgyPGhclNOLKm zxMdY88s1p.I0DnNWrC.IhdHlOSm40gP7Ch5I_yBFR.OOSJo1UZinbI5AgP.8deNwYgzNh_IH4ud A5.AeVPbaUfQA728GpBlyRw4qluFZpCsUQc1whRxd3ZDcYr8KIPyndcCyY.sxBgaIL6YNTMwT8Ft cFbcw2RHn_PrVZspVbtGK_HHwARz2yLau8qa38MXmWnDqlx6kfxR56tMm2rTCsWB531n4jk4sCga 7KwJeoDEcEwavpN0hFaxBHkWKzQEKpvPS.kY5ZOuH_.KRpoBU3HfMCWxUsdb8mqC9zGSvQHQdPcH 9vbiDDxwbeIt3Hs4_0T6w.ikA6icBtRALixNPdubdFFAEO2zc7ozujMb_jiUXq.LFY0RsjvTc.Oi lSA1xq94ZxXmZ90hvRLh8ug-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.bf2.yahoo.com with HTTP; Sun, 12 Aug 2018 23:23:45 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp405.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 215759bf1a45537ff59cbaf59bc7e6c5; Sun, 12 Aug 2018 23:23:40 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] From: Mark Millard In-Reply-To: <20180812224021.GA46372@www.zefox.net> Date: Sun, 12 Aug 2018 16:23:31 -0700 Cc: John Kennedy , freebsd-arm , fbssd@www.zefox.net Content-Transfer-Encoding: quoted-printable Message-Id: References: <20180802002841.GB99523@www.zefox.net> <20180802015135.GC99523@www.zefox.net> <20180806155837.GA6277@raichu> <20180808153800.GF26133@www.zefox.net> <20180808204841.GA19379@raichu> <2DC1A479-92A0-48E6-9245-3FF5CFD89DEF@yahoo.com> <20180809033735.GJ30738@phouka1.phouka.net> <20180809175802.GA32974@www.zefox.net> <20180812173248.GA81324@phouka1.phouka.net> <20180812224021.GA46372@www.zefox.net> To: bob prohaska , Mark Johnston X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Aug 2018 23:23:52 -0000 On 2018-Aug-12, at 3:40 PM, bob prohaska wrote: > On Sun, Aug 12, 2018 at 10:32:48AM -0700, John Kennedy wrote: >> . . . > Setting vm.pageout_oom_seq to 120 made a decisive improvement, almost = allowing > buildworld to finish. By the time I tried CAM_IOSCHED_DYNAMIC = buildworld was > getting only about half as far, so it seems the patches were harmful = to a degree. > Changes were applied in the order=20 You could experiment with figures bigger than 120 for vm.pageout_oom_seq . I'll note that the creation of this mechanism seems to be shown for -r290920 at: = https://lists.freebsd.org/pipermail/svn-src-head/2015-November/078968.html= In part is says: . . . only raise OOM when pagedaemon is unable to produce a free page in several back-to-back passes. Track the failed passes per pagedaemon thread. =20 The number of passes to trigger OOM was selected empirically and tested both on small (32M-64M i386 VM) and large (32G amd64) configurations. If the specifics of the load require tuning, sysctl vm.pageout_oom_seq sets the number of back-to-back passes which must fail before OOM is raised. Each pass takes 1/2 of seconds. Less the value, more sensible the pagedaemon is to the page shortage. The code shows: int vmd_oom_seq and it looks like fairly large values would be tolerated. You may be able to scale beyond the problem showing up in your context. > pageout=20 > batchqueue > slow_swap > iosched For my new Pine64+ 2GB experiments I've only applied the Mark J. reporting patches, not the #define one. Nor have I involved CAM_IOSCHED_DYNAMIC. But with 2 GiBytes of RAM and the default 12 for vm.pageout_oom_seq I got: v_free_count: 7773, v_inactive_count: 1 Aug 12 09:30:13 pine64 kernel: pid 80573 (c++), uid 0, was killed: out = of swap space with no other reports from Mark Johnston's reporting patches. It appears that long I/O latencies as seen by the subsystem are not necessary to ending up with OOM kills, even if they can contribute when they occur. (7773 * 4 KiBytes =3D 31,838,298 Bytes, by the way.) > My RPI3 is now updating to 337688 with no patches/config changes. I'll = start the > sequence over and would be grateful if anybody could suggest a better = sequence. Side note: more text from -r290920 : In future, some heuristic to calculate the value of the tunable might be designed based on the system configuration and load. But before it can be done, the i/o system must be fixed to reliably time-out pagedaemon writes, even if waiting for the memory to proceed. Then, code can account for the in-flight page-outs and postpone OOM until all of them finished, which should reduce the need in tuning. Right now, ignoring the in-flight writes and the counter allows to break deadlocks due to write path doing sleepable memory allocations. I've no clue if this ever progressed after -r290920 . =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)