From owner-freebsd-arm@freebsd.org Mon Aug 6 01:43:51 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F9441065BA2 for ; Mon, 6 Aug 2018 01:43:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.ne1.yahoo.com (sonic314-21.consmr.mail.ne1.yahoo.com [66.163.189.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 94EC181470 for ; Mon, 6 Aug 2018 01:43:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: VVlMsmAVM1k5YIsL4zmk_XoQXvQUUWCUkaBReByQfdoAktjmSDHDtwhzG1lZMrU QKBfvnjdY.i1Z1X.aPYcSszdp.6MOgRUT5kETzk_BfXgFlqvvc.uwD_7j41hzXNyPLPhPpM_pVfR GosDNysxAMLIyWoWW2_cdGzZqA3B2yHefJvxZRAlHfZKsf1mD27rlsbhfN8WrYF79CVra1tfXoGJ _.lCWsv5BiD3L7iQROT4NxhT2saO_sG3Uzsw88LVGEUsV.e3gEPDVdoS2mP8ZDEEB6Necl48jTu3 kguvTktWpg1Cp41c_F9zlW5YxOAfQhrhR1mh34Y50LBVdX4HIYkSfrnWjimK7K0T9TwN0MUCgv0f TagYLFzKfZ0ehYJu1Qhz5isR5fVGd37gsbugRk1sw_uUITJxT3ChvF9dzsMhmzagHolZZUh5zAVU 9OrQ6Vo68wzZ9XiaCJkUQHsNYo5LjictdXLqOpHSvzCbUgWhJrtF0im3s9lN8xFcbxC51hCIvf3w oa.FeB9hEjRYZXq3v_Am0zgarifSdtXqaizMTKfR52rBr_L7hUktnVYNzzj1ZWmNeFobVUMm4E5L E_6xL.22SoaD0UtOr21ua4pLULt7s3HQtmd3mdy4FumkrKPezT5BM0VxbuF8QN.aTEspZOq76arz 2USx.eOJtOsVhLg5oP0G4x77foKejjBr9GOiTbqY0oHw0ATY4NF66SZGwh0gkZrmAvgxg1OCF09Y c4RLrxXdNR0KkoeR06H4Q_cRVM5VCoKfWaTe1qhpLKM.qJE4PbDIBzgdkJP3BUhBU1u.bwcE24.v 2dYHc3XYd0cEmN.yxxaadsNV28knenpItTy9L3tTgS8GMTXJ1bRiK_z.YJqQYkbJc32CwuJD3e_m TQv4vOh8DA7oqvSrOl7O0ofFs9LfOGWPIuy38qhMvZobMksyYus1pBRIMVY6DTDhyVch417MXDqY vbpWm6wrpd6gv5v67eDSG4T7eZ_JZm43fu2vP0OvBIp04zgo2sWK1sX.1BoyMW.pDEwcgudL7no8 xx4nqFAtbbFbb Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.ne1.yahoo.com with HTTP; Mon, 6 Aug 2018 01:43:49 +0000 Received: from ip70-189-131-151.lv.lv.cox.net (EHLO [192.168.0.105]) ([70.189.131.151]) by smtp424.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 8d322980fb85c58131395b33268e8a82; Mon, 06 Aug 2018 01:43:48 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] From: Mark Millard In-Reply-To: <20180805231508.GA12753@www.zefox.net> Date: Sun, 5 Aug 2018 18:43:46 -0700 Cc: John-Mark Gurney , markj@freebsd.org, Warner Losh , Jamie Landeg-Jones , freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: References: <20180802015135.GC99523@www.zefox.net> <201808030034.w730YURL034270@donotpassgo.dyslexicfish.net> <201808040355.w743tPsF039729@donotpassgo.dyslexicfish.net> <8CC5DF53-F950-495C-9DC8-56FCA0087259@yahoo.com> <20180804140816.GJ2884@funkthat.com> <16ABD9F0-C908-479C-960D-0C1AEDE89053@yahoo.com> <20180805014545.GK2884@funkthat.com> <2C0383B0-3292-48BE-90BC-24FCF2A137D4@yahoo.com> <20180805231508.GA12753@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Aug 2018 01:43:51 -0000 On 2018-Aug-5, at 4:15 PM, bob prohaska wrote: > On Sun, Aug 05, 2018 at 10:39:27AM -0700, Mark Millard wrote: >>=20 >> "Total Active Working Set too large" (with lots of swap left) seems = to be what folks >> are running into in these rpi3/rpi2 examples. >>=20 >>=20 > Does the size of the working set vary with how the swap is apportioned = between > storage devices? In my observations, 1 GB plus 2 GB swap partitions = both on microSD=20 > allows a -j4 buildworld to run to completion without OOMA = intervention. It looks as > if both partitions are burdened equally, so usable swap is actually 2 = GB. The less memory in use for other things (especially less wired memory), the more for the working set. The overallocation of swap beyond the maximum recommended might have some contribution to how much the working set can be. But I've no clue of any specific tradeoffs. Describing 2 or 3 in-use swap partitions instead of just one might. But, again, I've no clue of any specific tradeoffs. > A pair of 1 GB swap partitions, one on microSD and one on USB flash, = invites OOMA=20 > to intervene prematurely. As I remember you have examples that involve only one device and, separately, you have some problematic devices. Others seem to have avoided some of the complications: They got the problem in simpler contexts. > Stranger still, 1 GB of swap on microSD, which is insufficient (~1.4 = GB required),=20 > does not trigger OOMA at all, resulting in a hung system. Part of the problem is that messages like: (taken from a John Kennedy message) Aug 5 01:34:24 rpi3 kernel: pid 63223 (ld.lld), uid 0, was = killed: out of swap space Aug 5 01:34:26 rpi3 kernel: pid 63360 (c++), uid 0, was killed: = out of swap space Aug 5 01:34:26 rpi3 kernel: pid 846 (ntpd), uid 123, was = killed: out of swap space are misleading: the issue is not necessarily "out of swap space" but what the book described (when there is lots of swap around): QUOTE: The FreeBSD swap-out daemon will not select a runnable processes to swap out. So, if the set of runnable processes do not fit in memory, the machine will effectively deadlock. Current machines have enough memory that this condition usually does not arise. If it does, FreeBSD avoids deadlock by killing the largest process. If the condition begins to = arise in normal operation, the 4.4BSD algorithm will need to be restored. END QUOTE. (The 4.4BSD algorithm is not in use [or even present].) The truly "out of swap space" context is explicitly different code doing different things (to some extent). Trev has reported that a message that can (sometimes?) be produced for this is like: Aug 5 17:54:01 sentinel kernel: swap_pager_getswapspace(32): failed (at least if it does not hang up first). "was killed: out of swap space" without prior "swap_pager_getswapspace(??): failed" notices look to not be trusted for the "out of" wording or its implications. > The RPi3 right now has three swap partitions; one of 1 GB on microSD, = one > of 2 GB on microSD and one of 1 GB on USB. Might it shed any light to = try=20 > sysctl vm.swap_idle_enabled=3D1 > just to see if anything changes? My guess is no, but it wouldn't be = the first=20 > time if I'm wrong. While I have expectation based on my understanding of what I've read (including some code) and seen in top, I think you should do such experiments if you are interested in taking the time. (The same goes for others with simpler contexts.) Doing so even for the "single device" type of context that showed the problem likely would be good to do as well. You might even want to see how it goes for a context that worked. ("Does it then still work?" is also good to know.) I continue to encourage you to use total swap space figures that are somewhat under what that message happens to be listing as the maximum at the time (if you are getting such messages). (You might well be doing so. I've not been monitoring for this.) Being "over the maximum recommended" just seems to be a complication of the context that is fairly easily avoided. There is already a lot involved. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)