From owner-freebsd-arm@freebsd.org Sun Aug 12 22:40:14 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 882DC1055F53 for ; Sun, 12 Aug 2018 22:40:14 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E45EE7D535 for ; Sun, 12 Aug 2018 22:40:13 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id w7CMeMIc046486 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sun, 12 Aug 2018 15:40:23 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id w7CMeMki046485; Sun, 12 Aug 2018 15:40:22 -0700 (PDT) (envelope-from fbsd) Date: Sun, 12 Aug 2018 15:40:21 -0700 From: bob prohaska To: John Kennedy Cc: freebsd-arm , fbssd@www.zefox.net Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] Message-ID: <20180812224021.GA46372@www.zefox.net> References: <20180802002841.GB99523@www.zefox.net> <20180802015135.GC99523@www.zefox.net> <20180806155837.GA6277@raichu> <20180808153800.GF26133@www.zefox.net> <20180808204841.GA19379@raichu> <2DC1A479-92A0-48E6-9245-3FF5CFD89DEF@yahoo.com> <20180809033735.GJ30738@phouka1.phouka.net> <20180809175802.GA32974@www.zefox.net> <20180812173248.GA81324@phouka1.phouka.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20180812173248.GA81324@phouka1.phouka.net> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 12 Aug 2018 22:40:14 -0000 On Sun, Aug 12, 2018 at 10:32:48AM -0700, John Kennedy wrote: > On Thu, Aug 09, 2018 at 10:58:02AM -0700, bob prohaska wrote: > > ... I've gathered up what I've been given and put it in > > http://www.zefox.net/~fbsd/rpi3/swaptests/tools/ > > in hopes it'll be useful. ... > > Bob, are you doing anything clever to track/graph resources over time? > The closest thing to clever is the logging script, started by Mark M., containing now #!/bin/sh while true do gstat -abd -I 10s ; date ; swapinfo ; tail -n 2 /var/log/messages done Output goes to a file, and sort can we used to sift out values of interest, for example sort -n -u -k 5 swapscript.log > readdelay.sort will push the biggest ms/read values to the end of the file, where they can be examined. Things like total swap can be fished out in a similar way, with some cruft in the output. Then the values of interest can be located in the original file using a browser's find feature, to see them in context. Error messages can be found the same way, though the presistence of stale messages can clutter the view signficantly. Timestamps, if present, are usually a better way to match console errors to the log file. The script surely isn't "lightweight", but in my case the crashes came before the script and haven't changed much since it arrived. Still, you make a good point and I should do a test occasionally to see if the script contributes to the crashes. I don't think the script has ever been killed by OOMA. > I've got my box patched up to 12.0-ALPHA1 (r337589+2a27f5c34b07) + patches, and > haven't seen any logged output (assuming it'll end up in /var/log/messages) > although I'm certainly exercising things. I typically beat things up with > a buildworld-then-buildkernel. > > Patches I've applied: > http://www.zefox.net/~fbsd/rpi3/swaptests/tools/pageout.patch > http://www.zefox.net/~fbsd/rpi3/swaptests/tools/slow_swap.patch > > I haven't used the CAM_IOSCHED_DYNAMIC option or adjusted vm.pageout_oom_seq. > Setting vm.pageout_oom_seq to 120 made a decisive improvement, almost allowing buildworld to finish. By the time I tried CAM_IOSCHED_DYNAMIC buildworld was getting only about half as far, so it seems the patches were harmful to a degree. Changes were applied in the order pageout batchqueue slow_swap iosched My RPI3 is now updating to 337688 with no patches/config changes. I'll start the sequence over and would be grateful if anybody could suggest a better sequence. Thanks for reading, bob prohaska