From owner-freebsd-arm@freebsd.org Wed Aug 8 22:42:20 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 96278106E1D6 for ; Wed, 8 Aug 2018 22:42:20 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (www.zefox.net [50.1.20.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.zefox.org", Issuer "www.zefox.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0226E7038B; Wed, 8 Aug 2018 22:42:19 +0000 (UTC) (envelope-from fbsd@www.zefox.net) Received: from www.zefox.net (localhost [127.0.0.1]) by www.zefox.net (8.15.2/8.15.2) with ESMTPS id w78MgScY029994 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 8 Aug 2018 15:42:29 -0700 (PDT) (envelope-from fbsd@www.zefox.net) Received: (from fbsd@localhost) by www.zefox.net (8.15.2/8.15.2/Submit) id w78MgSvb029993; Wed, 8 Aug 2018 15:42:28 -0700 (PDT) (envelope-from fbsd) Date: Wed, 8 Aug 2018 15:42:27 -0700 From: bob prohaska To: Warner Losh Cc: Mark Johnston , "freebsd-arm@freebsd.org" , bob prohaska Subject: Re: RPI3 swap experiments ["was killed: out of swap space" with: "v_free_count: 5439, v_inactive_count: 1"] Message-ID: <20180808224227.GB29312@www.zefox.net> References: <2222ABBD-E689-4C3B-A7D3-50AECCC5E7B2@yahoo.com> <20180801034511.GA96616@www.zefox.net> <201808010405.w7145RS6086730@donotpassgo.dyslexicfish.net> <6BFE7B77-A0E2-4FAF-9C68-81951D2F6627@yahoo.com> <20180802002841.GB99523@www.zefox.net> <20180802015135.GC99523@www.zefox.net> <20180806155837.GA6277@raichu> <20180808214224.GA29312@www.zefox.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Aug 2018 22:42:20 -0000 On Wed, Aug 08, 2018 at 03:55:52PM -0600, Warner Losh wrote: > On Wed, Aug 8, 2018 at 3:42 PM, bob prohaska wrote: > > > On Mon, Aug 06, 2018 at 11:58:37AM -0400, Mark Johnston wrote: > > > > > > If the above suggestion doesn't help, the next thing to try would be to > > > revert the oom_seq value to the default, apply this patch, and see if > > > the problem continues to occur. If this doesn't help, please try > > > applying both measures, i.e., set oom_seq to 120 _and_ apply the patch. > > > > > > > Applying both measures resulted in a panic, not entirely unlike the effect > > of deliberately running the machine out of swap on microSD. The first > > obvious > > sign of trouble was errors attributed to da0: > > > > (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 25 bc b8 00 00 80 00 > > (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error > > (da0:umass-sim0:0:0:0): Retrying command, 3 more tries remain > > > > So we're pushing so hard that the writes are actively failing... > > With the da driver there's some hope. Add options IOSCHED to the kernel > config file and reboot. This will give you some detailed statistics, as > well as power-of-two bucketized latency histograms. It may even be a vector > forward to slow the writes / trims down, though there's some issues when > you slow writes down TOO much, it helps *A*LOT* keep the system responsive. > We do that at work to make our consumer SSDs not suck for serving content > (reading) while we're doing some writes to them... The thumb drives are > like the consumer SSDs we buy, only crappier... > I've added some sorted versions of the swapscript output keyed on ms/read and ms/write to the directory at http://www.zefox.net/~fbsd/rpi3/swaptests/r337226M/1gbsdflash_1gbusbflash/batchqueue/pageout120/ and randomly through the older testcases. Some of the delays approach a minute, and that's been the case for as long as I've been keeping track. However, the big delays tend to be scattered through the gstat log and aren't necessarily associated with trouble, nor always with high levels of swap usage. Mark Johnston's slow_swap.diff patch is compiling now, if that succeeds I'll try it first. If it fails (I may have damaged /usr/obj in the last test run) I'll add options IOSCHED to the GENERIC config file before trying again. Thanks for reading! bob prohaska