From owner-freebsd-arm@freebsd.org Sun Mar 4 18:46:25 2018 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9C84F26C3F for ; Sun, 4 Mar 2018 18:46:25 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound1a.eu.mailhop.org (outbound1a.eu.mailhop.org [52.58.109.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5380869306 for ; Sun, 4 Mar 2018 18:46:25 +0000 (UTC) (envelope-from ian@freebsd.org) X-MHO-User: 5150f898-1fdc-11e8-91c6-33ffc249f3e8 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound1.eu.mailhop.org (Halon) with ESMTPSA id 5150f898-1fdc-11e8-91c6-33ffc249f3e8; Sun, 04 Mar 2018 18:46:17 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id w24IkBhx093540; Sun, 4 Mar 2018 11:46:11 -0700 (MST) (envelope-from ian@freebsd.org) Message-ID: <1520189171.38056.2.camel@freebsd.org> Subject: Re: Is maximum swap usage tunable? From: Ian Lepore To: Warner Losh , bob prohaska Cc: "freebsd-arm@freebsd.org" Date: Sun, 04 Mar 2018 11:46:11 -0700 In-Reply-To: References: <20180228170311.GA26187@www.zefox.net> <20180228185517.GB26187@www.zefox.net> <8f422161-885e-aa91-eacd-018540222d65@mgm51.com> <20180228214301.GA29481@www.zefox.net> <20180303162605.GA41874@www.zefox.net> <20180304182831.GA44154@www.zefox.net> Content-Type: text/plain; charset="ISO-8859-1" X-Mailer: Evolution 3.18.5.1 FreeBSD GNOME Team Port Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Mar 2018 18:46:26 -0000 On Sun, 2018-03-04 at 11:35 -0700, Warner Losh wrote: > On Sun, Mar 4, 2018 at 11:28 AM, bob prohaska wrote: > > > > > On Sat, Mar 03, 2018 at 08:26:05AM -0800, bob prohaska wrote: > > > > > > > > > Is there some sort of experiment which can distinguish hardware delays > > > from software delays? For example, would logging gstat output shed any > > > light? > > > > > For lack of any better ideas, I tried running > > make -j2 -DNO_CLEAN buildworld > buildworld.log && make -j2 -DNO_CLEAN > > KERNCONF= > > ZEFOX buildkernel > buildkernel.log > > > > while also running > > gstat -a -B -I 10s > j2_gstat.log & in another ssh session > > > > In due course the console reported > > > > FreeBSD/arm64 (www.zefox.org) (ttyu0) > > > > login: Mar  4 09:28:30 www kernel: pid 9310 (c++), uid 0, was killed: out > > of swap space > > > > as expected. > > > > However, a grep of j2_gstat  revealed a maximum write delay of 30ms/w > > for swap on microSD. > > > > Swap on USB flash is slower, but still generally under 100 ms. > > Only a handful of widely spaced delays exceeded 200 ms/w. > > > gstat doesn't tell you the worst-case. It tells you the average of the > requests. This can (and does) hide very long outlier behavior which drives > the crazy delayed swap messages. > > The worst-case events were > > > > dT: 10.002s  w: 10.000s > >  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name > > > >     0      6      0      0    0.0      6    113   14.6    3.3  da0b > >     0      4      0      0    0.0      4     48   29.0    3.1  da0b > >     0      9      5     79    3.0      5     47    7.9    2.6  da0b > >     4      8      0      0    0.0      8     99   67.5   32.5  da0b > >     0      1      0     13    5.6      1     28   5674   88.3  da0b > >     0      0      0      0    0.0      0     38   18.6    0.3  da0b > >     0      1      1      9    2.8      0      0    0.0    0.2  da0b > >     0      1      1     26    5.1      0      0    0.0    0.4  da0b > >     0      0      0      3    2.6      0      0    0.0    0.1  da0b > >     0      1      1      9  161.8      0      0    0.0   14.3  da0b > > > > No "indefinite delay" warnings were presented on the console. > > uname -a reports r329893, sources are at 330383. > > > > I hope this is useful information, > > > Despite my quibble over what you're measuring (I look at this stuff all the > time for Netflix), I think this is quite useful.... Thanks! > I think it would be more useful with a -I1 instead of 10.  Including the -d flag might also be useful, since deleting is what's so slow on flash-based devices (I'm not sure if swapping triggers BIO_DELETE stuff, but other filesystem activity does). Also, I see a 5-second write latency there at one point.  -- Ian