Date: Sun, 4 Mar 2018 13:40:03 -0800 From: bob prohaska <fbsd@www.zefox.net> To: Ian Lepore <ian@freebsd.org> Cc: Warner Losh <imp@bsdimp.com>, "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org>, bob prohaska <fbsd@www.zefox.net> Subject: Re: Is maximum swap usage tunable? Message-ID: <20180304214003.GB44154@www.zefox.net> In-Reply-To: <1520189171.38056.2.camel@freebsd.org> References: <a759ecea-83f4-b0b2-7113-c39633f68637@mgm51.com> <20180228185517.GB26187@www.zefox.net> <8f422161-885e-aa91-eacd-018540222d65@mgm51.com> <20180228214301.GA29481@www.zefox.net> <b82801b8-bc29-414c-1170-621bb4a5d937@mgm51.com> <CANCZdfq_MpxjUyVd-=%2BMiAQAER4TeDh9irhx_evdXwa3yt3h0g@mail.gmail.com> <20180303162605.GA41874@www.zefox.net> <20180304182831.GA44154@www.zefox.net> <CANCZdfpcccDuhJh4EF0C4KFz=dcN5mWvQ9aMiyF53kD=hMzy3Q@mail.gmail.com> <1520189171.38056.2.camel@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Mar 04, 2018 at 11:46:11AM -0700, Ian Lepore wrote: > On Sun, 2018-03-04 at 11:35 -0700, Warner Losh wrote: > > On Sun, Mar 4, 2018 at 11:28 AM, bob prohaska <fbsd@www.zefox.net> wrote: > > > > > > > > On Sat, Mar 03, 2018 at 08:26:05AM -0800, bob prohaska wrote: > > > > > > > > > > > > Is there some sort of experiment which can distinguish hardware delays > > > > from software delays? For example, would logging gstat output shed any > > > > light? > > > > > > > For lack of any better ideas, I tried running > > > make -j2 -DNO_CLEAN buildworld > buildworld.log && make -j2 -DNO_CLEAN > > > KERNCONF= > > > ZEFOX buildkernel > buildkernel.log > > > > > > while also running > > > gstat -a -B -I 10s > j2_gstat.log & in another ssh session > > > > > > In due course the console reported > > > > > > FreeBSD/arm64 (www.zefox.org) (ttyu0) > > > > > > login: Mar??4 09:28:30 www kernel: pid 9310 (c++), uid 0, was killed: out > > > of swap space > > > > > > as expected. > > > > > > However, a grep of j2_gstat??revealed a maximum write delay of 30ms/w > > > for swap on microSD. > > > > > > Swap on USB flash is slower, but still generally under 100 ms. > > > Only a handful of widely spaced delays exceeded 200 ms/w. > > > > > gstat doesn't tell you the worst-case. It tells you the average of the > > requests. This can (and does) hide very long outlier behavior which drives > > the crazy delayed swap messages. > > > > The worst-case events were > > > > > > dT: 10.002s??w: 10.000s > > > ?L(q)??ops/s????r/s???kBps???ms/r????w/s???kBps???ms/w???%busy Name > > > > > > ????0??????6??????0??????0????0.0??????6????113???14.6????3.3??da0b > > > ????0??????4??????0??????0????0.0??????4?????48???29.0????3.1??da0b > > > ????0??????9??????5?????79????3.0??????5?????47????7.9????2.6??da0b > > > ????4??????8??????0??????0????0.0??????8?????99???67.5???32.5??da0b > > > ????0??????1??????0?????13????5.6??????1?????28???5674???88.3??da0b > > > ????0??????0??????0??????0????0.0??????0?????38???18.6????0.3??da0b > > > ????0??????1??????1??????9????2.8??????0??????0????0.0????0.2??da0b > > > ????0??????1??????1?????26????5.1??????0??????0????0.0????0.4??da0b > > > ????0??????0??????0??????3????2.6??????0??????0????0.0????0.1??da0b > > > ????0??????1??????1??????9??161.8??????0??????0????0.0???14.3??da0b > > > [where did all the question marks come from?] > > > No "indefinite delay" warnings were presented on the console. > > > uname -a reports r329893, sources are at 330383. > > > > > > I hope this is useful information, > > > > > Despite my quibble over what you're measuring (I look at this stuff all the > > time for Netflix), I think this is quite useful.... Thanks! > > > > I think it would be more useful with a -I1 instead of 10. ?Including > the -d flag might also be useful, since deleting is what's so slow on > flash-based devices (I'm not sure if swapping triggers BIO_DELETE > stuff, but other filesystem activity does). > A few minutes before reading your reply a make -j4 buildworld was started with the usb-hosted swap partition turned off, leaving only the microSD-hosted swap running. It got past the first cycle of llvm-tblgen showing only half of the one GB available swap in use, with no errors or warnings. There is some decent chance it'll run to completion, probably late tomorrow. For now I'll just see how far it gets. > Also, I see a 5-second write latency there at one point.? > Yes, just one. Are there perhaps more specific monitoring tests that could distinguish between swap-related, flash-related and usb-related sources of delay? Armv7 seems to have some USB debugging options in the kernel config, is there something similar for arm64? I looked but didn't see any. Thanks for reading, and any guidance! bob prohaska
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180304214003.GB44154>