Date: Fri, 29 Aug 2014 09:32:47 +0100 From: "Steven Hartland" <smh@freebsd.org> To: "Peter Wemm" <peter@wemm.org>, "Alan Cox" <alc@rice.edu> Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org, Dmitry Morozovsky <marck@rinet.ru>, "Matthew D. Fuller" <fullermd@over-yonder.net> Subject: Re: svn commit: r270759 - in head/sys: cddl/compat/opensolaris/kern cddl/compat/opensolaris/sys cddl/contrib/opensolaris/uts/common/fs/zfs vm Message-ID: <4A4B2C2D36064FD9840E3603D39E58E0@multiplay.co.uk> References: <201408281950.s7SJo90I047213@svn.freebsd.org> <20140828211508.GK46031@over-yonder.net> <53FFAD79.7070106@rice.edu> <1617817.cOUOX4x8n2@overcee.wemm.org>
next in thread | previous in thread | raw e-mail | index | archive | help
> On Thursday 28 August 2014 17:30:17 Alan Cox wrote:
> > On 08/28/2014 16:15, Matthew D. Fuller wrote:
> > > On Thu, Aug 28, 2014 at 10:11:39PM +0100 I heard the voice of
> > >
> > > Steven Hartland, and lo! it spake thus:
> > >> Its very likely applicable to stable/9 although I've never used 9
> > >> myself, we jumped from 9 direct to 10.
> > >
> > > This is actually hitting two different issues from the two bugs:
> > >
> > > - 191510 is about "ARC isn't greedy enough" on huge-memory
> > > machines,
> > >
> > > and from the osreldate that bug was filed on 9.2, so presumably
> > > is
> > > applicable.
> > >
> > > - 187594 is about "ARC is too greedy" (probably mostly on
> > > not-so-huge
> > >
> > > machines) and starves/drives the rest of the system into swap.
> > > That
> > > I believe came about as a result of some unrelated change in the
> > > 10.x stream that upset the previous balance between ARC and the
> > > rest
> > > of the VM, so isn't a problem on 9.x.
> >
> > 10.0 had a bug in the page daemon that was fixed in 10-STABLE about
> > three months ago (r265945). The ARC was not the only thing affected
> > by
> this bug.
>
> I'm concerned about potential unintended consequences of this change.
>
> Before, arc reclaim was driven by vm_paging_needed(), which was:
> vm_paging_needed(void)
> {
> return (vm_cnt.v_free_count + vm_cnt.v_cache_count <
> vm_pageout_wakeup_thresh);
> }
>
> Now it's ignoring the v_cache_count and looking exclusively at
> v_free_count.
> "cache" pages are free pages that just happen to have known contents.
> If I
> read this change right, zfs arc will now discard checksummed cache
> pages to
> make room for non-checksummed pages:
That test is still there so if it needs to it will still trigger.
However that often a lower level as vm_pageout_wakeup_thresh is only
110%
of min free, where as zfs_arc_free_target is based of target free which
is
4 * (min free + reserved).
> + if (kmem_free_count() < zfs_arc_free_target) {
> + return (1);
> + }
> ...
> +kmem_free_count(void)
> +{
> + return (vm_cnt.v_free_count);
> +}
>
> This seems like a pretty substantial behavior change. I'm concerned
> that it
> doesn't appear to count all the forms of "free" pages.
>
> I haven't seen the problems with the over-aggressive ARC since the
> page daemon
> bug was fixed. It's been working fine under pretty abusive loads in
> the freebsd
> cluster after that fix.
Others have also confirmed that even with r265945 they can still trigger
performance issue.
In addition without it we still have loads of RAM sat their unused, in
my
particular experience we have 40GB of 192GB sitting their unused and
that
was with a stable build from last weekend.
With the patch we confirmed that both RAM usage and performance for
those
seeing that issue are resolved, with no reported regressions.
> (I should know better than to fire a reply off before full fact
> checking, but
> this commit worries me..)
Not a problem, its great to know people pay attention to changes, and
raise
their concerns. Always better to have a discussion about potential
issues
than to wait for a problem to occur.
Hopefully the above gives you some piece of mind, but if you still have
any
concerns I'm all ears.
Out of interest would it be possible to update machines in the cluster
to
see how their workload reacts to the change?
Regards
Steve
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4A4B2C2D36064FD9840E3603D39E58E0>