Date: Thu, 30 Aug 2018 10:13:41 -0700 From: Richard Elling <richard.elling@richardelling.com> To: openzfs-developer <developer@lists.open-zfs.org> Cc: Paul <devgs@ukr.net>, freebsd-fs@freebsd.org, developer@open-zfs.org Subject: Re: [developer] Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size Message-ID: <2AFA47E8-D069-418D-9302-1468329A969C@richardelling.com> In-Reply-To: <20180830144004.GD15740@raichu> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> <1535611885.624706680.h132ex07@frv33.fwdcdn.com> <20180830144004.GD15740@raichu>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Mark,=20 yes, this is the change I've tested on ZoL. It is a trivial, low-risk = change that is needed to restore the=20 previous behaviour. -- richard > On Aug 30, 2018, at 7:40 AM, Mark Johnston <markj@freebsd.org> wrote: >=20 > On Thu, Aug 30, 2018 at 09:55:27AM +0300, Paul wrote: >> 30 August 2018, 00:22:14, by "Mark Johnston" <markj@freebsd.org>: >>=20 >>> On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: >>>> Hello team, >>>>=20 >>>>=20 >>>> It seems like a commit on Mar 23 introduced a bug: if during = execution of arc_adjust() >>>> target is reached after MRU is evicted current code continues = evicting MFU. Before said >>>> commit, on the step prior to MFU eviction, target value was = recalculated as: >>>>=20 >>>> target =3D arc_size - arc_c; >>>>=20 >>>> arc_size here is a global variable that was being updated = accordingly, during MRU eviction, >>>> hence this expression, resulted in zero or negative target if MRU = eviction was enough >>>> to reach the original goal. >>>>=20 >>>> Modern version uses cached value of arc_size, called asize: >>>>=20 >>>> target =3D asize - arc_c; >>>>=20 >>>> Because asize stays constant during execution of whole body of = arc_adjust() it means that >>>> above expression will always be evaluated to value > 0, causing MFU = to be evicted every=20 >>>> time, even if MRU eviction has reached the goal already. Because of = the difference in=20 >>>> nature of MFU and MRU, globally it leads to eventual reduction of = amount of MFU in ARC=20 >>>> to dramatic numbers. >>>=20 >>> Hi Paul, >>>=20 >>> Your analysis does seem right to me. I cc'ed the openzfs mailing = list >>> so that an actual ZFS expert can chime in; it looks like this = behaviour >>> is consistent between FreeBSD, illumos and ZoL. >>>=20 >>> Have you already tried the obvious "fix" of subtracting = total_evicted >>> from the MFU target? >>=20 >> We are going to apply the asize patch (plus the ameta, as suggested = by Richard) and reboot=20 >> one of our production servers this night or the following. >=20 > Just to be explicit, are you testing something equivalent to the patch > at the end of this email? >=20 >> Then we have to wait a few days and observer the ARC behaviour. >=20 > Thanks! Please let us know how it goes: we're preparing to release > FreeBSD 12.0 shortly and I'd like to get this fixed in head/ as soon = as > possible. >=20 > diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c = b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > index 1387925c4607..882c04dba50a 100644 > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > @@ -4446,6 +4446,12 @@ arc_adjust(void) > arc_adjust_impl(arc_mru, 0, target, = ARC_BUFC_METADATA); > } >=20 > + /* > + * Re-sum ARC stats after the first round of evictions. > + */ > + asize =3D aggsum_value(&arc_size); > + ameta =3D aggsum_value(&arc_meta_used); > + > /* > * Adjust MFU size > * >=20 > ------------------------------------------ > openzfs: openzfs-developer > Permalink: = https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M1c45cd091= 14d2ce2e8c9dd26 = <https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M1c45cd09= 114d2ce2e8c9dd26> > Delivery options: = https://openzfs.topicbox.com/groups/developer/subscription = <https://openzfs.topicbox.com/groups/developer/subscription>;
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?2AFA47E8-D069-418D-9302-1468329A969C>