From owner-freebsd-fs@freebsd.org Fri Aug 31 07:05:20 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CCD3F75DC8 for ; Fri, 31 Aug 2018 07:05:20 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv196.fwdcdn.com (frv196.fwdcdn.com [212.42.77.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D2FD185211 for ; Fri, 31 Aug 2018 07:05:19 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-Id:Cc:To:Subject:From:Date:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=dV5AD2cO8sIY6y2VnOwBI0VYdJgusN1EpltFKgiLM/k=; b=cDuSHsqcQE8rr73/LsqD7pKcwa k2mRSVfSWr9ndtGt79vN8wKtZUD/LlasrUawus7rnWp64Fe4fW8PFMcLKYEaFfFtcr5vCH9Jll/rP dJ5w3wKMVSjp8l+X3f5genqifM52CjSWt3JJMbQVHr5HvKU0T3EJVciwEQMkvgre8lhE=; Received: from [10.10.10.33] (helo=frv33.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1fvdUU-000G2q-Pa for freebsd-fs@freebsd.org; Fri, 31 Aug 2018 10:05:10 +0300 Date: Fri, 31 Aug 2018 10:05:10 +0300 From: Paul Subject: Re[2]: [developer] Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size To: Richard Elling Cc: openzfs-developer , freebsd-fs@freebsd.org, developer@open-zfs.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1535697386.721828038.3yjli8h7@frv33.fwdcdn.com> In-Reply-To: <2AFA47E8-D069-418D-9302-1468329A969C@richardelling.com> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> <1535611885.624706680.h132ex07@frv33.fwdcdn.com> <20180830144004.GD15740@raichu> <2AFA47E8-D069-418D-9302-1468329A969C@richardelling.com> X-Reply-Action: reply Received: from devgs@ukr.net by frv33.fwdcdn.com; Fri, 31 Aug 2018 10:05:10 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Aug 2018 07:05:20 -0000 I can confirm that the old behaviour is restored. Though it's only 16 hours in we can see a stable, healthy grown of MFU, unlike before the patch. Most importantly the MRU is now pushed below the arc_p limit, that was never the case before patch. Just as an example, the current ARC picture is as such: ARC: 360G Total, 179G MFU, 158G MRU, 361M Anon, 3398M Header, 20G Other And yes, our version of patch was exactly what Mark Johnston has assumed. 30 August 2018, 20:13:47, by "Richard Elling" : > Hi Mark, > yes, this is the change I've tested on ZoL. It is a trivial, low-risk change that is needed to restore the > previous behaviour. -- richard > > > On Aug 30, 2018, at 7:40 AM, Mark Johnston wrote: > > On Thu, Aug 30, 2018 at 09:55:27AM +0300, Paul wrote: > > > 30 August 2018, 00:22:14, by "Mark Johnston" : > > > > > > > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > > > > > Hello team, > > > > > > > > > > > > > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > > > > > target is reached after MRU is evicted current code continues evicting MFU. Before said > > > > > commit, on the step prior to MFU eviction, target value was recalculated as: > > > > > > > > > > target = arc_size - arc_c; > > > > > > > > > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > > > > > hence this expression, resulted in zero or negative target if MRU eviction was enough > > > > > to reach the original goal. > > > > > > > > > > Modern version uses cached value of arc_size, called asize: > > > > > > > > > > target = asize - arc_c; > > > > > > > > > > Because asize stays constant during execution of whole body of arc_adjust() it means that > > > > > above expression will always be evaluated to value > 0, causing MFU to be evicted every > > > > > time, even if MRU eviction has reached the goal already. Because of the difference in > > > > > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > > > > > to dramatic numbers. > > > > > > > > Hi Paul, > > > > > > > > Your analysis does seem right to me. I cc'ed the openzfs mailing list > > > > so that an actual ZFS expert can chime in; it looks like this behaviour > > > > is consistent between FreeBSD, illumos and ZoL. > > > > > > > > Have you already tried the obvious "fix" of subtracting total_evicted > > > > from the MFU target? > > > > > > We are going to apply the asize patch (plus the ameta, as suggested by Richard) and reboot > > > one of our production servers this night or the following. > > > > Just to be explicit, are you testing something equivalent to the patch > > at the end of this email? > > > > > Then we have to wait a few days and observer the ARC behaviour. > > > > Thanks! Please let us know how it goes: we're preparing to release > > FreeBSD 12.0 shortly and I'd like to get this fixed in head/ as soon as > > possible. > > > > diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > > index 1387925c4607..882c04dba50a 100644 > > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > > @@ -4446,6 +4446,12 @@ arc_adjust(void) > > arc_adjust_impl(arc_mru, 0, target, ARC_BUFC_METADATA); > > } > > > > + /* > > + * Re-sum ARC stats after the first round of evictions. > > + */ > > + asize = aggsum_value(&arc_size); > > + ameta = aggsum_value(&arc_meta_used); > > + > > /* > > * Adjust MFU size > > * > > > > ------------------------------------------ > > openzfs: openzfs-developer > > Permalink: https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M1c45cd09114d2ce2e8c9dd26 > > Delivery options: https://openzfs.topicbox.com/groups/developer/subscription