From owner-freebsd-fs@freebsd.org Thu Aug 30 17:13:46 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9B0A910A451F for ; Thu, 30 Aug 2018 17:13:46 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0B4458A935 for ; Thu, 30 Aug 2018 17:13:46 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: by mail-pf1-x441.google.com with SMTP id u24-v6so4157406pfn.13 for ; Thu, 30 Aug 2018 10:13:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=richardelling.com; s=google; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=EGSGA8Ru8xsRnAULkIZALCQayEYQapw0SVxJNwOXmOo=; b=RkdJRt/6Y3xrxQPLrqWvf/vuI/4TXsGaQvvVYC84mmtjvgqyZFhh0cyZqQVkha+ebr CV/Yb2R44T0aLwqOr2Hvym8QmOPYyaMWaM3AnNxE57AcZKcWLnsLVf8z4VVCcI0/tkRA 7FbXzr32SdXWaj4Or7rhN3yg4+ckSVFFWyE6s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=EGSGA8Ru8xsRnAULkIZALCQayEYQapw0SVxJNwOXmOo=; b=FVtzcKafe+039TGayQLdwSrby1oJjzl3VfazBrm8FizE9aXvjiAwKFPo7Z0m4mcNbj gTIiX2hhAVagT9+OsgzWLbAtBwTE9xCRYobgnEF6BsmngSc89uccoiErJ0OxUhfOV/f+ rnMliXZngZb4R8G6s5rTKjZXyc+gKadFVksLOchJ10Ikm6hrAy9REDEFQkGw0DR9u7X8 GHOLslI/iSX6MbTLv43fjYAlDFQH/AwnBcfMH1b8hbLnz2gfFBVO5woVni2lviSxY6MR j230fTDmCvIAMcn3/lfu39QYdOKabRI6L/XMdLHTiemcSiAZU3DOhSQFYY3rbhHqYTyK /dmA== X-Gm-Message-State: APzg51Bu3Vgya7w+tJHsnDtryelFT/OKjAFekP/jCEIQ52nVqk9VqKlD cLfuVOxOCxUHsCN+rJLAdTWZc7DfE9Y= X-Google-Smtp-Source: ANB0VdbJiujtgZ9YSohffPdPwF6FaOh+bHi/PMolSeOSGko9jmDZLlbBvqLnM5y3EZwxoIaQei8fyA== X-Received: by 2002:a63:d09:: with SMTP id c9-v6mr10737930pgl.314.1535649224914; Thu, 30 Aug 2018 10:13:44 -0700 (PDT) Received: from [172.20.10.3] ([172.56.41.147]) by smtp.gmail.com with ESMTPSA id b18-v6sm15196060pgk.15.2018.08.30.10.13.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 10:13:44 -0700 (PDT) From: Richard Elling Message-Id: <2AFA47E8-D069-418D-9302-1468329A969C@richardelling.com> Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: [developer] Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size Date: Thu, 30 Aug 2018 10:13:41 -0700 In-Reply-To: <20180830144004.GD15740@raichu> Cc: Paul , freebsd-fs@freebsd.org, developer@open-zfs.org To: openzfs-developer References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> <1535611885.624706680.h132ex07@frv33.fwdcdn.com> <20180830144004.GD15740@raichu> X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 17:13:47 -0000 Hi Mark,=20 yes, this is the change I've tested on ZoL. It is a trivial, low-risk = change that is needed to restore the=20 previous behaviour. -- richard > On Aug 30, 2018, at 7:40 AM, Mark Johnston wrote: >=20 > On Thu, Aug 30, 2018 at 09:55:27AM +0300, Paul wrote: >> 30 August 2018, 00:22:14, by "Mark Johnston" : >>=20 >>> On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: >>>> Hello team, >>>>=20 >>>>=20 >>>> It seems like a commit on Mar 23 introduced a bug: if during = execution of arc_adjust() >>>> target is reached after MRU is evicted current code continues = evicting MFU. Before said >>>> commit, on the step prior to MFU eviction, target value was = recalculated as: >>>>=20 >>>> target =3D arc_size - arc_c; >>>>=20 >>>> arc_size here is a global variable that was being updated = accordingly, during MRU eviction, >>>> hence this expression, resulted in zero or negative target if MRU = eviction was enough >>>> to reach the original goal. >>>>=20 >>>> Modern version uses cached value of arc_size, called asize: >>>>=20 >>>> target =3D asize - arc_c; >>>>=20 >>>> Because asize stays constant during execution of whole body of = arc_adjust() it means that >>>> above expression will always be evaluated to value > 0, causing MFU = to be evicted every=20 >>>> time, even if MRU eviction has reached the goal already. Because of = the difference in=20 >>>> nature of MFU and MRU, globally it leads to eventual reduction of = amount of MFU in ARC=20 >>>> to dramatic numbers. >>>=20 >>> Hi Paul, >>>=20 >>> Your analysis does seem right to me. I cc'ed the openzfs mailing = list >>> so that an actual ZFS expert can chime in; it looks like this = behaviour >>> is consistent between FreeBSD, illumos and ZoL. >>>=20 >>> Have you already tried the obvious "fix" of subtracting = total_evicted >>> from the MFU target? >>=20 >> We are going to apply the asize patch (plus the ameta, as suggested = by Richard) and reboot=20 >> one of our production servers this night or the following. >=20 > Just to be explicit, are you testing something equivalent to the patch > at the end of this email? >=20 >> Then we have to wait a few days and observer the ARC behaviour. >=20 > Thanks! Please let us know how it goes: we're preparing to release > FreeBSD 12.0 shortly and I'd like to get this fixed in head/ as soon = as > possible. >=20 > diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c = b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > index 1387925c4607..882c04dba50a 100644 > --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c > @@ -4446,6 +4446,12 @@ arc_adjust(void) > arc_adjust_impl(arc_mru, 0, target, = ARC_BUFC_METADATA); > } >=20 > + /* > + * Re-sum ARC stats after the first round of evictions. > + */ > + asize =3D aggsum_value(&arc_size); > + ameta =3D aggsum_value(&arc_meta_used); > + > /* > * Adjust MFU size > * >=20 > ------------------------------------------ > openzfs: openzfs-developer > Permalink: = https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M1c45cd091= 14d2ce2e8c9dd26 = > Delivery options: = https://openzfs.topicbox.com/groups/developer/subscription =