From owner-freebsd-fs@freebsd.org Thu Aug 30 14:40:10 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0982010A05A8 for ; Thu, 30 Aug 2018 14:40:10 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-pf1-x444.google.com (mail-pf1-x444.google.com [IPv6:2607:f8b0:4864:20::444]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7C43883E4F for ; Thu, 30 Aug 2018 14:40:09 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-pf1-x444.google.com with SMTP id k21-v6so3966045pff.11 for ; Thu, 30 Aug 2018 07:40:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=jOeYtncPj+q7rD29er3ysomxyCDKrHmZNmoj97Ukbg8=; b=XpEzblbtfTajM6wW21hAKrM9uN6QQOQERebGqYnJgV1YR9r11wy3AN1GBdQaxqwqpJ kDzVIcx+qajViIk8nJpUv7eRnUAz3sKiFR+YJdumSgyrlD9t78yP12BmfTK9C1y5wY2p aCFOnQM5X2P6lytAXPNSf8tUCtCgW2YR59CAo8qck2XA8npqItBY2tEGNVLXd6361Tu9 5QDMW92MmHyH4N3aLrsS2PskrTlXWSoD7Fa05y/8PLsbGYkNoaZ9UDzOfQI9DIWHdiCu b3HHLYK/tGJKAfB2bNfOKz8z6/r7kofG5YgAi3vOoALxwNpTv9EAjZwfnG6TcubIhEOX AC5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to:user-agent; bh=jOeYtncPj+q7rD29er3ysomxyCDKrHmZNmoj97Ukbg8=; b=hU3TrfcEI0wWcIJs5I5PWVScJHSbLLfXH4FEh8lkmSaLKuuaTqGiFmoxJ4UHsN46Hr SVneO8Kp1K9u8YVl0+VSOvcdNSW+9OrwqOKT6f2MKdbb4DUCV5D919zK9+xBmo+CNpdb n9bbCzfzbRuDjKZyZkYnReHlXAMOysWvcqvrwqIz0ALThnVkW3jssop3L5KNa1iiXMX1 IHkcQLysCYpvjSZWGayr1+0dz2JeHnnbAOMCmINAoMkVH1fZ9PVN4lQPHGyIjtJUvBz/ WbZxMNZ6Ty6zxgal4X2hK2opUm72t/G3kzM5ZVzTHzYshalfsACImdgDpJfKZnbj9RHm Ndcw== X-Gm-Message-State: APzg51ByHqZGIiRd+lK5d59c6SW95KD6DnHDy0zKUvLCuwCFlMMwZ7uB bS0vc4sqj7eRi8mkUhhN0hM= X-Google-Smtp-Source: ANB0VdZszNhjqsFqlKgpKX9QV5diz1a1LJgyFtwV2DEDm+xqUE+q9CVVQ5Hbs3/l3TeDzfZkUTrAJA== X-Received: by 2002:aa7:860b:: with SMTP id p11-v6mr10760285pfn.247.1535640008468; Thu, 30 Aug 2018 07:40:08 -0700 (PDT) Received: from raichu (toroon0560w-lp130-09-70-52-224-239.dsl.bell.ca. [70.52.224.239]) by smtp.gmail.com with ESMTPSA id r87-v6sm12477270pfb.1.2018.08.30.07.40.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 30 Aug 2018 07:40:07 -0700 (PDT) Sender: Mark Johnston Date: Thu, 30 Aug 2018 10:40:04 -0400 From: Mark Johnston To: Paul Cc: freebsd-fs@freebsd.org, developer@open-zfs.org Subject: Re: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size Message-ID: <20180830144004.GD15740@raichu> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> <1535611885.624706680.h132ex07@frv33.fwdcdn.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1535611885.624706680.h132ex07@frv33.fwdcdn.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 14:40:10 -0000 On Thu, Aug 30, 2018 at 09:55:27AM +0300, Paul wrote: > 30 August 2018, 00:22:14, by "Mark Johnston" : > > > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > > > Hello team, > > > > > > > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > > > target is reached after MRU is evicted current code continues evicting MFU. Before said > > > commit, on the step prior to MFU eviction, target value was recalculated as: > > > > > > target = arc_size - arc_c; > > > > > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > > > hence this expression, resulted in zero or negative target if MRU eviction was enough > > > to reach the original goal. > > > > > > Modern version uses cached value of arc_size, called asize: > > > > > > target = asize - arc_c; > > > > > > Because asize stays constant during execution of whole body of arc_adjust() it means that > > > above expression will always be evaluated to value > 0, causing MFU to be evicted every > > > time, even if MRU eviction has reached the goal already. Because of the difference in > > > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > > > to dramatic numbers. > > > > Hi Paul, > > > > Your analysis does seem right to me. I cc'ed the openzfs mailing list > > so that an actual ZFS expert can chime in; it looks like this behaviour > > is consistent between FreeBSD, illumos and ZoL. > > > > Have you already tried the obvious "fix" of subtracting total_evicted > > from the MFU target? > > We are going to apply the asize patch (plus the ameta, as suggested by Richard) and reboot > one of our production servers this night or the following. Just to be explicit, are you testing something equivalent to the patch at the end of this email? > Then we have to wait a few days and observer the ARC behaviour. Thanks! Please let us know how it goes: we're preparing to release FreeBSD 12.0 shortly and I'd like to get this fixed in head/ as soon as possible. diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c index 1387925c4607..882c04dba50a 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c @@ -4446,6 +4446,12 @@ arc_adjust(void) arc_adjust_impl(arc_mru, 0, target, ARC_BUFC_METADATA); } + /* + * Re-sum ARC stats after the first round of evictions. + */ + asize = aggsum_value(&arc_size); + ameta = aggsum_value(&arc_meta_used); + /* * Adjust MFU size *