From owner-freebsd-fs@freebsd.org Thu Aug 30 06:55:31 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D23710967D4 for ; Thu, 30 Aug 2018 06:55:31 +0000 (UTC) (envelope-from devgs@ukr.net) Received: from frv196.fwdcdn.com (frv196.fwdcdn.com [212.42.77.196]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.ukr.net", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CB4D274B5E for ; Thu, 30 Aug 2018 06:55:30 +0000 (UTC) (envelope-from devgs@ukr.net) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=ukr.net; s=ffe; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-Id:Cc:To:Subject:From:Date:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=d2dgnriTyupDsC8f+s4uefxDlXwti3HKnwsimEuQNE8=; b=iC44mcJd6LvcLNyUGE9ZhZSs0J W9o5xNfIgkteyowXCPUAcj1H41/Zr2z2q/Ii3dcXz2Cem01wOznbQvyEAybvsZ5MSwRMuOI3eSSSC 9gkDLUDFFpYZ5LYGAEE+PJm6dyx5plqY3eHffkXFUAeoIvdWgKoIl2zZGfX5EI/VLsXg=; Received: from [10.10.10.33] (helo=frv33.fwdcdn.com) by frv196.fwdcdn.com with smtp ID 1fvGrX-000LtD-UQ for freebsd-fs@freebsd.org; Thu, 30 Aug 2018 09:55:27 +0300 Date: Thu, 30 Aug 2018 09:55:27 +0300 From: Paul Subject: Re[2]: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size To: Mark Johnston Cc: freebsd-fs@freebsd.org, developer@open-zfs.org X-Mailer: mail.ukr.net 5.0 Message-Id: <1535611885.624706680.h132ex07@frv33.fwdcdn.com> In-Reply-To: <20180829212207.GF2709@raichu> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> X-Reply-Action: reply Received: from devgs@ukr.net by frv33.fwdcdn.com; Thu, 30 Aug 2018 09:55:27 +0300 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: binary X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Aug 2018 06:55:31 -0000 30 August 2018, 00:22:14, by "Mark Johnston" : > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: > > Hello team, > > > > > > It seems like a commit on Mar 23 introduced a bug: if during execution of arc_adjust() > > target is reached after MRU is evicted current code continues evicting MFU. Before said > > commit, on the step prior to MFU eviction, target value was recalculated as: > > > > target = arc_size - arc_c; > > > > arc_size here is a global variable that was being updated accordingly, during MRU eviction, > > hence this expression, resulted in zero or negative target if MRU eviction was enough > > to reach the original goal. > > > > Modern version uses cached value of arc_size, called asize: > > > > target = asize - arc_c; > > > > Because asize stays constant during execution of whole body of arc_adjust() it means that > > above expression will always be evaluated to value > 0, causing MFU to be evicted every > > time, even if MRU eviction has reached the goal already. Because of the difference in > > nature of MFU and MRU, globally it leads to eventual reduction of amount of MFU in ARC > > to dramatic numbers. > > Hi Paul, > > Your analysis does seem right to me. I cc'ed the openzfs mailing list > so that an actual ZFS expert can chime in; it looks like this behaviour > is consistent between FreeBSD, illumos and ZoL. > > Have you already tried the obvious "fix" of subtracting total_evicted > from the MFU target? We are going to apply the asize patch (plus the ameta, as suggested by Richard) and reboot one of our production servers this night or the following. Then we have to wait a few days and observer the ARC behaviour. > > > Servers that run the version of FreeBSD prior to the issue have this picture of ARC: > > > > ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G Other > > > > As you can see, MFU dominates. This is a nature of our workload: we have a considerably > > small dataset that we use constantly and repeatedly; and a large dataset that we use > > rarely. > > > > But on the modern version of FreeBSD picture is dramatically different: > > > > ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G Other > > > > This leads to a much heavier burden on the disk sub-system. > > > > > > Commit that introduced a bug: > > https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233cf1f92b8