From owner-freebsd-fs@freebsd.org Wed Aug 29 22:03:33 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A40A81097CA8 for ; Wed, 29 Aug 2018 22:03:33 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44B96840E1 for ; Wed, 29 Aug 2018 22:03:33 +0000 (UTC) (envelope-from richard.elling@richardelling.com) Received: by mail-pf1-x441.google.com with SMTP id l9-v6so2880401pff.9 for ; Wed, 29 Aug 2018 15:03:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=richardelling.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=6L9BM6DqgWpDlLqacQ2FtbXQSEif8XfJkVkcBejCu0I=; b=ZO5ckVfUdb4c2izpqlRyLUGgvIawW+9VzRlkIGjEGbbMKTAIFLJSPxcVTM+VpLUxRu W+rcVs0iKsOMAQNJDtQbUTD7YSfMwtAbYQQMya6pDt/LdpBHqj2ewr1hzIFdkf7RMFcX gcgRH/7LjQfzmZ4XI8x2kd8IFcq3NpewkyNho= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=6L9BM6DqgWpDlLqacQ2FtbXQSEif8XfJkVkcBejCu0I=; b=AmTeZQCTwHzY7vy1bg52Zf5YW05xvhPDzV/NBjmAcQSOuYl2ZE5wJMt1ZIUbpVHjQ5 GDyK1ALL3yKyD1KCCr28ZItsxK2Qk6WNocMUJpMdgmeA0Lw9hzb2EyzaTqnphBNx9tX2 m1rtxWv1t7KYPDSQqwSxlGjAxYOKmDLAh6K+rdLnMHJJeKSRXcV34uRQreKDi1KLQEiE PTNU+axRqeu11H2vaFWFUHsvtZwBuFyf84RmZnbFhHyN5gbSx2iUSJmas3QNYXwX9uu+ 98vje7oPonOy8OibES7wXPnhPtVFVQms9vk4oiLPnliipgZpk+OVL3xlJ/9dlaFKH2qa 2pKA== X-Gm-Message-State: APzg51CA87wg/MBS6cy/SRlhxkmob4Zm+ksmYoJkPG/1FkFbMEr4xkNw 4ZVyp5eWVX/qcF7/U5sJQvzrvg== X-Google-Smtp-Source: ANB0VdYaIHFKPeFn4Qs98shVrsSJqZ4WEEkgnW7RoEV11wPC1ydAOz2En5MOF3EL//QLFumVgYl1nQ== X-Received: by 2002:a63:5fc8:: with SMTP id t191-v6mr7228851pgb.183.1535580211449; Wed, 29 Aug 2018 15:03:31 -0700 (PDT) Received: from [192.168.0.138] ([172.58.19.49]) by smtp.gmail.com with ESMTPSA id t21-v6sm8236877pfi.73.2018.08.29.15.03.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Aug 2018 15:03:30 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: [developer] Re: Potential bug recently introduced in arc_adjust() that leads to unintended pressure on MFU eventually leading to dramatic reduction in its size From: Richard Elling In-Reply-To: <20180829212207.GF2709@raichu> Date: Wed, 29 Aug 2018 15:03:27 -0700 Cc: Paul , freebsd-fs@freebsd.org, developer@open-zfs.org Content-Transfer-Encoding: quoted-printable Message-Id: <6569BAF8-1AC4-4CB4-A384-E5C8EFF129D8@richardelling.com> References: <1535534257.46692673.obgqw5dr@frv33.fwdcdn.com> <20180829212207.GF2709@raichu> To: openzfs-developer X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Aug 2018 22:03:33 -0000 Thanks for passing this along, Mark. Comments embedded > On Aug 29, 2018, at 2:22 PM, Mark Johnston wrote: >=20 > On Wed, Aug 29, 2018 at 12:42:33PM +0300, Paul wrote: >> Hello team, >>=20 >>=20 >> It seems like a commit on Mar 23 introduced a bug: if during = execution of arc_adjust() >> target is reached after MRU is evicted current code continues = evicting MFU. Before said >> commit, on the step prior to MFU eviction, target value was = recalculated as: arc_size is hot, so it was broken up into per-cpu counters and asize is = now a snapshot of the sum of the counters... >>=20 >> target =3D arc_size - arc_c; >>=20 >> arc_size here is a global variable that was being updated = accordingly, during MRU eviction, >> hence this expression, resulted in zero or negative target if MRU = eviction was enough >> to reach the original goal. >>=20 >> Modern version uses cached value of arc_size, called asize: >>=20 >> target =3D asize - arc_c; >>=20 >> Because asize stays constant during execution of whole body of = arc_adjust() it means that >> above expression will always be evaluated to value > 0, causing MFU = to be evicted every=20 >> time, even if MRU eviction has reached the goal already. Because of = the difference in=20 >> nature of MFU and MRU, globally it leads to eventual reduction of = amount of MFU in ARC=20 >> to dramatic numbers. >=20 > Hi Paul, >=20 > Your analysis does seem right to me. I cc'ed the openzfs mailing list > so that an actual ZFS expert can chime in; it looks like this = behaviour > is consistent between FreeBSD, illumos and ZoL. Agree. In the pre-aggsum code, arc_size would have changed after the MRU = adjustment. Now it does not. I have at least one correlation to this occuring in a = repeatable test that I can run on my ZoL test machine (when it is finished punishing some = other code). >=20 > Have you already tried the obvious "fix" of subtracting total_evicted > from the MFU target? ameta also needs to be re-aggsummed after the MRU adjustments. -- richard >=20 >> Servers that run the version of FreeBSD prior to the issue have this = picture of ARC: >>=20 >> ARC: 369G Total, 245G MFU, 97G MRU, 36M Anon, 3599M Header, 24G = Other >>=20 >> As you can see, MFU dominates. This is a nature of our workload: we = have a considerably=20 >> small dataset that we use constantly and repeatedly; and a large = dataset that we use >> rarely. >>=20 >> But on the modern version of FreeBSD picture is dramatically = different:=20 >>=20 >> ARC: 360G Total, 50G MFU, 272G MRU, 211M Anon, 7108M Header, 30G = Other >>=20 >> This leads to a much heavier burden on the disk sub-system. >>=20 >>=20 >> Commit that introduced a bug:=20 >> = https://github.com/freebsd/freebsd/commit/555f9563c9dc217341d4bb5129f5d233= cf1f92b8 >=20 > ------------------------------------------ > openzfs: openzfs-developer > Permalink: = https://openzfs.topicbox.com/groups/developer/T10a105c53bcce15c-M8152dc243= 0a5ea4e625ad564 > Delivery options: = https://openzfs.topicbox.com/groups/developer/subscription