From owner-freebsd-fs@freebsd.org Sun Mar 29 21:26:30 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BC2B2264F25 for ; Sun, 29 Mar 2020 21:26:30 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48r7tx3VMcz3x03 for ; Sun, 29 Mar 2020 21:26:16 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 5333440002 for ; Sun, 29 Mar 2020 23:26:04 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 3244140004; Sun, 29 Mar 2020 23:26:04 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7] (unknown [IPv6:2001:9b1:28ff:d901:6519:594e:bae9:86c7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 7A02640002 for ; Sun, 29 Mar 2020 23:26:03 +0200 (CEST) From: Peter Eriksson Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: ZFS ARC Metadata "sizing" for datasets&snapshots Date: Sun, 29 Mar 2020 23:26:03 +0200 References: <66AB88C0-12E8-48A0-9CD7-75B30C15123A@pk1048.com> <982F9A21-FF1C-4DAB-98B3-610D70714ED3@lysator.liu.se> To: FreeBSD Filesystems In-Reply-To: Message-Id: <25DADCD5-085B-4E85-B8DA-B3115472CB2D@lysator.liu.se> X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 48r7tx3VMcz3x03 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-4.13 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; MV_CASE(0.50)[]; IP_SCORE(-1.83)[ip: (-7.27), ipnet: 2001:6b0::/32(-1.04), asn: 1653(-0.85), country: EU(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2020 21:26:30 -0000 Hmm.. I wonder if someone knows how much ZFS ARC metadata one should = expect for a certain server. Just for fun, I did some tests. On a test server with 512GB RAM and arc_max set to 384GB, arc_meta_limit = set to 256GB, 12866 filesystems (datasets) and 430610 snapshots (spread = out over those filesystems), and an uptime of 1 day doing nothing = basically (taking some snapshots) I get these numbers: anon_size: 1.0 M arc_max: 412.3 G arc_meta_limit: 274.9 G arc_meta_max: 33.5 G arc_meta_used: 33.5 G compressed_size: 9.5 G data_size: 7.4 G hdr_size: 462.3 M metadata_size: 30.2 G mru_size: 16.7 G other_size: 2.8 G overhead_size: 28.2 G size: 40.9 G uncompressed_size: 45.9 G Ie ARC metadata_size at 30GB / arc_meta_used at 33.5G.=20 Doing a =E2=80=9Czfs list -t all=E2=80=9D takes ~100s and =E2=80=9Czfs = list=E2=80=9D takes ~3s. On production server (just booted) with just 256GB RAM doing =E2=80=9Czfs = list=E2=80=9D takes 0.4s but =E2=80=9Czfs list -t all=E2=80=9D is taking = a long time=E2=80=A6 anon_size: 3.3 M arc_max: 103.1 G arc_meta_limit: 51.5 G arc_meta_max: 2.2 G arc_meta_used: 2.2 G compressed_size: 1.3 G data_size: 2.7 G hdr_size: 17.9 M metadata_size: 2.0 G mru_size: 3.3 G other_size: 180.5 M overhead_size: 3.4 G size: 4.9 G uncompressed_size: 3.4 G The =E2=80=9Czfs list -t all=E2=80=9D took 2542 seconds (42 minutes) - = 131256 datasets+snapshots (1600 filesystems). That is ~50 = snapshots/filesystems per second. After that command has executed metadata_size has increased with 5GB and = a new =E2=80=9Czfs list -t all=E2=80=9D just takes 37 seconds. anon_size: 5.1 M arc_max: 103.1 G arc_meta_limit: 51.5 G arc_meta_max: 7.9 G arc_meta_used: 7.8 G compressed_size: 2.5 G data_size: 1.5 G hdr_size: 98.0 M metadata_size: 7.1 G mru_size: 7.3 G other_size: 660.9 M overhead_size: 6.1 G size: 9.4 G uncompressed_size: 8.2 G So... perhaps ~40KB (5G/131256) per dataset/snapshot on average. (Yes, = oversimplificated but anyway) Hmm.. Perhaps one should regularly do a =E2=80=9Czfs list -t all = >/dev/null=E2=80=9D just to prime the ARC metadata cache (and keep it = primed) :-) - Peter