From owner-freebsd-fs@freebsd.org Tue Sep 8 17:06:36 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 99CA73CFC6E for ; Tue, 8 Sep 2020 17:06:36 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: from mail-ej1-x642.google.com (mail-ej1-x642.google.com [IPv6:2a00:1450:4864:20::642]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmBQ36Wc9z42nq for ; Tue, 8 Sep 2020 17:06:35 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: by mail-ej1-x642.google.com with SMTP id p9so23594172ejf.6 for ; Tue, 08 Sep 2020 10:06:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=IYM6iJdqqYhINX0fwgEzq4yf7PEghQh6BisS+AyKgZQ=; b=Bv+GBn8Nju0TZM1fgdpQqRKmOCryXj9oAN9fAhZiPYCVFTxku+WlFDQEi7txtwdzkZ gZGGPJXBxOGhndem5uo1DmDqFxiXE4wuWeU2d964RSvAiadX1IiSbcs2kf446y8QBHaO ymdIdO0asUbgcXuUzEQ6xmpirjwM5PvZcTha/Xn5mqKWSzD6aT1x2Ex2hCVrkrxnhX17 gix3MRRSB2HXNGH9BneAdEnaxte9kKLC49rG5ampDxcVXGBDs43ZGjvSivcaxi2KM3AT 8jmTanH4LQ882MrgDclV/yceid3ORnRxem3SOfm4mfW5JWdeRhQC4vWBouXQ0MsyxplA Y9gQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=IYM6iJdqqYhINX0fwgEzq4yf7PEghQh6BisS+AyKgZQ=; b=RqyDN//GT11Hr1mV/vGXKAZSLyy77RV9PlsQ5kk0NqSQ07SF9gsS3bUagM+bySe/KP CwWXMcLvuLB2txfFmTuqi+2uUJMptdCNfW2FI30eGomHLDSEv1bDtwYeGclka6aYpRsS 8WvJLNLC/fIDws1avMshUmkifZRjkS7ECZ7xWgv6GNPPxRzQMZ+jH3Cwl9aPlJFjDGFs k8KuYp7QEvagXXXF/8lrNk3PEuVFIP7WUZvRMIl4EXPUE60WMnMOD/GcBKkHJKTrDBfB nboGUWMNOayCK+mGf4py/0n1F/XJA1NJffhJZ4neHCsH1raH9QtxPiUXWydehVdURQzS Z2sQ== X-Gm-Message-State: AOAM532NzAQYmGe0FtI7tuSNuw7974jqoAWigi44rzNmLFX9vdTNmyEn 44D3rRRNz3CkjDTtkHeFKh7dwaGN0Q/ca0OEVOwMLw== X-Google-Smtp-Source: ABdhPJwjkS4VnbtXEJJD7pVF86l1O7tbK9Q16qZkXwOES08yVNyjhHf6eVl2FIwTcdwkUre3x3jLk5cFXY/lSzorneE= X-Received: by 2002:a17:906:4e54:: with SMTP id g20mr27898481ejw.252.1599584794190; Tue, 08 Sep 2020 10:06:34 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ryan Moeller Date: Tue, 8 Sep 2020 13:06:23 -0400 Message-ID: Subject: Re: OpenZFS and L2ARC To: Stefan Esser Cc: FreeBSD CURRENT , freebsd-fs Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4BmBQ36Wc9z42nq X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ixsystems.com header.s=google header.b=Bv+GBn8N; dmarc=pass (policy=none) header.from=ixsystems.com; spf=pass (mx1.freebsd.org: domain of ryan@ixsystems.com designates 2a00:1450:4864:20::642 as permitted sender) smtp.mailfrom=ryan@ixsystems.com X-Spamd-Result: default: False [-3.43 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[ixsystems.com:s=google]; FREEFALL_USER(0.00)[ryan]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-0.99)[-0.993]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_MEDIUM(-0.98)[-0.981]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[ixsystems.com:+]; DMARC_POLICY_ALLOW(-0.50)[ixsystems.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::642:from]; NEURAL_HAM_SHORT(-0.46)[-0.456]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-fs] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 17:06:36 -0000 On Tue, Sep 8, 2020 at 3:43 AM Stefan Esser wrote: > > OpenZFS seems to work quite well for me, in general, but I have > questions regarding the L2ARC statistics. > > The system uses a 3 * 6 TB raidz1 (plus further ZFS volumes that > are not relevant here, since without level 2 ARC) and an 1 TB M.2 > SSD with a 256 GB partition for the L2ARC (and most of it currently > unused, else). > > The L2ARC seems to have filled to the limit of 256 GB, but after > several reboots, sysctl reports a L2ARC size of nearly twice the > allocated space: > > kstat.zfs.misc.arcstats.l2_size: 534620858880 > > That is 497 GiB, and might be possible with a lz4 compression > factor of 2 - if the value reported is not the space allocated, > but the actual (uncompressed) data held by the L2ARC. > > > The sysutils/zfs-stats port reports the following values for > this system, BTW: > > ------------------------------------------------------------------------ > ZFS Subsystem Report Tue Sep 8 09:02:46 2020 > ------------------------------------------------------------------------ > > L2 ARC Summary: (HEALTHY) > Passed Headroom: 0 > Tried Lock Failures: 0 > IO In Progress: 0 > Low Memory Aborts: 7 > Free on Write: 123 > Writes While Full: 0 > R/W Clashes: 0 > Bad Checksums: 0 > IO Errors: 0 > SPA Mismatch: 0 > > L2 ARC Size: (Adaptive) 497.91 GiB > Header Size: 0.11% 558.83 MiB > > L2 ARC Evicts: > Lock Retries: 6 > Upon Reading: 0 > > L2 ARC Breakdown: 5.75 m > Hit Ratio: 81.94% 4.71 m > Miss Ratio: 18.06% 1.04 m > Feeds: 235.04 k > > L2 ARC Buffer: > Bytes Scanned: 0 Bytes > Buffer Iterations: 0 > List Iterations: 0 > NULL List Iterations: 0 > > L2 ARC Writes: > Writes Sent: 100.00% 22.67 k > > ------------------------------------------------------------------------ > > With the FreeBSD ZFS (without persistent L2ARC) I never got more > than 20% hit ratio on the L2ARC between reboots. > > Quite a number of sysctl variable names have changed, and the port > needs to be adapted to the new names (therefore there are lots of 0 > values in the -L output). > > The following names used by zfs-stats do not exist in OpenZFS: > > kstat.zfs.misc.arcstats.recycle_miss > kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned > kstat.zfs.misc.arcstats.l2_write_buffer_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter > kstat.zfs.misc.arcstats.l2_write_full > kstat.zfs.misc.arcstats.l2_write_in_l2 > kstat.zfs.misc.arcstats.l2_write_io_in_progress > kstat.zfs.misc.arcstats.l2_write_not_cacheable > kstat.zfs.misc.arcstats.l2_write_passed_headroom > kstat.zfs.misc.arcstats.l2_write_pios > kstat.zfs.misc.arcstats.l2_write_spa_mismatch > kstat.zfs.misc.arcstats.l2_write_trylock_fail > kstat.zfs.misc.arcstats.l2_writes_hdr_miss > vfs.zfs.vdev.cache.size > > The existence of vfs.zfs.vdev.cache.size vs vfs.zfs.vdev.cache_size > can be used to detect OpenZFS, and is easily fixed. > > But the above listed L2ARC values seem to have been removed from or > have never existed in OpenZFS, and I did not find any substitutes. > > Are there any plans to re-create them in OpenZFS on FreeBSD or are > they gone for good? > > I'd like to update the zfs-stats port for compatibil=C3=ADty with OpenZFS= ... Improved L2ARC stats is something we would like to look into, yes. There is currently one PR open to add a few more arcstats for L2: * evict_l2_eligible_mfu * evict_l2_eligible_mru * l2_prefetch_asize * l2_mru_asize * l2_mfu_asize * l2_bufc_data_asize * l2_bufc_metadata_asize Adding the other L2ARC arcstats from FreeBSD would be a welcome change, as well. It's *somewhere* on the todo list. -- Ryan Moeller iXsystems, Inc. OS Developer Email: ryan@iXsystems.com