From owner-freebsd-fs@freebsd.org Tue Sep 8 17:52:49 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5391E3D1FCE for ; Tue, 8 Sep 2020 17:52:49 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BmCRL0dtzz4B8N for ; Tue, 8 Sep 2020 17:52:44 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 9E1C014AA7 for ; Tue, 8 Sep 2020 17:52:35 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 9E1C014AA7 Subject: Re: OpenZFS and L2ARC To: freebsd-fs@freebsd.org References: From: Allan Jude Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Message-ID: <4f30a8a5-b7fb-e4bb-a5e6-b59fa4ec10c2@freebsd.org> Date: Tue, 8 Sep 2020 13:52:34 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD" X-Rspamd-Queue-Id: 4BmCRL0dtzz4B8N X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 17:52:50 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD Content-Type: multipart/mixed; boundary="m4BJR1mD8qd0ajS7YJhMBnWcxgtMpCAfA"; protected-headers="v1" From: Allan Jude To: freebsd-fs@freebsd.org Message-ID: <4f30a8a5-b7fb-e4bb-a5e6-b59fa4ec10c2@freebsd.org> Subject: Re: OpenZFS and L2ARC References: In-Reply-To: --m4BJR1mD8qd0ajS7YJhMBnWcxgtMpCAfA Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2020-09-08 03:43, Stefan Esser wrote: > OpenZFS seems to work quite well for me, in general, but I have > questions regarding the L2ARC statistics. >=20 > The system uses a 3 * 6 TB raidz1 (plus further ZFS volumes that > are not relevant here, since without level 2 ARC) and an 1 TB M.2 > SSD with a 256 GB partition for the L2ARC (and most of it currently > unused, else). >=20 > The L2ARC seems to have filled to the limit of 256 GB, but after > several reboots, sysctl reports a L2ARC size of nearly twice the > allocated space: >=20 > kstat.zfs.misc.arcstats.l2_size: 534620858880 >=20 > That is 497 GiB, and might be possible with a lz4 compression > factor of 2 - if the value reported is not the space allocated, > but the actual (uncompressed) data held by the L2ARC. >=20 >=20 > The sysutils/zfs-stats port reports the following values for > this system, BTW: >=20 > -----------------------------------------------------------------------= - > ZFS Subsystem Report=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Tue S= ep=A0 8 09:02:46 2020 > -----------------------------------------------------------------------= - >=20 > L2 ARC Summary: (HEALTHY) > =A0=A0=A0=A0Passed Headroom:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0Tried Lock Failures:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0IO In Progress:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 0 > =A0=A0=A0=A0Low Memory Aborts:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 7 > =A0=A0=A0=A0Free on Write:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 123 > =A0=A0=A0=A0Writes While Full:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0R/W Clashes:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0= > =A0=A0=A0=A0Bad Checksums:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 0 > =A0=A0=A0=A0IO Errors:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0SPA Mismatch:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 = 0 >=20 > L2 ARC Size: (Adaptive)=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 49= 7.91=A0=A0=A0 GiB > =A0=A0=A0=A0Header Size:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0.11%=A0=A0=A0= 558.83=A0=A0=A0 MiB >=20 > L2 ARC Evicts: > =A0=A0=A0=A0Lock Retries:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 = 6 > =A0=A0=A0=A0Upon Reading:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 = 0 >=20 > L2 ARC Breakdown:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 5.75=A0=A0= =A0 m > =A0=A0=A0=A0Hit Ratio:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 81.94%=A0=A0=A0= 4.71=A0=A0=A0 m > =A0=A0=A0=A0Miss Ratio:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 18.06%=A0=A0=A0= 1.04=A0=A0=A0 m > =A0=A0=A0=A0Feeds:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= =A0 235.04=A0=A0=A0 k >=20 > L2 ARC Buffer: > =A0=A0=A0=A0Bytes Scanned:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0= 0=A0=A0=A0 Bytes > =A0=A0=A0=A0Buffer Iterations:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0List Iterations:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 > =A0=A0=A0=A0NULL List Iterations:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 0 >=20 > L2 ARC Writes: > =A0=A0=A0=A0Writes Sent:=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 100.00%=A0=A0= =A0 22.67=A0=A0=A0 k >=20 > -----------------------------------------------------------------------= - >=20 > With the FreeBSD ZFS (without persistent L2ARC) I never got more > than 20% hit ratio on the L2ARC between reboots. >=20 > Quite a number of sysctl variable names have changed, and the port > needs to be adapted to the new names (therefore there are lots of 0 > values in the -L output). >=20 > The following names used by zfs-stats do not exist in OpenZFS: >=20 > kstat.zfs.misc.arcstats.recycle_miss > kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned > kstat.zfs.misc.arcstats.l2_write_buffer_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_iter > kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter > kstat.zfs.misc.arcstats.l2_write_full > kstat.zfs.misc.arcstats.l2_write_in_l2 > kstat.zfs.misc.arcstats.l2_write_io_in_progress > kstat.zfs.misc.arcstats.l2_write_not_cacheable > kstat.zfs.misc.arcstats.l2_write_passed_headroom > kstat.zfs.misc.arcstats.l2_write_pios > kstat.zfs.misc.arcstats.l2_write_spa_mismatch > kstat.zfs.misc.arcstats.l2_write_trylock_fail > kstat.zfs.misc.arcstats.l2_writes_hdr_miss > vfs.zfs.vdev.cache.size >=20 > The existence of vfs.zfs.vdev.cache.size vs vfs.zfs.vdev.cache_size > can be used to detect OpenZFS, and is easily fixed. >=20 > But the above listed L2ARC values seem to have been removed from or > have never existed in OpenZFS, and I did not find any substitutes. >=20 > Are there any plans to re-create them in OpenZFS on FreeBSD or are > they gone for good? >=20 > I'd like to update the zfs-stats port for compatibil=EDty with OpenZFS = =2E.. > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" I think the kstats you are looking for are: kstat.zfs.misc.arcstats.l2_asize: 0 kstat.zfs.misc.arcstats.l2_size: 0 l2_asize is the 'allocated' size, that will be how much actual disk space is used. And l2_size will be the amount of data contained, so compression may make this quite a bit larger than the size of the physical device. --=20 Allan Jude --m4BJR1mD8qd0ajS7YJhMBnWcxgtMpCAfA-- --BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfV8TjAAoJEBmVNT4SmAt+UsAQAMHOPL/VVqRg1UlNzlHWR5MB aAALic6r+kRG1uI2sFHuLJFEW0KDNmSV0wS2yZGIoXdmEmGSGtk2vPZtdMGW8e4Y BCNY36zCaC9HPc5R6nL+PFdV2U0CU46WEtCQy51ESotjQxNR+EkVnETuRLyaDTMY vqAgs33ue/fVKSot8UfgYedjJlT8F1CtRGuby3cClZUr7HmBesJrstZoIv+0NJ8H MQhGhOvuSeeLQz81LOVsC9nj1PcAtAHBJbbffAhC0wLh6rQC92OaFNUQFj+4BXpm 7w3JzAbUi/+MpY8xIFNkxuYAQ9ve3KBAy7EtHFWY/vCNZIkoM8XSuCPilFFXiNLE paqZ2WCutckXpc5J3lYZIFWiEMgjuSUhut5IZaGd0CBa4AqGT2q1BPFxMndLehaP b5Aq8cAKIKRxL3++6vQgO7KUVAF2QBcrkI6GSUd5stE7NKPqi+5panCkQwd5AGFP NFSAfNIQrKQ3bl0pVq3oyPnxjTsnKvlpvRLXsJZhZC99jxx7qa+eXyUdr1HC9kLy VnP5sd3Xk37lce5UnQKa+pLHkR5JrWmvPiHXT24+aQJxhZfI546qfdOFnztzs68Z iJAYXAy7qINXRIscc0ytiaSabr77dlvgBAIkCu6VGrhAZMLQtgnh5nr6fXeND81W IG3TavEXUNFKfXSByYwn =YTk9 -----END PGP SIGNATURE----- --BHIyuXEo77xdHCnTHZ5IPrjW2EOEbP8XD--