From owner-freebsd-fs@freebsd.org Tue Aug 11 03:46:03 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 12449378988 for ; Tue, 11 Aug 2020 03:46:03 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BQdzG4xNpz3xhr for ; Tue, 11 Aug 2020 03:46:02 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 0994F20FEB; Tue, 11 Aug 2020 03:45:56 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 0994F20FEB From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: Date: Mon, 10 Aug 2020 23:45:49 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk" X-Rspamd-Queue-Id: 4BQdzG4xNpz3xhr X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 11 Aug 2020 03:46:03 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk Content-Type: multipart/mixed; boundary="JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> In-Reply-To: --JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is the eighth weekly status report on the project to complete the integration of ZSTD compression into OpenZFS. https://github.com/openzfs/zfs/pull/10692 - I created some new tests around the L2ARC to facilitate testing of ZSTD + L2ARC. These tests found an issue (even with just lz4 compression) where if the compressed_arc feature is disabled, the wrong size is used when calculating the checksum of the buffer read back from the L2ARC, resulting in a silent checksum failure. After the block from the L2ARC fails to checksum, it is re-read from the main pool. https://github.com/openzfs/zfs/pull/10693 - I have created a patch to fix the issue between L2ARC and compressed_arc. https://github.com/allanjude/zfs/commit/1f565ef0c6bd2e785fb3777c111184bb4= bc551c4 - A followup to the rewritten version of the ZSTD feature activation code. The handling of zfs_prop_set_special() was not actually setting the property, so we return -1 so that the normal property setting routine will be followed, in addition to the special handling. https://github.com/allanjude/zfs/commit/8eac845a221952b3c9c52b4caf9be4bdf= 401e2b9 - Fixed an issue where if compression failed (this can be triggered by "early abort", where the data is uncompressable and wont fit in the output buffer that is 12.5% smaller than the input), it would skip the encryption code block, which could result in data being written to the L2ARC uncompressed and unencrypted. Based on the above, I am considering that we might want to calculate the checksum of the block after we re-transform it, and make sure it matches the checksum in the blockpointer, if it does not, we just skip writing to the L2ARC as if the block was ineligible for one of the normal reasons. This would ensure we don't end up reading from the L2ARC only to re-read from the main pool because the block did not survive the trip.= That leaves just the future proofing bits left (L2ARC, nop-write, etc when newer ZSTD does not recompress the block in the same way), but that specific bit doesn't need to block merging ZSTD support. This project is sponsored by the FreeBSD Foundation. On 2020-08-05 22:49, Allan Jude wrote: > This is the seventh weekly status report on the project to integrate > ZSTD into OpenZFS. >=20 > The compatibility related changes I created last week were refined and > marged into the mainline branch. >=20 > Thanks to Brian Behlendorf for reviewing my proposed change for the zst= d > feature flag activation, and pointing out a better approach. I have > reworked the patch based on his suggestion and prototype: >=20 > https://github.com/allanjude/zfs/commit/2508dafcec0a05d61afc5fbd5da356e= 201afbe97 > - Activate the per-dataset ZSTD feature flag as soon as the property is= > set to ZSTD. Before, simply doing `zfs set compression=3Dzstd dataset` > would not activate the feature flag. The feature flag would be activate= d > when the first block that used ZSTD compression was written (see > dsl_dataset_block_born()). This meant that if you set the property, > exported the pool, the pool would import on systems with older versions= > of ZFS that did not support ZSTD, but would crash their userspace tools= , > because the property value was out of bounds. >=20 >=20 > https://github.com/allanjude/zfs/commit/b8bec3fd2a8feb3a4de572eb15515d3= 764f92a35 > - I created a test that ensures that the feature flag is activated by > `zfs set compression=3Dzstd` and also ensures that the feature flag > reverts to the 'enabled' state once the last dataset using zstd is > destroyed. >=20 >=20 > The next step is ensuring that ZSTD compression inter-operates properly= > with the L2ARC and Encryption etc. >=20 > I've also been discussing ideas with Brian about future-proofing, to > handle the case where a newer version of ZSTD might compression the sam= e > input differently (better ratio), and how that would impact L2ARC, > nop-write, etc. One idea (originally from Pawel Dawidek) is to do > something similar to what encryption does, and split the checksum field= =2E > Using half to checksum the original data, and half the compressed > version. This would allow ZFS to detect when the same content compresse= d > differently (combined with the ZSTD version header in the compressed > data), giving better compatibility as we upgrade ZSTD. >=20 >=20 > This project is sponsored by the FreeBSD Foundation. >=20 >=20 >=20 --=20 Allan Jude --JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3-- --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIbBAEBAgAGBQJfMhRzAAoJEBmVNT4SmAt+TkIP+Inmj5M3TkOm4G1Rc2rgEEDz 0BXygiUCbrFHn3aeadgeImYIGmZfATH4USFGpBt9FcyETeUCp/9fTlHPnd2PIttY Y6TlVi9jqa8Gnb7M/HG3egArF3oYODvc/oJk+AurRNX+45mVWSkmfddc7YRwgZta /AQVVgAyYSukgASD8MgRyQV1FNe0OhFJWk3o5Xqc6qUhSpO3K3PfWgzkzBgVtLUa XD6T5EgxQo632dbBG3V/S8a6nhKht21rV35FyrUjxLWCWh5bibAJyY9CZNXdi+F5 sdEyoBuDWB1KwUJDR+OA4RO93cL4l6Ai3Km4UbeA12HTp7jmFwah/z32M5zPy0xg L3mQBVOCfn+IuFb2aNS/U/JAu9ydy/vnY0oX9YKD1kLoYEkRznnSlms/+EBryMUm TS6HRzOHoHNMjCVbuGfc0CNt1CVzEywJvqX2Mrj/HvTlsx89hfbxkdgE6XMJ8C57 U5WBKt3qLWNRTCiz65/CnY/ZB4TJucdBEfe3g3IEmyOSR6fFBMaSOlqe8oXYDmeT BoIcoR51CUf5TZLrBuTC0BBY/2q54M89Ad4wwO5/Se6EPCxngm8707tB/lDdHM9S Tn47VhhCsIq/IFVIae0/TqBWQ43bvqJjWyj30SBvSRlDvBYN8DdVMna85bV3oR+d nQXzKwfuDffLL+v1+lY= =uF1Q -----END PGP SIGNATURE----- --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk--