From owner-freebsd-fs@freebsd.org Tue Aug 18 22:51:34 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DA5973AA205 for ; Tue, 18 Aug 2020 22:51:34 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BWR3p0Ch1z3YV0 for ; Tue, 18 Aug 2020 22:51:33 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 36FC720AD2; Tue, 18 Aug 2020 22:51:27 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 36FC720AD2 From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> Date: Tue, 18 Aug 2020 18:51:23 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8KetLi2IYIuXZfnxT4bnEOAmPOEQ97Hg2" X-Rspamd-Queue-Id: 4BWR3p0Ch1z3YV0 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Aug 2020 22:51:34 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --8KetLi2IYIuXZfnxT4bnEOAmPOEQ97Hg2 Content-Type: multipart/mixed; boundary="OT6eAyJFIxS85GxYmTZUvBufpp1SgTqHy"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> In-Reply-To: --OT6eAyJFIxS85GxYmTZUvBufpp1SgTqHy Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is the ninth weekly status report on my FreeBSD Foundation sponsored project to complete the integration of ZSTD compression into OpenZFS. https://github.com/openzfs/zfs/pull/10693 - The L2ARC fixes (for when compressed ARC is disabled) have been merged. https://github.com/openzfs/zfs/pull/10278/ - A number of other cleanups and fixes for the ZSTD have been integrated and squashed, and it looks like the completed ZSTD feature will be merged very soon. This included a bunch of fixes for makefiles and runfiles to hook the tests I added up to the ZFS test suite so they are run properly. It looks like this will mean that the ZSTD feature will be included in OpenZFS 2.0. Thanks for everyone who has tested, reviewed, or contributed to this effort, especially those who kept it alive while I was working on other things. Post-merge, the remaining work is to develop future-proofing around ZSTD so that we will be able to more seamlessly upgrade to newer releases of ZSTD. Recompression of the same input resulting in the same on-disk checksum is the main concern, as without this upgrading the compression algorithm will break features like nop-write. This project is sponsored by the FreeBSD Foundation. On 2020-08-10 23:45, Allan Jude wrote: > This is the eighth weekly status report on the project to complete the > integration of ZSTD compression into OpenZFS. >=20 > https://github.com/openzfs/zfs/pull/10692 - I created some new tests > around the L2ARC to facilitate testing of ZSTD + L2ARC. These tests > found an issue (even with just lz4 compression) where if the > compressed_arc feature is disabled, the wrong size is used when > calculating the checksum of the buffer read back from the L2ARC, > resulting in a silent checksum failure. After the block from the L2ARC > fails to checksum, it is re-read from the main pool. >=20 > https://github.com/openzfs/zfs/pull/10693 - I have created a patch to > fix the issue between L2ARC and compressed_arc. >=20 > https://github.com/allanjude/zfs/commit/1f565ef0c6bd2e785fb3777c111184b= b4bc551c4 > - A followup to the rewritten version of the ZSTD feature activation > code. The handling of zfs_prop_set_special() was not actually setting > the property, so we return -1 so that the normal property setting > routine will be followed, in addition to the special handling. >=20 > https://github.com/allanjude/zfs/commit/8eac845a221952b3c9c52b4caf9be4b= df401e2b9 > - Fixed an issue where if compression failed (this can be triggered by > "early abort", where the data is uncompressable and wont fit in the > output buffer that is 12.5% smaller than the input), it would skip the > encryption code block, which could result in data being written to the > L2ARC uncompressed and unencrypted. >=20 > Based on the above, I am considering that we might want to calculate th= e > checksum of the block after we re-transform it, and make sure it matche= s > the checksum in the blockpointer, if it does not, we just skip writing > to the L2ARC as if the block was ineligible for one of the normal > reasons. This would ensure we don't end up reading from the L2ARC only > to re-read from the main pool because the block did not survive the tri= p. >=20 > That leaves just the future proofing bits left (L2ARC, nop-write, etc > when newer ZSTD does not recompress the block in the same way), but tha= t > specific bit doesn't need to block merging ZSTD support. >=20 > This project is sponsored by the FreeBSD Foundation. >=20 >=20 > On 2020-08-05 22:49, Allan Jude wrote: >> This is the seventh weekly status report on the project to integrate >> ZSTD into OpenZFS. >> >> The compatibility related changes I created last week were refined and= >> marged into the mainline branch. >> >> Thanks to Brian Behlendorf for reviewing my proposed change for the zs= td >> feature flag activation, and pointing out a better approach. I have >> reworked the patch based on his suggestion and prototype: >> >> https://github.com/allanjude/zfs/commit/2508dafcec0a05d61afc5fbd5da356= e201afbe97 >> - Activate the per-dataset ZSTD feature flag as soon as the property i= s >> set to ZSTD. Before, simply doing `zfs set compression=3Dzstd dataset`= >> would not activate the feature flag. The feature flag would be activat= ed >> when the first block that used ZSTD compression was written (see >> dsl_dataset_block_born()). This meant that if you set the property, >> exported the pool, the pool would import on systems with older version= s >> of ZFS that did not support ZSTD, but would crash their userspace tool= s, >> because the property value was out of bounds. >> >> >> https://github.com/allanjude/zfs/commit/b8bec3fd2a8feb3a4de572eb15515d= 3764f92a35 >> - I created a test that ensures that the feature flag is activated by >> `zfs set compression=3Dzstd` and also ensures that the feature flag >> reverts to the 'enabled' state once the last dataset using zstd is >> destroyed. >> >> >> The next step is ensuring that ZSTD compression inter-operates properl= y >> with the L2ARC and Encryption etc. >> >> I've also been discussing ideas with Brian about future-proofing, to >> handle the case where a newer version of ZSTD might compression the sa= me >> input differently (better ratio), and how that would impact L2ARC, >> nop-write, etc. One idea (originally from Pawel Dawidek) is to do >> something similar to what encryption does, and split the checksum fiel= d. >> Using half to checksum the original data, and half the compressed >> version. This would allow ZFS to detect when the same content compress= ed >> differently (combined with the ZSTD version header in the compressed >> data), giving better compatibility as we upgrade ZSTD. >> >> >> This project is sponsored by the FreeBSD Foundation. >> >> >> >=20 >=20 --=20 Allan Jude --OT6eAyJFIxS85GxYmTZUvBufpp1SgTqHy-- --8KetLi2IYIuXZfnxT4bnEOAmPOEQ97Hg2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfPFtuAAoJEBmVNT4SmAt+CB8P+wakK5EThIHQdLtzA3jrBbjr k98RRAnLkYw/lyeegACFv6bAe1MdBd7kJnxxffYYAJxkyr2X10k0PfuX1PoMo299 vbV22jOdC4NhD+AKrHL5ADrOYdUo7l/M6ONWY3TVRh4rHs3yQG+yS/GUHH8swaGG xA0ieJnPgtfQQi/wJXw84Dhw8fLQWj2hGR2YMjjuJ4q+85tYUIF1Z1w5kW/lB7rz j8/o5erQ5T+dU9oYrUAYntmNyzfvtawPYodQqiVhPon9T2b24cSmPXO1OzwkULE3 3/oIN9Y1JqgKaKa15gQTNcOiq33rnNzN1iZ2lZjLK7VtDAsf7iSz5yIZVXuaklvI jFX2JiVD5m2NqxvCwSsI+G+vnhbfR8MyC7j7EN3ItJzEcQPXz/lmiC4R65zTlLj1 t/62KXX5KKLbEF1ryYK6YWn2F7h9kgdPFIBjUrKcg8ncWEqnq46Uc2MzKXTGJinV U5Wm1WkAqqMF4gJryirZCkUhJGMyGc0XWfXq/2qIIMrQ7OAmFwZx3UeD9NfqpSUf quUrTkDAAYHUUFChX4Oxlry5ibGepqGiXm0AIjs0PHBVuT0k8AkD3Tnzch6P0hrk Si7tXAYGTTS40VamxXfBh331gi9jFB+D9mby26ZU8meOVOSezA98pL4WKdslagLr KH1rhvF27tLtxE3oszH3 =zQcI -----END PGP SIGNATURE----- --8KetLi2IYIuXZfnxT4bnEOAmPOEQ97Hg2--