Date: Mon, 10 Aug 2020 23:45:49 -0400 From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Subject: Re: ZSTD Project Weekly Status Update Message-ID: <e9597d9b-88e0-334f-d266-6cbbaf746855@freebsd.org> In-Reply-To: <db71835b-9bb7-2722-fd02-194b97f1564e@freebsd.org> References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <db71835b-9bb7-2722-fd02-194b97f1564e@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk Content-Type: multipart/mixed; boundary="JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Message-ID: <e9597d9b-88e0-334f-d266-6cbbaf746855@freebsd.org> Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <db71835b-9bb7-2722-fd02-194b97f1564e@freebsd.org> In-Reply-To: <db71835b-9bb7-2722-fd02-194b97f1564e@freebsd.org> --JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is the eighth weekly status report on the project to complete the integration of ZSTD compression into OpenZFS. https://github.com/openzfs/zfs/pull/10692 - I created some new tests around the L2ARC to facilitate testing of ZSTD + L2ARC. These tests found an issue (even with just lz4 compression) where if the compressed_arc feature is disabled, the wrong size is used when calculating the checksum of the buffer read back from the L2ARC, resulting in a silent checksum failure. After the block from the L2ARC fails to checksum, it is re-read from the main pool. https://github.com/openzfs/zfs/pull/10693 - I have created a patch to fix the issue between L2ARC and compressed_arc. https://github.com/allanjude/zfs/commit/1f565ef0c6bd2e785fb3777c111184bb4= bc551c4 - A followup to the rewritten version of the ZSTD feature activation code. The handling of zfs_prop_set_special() was not actually setting the property, so we return -1 so that the normal property setting routine will be followed, in addition to the special handling. https://github.com/allanjude/zfs/commit/8eac845a221952b3c9c52b4caf9be4bdf= 401e2b9 - Fixed an issue where if compression failed (this can be triggered by "early abort", where the data is uncompressable and wont fit in the output buffer that is 12.5% smaller than the input), it would skip the encryption code block, which could result in data being written to the L2ARC uncompressed and unencrypted. Based on the above, I am considering that we might want to calculate the checksum of the block after we re-transform it, and make sure it matches the checksum in the blockpointer, if it does not, we just skip writing to the L2ARC as if the block was ineligible for one of the normal reasons. This would ensure we don't end up reading from the L2ARC only to re-read from the main pool because the block did not survive the trip.= That leaves just the future proofing bits left (L2ARC, nop-write, etc when newer ZSTD does not recompress the block in the same way), but that specific bit doesn't need to block merging ZSTD support. This project is sponsored by the FreeBSD Foundation. On 2020-08-05 22:49, Allan Jude wrote: > This is the seventh weekly status report on the project to integrate > ZSTD into OpenZFS. >=20 > The compatibility related changes I created last week were refined and > marged into the mainline branch. >=20 > Thanks to Brian Behlendorf for reviewing my proposed change for the zst= d > feature flag activation, and pointing out a better approach. I have > reworked the patch based on his suggestion and prototype: >=20 > https://github.com/allanjude/zfs/commit/2508dafcec0a05d61afc5fbd5da356e= 201afbe97 > - Activate the per-dataset ZSTD feature flag as soon as the property is= > set to ZSTD. Before, simply doing `zfs set compression=3Dzstd dataset` > would not activate the feature flag. The feature flag would be activate= d > when the first block that used ZSTD compression was written (see > dsl_dataset_block_born()). This meant that if you set the property, > exported the pool, the pool would import on systems with older versions= > of ZFS that did not support ZSTD, but would crash their userspace tools= , > because the property value was out of bounds. >=20 >=20 > https://github.com/allanjude/zfs/commit/b8bec3fd2a8feb3a4de572eb15515d3= 764f92a35 > - I created a test that ensures that the feature flag is activated by > `zfs set compression=3Dzstd` and also ensures that the feature flag > reverts to the 'enabled' state once the last dataset using zstd is > destroyed. >=20 >=20 > The next step is ensuring that ZSTD compression inter-operates properly= > with the L2ARC and Encryption etc. >=20 > I've also been discussing ideas with Brian about future-proofing, to > handle the case where a newer version of ZSTD might compression the sam= e > input differently (better ratio), and how that would impact L2ARC, > nop-write, etc. One idea (originally from Pawel Dawidek) is to do > something similar to what encryption does, and split the checksum field= =2E > Using half to checksum the original data, and half the compressed > version. This would allow ZFS to detect when the same content compresse= d > differently (combined with the ZSTD version header in the compressed > data), giving better compatibility as we upgrade ZSTD. >=20 >=20 > This project is sponsored by the FreeBSD Foundation. >=20 >=20 >=20 --=20 Allan Jude --JZRdp68DXAPxHE95DlB0LrH2lLIWkdpt3-- --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIbBAEBAgAGBQJfMhRzAAoJEBmVNT4SmAt+TkIP+Inmj5M3TkOm4G1Rc2rgEEDz 0BXygiUCbrFHn3aeadgeImYIGmZfATH4USFGpBt9FcyETeUCp/9fTlHPnd2PIttY Y6TlVi9jqa8Gnb7M/HG3egArF3oYODvc/oJk+AurRNX+45mVWSkmfddc7YRwgZta /AQVVgAyYSukgASD8MgRyQV1FNe0OhFJWk3o5Xqc6qUhSpO3K3PfWgzkzBgVtLUa XD6T5EgxQo632dbBG3V/S8a6nhKht21rV35FyrUjxLWCWh5bibAJyY9CZNXdi+F5 sdEyoBuDWB1KwUJDR+OA4RO93cL4l6Ai3Km4UbeA12HTp7jmFwah/z32M5zPy0xg L3mQBVOCfn+IuFb2aNS/U/JAu9ydy/vnY0oX9YKD1kLoYEkRznnSlms/+EBryMUm TS6HRzOHoHNMjCVbuGfc0CNt1CVzEywJvqX2Mrj/HvTlsx89hfbxkdgE6XMJ8C57 U5WBKt3qLWNRTCiz65/CnY/ZB4TJucdBEfe3g3IEmyOSR6fFBMaSOlqe8oXYDmeT BoIcoR51CUf5TZLrBuTC0BBY/2q54M89Ad4wwO5/Se6EPCxngm8707tB/lDdHM9S Tn47VhhCsIq/IFVIae0/TqBWQ43bvqJjWyj30SBvSRlDvBYN8DdVMna85bV3oR+d nQXzKwfuDffLL+v1+lY= =uF1Q -----END PGP SIGNATURE----- --YS1XubqmixOoUKSoGiUVjrTm02zNNWNzk--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e9597d9b-88e0-334f-d266-6cbbaf746855>