Date: Mon, 6 Jul 2020 20:07:05 -0400 From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Subject: Re: ZSTD Project Weekly Status Update Message-ID: <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> In-Reply-To: <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO Content-Type: multipart/mixed; boundary="3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Message-ID: <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> In-Reply-To: <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> --3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable In my continuing effort to complete the integration of ZSTD into OpenZFS, here is my third weekly status report: https://github.com/allanjude/zfs/commit/87bb538bdbc4bb8848ed5791b7b0de84a= 026ebbe - Completed the rewrite of the way the compression property is handled, moving away from the initial approach of storing the compression property (enum zio_compress) and the level (uint64_t) separately. Previously we exposed the list of compression algorithms and levels to userland as the corresponding value from the enum in the lower 7 bits, and the level in the remaining upper bits. Then, as part of the property GET and SET IOCTLs, we read the separate compression=3D and compress_level=3D properties from the ZAP and returned the combined value= , or split the combined value into those two separate properties. This worked but caused a lot of headache around property inheritance. Instead I've changed to doing the combine/split when reading/writing from the dataset properties ZAP, via the compression_changed_cb() function. So the properties ZAP contains the combined value (lower 7 bits are the compression algorithm, as defined in the enum zio_compress, and the upper bits are the compression level). Elsewhere in ZFS we keep the two values separate (os_compress and os_complevel, and related variables in all of the different parts of ZFS). So now, inheritance of the property is handled correctly, and avoids issues where a dataset with compression=3Dzstd-12, would say 'inherited from' a dataset with zstd at some other compression level (since both actually just had compression=3Dzstd, but different compress_level=3D val= ues). I have also further extended zdb to inspect the compression settings when looking at an object: https://github.com/allanjude/zfs/commit/3fef3c83b8ce90149110ed989bd9fd3e2= 89798e0 I am still working on a solution for setting the zstd feature flag to 'active' as soon as it is set, rather than only once a block is born. Additionally, I am investigating how to best handle the fact that embedded block pointers compressed with ZSTD will make 'zfs send -e' streams backwards incompatible, without a way for the user to opt-out of sending a stream that contains zstd compressed blocks that the receiving side may not be able to read. The same can be said for 'zfs send -c' as well. I am open to ideas on how best to handle this. I have thought about only sending ZSTD compressed blocks if the user specifies the -Z (--zstd) flag, but this can lead to confusion where using -c without -Z would have to either error out, or send the ZSTD compressed blocks uncompressed. This project is sponsored by the FreeBSD Foundation. --=20 Allan Jude --3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P-- --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfA7ysAAoJEBmVNT4SmAt+6vIP/A0FQvM5pFtur2ReGoFt0HxC Hj58PfJCBIPDfcwrdkBN4LsfRzb2jymlF8PyVqzvHkC0scGyaDiOSWu2EjEgtYlD xpCKByvfguEiME2vZ0Dlh1571SA0go0jJdS76RkehvKDX6i6EekWksCyUbIX4LDs ft2am63KFz9WKAG5PxOfZn94mKLODbfxzwf9KCpUAz0zwvz5ej07kyFblhJCiR35 cGvQOPH9XlOibos97XZsL2XEWt2gsxJv9qFslvAjOzK6XAtQfjLw52XaOSOp3Abe iEFxMLWm8i0s14BkgPh/bHmv7wB/TN/nwgIwoNeLMNQpCPZLYLg069sBY2gwNaWo yZ9LK836fzbJ/DbBRlaZzrI+Nrs8Bp28ZSmCGDD6WX/wpt2gyPKLJnrAKd9XNm0E WpqD1omGS1pCU/NiKoXpFbbid5S1O40bF4NNF0hIAVaPYlEES/M3mcdhg4ONwxa9 LdLcALGsyGrAzXwG6UmZQfcr80Tf6IGwiNsGEtnV4+FOK2yZu6MB1KA3v3FpLjqZ uASOqHxLMwILWWoHtAD7ggeX2BelfFfkDU3PVMhPs75SNEBNtAD6eouIxKzrIpOG /RMHbXEFr1Grba7qJ96f9avGvqBSndYUbfL4+3C9L9Rr+uhwiUBju0VQBZtT3IP5 PvEjG3n4Ty/q0/jJApDF =BenT -----END PGP SIGNATURE----- --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bebcc0bb-7590-a04b-09ae-fa04e22d27dc>