From owner-freebsd-fs@freebsd.org Tue Jul 7 00:07:16 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C79C834D30C for ; Tue, 7 Jul 2020 00:07:16 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [IPv6:2001:470:1:474::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B12mz6CyRz3W0G for ; Tue, 7 Jul 2020 00:07:15 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 5AE21276B9; Tue, 7 Jul 2020 00:07:09 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 5AE21276B9 From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: Date: Mon, 6 Jul 2020 20:07:05 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0 MIME-Version: 1.0 In-Reply-To: <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO" X-Rspamd-Queue-Id: 4B12mz6CyRz3W0G X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Jul 2020 00:07:16 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO Content-Type: multipart/mixed; boundary="3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> In-Reply-To: <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> --3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable In my continuing effort to complete the integration of ZSTD into OpenZFS, here is my third weekly status report: https://github.com/allanjude/zfs/commit/87bb538bdbc4bb8848ed5791b7b0de84a= 026ebbe - Completed the rewrite of the way the compression property is handled, moving away from the initial approach of storing the compression property (enum zio_compress) and the level (uint64_t) separately. Previously we exposed the list of compression algorithms and levels to userland as the corresponding value from the enum in the lower 7 bits, and the level in the remaining upper bits. Then, as part of the property GET and SET IOCTLs, we read the separate compression=3D and compress_level=3D properties from the ZAP and returned the combined value= , or split the combined value into those two separate properties. This worked but caused a lot of headache around property inheritance. Instead I've changed to doing the combine/split when reading/writing from the dataset properties ZAP, via the compression_changed_cb() function. So the properties ZAP contains the combined value (lower 7 bits are the compression algorithm, as defined in the enum zio_compress, and the upper bits are the compression level). Elsewhere in ZFS we keep the two values separate (os_compress and os_complevel, and related variables in all of the different parts of ZFS). So now, inheritance of the property is handled correctly, and avoids issues where a dataset with compression=3Dzstd-12, would say 'inherited from' a dataset with zstd at some other compression level (since both actually just had compression=3Dzstd, but different compress_level=3D val= ues). I have also further extended zdb to inspect the compression settings when looking at an object: https://github.com/allanjude/zfs/commit/3fef3c83b8ce90149110ed989bd9fd3e2= 89798e0 I am still working on a solution for setting the zstd feature flag to 'active' as soon as it is set, rather than only once a block is born. Additionally, I am investigating how to best handle the fact that embedded block pointers compressed with ZSTD will make 'zfs send -e' streams backwards incompatible, without a way for the user to opt-out of sending a stream that contains zstd compressed blocks that the receiving side may not be able to read. The same can be said for 'zfs send -c' as well. I am open to ideas on how best to handle this. I have thought about only sending ZSTD compressed blocks if the user specifies the -Z (--zstd) flag, but this can lead to confusion where using -c without -Z would have to either error out, or send the ZSTD compressed blocks uncompressed. This project is sponsored by the FreeBSD Foundation. --=20 Allan Jude --3WEGNZX5Y0W0zU3jKHglOROtcMQeUWE6P-- --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfA7ysAAoJEBmVNT4SmAt+6vIP/A0FQvM5pFtur2ReGoFt0HxC Hj58PfJCBIPDfcwrdkBN4LsfRzb2jymlF8PyVqzvHkC0scGyaDiOSWu2EjEgtYlD xpCKByvfguEiME2vZ0Dlh1571SA0go0jJdS76RkehvKDX6i6EekWksCyUbIX4LDs ft2am63KFz9WKAG5PxOfZn94mKLODbfxzwf9KCpUAz0zwvz5ej07kyFblhJCiR35 cGvQOPH9XlOibos97XZsL2XEWt2gsxJv9qFslvAjOzK6XAtQfjLw52XaOSOp3Abe iEFxMLWm8i0s14BkgPh/bHmv7wB/TN/nwgIwoNeLMNQpCPZLYLg069sBY2gwNaWo yZ9LK836fzbJ/DbBRlaZzrI+Nrs8Bp28ZSmCGDD6WX/wpt2gyPKLJnrAKd9XNm0E WpqD1omGS1pCU/NiKoXpFbbid5S1O40bF4NNF0hIAVaPYlEES/M3mcdhg4ONwxa9 LdLcALGsyGrAzXwG6UmZQfcr80Tf6IGwiNsGEtnV4+FOK2yZu6MB1KA3v3FpLjqZ uASOqHxLMwILWWoHtAD7ggeX2BelfFfkDU3PVMhPs75SNEBNtAD6eouIxKzrIpOG /RMHbXEFr1Grba7qJ96f9avGvqBSndYUbfL4+3C9L9Rr+uhwiUBju0VQBZtT3IP5 PvEjG3n4Ty/q0/jJApDF =BenT -----END PGP SIGNATURE----- --oaUCGgfwn4lMV2XqNmgjBcqyg8XSYoMfO--