From owner-freebsd-fs@freebsd.org Wed Jul 15 02:26:15 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2ED45356EAC for ; Wed, 15 Jul 2020 02:26:15 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4B61Td1gW6z3YKt for ; Wed, 15 Jul 2020 02:26:12 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 5686A24C48; Wed, 15 Jul 2020 02:26:06 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 5686A24C48 From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> Date: Tue, 14 Jul 2020 22:26:01 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="11rZcGtLSc3Dz7SCZJTTfFXDO0W11n4If" X-Rspamd-Queue-Id: 4B61Td1gW6z3YKt X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 15 Jul 2020 02:26:15 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --11rZcGtLSc3Dz7SCZJTTfFXDO0W11n4If Content-Type: multipart/mixed; boundary="66UlQ09ydSSLlog0hd3djPINvizOVW1Iw"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> In-Reply-To: --66UlQ09ydSSLlog0hd3djPINvizOVW1Iw Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable In my continuing effort to complete the integration of ZSTD into OpenZFS, here is my fourth weekly status report: https://github.com/allanjude/zfs/commit/b0b1270d4e7835ecff413208301375e3d= e2a4153 - Create a new test case to make sure that the ZSTD header we write along with the data is correct. Verify that the physical size of the compressed data is less than the psize for the block pointer, and verify that the level matches. It uses a random level between 1 and 19 and then verifies with zdb that the block was compressed with that level. I am still working on a solution for setting the zstd feature flag to 'active' as soon as it is set, rather than only once a block is born. As well as fixing up compatibility around zfs send/recv with the embedded block points flag. This project is sponsored by the FreeBSD Foundation. On 2020-07-06 20:07, Allan Jude wrote: > In my continuing effort to complete the integration of ZSTD into > OpenZFS, here is my third weekly status report: >=20 > https://github.com/allanjude/zfs/commit/87bb538bdbc4bb8848ed5791b7b0de8= 4a026ebbe > - Completed the rewrite of the way the compression property is handled,= > moving away from the initial approach of storing the compression > property (enum zio_compress) and the level (uint64_t) separately. >=20 > Previously we exposed the list of compression algorithms and levels to > userland as the corresponding value from the enum in the lower 7 bits, > and the level in the remaining upper bits. Then, as part of the propert= y > GET and SET IOCTLs, we read the separate compression=3D and > compress_level=3D properties from the ZAP and returned the combined val= ue, > or split the combined value into those two separate properties. This > worked but caused a lot of headache around property inheritance. >=20 > Instead I've changed to doing the combine/split when reading/writing > from the dataset properties ZAP, via the compression_changed_cb() > function. So the properties ZAP contains the combined value (lower 7 > bits are the compression algorithm, as defined in the enum zio_compress= , > and the upper bits are the compression level). Elsewhere in ZFS we keep= > the two values separate (os_compress and os_complevel, and related > variables in all of the different parts of ZFS). >=20 > So now, inheritance of the property is handled correctly, and avoids > issues where a dataset with compression=3Dzstd-12, would say 'inherited= > from' a dataset with zstd at some other compression level (since both > actually just had compression=3Dzstd, but different compress_level=3D v= alues). >=20 >=20 > I have also further extended zdb to inspect the compression settings > when looking at an object: > https://github.com/allanjude/zfs/commit/3fef3c83b8ce90149110ed989bd9fd3= e289798e0 >=20 >=20 > I am still working on a solution for setting the zstd feature flag to > 'active' as soon as it is set, rather than only once a block is born. >=20 >=20 > Additionally, I am investigating how to best handle the fact that > embedded block pointers compressed with ZSTD will make 'zfs send -e' > streams backwards incompatible, without a way for the user to opt-out o= f > sending a stream that contains zstd compressed blocks that the receivin= g > side may not be able to read. The same can be said for 'zfs send -c' as= > well. I am open to ideas on how best to handle this. I have thought > about only sending ZSTD compressed blocks if the user specifies the -Z > (--zstd) flag, but this can lead to confusion where using -c without -Z= > would have to either error out, or send the ZSTD compressed blocks > uncompressed. >=20 >=20 > This project is sponsored by the FreeBSD Foundation. >=20 --=20 Allan Jude --66UlQ09ydSSLlog0hd3djPINvizOVW1Iw-- --11rZcGtLSc3Dz7SCZJTTfFXDO0W11n4If Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfDmk9AAoJEBmVNT4SmAt+2/wP/RjlyPLcqL7mOplPW54P2uuN q6xgpWvk2odbZz6LOuG4qvo45+WU27rOJpDc1M/U8vs++kYHNfPt/CO9IjHkgMod o7FPPri+evJhnxpoIuK7c7wKfgUvyyzzxgT1MRfVjH0G3f/323Z22Bu5tXYPaN5d PnhCRQlcZyiaXwYneh1eTBFcXh1g3v0mUdm4o0gSrpAJCWCUDyx/6J5AKlZQRAnW kAUXOHrQ7vmA8uZeo6A4hkLiVKszLpTEeDuSxyTWpEXboUUyQYyTGvGAkc1xfLFO DGTMO66hjIfmdkmJAd4Pq8hx4GorX2gZWx21nWxpaqHdDnSj5fLCP55+zrL3FqEt E8IOUzxV9IbRW2IQAuPU2a2/BUEe7oOsaiIpDflb+qi0TGsn9rAGHvBLdKvwmK/f 1Gudt64woGHAVYd2ikCV0NdmwWxTw0r3YULbC7FhGRGMsWMBzWrh4QzkUNkcnVT1 MAV+NjewPXAKe+BIOEQJnxR3xT+/SCh0ZRkZ3dI9jgLPRpgyURzGDXkDajP9IB56 /3KV4vtpZwT3MplCpEKi9qpi7pfPqOCbAzZKFRt9tECnJ0Qv7cP0rb3JxtLEqhgv /Hopl/cjoum2EpLoSl60OlQnUfRStGRIRc/mp/5k/yYm9oR2EE+DUh4ojSGEhqZY eJyDc+7hMFnwh66N9K4U =ge0w -----END PGP SIGNATURE----- --11rZcGtLSc3Dz7SCZJTTfFXDO0W11n4If--