From owner-freebsd-fs@freebsd.org Wed Aug 26 02:22:44 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BC6913C6242 for ; Wed, 26 Aug 2020 02:22:44 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [209.51.186.6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BbqQD2QMNz4WkY for ; Wed, 26 Aug 2020 02:22:44 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id 6659B1E6CE; Wed, 26 Aug 2020 02:22:37 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net 6659B1E6CE From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: Date: Tue, 25 Aug 2020 22:22:33 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="oCsWTS9EN6xchYXf4VnVcjAQoSXZYLuGe" X-Rspamd-Queue-Id: 4BbqQD2QMNz4WkY X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; local_wl_from(0.00)[freebsd.org]; ASN(0.00)[asn:6939, ipnet:209.51.160.0/19, country:US] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Aug 2020 02:22:44 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --oCsWTS9EN6xchYXf4VnVcjAQoSXZYLuGe Content-Type: multipart/mixed; boundary="wv7SQW4dBTHm9wc9shJIOMU1u6Rdt0du3"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> In-Reply-To: <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> --wv7SQW4dBTHm9wc9shJIOMU1u6Rdt0du3 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is the tenth weekly status report on my FreeBSD Foundation sponsored project to complete the integration of ZSTD compression into OpenZFS. Late last week the main pull request was merged, and ZSTD support is now part of OpenZFS's trunk branch. Last night, OpenZFS with ZSTD was imported into FreeBSD's -current branch= =2E I am continuing to work on a number of things related to ZSTD, including future-proofing support (so upgrading ZSTD won't cause problems with features like nopwrite), and improving the integration of ZSTD into FreeBSD, including enabling support for booting from ZSTD compressed datasets, and improving the performance of ZSTD on FreeBSD. I'll also be adding some additional tests to make sure we detect any issues when we do look at updating ZSTD. Additionally, I am working on a bunch of documentation around using ZSTD in ZFS. For my benchmarking of ZSTD, I have been using a zfs recv of a stream in a file on a tmpfs, and recording how long it takes to receive and sync the data. The test data is a copy of the FreeBSD 12.1 source code, since that is easily reproducible. Does anyone have experience or a better suggestion on how to get the most consistent and repeatable results when benchmarking like this? On 2020-08-18 18:51, Allan Jude wrote: > This is the ninth weekly status report on my FreeBSD Foundation > sponsored project to complete the integration of ZSTD compression into > OpenZFS. >=20 > https://github.com/openzfs/zfs/pull/10693 - The L2ARC fixes (for when > compressed ARC is disabled) have been merged. >=20 > https://github.com/openzfs/zfs/pull/10278/ - A number of other cleanups= > and fixes for the ZSTD have been integrated and squashed, and it looks > like the completed ZSTD feature will be merged very soon. >=20 > This included a bunch of fixes for makefiles and runfiles to hook the > tests I added up to the ZFS test suite so they are run properly. >=20 > It looks like this will mean that the ZSTD feature will be included in > OpenZFS 2.0. Thanks for everyone who has tested, reviewed, or > contributed to this effort, especially those who kept it alive while I > was working on other things. >=20 > Post-merge, the remaining work is to develop future-proofing around ZST= D > so that we will be able to more seamlessly upgrade to newer releases of= > ZSTD. Recompression of the same input resulting in the same on-disk > checksum is the main concern, as without this upgrading the compression= > algorithm will break features like nop-write. >=20 > This project is sponsored by the FreeBSD Foundation. >=20 --=20 Allan Jude --wv7SQW4dBTHm9wc9shJIOMU1u6Rdt0du3-- --oCsWTS9EN6xchYXf4VnVcjAQoSXZYLuGe Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfRcdsAAoJEBmVNT4SmAt+tpoQAN1PAjBf3b9hIyhRZ0foAiSP 6Yz7KeNNwvUHb7tlYrEpKIMiYftOzqVRM1hzkn1fZvjcbET0avAXFP7tlfKGADiK VVOZJuT4QcF3FQazG7eeOG1Qe6kQNXmqqMsV7kipJtHAdGGy3pjS701K6K2wg7ka J1vLcyDkMk+w++IVQlrQwI/WocActnaraeBLqk8aj8SX0tsj4NpU9BEaSslKeiZU 873MyiY/QVfITahuKjoKXq7Hj6cBQPW5KGVZ9LBlXeYXp9usIZZ+i8XR9xHFpXa6 dSVL79o4p4nTlNghZ9wLKfkGtN+Isd7atCQkdHZK9I9Xdb6enlYw8nkpVsvheFWP vOw+cE25fTAYAaXKa7X5UTV0+Ns9H6by5/fBlblqpkE4ZJK+dERfiQCRaYdti02U blH3MbfC8vFk485HMKK2M43/IJyjNrbca+NUIpI4ry3ipxTE3MsFJYf5iZNpSzcC RlF9QJpFJWkI6ewjYxijNtw/74441H+rUSFQ00GKTlLiJUKzy+sLg4rPjRjgku1W HJ1ffF/ZROiL+sH2KaPSUXukTiNJ8DMMmLvvb1nBoJ1cjPJUIV3NbedsmyQ8Z8VK gCz1mrjASM4GfFwaWG3gW4FWGVXxREwJpU2YqlSY0DZkDjWXhq5X77F0LG5d/Eli 8vbIHl4QCJafj4+FIvSX =MofC -----END PGP SIGNATURE----- --oCsWTS9EN6xchYXf4VnVcjAQoSXZYLuGe--