Date: Mon, 20 Jul 2020 23:40:14 -0400 From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Subject: Re: ZSTD Project Weekly Status Update Message-ID: <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> In-Reply-To: <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --BzphuFD49KUoqTjS8pR7SJVV9MAdN0UlC Content-Type: multipart/mixed; boundary="7UCzvkUxDyNlbe9CvgYPdX9ObTi9Pk87U"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Message-ID: <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> In-Reply-To: <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> --7UCzvkUxDyNlbe9CvgYPdX9ObTi9Pk87U Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is the fifth weekly status report on the project to integrate ZSTD into OpenZFS. https://github.com/c0d3z3r0/zfs/pull/14/commits/9807c99169e5931a754bb0df6= 8267ffa2f289474 - Created a new test case to ensure that ZSTD compressed blocks survive replication with the -c flag. We wanted to make sure the on-disk compression header survived the trip. https://github.com/c0d3z3r0/zfs/pull/14/commits/94bef464fc304e9d6f5850391= e41720c3955af11 - I split the zstd.c file into OS specific bits (module/os/{linux,freebsd}/zstd_os.c) and also split the .h file into zstd.h and zstd_impl.h. This was done so that FreeBSD can use the existing kmem_cache mechanism, while Linux can use the vmem_alloc pool created in the earlier versions of this patch. I significantly changed the FreeBSD implementation from my earlier work, to reuse the power of 2 zio_data_buf_cache[]'s that already exist, only adding a few additional kmem_caches for large blocks with high compression levels. This should avoid creating as many unnecessary kmem caches. https://github.com/c0d3z3r0/zfs/pull/14/commits/3d48243b77e6c8c3bf562c7a2= 315dd6cc571f28c - Lastly, in my testing I was seeing a lot of hits on the new compression failure kstat I added. This was caused by the ZFS "early abort" feature, where we give the compressor an output buffer that is smaller than the input, so it will fail if the block will not compress enough to be worth it. This helps avoid wasting CPU on uncompressible blocks. However, it seems the 'one-file' version of zstd we are using does not expose the ZSTD_ErrorCode enum. This needs to be investigated further to avoid issues if the value changes (although it is apparently stable after version 1.3.1). I am still working on a solution for zfs send stream compatibility. I am leaning towards creating a new flag, --zstd, to enable ZSTD compressed output. If the -e or -c flag are used without the --zstd flag, and the dataset has the zstd feature active, the idea would be to emit a warning but send the blocks uncompressed, so that the stream remains compatible with older versions of ZFS. I will be discussing this on the OpenZFS Leadership call tomorrow, and am open to suggestions on how to best handle this. On 2020-07-14 22:26, Allan Jude wrote: > In my continuing effort to complete the integration of ZSTD into > OpenZFS, here is my fourth weekly status report: >=20 > https://github.com/allanjude/zfs/commit/b0b1270d4e7835ecff413208301375e= 3de2a4153 > - Create a new test case to make sure that the ZSTD header we write > along with the data is correct. Verify that the physical size of the > compressed data is less than the psize for the block pointer, and verif= y > that the level matches. It uses a random level between 1 and 19 and the= n > verifies with zdb that the block was compressed with that level. >=20 > I am still working on a solution for setting the zstd feature flag to > 'active' as soon as it is set, rather than only once a block is born. A= s > well as fixing up compatibility around zfs send/recv with the embedded > block points flag. >=20 > This project is sponsored by the FreeBSD Foundation. >=20 >=20 --=20 Allan Jude --7UCzvkUxDyNlbe9CvgYPdX9ObTi9Pk87U-- --BzphuFD49KUoqTjS8pR7SJVV9MAdN0UlC Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfFmOhAAoJEBmVNT4SmAt+aoMP/A5oJmKyp7lEkG7yFf+glHb3 IYKF2eWjVcCogLGMlQKdFlxXq+D6osHHpUacAsN2W/E8/3fXWPka9ybdKRfXN6Km v40PvpixAHN0GrOT4bnspUj/SrwVPiglj/nwy/K0Nv7vZdLq6Xw9/dQoElYWapKd Ht3zYH3zBTJQlxVEyvmvdaQAwzuVSJ4RtyHG2F07A0+ojQmpmrS3ZwlqiWRov2/w rLVemzC9iL7GkCsL6L6qCQddTcmYoZoIhk5UzmI4eN4jXCJZ3jWUfWYG1u3W7qPv PtHEyPBdEvafzyEfojKXhNccoHmY0L7kE8zq2URIpcWc51tsfyvEcXHv4XT1aPBF 7Pm1DVGO3uf6uQkwyFcNUAFz1jggnXD+F6FiFhRHdzplQICeJlSGvtUvIu/jdmHM NReTuOiF8DvnB2RE67V4gHpXfx6DuDaCIvoakiO7FmFdU7LufrmEtXDoivhsRR0M wr7O4JWWDZnJ+fibRTIuPaLTXUfMmOKUX9om7idjiY5cyU43QzKieLcspkoIrBKB xGo5QLwK5NSMgJF8vkW7Rt565cGzteEPbqv6ik4aDwjMnLMQhqg8kVnHpNVS4QC1 ZerihdbyNdnBw5nLW547bAJwIW6PGAi92IsfqZXCi0Hi1IZ9+QjncMqm2Lje77nG 0uNVQcvCJv9xUJV1c+wT =kdXF -----END PGP SIGNATURE----- --BzphuFD49KUoqTjS8pR7SJVV9MAdN0UlC--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9d77cb73-c8e8-cca0-b4b8-28e6790268d6>