Date: Wed, 23 Sep 2020 22:03:20 -0400 From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Subject: Re: ZSTD Project Weekly Status Update Message-ID: <4a2cd277-cde6-69b7-a0e4-0af03f37be9a@freebsd.org> In-Reply-To: <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org> References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <db71835b-9bb7-2722-fd02-194b97f1564e@freebsd.org> <e9597d9b-88e0-334f-d266-6cbbaf746855@freebsd.org> <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> <ce721076-962a-ddf4-6886-0eafbbb418b1@freebsd.org> <9f4ff5f0-9b6c-7299-98ee-988964a11ade@freebsd.org> <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --wzZBuWVSCdIIPHAHLNDEwlbiIt18woj3S Content-Type: multipart/mixed; boundary="euJYirddoC7x01r1Q5f4ETxuDFYh9fozp"; protected-headers="v1" From: Allan Jude <allanjude@freebsd.org> To: status-updates@freebsdfoundation.org, freebsd-fs <freebsd-fs@freebsd.org>, openzfs-developer <developer@open-zfs.org> Message-ID: <4a2cd277-cde6-69b7-a0e4-0af03f37be9a@freebsd.org> Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <bebcc0bb-7590-a04b-09ae-fa04e22d27dc@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <db71835b-9bb7-2722-fd02-194b97f1564e@freebsd.org> <e9597d9b-88e0-334f-d266-6cbbaf746855@freebsd.org> <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> <ce721076-962a-ddf4-6886-0eafbbb418b1@freebsd.org> <9f4ff5f0-9b6c-7299-98ee-988964a11ade@freebsd.org> <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org> In-Reply-To: <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org> --euJYirddoC7x01r1Q5f4ETxuDFYh9fozp Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is another weekly status report on my FreeBSD Foundation sponsored project to complete the integration of ZSTD compression into OpenZFS. I have continued to work on the benchmarks, including attempting to compare compression and performance across various settings with pgbench (PostgreSQL). However the results are not as meaningful as I would like so far. I am still trying some other configurations to try to get clearer results. Using multiple md(4) devices has helped avoid that being the bottleneck. https://docs.google.com/spreadsheets/d/1TvCAIDzFsjuLuea7124q-1UtMd0C9amTg= nXm2yPtiUQ/edit?usp=3Dsharing Additionally, I completed rewriting my earlier commit to convert the FreeBSD versions of zstd to use FreeBSD's UMA kernel memory allocator, rather than the custom mempool used on Linux due to its kmem_cache abstraction not being able to handle large allocations: https://github.com/openzfs/zfs/pull/10975 I am working on some benchmarks of this change as well. Thank you. On 2020-09-15 22:11, Allan Jude wrote: > This is another weekly status report on my FreeBSD Foundation sponsored= > project to complete the integration of ZSTD compression into OpenZFS. >=20 > The first batch of benchmarks are complete, although they took longer > than expected to get good data. >=20 > I am still not entirely pleased with the data, as in some cases I am > running up against limitations of my device-under-test rather than the > performance limits of ZFS. >=20 > Here is what I have so far: >=20 > https://docs.google.com/spreadsheets/d/1TvCAIDzFsjuLuea7124q-1UtMd0C9am= TgnXm2yPtiUQ/edit?usp=3Dsharing >=20 > A number of these tests were initially done on both FreeBSD and Linux o= n > the same machine, and the results were consistent within a 2% margin of= > error, so I've taken to doing most of the tests only on FreeBSD, becaus= e > it is easier. I've struggled to get a good ramdisk solution on Ubuntu e= tc. >=20 > To walk you through the different tabs in the spreadsheet so far: >=20 > #1: fio SSD > This is a random write test to my pool made of 4 SSDs. This ran into th= e > performance limitations of the SSDs when testing the very fast > algorithms. Since the data generated by fio is completely > uncompressible, there is no gain from the higher compression levels. >=20 > #2: fio to ramdisk > To overcome the limitations of the first test, I did it again with a > ramdisk. Obviously this had to be a smaller dataset, since there is > limited memory available, but it does a much better job of showing how > the zstd-fast levels scale, and how they outperform LZ4, although you > cannot compare the compression, because the data is uncompressible. >=20 > #3: zfs recv to SSD > For this test, I created a dataset by extracting the FreeBSD src.txz > file 8 times (each to a different directory), then created a snapshot o= f > that, and send it to a file on a tmpfs. >=20 > I then timed zfs recv < /tmpfs/snapshot.zfs with each compression > algorithm. This allows you to compare the compression gain for the time= > trade-off, but again ran into the throughput limitations of the SSDs, s= o > provides a bit less information about the performance of the higher > zstd-fast levels, but you can see the compression tradeoff. >=20 > I need to reconfigure my setup to re-do this benchmark using a ramdisk.= >=20 > #4: large image file 128k > For this, i created an approximately 20GB tar file, by unxz'ding the > FreeBSD 12.1 src.txz and concatenating it 16 times. This provides the > best possible case for compression. >=20 > One of the major advantages of ZSTD is that the decompression throughpu= t > stays relatively the same even as the compression level is increased. S= o > while writing a zstd-19 compressed block takes a lot longer than a > zstd-3 compressed block, both decompress at nearly the same speed. >=20 > This time I measured fio random read performance. Putting the > limitations of the SSDs to good use, this test shows the read > performance gains from reading compressed. Even though the disks top ou= t > around 1.5 GB/sec, zstd-compressed data can be read at an effective rat= e > of over 5 GB/sec. >=20 > #5: large image file 1m > This is the same test, but done with zfs recordsize=3D1m >=20 > The larger record size unlocks higher compression ratios, and achieves > throughputs in excess of 6 GB/sec. >=20 > #6: large image file 16k > This is again the same test, but with zfs recordsize=3D16k > This is an approximation of reading from a large database with a 16k > page size. > The lower record size provides much less compression, and the smaller > blocks result in more overhead, but, there are still large performance > gains to be had from the compression, although they are much less drast= ic. >=20 > I would be interested in what other tests people might be interested in= > seeing before I finish wearing these SSDs out. >=20 >=20 > Thanks again to the FreeBSD Foundation for sponsoring this work. >=20 >=20 --=20 Allan Jude --euJYirddoC7x01r1Q5f4ETxuDFYh9fozp-- --wzZBuWVSCdIIPHAHLNDEwlbiIt18woj3S Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfa/52AAoJEBmVNT4SmAt+UVwP/jrsSIz9z+MQtXfnPU4OjDgT gq5q6B3JgRmvoTuSd265ZxcoQCyN3rQrOqoHZDZsNgxX/kuu5XtxlTOGOqHo5yUo jwTU66s6qYVbmsnLui0kS3gvt9cFyzyfqG28n1wIt4KQXz3zlunH4LNDFzGqd2V4 RZyHioQ2cYzRlGfvSSiNXscXvB7tJi7M2/4y3tWY2pI9hbcenSbrdbLJBXwqDXtc kbt6rqG61xlCdk5Kd4IlVS755C6aYC+teq8qyrxvRMuOOTXku71HjUJK9nhAOsG+ hm34qSYI3CReegX3EoWHQcvzVsino1DzG82W+rqFAUXEwxrLzhAPwEdHLezDiJz6 uLY6EELUbvX3Cc6mWoPRwYXDh/DvWez0qk6JF9a/zuYVOYJ3Nvg5/AgWAxgzlHi9 ywHU6RhurFQI22C8TrXKjrBPYr5kYA+gTwiZTc+xsf+ezoT5GhqN4uSrWb6wDBU1 plSQmd7IZBWpMXSQP2gXc/Xi+0rQ/m8bG9AM8isfI/jyhUvf/B6nnHPWCoGn9jnY QS6sQPQHXJPwzL5E2c75B5ZWoYUGwzRDolUgCl88PkWILuFlEaawUKSr06g302hA JYM+RJ2EC8h7jyRgr2nZNohAb82XVZMY3IHF1E33/TRnYw38izJ0Kojg5fJYEWho RCnJbA5vmz7cKdJKw2ky =Fz6/ -----END PGP SIGNATURE----- --wzZBuWVSCdIIPHAHLNDEwlbiIt18woj3S--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4a2cd277-cde6-69b7-a0e4-0af03f37be9a>