From owner-freebsd-fs@freebsd.org Thu Sep 24 02:03:42 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 40B403E8D54 for ; Thu, 24 Sep 2020 02:03:42 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from tor1-11.mx.scaleengine.net (tor1-11.mx.scaleengine.net [IPv6:2001:470:1:474::25]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bxdcs5Gp3z3VsC for ; Thu, 24 Sep 2020 02:03:41 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by tor1-11.mx.scaleengine.net (Postfix) with ESMTPSA id C374EE6B6; Thu, 24 Sep 2020 02:03:35 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 tor1-11.mx.scaleengine.net C374EE6B6 From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> <9f4ff5f0-9b6c-7299-98ee-988964a11ade@freebsd.org> <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org> Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Subject: Re: ZSTD Project Weekly Status Update Message-ID: <4a2cd277-cde6-69b7-a0e4-0af03f37be9a@freebsd.org> Date: Wed, 23 Sep 2020 22:03:20 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.12.0 MIME-Version: 1.0 In-Reply-To: <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wzZBuWVSCdIIPHAHLNDEwlbiIt18woj3S" X-Rspamd-Queue-Id: 4Bxdcs5Gp3z3VsC X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Sep 2020 02:03:42 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --wzZBuWVSCdIIPHAHLNDEwlbiIt18woj3S Content-Type: multipart/mixed; boundary="euJYirddoC7x01r1Q5f4ETxuDFYh9fozp"; protected-headers="v1" From: Allan Jude To: status-updates@freebsdfoundation.org, freebsd-fs , openzfs-developer Message-ID: <4a2cd277-cde6-69b7-a0e4-0af03f37be9a@freebsd.org> Subject: Re: ZSTD Project Weekly Status Update References: <7b8842ad-d520-c575-22ee-2cd77244f2c6@freebsd.org> <708ec9f2-3c5c-6452-f6e6-bfb11a7f7eb2@freebsd.org> <528ca743-7889-d1fd-ca95-a17cd430725b@freebsd.org> <9d77cb73-c8e8-cca0-b4b8-28e6790268d6@freebsd.org> <327f4b10-9727-331e-2dc9-641dad96dd2a@freebsd.org> <738e1ca9-05b6-bc1f-468c-b5eee03643ab@freebsd.org> <9f4ff5f0-9b6c-7299-98ee-988964a11ade@freebsd.org> <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org> In-Reply-To: <761f6571-87ae-679c-a3e3-316dbb16200b@freebsd.org> --euJYirddoC7x01r1Q5f4ETxuDFYh9fozp Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable This is another weekly status report on my FreeBSD Foundation sponsored project to complete the integration of ZSTD compression into OpenZFS. I have continued to work on the benchmarks, including attempting to compare compression and performance across various settings with pgbench (PostgreSQL). However the results are not as meaningful as I would like so far. I am still trying some other configurations to try to get clearer results. Using multiple md(4) devices has helped avoid that being the bottleneck. https://docs.google.com/spreadsheets/d/1TvCAIDzFsjuLuea7124q-1UtMd0C9amTg= nXm2yPtiUQ/edit?usp=3Dsharing Additionally, I completed rewriting my earlier commit to convert the FreeBSD versions of zstd to use FreeBSD's UMA kernel memory allocator, rather than the custom mempool used on Linux due to its kmem_cache abstraction not being able to handle large allocations: https://github.com/openzfs/zfs/pull/10975 I am working on some benchmarks of this change as well. Thank you. On 2020-09-15 22:11, Allan Jude wrote: > This is another weekly status report on my FreeBSD Foundation sponsored= > project to complete the integration of ZSTD compression into OpenZFS. >=20 > The first batch of benchmarks are complete, although they took longer > than expected to get good data. >=20 > I am still not entirely pleased with the data, as in some cases I am > running up against limitations of my device-under-test rather than the > performance limits of ZFS. >=20 > Here is what I have so far: >=20 > https://docs.google.com/spreadsheets/d/1TvCAIDzFsjuLuea7124q-1UtMd0C9am= TgnXm2yPtiUQ/edit?usp=3Dsharing >=20 > A number of these tests were initially done on both FreeBSD and Linux o= n > the same machine, and the results were consistent within a 2% margin of= > error, so I've taken to doing most of the tests only on FreeBSD, becaus= e > it is easier. I've struggled to get a good ramdisk solution on Ubuntu e= tc. >=20 > To walk you through the different tabs in the spreadsheet so far: >=20 > #1: fio SSD > This is a random write test to my pool made of 4 SSDs. This ran into th= e > performance limitations of the SSDs when testing the very fast > algorithms. Since the data generated by fio is completely > uncompressible, there is no gain from the higher compression levels. >=20 > #2: fio to ramdisk > To overcome the limitations of the first test, I did it again with a > ramdisk. Obviously this had to be a smaller dataset, since there is > limited memory available, but it does a much better job of showing how > the zstd-fast levels scale, and how they outperform LZ4, although you > cannot compare the compression, because the data is uncompressible. >=20 > #3: zfs recv to SSD > For this test, I created a dataset by extracting the FreeBSD src.txz > file 8 times (each to a different directory), then created a snapshot o= f > that, and send it to a file on a tmpfs. >=20 > I then timed zfs recv < /tmpfs/snapshot.zfs with each compression > algorithm. This allows you to compare the compression gain for the time= > trade-off, but again ran into the throughput limitations of the SSDs, s= o > provides a bit less information about the performance of the higher > zstd-fast levels, but you can see the compression tradeoff. >=20 > I need to reconfigure my setup to re-do this benchmark using a ramdisk.= >=20 > #4: large image file 128k > For this, i created an approximately 20GB tar file, by unxz'ding the > FreeBSD 12.1 src.txz and concatenating it 16 times. This provides the > best possible case for compression. >=20 > One of the major advantages of ZSTD is that the decompression throughpu= t > stays relatively the same even as the compression level is increased. S= o > while writing a zstd-19 compressed block takes a lot longer than a > zstd-3 compressed block, both decompress at nearly the same speed. >=20 > This time I measured fio random read performance. Putting the > limitations of the SSDs to good use, this test shows the read > performance gains from reading compressed. Even though the disks top ou= t > around 1.5 GB/sec, zstd-compressed data can be read at an effective rat= e > of over 5 GB/sec. >=20 > #5: large image file 1m > This is the same test, but done with zfs recordsize=3D1m >=20 > The larger record size unlocks higher compression ratios, and achieves > throughputs in excess of 6 GB/sec. >=20 > #6: large image file 16k > This is again the same test, but with zfs recordsize=3D16k > This is an approximation of reading from a large database with a 16k > page size. > The lower record size provides much less compression, and the smaller > blocks result in more overhead, but, there are still large performance > gains to be had from the compression, although they are much less drast= ic. >=20 > I would be interested in what other tests people might be interested in= > seeing before I finish wearing these SSDs out. >=20 >=20 > Thanks again to the FreeBSD Foundation for sponsoring this work. >=20 >=20 --=20 Allan Jude --euJYirddoC7x01r1Q5f4ETxuDFYh9fozp-- --wzZBuWVSCdIIPHAHLNDEwlbiIt18woj3S Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJfa/52AAoJEBmVNT4SmAt+UVwP/jrsSIz9z+MQtXfnPU4OjDgT gq5q6B3JgRmvoTuSd265ZxcoQCyN3rQrOqoHZDZsNgxX/kuu5XtxlTOGOqHo5yUo jwTU66s6qYVbmsnLui0kS3gvt9cFyzyfqG28n1wIt4KQXz3zlunH4LNDFzGqd2V4 RZyHioQ2cYzRlGfvSSiNXscXvB7tJi7M2/4y3tWY2pI9hbcenSbrdbLJBXwqDXtc kbt6rqG61xlCdk5Kd4IlVS755C6aYC+teq8qyrxvRMuOOTXku71HjUJK9nhAOsG+ hm34qSYI3CReegX3EoWHQcvzVsino1DzG82W+rqFAUXEwxrLzhAPwEdHLezDiJz6 uLY6EELUbvX3Cc6mWoPRwYXDh/DvWez0qk6JF9a/zuYVOYJ3Nvg5/AgWAxgzlHi9 ywHU6RhurFQI22C8TrXKjrBPYr5kYA+gTwiZTc+xsf+ezoT5GhqN4uSrWb6wDBU1 plSQmd7IZBWpMXSQP2gXc/Xi+0rQ/m8bG9AM8isfI/jyhUvf/B6nnHPWCoGn9jnY QS6sQPQHXJPwzL5E2c75B5ZWoYUGwzRDolUgCl88PkWILuFlEaawUKSr06g302hA JYM+RJ2EC8h7jyRgr2nZNohAb82XVZMY3IHF1E33/TRnYw38izJ0Kojg5fJYEWho RCnJbA5vmz7cKdJKw2ky =Fz6/ -----END PGP SIGNATURE----- --wzZBuWVSCdIIPHAHLNDEwlbiIt18woj3S--