From nobody Tue Jan 18 16:51:49 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 2BF7619521D6 for <freebsd-fs@mlmmj.nyi.freebsd.org>; Tue, 18 Jan 2022 16:51:55 +0000 (UTC) (envelope-from se@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JdZYl05h0z4l2X; Tue, 18 Jan 2022 16:51:55 +0000 (UTC) (envelope-from se@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642524715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=VyxaVoZ+Om1tR+ra5OGiHvXOOcuNK7QfhjONwEpyWOk=; b=NRwBkY9bW7NU4zolTgtarV887oGDT/bFkSTpoxhQ0gPK1N/aLXIvZWGV+beputxd0NzqUx Vm5M1hMwYRZIDxNHJ+LDmRE2CS+DpUDXhih6KV++9wUrFleTN5D94WXUvn6B0Nbuxyxa2q XAmtbCCGl7RnqQAU/YoNFWHbmbbULMzcMYAYdV7vvBWCKuYHlM2cFSTuaVUEYjDb34Jxts UrCl0nvoNeoVsSRnlMdxL4tzwo7TFFbKpLJ7dWuPlG/Jt34XJ8uNKjjzwVatyPEHplsfy9 eAakLkHk33GLU2t80u5yUJuG7DJciC63ebIPeA1dRDpAGbppqe85eeuumIluZw== Received: from [IPV6:2003:cd:5f13:2400:7e:3a19:e7c9:e2e1] (p200300cd5f132400007e3a19e7c9e2e1.dip0.t-ipconnect.de [IPv6:2003:cd:5f13:2400:7e:3a19:e7c9:e2e1]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: se/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 3EBE729518; Tue, 18 Jan 2022 16:51:54 +0000 (UTC) (envelope-from se@FreeBSD.org) Message-ID: <c8db3f82-72e2-1c7b-2e77-055edb5043e4@FreeBSD.org> Date: Tue, 18 Jan 2022 17:51:49 +0100 List-Id: Filesystems <freebsd-fs.freebsd.org> List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: <mailto:freebsd-fs+help@freebsd.org> List-Post: <mailto:freebsd-fs@freebsd.org> List-Subscribe: <mailto:freebsd-fs+subscribe@freebsd.org> List-Unsubscribe: <mailto:freebsd-fs+unsubscribe@freebsd.org> Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Subject: Re: [zfs] recordsize: unexpected increase of disk usage when increasing it Content-Language: en-US To: Florent Rivoire <florent@rivoire.fr> Cc: Rich <rincebrain@gmail.com>, freebsd-fs <freebsd-fs@freebsd.org>, Alan Somers <asomers@freebsd.org> References: <CADzRhsEsZMGE-SoeWLMG9NTtkwhhy6OGQQ046m9AxGFbp5h_kQ@mail.gmail.com> <CAOeNLuopaY3j7P030KO4LMwU3BOU5tXiu6gRsSKsDrFEuGKuaA@mail.gmail.com> <CAOtMX2h=miZt=6__oAhPVzsK9ReShy6nG+aTiudvK_jp2sQKJQ@mail.gmail.com> <CADzRhsF+s1NHudToY0J7Wn90D8gwaM16Ym43XXopoaWVQGS8CA@mail.gmail.com> From: Stefan Esser <se@FreeBSD.org> In-Reply-To: <CADzRhsF+s1NHudToY0J7Wn90D8gwaM16Ym43XXopoaWVQGS8CA@mail.gmail.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------E05brz3x5OqYb0hq0WY09h7h" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1642524715; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=VyxaVoZ+Om1tR+ra5OGiHvXOOcuNK7QfhjONwEpyWOk=; b=TuiW5JfjfQXyzeueoUfHDBCgHrNph+zwEQTM7M1kZsE9ouDxZz5A31l/KhNoWBXBVj8JnZ en3oXi4W4UbK+66POb53SdtRYH2G7mAnOMEljrzJ3IqGFi4d3d/bKNd7ZRj94K0MouaKT8 fzGuZOAKYIyorw37jVdOLnMLAfQ3qQUqpj4y534/SHkLbmYWhdeFF1qzF29WkLocJ6S5DF tlQ1OBL8FUgItelitI35tvpMtDTcmHLpq4vKacU+hITxVOnc1XPMuOvT/9/1s51wm8ZVF5 k4OeSWzRTHYkGDApWFZ1na+FapOYSpuCT3MahkZC9f5JRSdRBoE/1Y6PL7T4jA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1642524715; a=rsa-sha256; cv=none; b=asP2ArMvYhckUMoljPVQvq/ioZB5Fot8si31afAqtC3JpquxMlNaNw3VcfPpG1lQ8LpDQJ VXvadO2DErAqFTby73WFJn18GEIikXN/Xbik+xMGWcPhPeZI20eiVI0Edkjgv7laxUWJPc vFPG9ZTBYyDnXiyIemaJRH4YRnwiU1dthhLBLzPeIk2nq8koQziJAlRO7mXrXpfFvJcCZY 0Oi4p/6Xjr6IEUZWQWHxlhOvOvN0QHfhpyhLwe909sGccLa2s2RxNA+Pm90blL0o8GNf44 fJvLoyqmBkjF/8yWFpYKl+15LEvwocMGQ1YR0UcrdjP0n/rX9OVivv7tKueirA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------E05brz3x5OqYb0hq0WY09h7h Content-Type: multipart/mixed; boundary="------------zQpsYyDXtdhILUIXo9qcTpjX"; protected-headers="v1" From: Stefan Esser <se@FreeBSD.org> To: Florent Rivoire <florent@rivoire.fr> Cc: Rich <rincebrain@gmail.com>, freebsd-fs <freebsd-fs@freebsd.org>, Alan Somers <asomers@freebsd.org> Message-ID: <c8db3f82-72e2-1c7b-2e77-055edb5043e4@FreeBSD.org> Subject: Re: [zfs] recordsize: unexpected increase of disk usage when increasing it References: <CADzRhsEsZMGE-SoeWLMG9NTtkwhhy6OGQQ046m9AxGFbp5h_kQ@mail.gmail.com> <CAOeNLuopaY3j7P030KO4LMwU3BOU5tXiu6gRsSKsDrFEuGKuaA@mail.gmail.com> <CAOtMX2h=miZt=6__oAhPVzsK9ReShy6nG+aTiudvK_jp2sQKJQ@mail.gmail.com> <CADzRhsF+s1NHudToY0J7Wn90D8gwaM16Ym43XXopoaWVQGS8CA@mail.gmail.com> In-Reply-To: <CADzRhsF+s1NHudToY0J7Wn90D8gwaM16Ym43XXopoaWVQGS8CA@mail.gmail.com> --------------zQpsYyDXtdhILUIXo9qcTpjX Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Am 18.01.22 um 17:07 schrieb Florent Rivoire: > On Tue, Jan 18, 2022 at 3:23 PM Alan Somers <asomers@freebsd.org> wrote= : >> However, I would suggest that you don't bother. With a 128kB recsize,= >> ZFS has something like a 1000:1 ratio of data:metadata. In other >> words, increasing your recsize can save you at most 0.1% of disk >> space. Basically, it doesn't matter. What it _does_ matter for is >> the tradeoff between write amplification and RAM usage. 1000:1 is >> comparable to the disk:ram of many computers. And performance is more= >> sensitive to metadata access times than data access times. So >> increasing your recsize can help you keep a greater fraction of your >> metadata in ARC. OTOH, as you remarked increasing your recsize will >> also increase write amplification. >=20 > In the attached zdb files (for 128K recordsize), we can see that the > "L0 ZFS plain file" objects are using 99.89% in my test zpool. So the > ratio in my case is exactly 1000:1 like you said. > I had that rule-of-thumb in mind, but thanks for reminding me ! >=20 > As quickly mentioned in my first email, the context is that I'm > considering using a mirror of SSDs as "special devices" for a new > zpool which will still be mainly made of magnetic HDDs (raidz2 of > 5x3TB). Why don't you use the SSDs as L2ARC and let the system place often used data and meta-data in that cache? I'm using 1/2 TB of a 1 TB SSD as L2ARC for a pool of comparable size, and with the persistence offered by OpenZFS it is already filled with relevant data when the system boots. With zstd-2 compression I get a compression ratio of typically 2 to 3 for my data, which makes the SSD effectively hold more than 1 TB, since the L2ARC stores compressed blocks. (BTW: The L2ARC meta-data is kept in the ARC, therefore too big an L2ARC can reduce the amount of RAM available for other data in the ARC and for programs more than it helps ...) After 1,5 years, smartcl reports 14 TB written, which is a small fraction of the specified write resilience of my SSD (less than 3%). In my work-loads I see an ARC efficiency of about 98,5% and a L2ARC efficiency of 80%, resulting in 1/5 of 1.5% =3D 0.3% of disk reads actually being served by the rotating disks. Regards, STefan --------------zQpsYyDXtdhILUIXo9qcTpjX-- --------------E05brz3x5OqYb0hq0WY09h7h Content-Type: application/pgp-signature; name="OpenPGP_signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="OpenPGP_signature" -----BEGIN PGP SIGNATURE----- wsB5BAABCAAjFiEEo3HqZZwL7MgrcVMTR+u171r99UQFAmHm8CUFAwAAAAAACgkQR+u171r99UQR 7wf+Mz5+XiTE/GvC5UPPD9EZW7YmYQLeoSHBZ5mbBi+r1Je8n+nPler6eZbHvuD86YfI41FzVALP 2lRyXDZFb/CymamDbVAiZaFAWpQNoXho/2l1dzZOwT3zvYbO5VYuNvXZlqvGRYmR71DfSdp7HnZA 89si23od13P+eE+NdCj66CLR9KzXX0+J/EbgwGE4tHSrwRYj9y6AOjmZo/F93AdGo1TTHbAGyAc0 +lPGNbTreXfyxN+Mc7+GAqWn7G1qemOH1dAGn7k1FyRjXa7BLtIhs1TszmzAkATNI5w2ZJiTSU7S NRvpmNQp3CYc6Rz7xdjfWtTle4/imog+B/r3sUvsmQ== =+381 -----END PGP SIGNATURE----- --------------E05brz3x5OqYb0hq0WY09h7h--