From nobody Tue Jan 18 14:12:50 2022 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 1FB691977164 for ; Tue, 18 Jan 2022 14:13:01 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from mail-ua1-x92a.google.com (mail-ua1-x92a.google.com [IPv6:2607:f8b0:4864:20::92a]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JdW2N1xNbz4Yyf for ; Tue, 18 Jan 2022 14:13:00 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: by mail-ua1-x92a.google.com with SMTP id 2so12100437uax.10 for ; Tue, 18 Jan 2022 06:13:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=4dAVwuHogkY5EYr5/ZA/bu3BKoyeGGujBYB1rGPWgMc=; b=mQGhEEHsqNx1+IRLu+KpGCtouYDhbfovCKVhSgC6FYylTJREborD5nMm6u/cg6nzLQ +KhwvHNBccUuhVnp92BIgCdHmJvBMz+pHpP0JUQPzH4bYGyNbVplSBdRwchLAw9AE/kQ eFt0Y+X9uOa+NOqInYfGfhWFS7tdVrfCfHrblet3JbibZnrrSigcKJbrd5vVEfZzCZA1 cfkf+E+WeSMpZlibh15hW0oZM0to+r0pHWIpvRkRDZW7Cq6IQCFau0xC9EDwKL3Vc5YC /7EaDIIuJBv2jW9EokgeChG0OP2WQMJxHCf4lTvptycSk4sDhkd9KAE3bktkCauRz9/v 44zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=4dAVwuHogkY5EYr5/ZA/bu3BKoyeGGujBYB1rGPWgMc=; b=sELTQwq1dlDZHJHlWkqcSeUSW5/XIrVAivQo5xa4P9wjz0+zweTB2c95E8bGljTZjx 8PecSorpFWJyrqFsK42pkaCAPU5zKvRuCLV2AB2kZ/SN5Rr58A9phISPWplIFux+mFbv g4sjVHJ/pxPN7BlJ33wdL52QadO/YiFO3/SmolhzjdkgFDgD5SV/9mCi8N/GJiCqLLAk Sl/0Wrc3uvwuK6EPo8+7NlcSqYLfrNvNVqWkgP/sxA4bMmKhzm0BzZ90q+FQhrf4RRHe FdNqM9+atTx1tbNBYQ7mcNcFlqeKl4++wx5t3wvVfEB7LhQfQcgrQ2ZxHCIx0AxtX46z x8yw== X-Gm-Message-State: AOAM532IZrCkZPmR6zeeKqpkBH/0MBO5L5DntEtEahALp/eX2UBt5lb8 3qr3PZCJqcqeOeN471nmekx1sF4WTuERlNraSYWqGtFd X-Google-Smtp-Source: ABdhPJy9Lcfkr+rN1gAPyd6soHLWsBsRDyoxwJSkFhtsfMMVWuViy9HPX5tyHRKW6Yt8hi3r6fvaJ/kzQ+rYOjFp8uc= X-Received: by 2002:a1f:c18c:: with SMTP id r134mr1270267vkf.6.1642515179745; Tue, 18 Jan 2022 06:12:59 -0800 (PST) List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Rich Date: Tue, 18 Jan 2022 09:12:50 -0500 Message-ID: Subject: Re: [zfs] recordsize: unexpected increase of disk usage when increasing it To: Florent Rivoire Cc: freebsd-fs Content-Type: multipart/alternative; boundary="000000000000d3b3f605d5dbdd98" X-Rspamd-Queue-Id: 4JdW2N1xNbz4Yyf X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=mQGhEEHs; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rincebrain@gmail.com designates 2607:f8b0:4864:20::92a as permitted sender) smtp.mailfrom=rincebrain@gmail.com X-Spamd-Result: default: False [-4.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::92a:from]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-fs]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-ThisMailContainsUnwantedMimeParts: N --000000000000d3b3f605d5dbdd98 Content-Type: text/plain; charset="UTF-8" Compression would have made your life better here, and possibly also made it clearer what's going on. All records in a file are going to be the same size pre-compression - so if you set the recordsize to 1M and save a 131.1M file, it's going to take up 132M on disk before compression/raidz overhead/whatnot. Usually compression saves you from the tail padding actually requiring allocation on disk, which is one reason I encourage everyone to at least use lz4 (or, if you absolutely cannot for some reason, I guess zle should also work for this one case...) But I would say it's probably the sum of last record padding across the whole dataset, if you don't have compression on. - Rich On Tue, Jan 18, 2022 at 8:57 AM Florent Rivoire wrote: > TLDR: I rsync-ed the same data twice: once with 128K recordsize and > once with 1M, and the allocated size on disk is ~3% bigger with 1M. > Why not smaller ? > > > Hello, > > I would like some help to understand how the disk usage evolves when I > change the recordsize. > > I've read several articles/presentations/forums about recordsize in > ZFS, and if I try to summarize, I mainly understood that: > - recordsize is the "maximum" size of "objects" (so "logical blocks") > that zfs will create for both -data & metadata, then each object is > compressed and allocated to one vdev, splitted into smaller (ashift > size) "physical" blocks and written on disks > - increasing recordsize is usually good when storing large files that > are not modified, because it limits the nb of metadata objects > (block-pointers), which has a positive effect on performance > - decreasing recordsize is useful for "databases-like" workloads (ie: > small random writes inside existing objects), because it avoids write > amplification (read-modify-write a large object for a small update) > > Today, I'm trying to observe the effect of increasing recordsize for > *my* data (because I'm also considering defining special_small_blocks > & using SSDs as "special", but not tested nor discussed here, just > recordsize). > So, I'm doing some benchmarks on my "documents" dataset (details in > "notes" below), but the results are really strange to me. > > When I rsync the same data to a freshly-recreated zpool: > A) with recordsize=128K : 226G allocated on disk > B) with recordsize=1M : 232G allocated on disk => bigger than 128K ?!? > > I would clearly expect the other way around, because bigger recordsize > generates less metadata so smaller disk usage, and there shouldn't be > any overhead because 1M is just a maximum and not a forced size to > allocate for every object. > I don't mind the increased usage (I can live with a few GB more), but > I would like to understand why it happens. > > I tried to give all the details of my tests below. > Did I do something wrong ? Can you explain the increase ? > > Thanks ! > > > > =============================================== > A) 128K > ========== > > # zpool destroy bench > # zpool create -o ashift=12 bench > /dev/gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4 > > # rsync -av --exclude '.zfs' /mnt/tank/docs-florent/ /bench > [...] > sent 241,042,476,154 bytes received 353,838 bytes 81,806,492.45 bytes/sec > total size is 240,982,439,038 speedup is 1.00 > > # zfs get recordsize bench > NAME PROPERTY VALUE SOURCE > bench recordsize 128K default > > # zpool list -v bench > NAME SIZE ALLOC FREE > CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > bench 2.72T 226G 2.50T > - - 0% 8% 1.00x ONLINE - > gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4 2.72T 226G 2.50T > - - 0% 8.10% - ONLINE > > # zfs list bench > NAME USED AVAIL REFER MOUNTPOINT > bench 226G 2.41T 226G /bench > > # zfs get all bench |egrep "(used|referenced|written)" > bench used 226G - > bench referenced 226G - > bench usedbysnapshots 0B - > bench usedbydataset 226G - > bench usedbychildren 1.80M - > bench usedbyrefreservation 0B - > bench written 226G - > bench logicalused 226G - > bench logicalreferenced 226G - > > # zdb -Lbbbs bench > zpool-bench-rcd128K.zdb > > > > =============================================== > B) 1M > ========== > > # zpool destroy bench > # zpool create -o ashift=12 bench > /dev/gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4 > # zfs set recordsize=1M bench > > # rsync -av --exclude '.zfs' /mnt/tank/docs-florent/ /bench > [...] > sent 241,042,476,154 bytes received 353,830 bytes 80,173,899.88 bytes/sec > total size is 240,982,439,038 speedup is 1.00 > > # zfs get recordsize bench > NAME PROPERTY VALUE SOURCE > bench recordsize 1M local > > # zpool list -v bench > NAME SIZE ALLOC FREE > CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT > bench 2.72T 232G 2.49T > - - 0% 8% 1.00x ONLINE - > gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4 2.72T 232G 2.49T > - - 0% 8.32% - ONLINE > > # zfs list bench > NAME USED AVAIL REFER MOUNTPOINT > bench 232G 2.41T 232G /bench > > # zfs get all bench |egrep "(used|referenced|written)" > bench used 232G - > bench referenced 232G - > bench usedbysnapshots 0B - > bench usedbydataset 232G - > bench usedbychildren 1.96M - > bench usedbyrefreservation 0B - > bench written 232G - > bench logicalused 232G - > bench logicalreferenced 232G - > > # zdb -Lbbbs bench > zpool-bench-rcd1M.zdb > > > > =============================================== > Notes: > ========== > > - the source dataset contains ~50% of pictures (raw files and jpg), > and also some music, various archived documents, zip, videos > - no change on the source dataset while testing (cf size logged by resync) > - I repeated the tests twice (128K, then 1M, then 128K, then 1M), and > same results > - probably not important here, but: > /dev/gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4 is a Red 3TB CMR > (WD30EFRX), and /mnt/tank/docs-florent/ is a 128K-recordsize dataset > on another zpool that I never tweaked except ashit=12 (because using > the same model of Red 3TB) > > # zfs --version > zfs-2.0.6-1 > zfs-kmod-v2021120100-zfs_a8c7652 > > # uname -a > FreeBSD xxxxxxxxx 12.2-RELEASE-p11 FreeBSD 12.2-RELEASE-p11 > 75566f060d4(HEAD) TRUENAS amd64 > --000000000000d3b3f605d5dbdd98 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Compression would have made your life better here, and pos= sibly also made it clearer what's going on.

All reco= rds in a file are going to be the same size pre-compression - so if you set= the recordsize to 1M and save a 131.1M file, it's going to take up 132= M on disk before compression/raidz overhead/whatnot.

Usually compression saves you from the tail padding actually requiring a= llocation on disk, which is one reason I encourage everyone to at least use= lz4 (or, if you absolutely cannot for some reason, I guess zle should also= work for this one case...)

But I would say it'= ;s probably the sum of last record padding across the whole dataset, if you= don't have compression on.

- Rich
=
On Tue= , Jan 18, 2022 at 8:57 AM Florent Rivoire <florent@rivoire.fr> wrote:
TLDR: I rsync-ed the same data twice: once with = 128K recordsize and
once with 1M, and the allocated size on disk is ~3% bigger with 1M.
Why not smaller ?


Hello,

I would like some help to understand how the disk usage evolves when I
change the recordsize.

I've read several articles/presentations/forums about recordsize in
ZFS, and if I try to summarize, I mainly understood that:
- recordsize is the "maximum" size of "objects" (so &qu= ot;logical blocks")
that zfs will create for both=C2=A0 -data & metadata, then each object = is
compressed and allocated to one vdev, splitted into smaller (ashift
size) "physical" blocks and written on disks
- increasing recordsize is usually good when storing large files that
are not modified, because it limits the nb of metadata objects
(block-pointers), which has a positive effect on performance
- decreasing recordsize is useful for "databases-like" workloads = (ie:
small random writes inside existing objects), because it avoids write
amplification (read-modify-write a large object for a small update)

Today, I'm trying to observe the effect of increasing recordsize for *my* data (because I'm also considering defining special_small_blocks & using SSDs as "special", but not tested nor discussed here,= just
recordsize).
So, I'm doing some benchmarks on my "documents" dataset (deta= ils in
"notes" below), but the results are really strange to me.

When I rsync the same data to a freshly-recreated zpool:
A) with recordsize=3D128K : 226G allocated on disk
B) with recordsize=3D1M : 232G allocated on disk =3D> bigger than 128K ?= !?

I would clearly expect the other way around, because bigger recordsize
generates less metadata so smaller disk usage, and there shouldn't be any overhead because 1M is just a maximum and not a forced size to
allocate for every object.
I don't mind the increased usage (I can live with a few GB more), but I would like to understand why it happens.

I tried to give all the details of my tests below.
Did I do something wrong ? Can you explain the increase ?

Thanks !



=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
A) 128K
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

# zpool destroy bench
# zpool create -o ashift=3D12 bench
/dev/gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4

# rsync -av --exclude '.zfs' /mnt/tank/docs-florent/ /bench
[...]
sent 241,042,476,154 bytes=C2=A0 received 353,838 bytes=C2=A0 81,806,492.45= bytes/sec
total size is 240,982,439,038=C2=A0 speedup is 1.00

# zfs get recordsize bench
NAME=C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0 VALUE=C2=A0 =C2=A0 SOURCE
bench=C2=A0 recordsize=C2=A0 128K=C2=A0 =C2=A0 =C2=A0default

# zpool list -v bench
NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE
CKPOINT=C2=A0 EXPANDSZ=C2=A0 =C2=A0FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP=C2=A0 = =C2=A0 HEALTH=C2=A0 ALTROOT
bench=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A02.72T=C2=A0 =C2=A0226G=C2=A0 2.50T
=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A00%=C2=A0 =C2= =A0 =C2=A08%=C2=A0 1.00x=C2=A0 =C2=A0 ONLINE=C2=A0 -
=C2=A0 gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4=C2=A0 2.72T=C2=A0 =C2=A02= 26G=C2=A0 2.50T
=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A00%=C2=A0 8.1= 0%=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE

# zfs list bench
NAME=C2=A0 =C2=A0 USED=C2=A0 AVAIL=C2=A0 =C2=A0 =C2=A0REFER=C2=A0 MOUNTPOIN= T
bench=C2=A0 =C2=A0226G=C2=A0 2.41T=C2=A0 =C2=A0 =C2=A0 226G=C2=A0 /bench
# zfs get all bench |egrep "(used|referenced|written)"
bench=C2=A0 used=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 226G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0-
bench=C2=A0 referenced=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 226G=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 usedbysnapshots=C2=A0 =C2=A0 =C2=A0 =C2=A00B=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 usedbydataset=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0226G=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 usedbychildren=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.80M=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
bench=C2=A0 usedbyrefreservation=C2=A0 0B=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 written=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02= 26G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0- bench=C2=A0 logicalused=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0226G=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 logicalreferenced=C2=A0 =C2=A0 =C2=A0226G=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-

# zdb -Lbbbs bench > zpool-bench-rcd128K.zdb



=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
B) 1M
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

# zpool destroy bench
# zpool create -o ashift=3D12 bench
/dev/gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4
# zfs set recordsize=3D1M bench

# rsync -av --exclude '.zfs' /mnt/tank/docs-florent/ /bench
[...]
sent 241,042,476,154 bytes=C2=A0 received 353,830 bytes=C2=A0 80,173,899.88= bytes/sec
total size is 240,982,439,038=C2=A0 speedup is 1.00

# zfs get recordsize bench
NAME=C2=A0 =C2=A0PROPERTY=C2=A0 =C2=A0 VALUE=C2=A0 =C2=A0 SOURCE
bench=C2=A0 recordsize=C2=A0 1M=C2=A0 =C2=A0 =C2=A0 =C2=A0local

# zpool list -v bench
NAME=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0SIZE=C2=A0 ALLOC=C2=A0 =C2=A0FREE
CKPOINT=C2=A0 EXPANDSZ=C2=A0 =C2=A0FRAG=C2=A0 =C2=A0 CAP=C2=A0 DEDUP=C2=A0 = =C2=A0 HEALTH=C2=A0 ALTROOT
bench=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A02.72T=C2=A0 =C2=A0232G=C2=A0 2.49T
=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A00%=C2=A0 =C2= =A0 =C2=A08%=C2=A0 1.00x=C2=A0 =C2=A0 ONLINE=C2=A0 -
=C2=A0 gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4=C2=A0 2.72T=C2=A0 =C2=A02= 32G=C2=A0 2.49T
=C2=A0 -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-=C2=A0 =C2=A0 =C2=A00%=C2=A0 8.3= 2%=C2=A0 =C2=A0 =C2=A0 -=C2=A0 =C2=A0 ONLINE

# zfs list bench
NAME=C2=A0 =C2=A0 USED=C2=A0 AVAIL=C2=A0 =C2=A0 =C2=A0REFER=C2=A0 MOUNTPOIN= T
bench=C2=A0 =C2=A0232G=C2=A0 2.41T=C2=A0 =C2=A0 =C2=A0 232G=C2=A0 /bench
# zfs get all bench |egrep "(used|referenced|written)"
bench=C2=A0 used=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 232G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0-
bench=C2=A0 referenced=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 232G=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 usedbysnapshots=C2=A0 =C2=A0 =C2=A0 =C2=A00B=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 usedbydataset=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0232G=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 usedbychildren=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.96M=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -
bench=C2=A0 usedbyrefreservation=C2=A0 0B=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 written=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02= 32G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0- bench=C2=A0 logicalused=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0232G=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-
bench=C2=A0 logicalreferenced=C2=A0 =C2=A0 =C2=A0232G=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0-

# zdb -Lbbbs bench > zpool-bench-rcd1M.zdb



=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Notes:
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

- the source dataset contains ~50% of pictures (raw files and jpg),
and also some music, various archived documents, zip, videos
- no change on the source dataset while testing (cf size logged by resync)<= br> - I repeated the tests twice (128K, then 1M, then 128K, then 1M), and
same results
- probably not important here, but:
/dev/gptid/3c0f5cbc-b0ce-11ea-ab91-c8cbb8cc3ad4 is a Red 3TB CMR
(WD30EFRX), and /mnt/tank/docs-florent/ is a 128K-recordsize dataset
on another zpool that I never tweaked except ashit=3D12 (because using
the same model of Red 3TB)

# zfs --version
zfs-2.0.6-1
zfs-kmod-v2021120100-zfs_a8c7652

# uname -a
FreeBSD xxxxxxxxx 12.2-RELEASE-p11 FreeBSD 12.2-RELEASE-p11
75566f060d4(HEAD) TRUENAS=C2=A0 amd64
--000000000000d3b3f605d5dbdd98--