Date: Mon, 21 Aug 2023 15:25:25 +0100 From: Kaya Saman <kayasaman@optiplex-networks.com> To: freebsd@vanderzwan.org Cc: questions@freebsd.org Subject: Re: ZFS Root size keeps going down after upgrade to 13.2-release Message-ID: <fa58ca88-8304-f2cc-7c52-eaa4e7e5b88c@optiplex-networks.com> In-Reply-To: <005285FF-8BCB-4C65-AFCF-61A06265C530@vanderzwan.org> References: <8f3b445d-3f55-f359-ce5e-0d86c57755fe@optiplex-networks.com> <20230821130132.50aeb96b51b2b71efa14cb94@sohara.org> <8dbe0d99-ca94-998d-0c4e-27642db1b11a@optiplex-networks.com> <362941E9-C354-4325-A691-6DC44E5F5AB6@vanderzwan.org> <ff0c28e6-154e-ca74-e8f0-1f36019f8617@optiplex-networks.com> <0178E106-F136-44F6-BB36-D94ABEE4BEDE@vanderzwan.org> <3caba337-1d4c-4555-d3ba-7d36278a4c9b@optiplex-networks.com> <0E0B068F-C160-445F-BCED-828694122C8C@vanderzwan.org> <f3894f18-2338-a8ee-65f0-5d53ef181993@optiplex-networks.com> <2842F607-E62B-4D83-95AE-B44AE769B77F@vanderzwan.org> <f68aaedf-9c53-5e45-78ae-0c2e0abd7b95@optiplex-networks.com> <005285FF-8BCB-4C65-AFCF-61A06265C530@vanderzwan.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8/21/23 15:15, freebsd@vanderzwan.org wrote:
>
>> On 21 Aug 2023, at 15:52, Kaya Saman <kayasaman@optiplex-networks.com>=
wrote:
>>
>>
>> On 8/21/23 14:49, freebsd@vanderzwan.org wrote:
>>>> On 21 Aug 2023, at 15:41, Kaya Saman <kayasaman@optiplex-networks.co=
m> wrote:
>>>>
>>>>
>>>> On 8/21/23 14:38, freebsd@vanderzwan.org wrote:
>>>>>> On 21 Aug 2023, at 15:34, Kaya Saman <kayasaman@optiplex-networks.=
com> wrote:
>>>>>>
>>>>>>
>>>>>> On 8/21/23 14:23, freebsd@vanderzwan.org wrote:
>>>>>>>> On 21 Aug 2023, at 15:14, Kaya Saman <kayasaman@optiplex-network=
s.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/21/23 14:07, freebsd@vanderzwan.org wrote:
>>>>>>>>>> On 21 Aug 2023, at 14:21, Kaya Saman <kayasaman@optiplex-netwo=
rks.com> wrote:
>>>>>>>>>> # zfs list -ro space zroot/ROOT
>>>>>>>>>> NAME AVAIL USED USE=
DSNAP USEDDS USEDREFRESERV USEDCHILD
>>>>>>>>>> zroot/ROOT 720M 88.5G 0B =
88K 0B 88.5G
>>>>>>>>>> zroot/ROOT/13.1-RELEASE-p5_2023-08-20_220010 720M 8K 0B =
8K 0B 0B
>>>>>>>>>> zroot/ROOT/13.2-RELEASE-p2_2023-08-20_224534 720M 8K 0B =
8K 0B 0B
>>>>>>>>>> zroot/ROOT/default 720M 88.5G 52.=
0G 36.5G 0B 0B
>>>>>>>>>>
>>>>>>>>> That last line seems to indicate there is 52.0 G used in snapsh=
ots of zroot/ROOT/default and 36.5G in the dataset itself.
>>>>>>>>> Output of zfs list -o space -t snapshot |grep zroot/ROOT/defau=
lt should give a list of the snapshots and the sizes.
>>>>>>>>>
>>>>>>>>> Paul
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Thanks so much Paul.... I feel like I'm spinning around in circl=
es currently...
>>>>>>>>
>>>>>>>>
>>>>>>>> There doesn't seem to be anything indicative of that space being=
used up?
>>>>>>>>
>>>>>>>>
>>>>>>>> zfs list -o space -t snapshot |grep zroot/ROOT/defaul
>>>>>>>> zroot/ROOT/default@2022-11-28-06:08:26-0 - 583M - -=
- -
>>>>>>>> zroot/ROOT/default@2022-11-28-06:38:48-0 - 11.4M - -=
- -
>>>>>>>> zroot/ROOT/default@2022-11-28-07:26:20-0 - 9.41M - -=
- -
>>>>>>>> zroot/ROOT/default@2023-08-20-22:00:10-0 - 34.5M - -=
- -
>>>>>>>> zroot/ROOT/default@2023-08-20-22:45:34-0 - 9.56M - -=
- -
>>>>>>>>
>>>>>>>>
>>>>>>>> certainly not 52GB?
>>>>>>>>
>>>>>>>>
>>>>>>> That=E2=80=99s definitely confusing.
>>>>>>> What=E2=80=99s the full output of 'zfs list -o space -r zroot=E2=80=
=99 and of 'zpool list -v zroot=E2=80=99
>>>>>>>
>>>>>>> Paul
>>>>>>>
>>>>>>>
>>>>>> Here are the outputs:
>>>>>>
>>>>>>
>>>>>> zfs list -o space -r zroot
>>>>>> NAME AVAIL USED USEDSNA=
P USEDDS USEDREFRESERV USEDCHILD
>>>>>> zroot 720M 107G 0B 2.=
20G 0B 105G
>>>>>> zroot/ROOT 720M 88.5G 0B =
88K 0B 88.5G
>>>>>> zroot/ROOT/13.1-RELEASE-p5_2023-08-20_220010 720M 8K 0B =
8K 0B 0B
>>>>>> zroot/ROOT/13.2-RELEASE-p2_2023-08-20_224534 720M 8K 0B =
8K 0B 0B
>>>>>> zroot/ROOT/default 720M 88.5G 52.0G =
36.5G 0B 0B
>>>>>> zroot/swap 3.78G 6.19G 0B 3.=
11G 3.08G 0B
>>>>>> zroot/tmp 720M 121M 0B 1=
21M 0B 0B
>>>>>> zroot/usr 720M 7.09G 0B =
88K 0B 7.09G
>>>>>> zroot/usr/home 720M 88K 0B =
88K 0B 0B
>>>>>> zroot/usr/obj 720M 4.75G 0B 4.=
75G 0B 0B
>>>>>> zroot/usr/ports 720M 432K 0B 1=
76K 0B 256K
>>>>>> zroot/usr/ports/distfiles 720M 168K 0B 1=
68K 0B 0B
>>>>>> zroot/usr/ports/packages 720M 88K 0B =
88K 0B 0B
>>>>>> zroot/usr/src 720M 2.35G 0B 2.=
35G 0B 0B
>>>>>> zroot/var 720M 2.60G 0B =
88K 0B 2.60G
>>>>>> zroot/var/audit 720M 88K 0B =
88K 0B 0B
>>>>>> zroot/var/backups 720M 141M 0B 1=
41M 0B 0B
>>>>>> zroot/var/crash 720M 88K 0B =
88K 0B 0B
>>>>>> zroot/var/db 720M 1.69G 0B 1.=
59G 0B 94.8M
>>>>>> zroot/var/db/pkg 720M 94.8M 0B 94=
.8M 0B 0B
>>>>>> zroot/var/empty 720M 88K 0B =
88K 0B 0B
>>>>>> zroot/var/log 720M 792M 0B 7=
92M 0B 0B
>>>>>> zroot/var/mail 720M 3.96M 0B 3.=
96M 0B 0B
>>>>>> zroot/var/run 720M 248K 0B 2=
48K 0B 0B
>>>>>> zroot/var/tmp 720M 672K 0B 6=
72K 0B 0B
>>>>>>
>>>>>>
>>>>>> zpool list -v zroot
>>>>>> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP D=
EDUP HEALTH ALTROOT
>>>>>> zroot 111G 104G 7.22G - - 82% 93% 1=
.00x ONLINE -
>>>>>> mirror-0 111G 104G 7.22G - - 82% 93.5% =
- ONLINE
>>>>>> ada0p3 111G - - - - - - =
- ONLINE
>>>>>> ada1p3 111G - - - - - - =
- ONLINE
>>>>>>
>>>>> Those look quite normal. What about 'zfs list -r -t all -o space zr=
oot/ROOT/default=E2=80=99 ?
>>>>>
>>>>> Paul
>>>>>
>>>>>
>>>>>
>>>> zfs list -r -t all -o space zroot/ROOT/default
>>>> NAME AVAIL USED USEDSNAP USE=
DDS USEDREFRESERV USEDCHILD
>>>> zroot/ROOT/default 720M 88.5G 52.0G 36.=
5G 0B 0B
>>>> zroot/ROOT/default@2022-11-28-06:08:26-0 - 583M - - =
- -
>>>> zroot/ROOT/default@2022-11-28-06:38:48-0 - 11.4M - - =
- -
>>>> zroot/ROOT/default@2022-11-28-07:26:20-0 - 9.41M - - =
- -
>>>> zroot/ROOT/default@2023-08-20-22:00:10-0 - 34.5M - - =
- -
>>>> zroot/ROOT/default@2023-08-20-22:45:34-0 - 9.56M - - =
- -
>>>>
>>>>
>>>> Strange! It shows a 52GB 'claimed' snapshot but where is it?
>>>>
>>>>
>>> Can you give output of 'zfs get all zroot/ROOT/default=E2=80=99 as we=
ll ? Maybe that gives a hint.
>>>
>>> Paul
>>>
>> Sure:
>>
>>
>> zfs get all zroot/ROOT/default
>> NAME PROPERTY VALUE SOURCE
>> zroot/ROOT/default type filesystem -
>> zroot/ROOT/default creation Tue Aug 7 2:53 2018 -
>> zroot/ROOT/default used 88.5G -
>> zroot/ROOT/default available 720M -
>> zroot/ROOT/default referenced 36.5G -
>> zroot/ROOT/default compressratio 1.29x -
>> zroot/ROOT/default mounted yes -
>> zroot/ROOT/default quota none default
>> zroot/ROOT/default reservation none default
>> zroot/ROOT/default recordsize 128K default
>> zroot/ROOT/default mountpoint / local
>> zroot/ROOT/default sharenfs off default
>> zroot/ROOT/default checksum on default
>> zroot/ROOT/default compression lz4 inherited from zroot
>> zroot/ROOT/default atime on default
>> zroot/ROOT/default devices on default
>> zroot/ROOT/default exec on default
>> zroot/ROOT/default setuid on default
>> zroot/ROOT/default readonly off default
>> zroot/ROOT/default jailed off default
>> zroot/ROOT/default snapdir hidden default
>> zroot/ROOT/default aclmode discard default
>> zroot/ROOT/default aclinherit restricted default
>> zroot/ROOT/default createtxg 93 -
>> zroot/ROOT/default canmount noauto local
>> zroot/ROOT/default xattr on default
>> zroot/ROOT/default copies 1 default
>> zroot/ROOT/default version 5 -
>> zroot/ROOT/default utf8only off -
>> zroot/ROOT/default normalization none -
>> zroot/ROOT/default casesensitivity sensitive -
>> zroot/ROOT/default vscan off default
>> zroot/ROOT/default nbmand off default
>> zroot/ROOT/default sharesmb off default
>> zroot/ROOT/default refquota none default
>> zroot/ROOT/default refreservation none default
>> zroot/ROOT/default guid 15509736917917171623 -
>> zroot/ROOT/default primarycache all default
>> zroot/ROOT/default secondarycache all default
>> zroot/ROOT/default usedbysnapshots 52.0G -
>> zroot/ROOT/default usedbydataset 36.5G -
>> zroot/ROOT/default usedbychildren 0B -
>> zroot/ROOT/default usedbyrefreservation 0B -
>> zroot/ROOT/default logbias latency default
>> zroot/ROOT/default objsetid 58 -
>> zroot/ROOT/default dedup off default
>> zroot/ROOT/default mlslabel none default
>> zroot/ROOT/default sync standard default
>> zroot/ROOT/default dnodesize legacy default
>> zroot/ROOT/default refcompressratio 1.32x -
>> zroot/ROOT/default written 1.53G -
>> zroot/ROOT/default logicalused 108G -
>> zroot/ROOT/default logicalreferenced 46.0G -
>> zroot/ROOT/default volmode default default
>> zroot/ROOT/default filesystem_limit none default
>> zroot/ROOT/default snapshot_limit none default
>> zroot/ROOT/default filesystem_count none default
>> zroot/ROOT/default snapshot_count none default
>> zroot/ROOT/default snapdev hidden default
>> zroot/ROOT/default acltype nfsv4 default
>> zroot/ROOT/default context none default
>> zroot/ROOT/default fscontext none default
>> zroot/ROOT/default defcontext none default
>> zroot/ROOT/default rootcontext none default
>> zroot/ROOT/default relatime off default
>> zroot/ROOT/default redundant_metadata all default
>> zroot/ROOT/default overlay on default
>> zroot/ROOT/default encryption off default
>> zroot/ROOT/default keylocation none default
>> zroot/ROOT/default keyformat none default
>> zroot/ROOT/default pbkdf2iters 0 default
>> zroot/ROOT/default special_small_blocks 0 default
>>
>>
> That looks consistent with the rest of the data.
>
> Man page for zfsprops mentions this for snapshots;
> The used space of a snapshot (see the Snaps=
hots
> section of zfsconcepts(7)) is space that is
> referenced exclusively by this snapshot. I=
f this
> snapshot is destroyed, the amount of used s=
pace
> will be freed. Space that is shared by mul=
tiple
> snapshots isn't accounted for in this metri=
c. When
> a snapshot is destroyed, space that was pre=
viously
> shared with this snapshot can become unique=
to
> snapshots adjacent to it, thus changing the=
used
> space of those snapshots. The used space o=
f the
> latest snapshot can also be affected by cha=
nges in
> the file system. Note that the used space =
of a
> snapshot is a subset of the written space o=
f the
> snapshot.
> Note : Space that is shared by multiple snapshots isn't accounted for =
in this metric.
>
> My theory is that the snapshots/boot environments share about 52G of w=
ith each other but not with the parent dataset.
> As there is no data unique to one snapshot it does not show
> What you could do is delete all except one bootenvironment.
> The snapshot data used should then be unique to that last snapshot and =
appear in the lists.
>
> Paul
>
>
>
I just responded to Olivier above but I have managed to free up the=20
necessary space by deleting all the snapshots:
df -h
Filesystem=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Size=C2=A0=C2=A0=C2=A0 Used=C2=A0=C2=
=A0 Avail Capacity=C2=A0=20
Mounted on
zroot/ROOT/default=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
89G=C2=A0=C2=A0=C2=A0=C2=A0 36G=C2=A0=C2=A0=C2=A0=C2=A0 53G 41%=C2=A0=C2=
=A0=C2=A0 /
For some reason some of the snapshots were automatically 'cloned' and=20
exactly as you suggested above there was a fair amount of space sharing=20
between snapshots. This of course resulted in data being removed from=20
the main snapshot but still remained on disk due another snapshot=20
referencing that data....
Very confusing for an automated process but anyway, I have space again :-=
)
Thank you so much everyone for your help and input!
Best Regards,
Kaya
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?fa58ca88-8304-f2cc-7c52-eaa4e7e5b88c>