From nobody Mon Aug 21 14:25:25 2023 X-Original-To: questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RTvsC2xHFz4qrL9 for ; Mon, 21 Aug 2023 14:25:35 +0000 (UTC) (envelope-from kayasaman@optiplex-networks.com) Received: from mail.optiplex-networks.com (mail.optiplex-networks.com [45.149.190.182]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RTvsC06zxz3fkD for ; Mon, 21 Aug 2023 14:25:35 +0000 (UTC) (envelope-from kayasaman@optiplex-networks.com) Authentication-Results: mx1.freebsd.org; none Received: from localhost (localhost [127.0.0.1]) by mail.optiplex-networks.com (Postfix) with ESMTP id 9038315C2EE8; Mon, 21 Aug 2023 15:25:33 +0100 (BST) Received: from mail.optiplex-networks.com ([127.0.0.1]) by localhost (mail.optiplex-networks.com [127.0.0.1]) (amavis, port 10032) with ESMTP id a5mRejJohq4D; Mon, 21 Aug 2023 15:25:32 +0100 (BST) Received: from localhost (localhost [127.0.0.1]) by mail.optiplex-networks.com (Postfix) with ESMTP id C975415C2E70; Mon, 21 Aug 2023 15:25:29 +0100 (BST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.optiplex-networks.com C975415C2E70 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=optiplex-networks.com; s=AE93A2AC-7F67-11EA-90AE-8A1FE64F6997; t=1692627929; bh=+RYyes3PyMtv8oED7oWuDqswGaKSGWuvzdfl48wum/o=; h=Message-ID:Date:MIME-Version:To:From; b=ZRAGTIf+vPs+AabHDx8YZrLAq/IX1pJny8GC6V4BMNGTCD0i/ezMfhTOufOeUIxf1 PtvdP40tLaXOIz2bKa/SExF5dIlN9T41eYGFC+MpwcnPiwgEQiu7xi30bcbLYfW1TG IeTW/GtB9HOVHVBiQ2loIP+fTB2ywkQenSR8AddsRltYIYwF0xSloAKbpkhxIEUzL/ Btv9XUsQoJPu2IrPQM6HDf9VAZh62A7r/0T4s5XhFqKFgNkHhXH1cVoYWVPUNHUcnj 1791M+JheiocC0JNVOGrS5LO8SH5AUjF2dfuM+1mxbiNEWpMrktOSRgSTfqlSw/om3 THzYKO2bXoZWg== X-Virus-Scanned: amavis at mail.optiplex-networks.com Received: from mail.optiplex-networks.com ([127.0.0.1]) by localhost (mail.optiplex-networks.com [127.0.0.1]) (amavis, port 10026) with ESMTP id LPkAT1iieSim; Mon, 21 Aug 2023 15:25:29 +0100 (BST) Received: from [192.168.20.23] (unknown [192.168.20.23]) by mail.optiplex-networks.com (Postfix) with ESMTPSA id 8986215C2EE3; Mon, 21 Aug 2023 15:25:25 +0100 (BST) Message-ID: Date: Mon, 21 Aug 2023 15:25:25 +0100 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Subject: Re: ZFS Root size keeps going down after upgrade to 13.2-release Content-Language: en-US To: freebsd@vanderzwan.org Cc: questions@freebsd.org References: <8f3b445d-3f55-f359-ce5e-0d86c57755fe@optiplex-networks.com> <20230821130132.50aeb96b51b2b71efa14cb94@sohara.org> <8dbe0d99-ca94-998d-0c4e-27642db1b11a@optiplex-networks.com> <362941E9-C354-4325-A691-6DC44E5F5AB6@vanderzwan.org> <0178E106-F136-44F6-BB36-D94ABEE4BEDE@vanderzwan.org> <3caba337-1d4c-4555-d3ba-7d36278a4c9b@optiplex-networks.com> <0E0B068F-C160-445F-BCED-828694122C8C@vanderzwan.org> <2842F607-E62B-4D83-95AE-B44AE769B77F@vanderzwan.org> <005285FF-8BCB-4C65-AFCF-61A06265C530@vanderzwan.org> From: Kaya Saman In-Reply-To: <005285FF-8BCB-4C65-AFCF-61A06265C530@vanderzwan.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4RTvsC06zxz3fkD X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:202596, ipnet:45.149.188.0/22, country:GB] On 8/21/23 15:15, freebsd@vanderzwan.org wrote: > >> On 21 Aug 2023, at 15:52, Kaya Saman = wrote: >> >> >> On 8/21/23 14:49, freebsd@vanderzwan.org wrote: >>>> On 21 Aug 2023, at 15:41, Kaya Saman wrote: >>>> >>>> >>>> On 8/21/23 14:38, freebsd@vanderzwan.org wrote: >>>>>> On 21 Aug 2023, at 15:34, Kaya Saman wrote: >>>>>> >>>>>> >>>>>> On 8/21/23 14:23, freebsd@vanderzwan.org wrote: >>>>>>>> On 21 Aug 2023, at 15:14, Kaya Saman wrote: >>>>>>>> >>>>>>>> >>>>>>>> On 8/21/23 14:07, freebsd@vanderzwan.org wrote: >>>>>>>>>> On 21 Aug 2023, at 14:21, Kaya Saman wrote: >>>>>>>>>> # zfs list -ro space zroot/ROOT >>>>>>>>>> NAME AVAIL USED USE= DSNAP USEDDS USEDREFRESERV USEDCHILD >>>>>>>>>> zroot/ROOT 720M 88.5G 0B = 88K 0B 88.5G >>>>>>>>>> zroot/ROOT/13.1-RELEASE-p5_2023-08-20_220010 720M 8K 0B = 8K 0B 0B >>>>>>>>>> zroot/ROOT/13.2-RELEASE-p2_2023-08-20_224534 720M 8K 0B = 8K 0B 0B >>>>>>>>>> zroot/ROOT/default 720M 88.5G 52.= 0G 36.5G 0B 0B >>>>>>>>>> >>>>>>>>> That last line seems to indicate there is 52.0 G used in snapsh= ots of zroot/ROOT/default and 36.5G in the dataset itself. >>>>>>>>> Output of zfs list -o space -t snapshot |grep zroot/ROOT/defau= lt should give a list of the snapshots and the sizes. >>>>>>>>> >>>>>>>>> Paul >>>>>>>>> >>>>>>>>> >>>>>>>> Thanks so much Paul.... I feel like I'm spinning around in circl= es currently... >>>>>>>> >>>>>>>> >>>>>>>> There doesn't seem to be anything indicative of that space being= used up? >>>>>>>> >>>>>>>> >>>>>>>> zfs list -o space -t snapshot |grep zroot/ROOT/defaul >>>>>>>> zroot/ROOT/default@2022-11-28-06:08:26-0 - 583M - -= - - >>>>>>>> zroot/ROOT/default@2022-11-28-06:38:48-0 - 11.4M - -= - - >>>>>>>> zroot/ROOT/default@2022-11-28-07:26:20-0 - 9.41M - -= - - >>>>>>>> zroot/ROOT/default@2023-08-20-22:00:10-0 - 34.5M - -= - - >>>>>>>> zroot/ROOT/default@2023-08-20-22:45:34-0 - 9.56M - -= - - >>>>>>>> >>>>>>>> >>>>>>>> certainly not 52GB? >>>>>>>> >>>>>>>> >>>>>>> That=E2=80=99s definitely confusing. >>>>>>> What=E2=80=99s the full output of 'zfs list -o space -r zroot=E2=80= =99 and of 'zpool list -v zroot=E2=80=99 >>>>>>> >>>>>>> Paul >>>>>>> >>>>>>> >>>>>> Here are the outputs: >>>>>> >>>>>> >>>>>> zfs list -o space -r zroot >>>>>> NAME AVAIL USED USEDSNA= P USEDDS USEDREFRESERV USEDCHILD >>>>>> zroot 720M 107G 0B 2.= 20G 0B 105G >>>>>> zroot/ROOT 720M 88.5G 0B = 88K 0B 88.5G >>>>>> zroot/ROOT/13.1-RELEASE-p5_2023-08-20_220010 720M 8K 0B = 8K 0B 0B >>>>>> zroot/ROOT/13.2-RELEASE-p2_2023-08-20_224534 720M 8K 0B = 8K 0B 0B >>>>>> zroot/ROOT/default 720M 88.5G 52.0G = 36.5G 0B 0B >>>>>> zroot/swap 3.78G 6.19G 0B 3.= 11G 3.08G 0B >>>>>> zroot/tmp 720M 121M 0B 1= 21M 0B 0B >>>>>> zroot/usr 720M 7.09G 0B = 88K 0B 7.09G >>>>>> zroot/usr/home 720M 88K 0B = 88K 0B 0B >>>>>> zroot/usr/obj 720M 4.75G 0B 4.= 75G 0B 0B >>>>>> zroot/usr/ports 720M 432K 0B 1= 76K 0B 256K >>>>>> zroot/usr/ports/distfiles 720M 168K 0B 1= 68K 0B 0B >>>>>> zroot/usr/ports/packages 720M 88K 0B = 88K 0B 0B >>>>>> zroot/usr/src 720M 2.35G 0B 2.= 35G 0B 0B >>>>>> zroot/var 720M 2.60G 0B = 88K 0B 2.60G >>>>>> zroot/var/audit 720M 88K 0B = 88K 0B 0B >>>>>> zroot/var/backups 720M 141M 0B 1= 41M 0B 0B >>>>>> zroot/var/crash 720M 88K 0B = 88K 0B 0B >>>>>> zroot/var/db 720M 1.69G 0B 1.= 59G 0B 94.8M >>>>>> zroot/var/db/pkg 720M 94.8M 0B 94= .8M 0B 0B >>>>>> zroot/var/empty 720M 88K 0B = 88K 0B 0B >>>>>> zroot/var/log 720M 792M 0B 7= 92M 0B 0B >>>>>> zroot/var/mail 720M 3.96M 0B 3.= 96M 0B 0B >>>>>> zroot/var/run 720M 248K 0B 2= 48K 0B 0B >>>>>> zroot/var/tmp 720M 672K 0B 6= 72K 0B 0B >>>>>> >>>>>> >>>>>> zpool list -v zroot >>>>>> NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP D= EDUP HEALTH ALTROOT >>>>>> zroot 111G 104G 7.22G - - 82% 93% 1= .00x ONLINE - >>>>>> mirror-0 111G 104G 7.22G - - 82% 93.5% = - ONLINE >>>>>> ada0p3 111G - - - - - - = - ONLINE >>>>>> ada1p3 111G - - - - - - = - ONLINE >>>>>> >>>>> Those look quite normal. What about 'zfs list -r -t all -o space zr= oot/ROOT/default=E2=80=99 ? >>>>> >>>>> Paul >>>>> >>>>> >>>>> >>>> zfs list -r -t all -o space zroot/ROOT/default >>>> NAME AVAIL USED USEDSNAP USE= DDS USEDREFRESERV USEDCHILD >>>> zroot/ROOT/default 720M 88.5G 52.0G 36.= 5G 0B 0B >>>> zroot/ROOT/default@2022-11-28-06:08:26-0 - 583M - - = - - >>>> zroot/ROOT/default@2022-11-28-06:38:48-0 - 11.4M - - = - - >>>> zroot/ROOT/default@2022-11-28-07:26:20-0 - 9.41M - - = - - >>>> zroot/ROOT/default@2023-08-20-22:00:10-0 - 34.5M - - = - - >>>> zroot/ROOT/default@2023-08-20-22:45:34-0 - 9.56M - - = - - >>>> >>>> >>>> Strange! It shows a 52GB 'claimed' snapshot but where is it? >>>> >>>> >>> Can you give output of 'zfs get all zroot/ROOT/default=E2=80=99 as we= ll ? Maybe that gives a hint. >>> >>> Paul >>> >> Sure: >> >> >> zfs get all zroot/ROOT/default >> NAME PROPERTY VALUE SOURCE >> zroot/ROOT/default type filesystem - >> zroot/ROOT/default creation Tue Aug 7 2:53 2018 - >> zroot/ROOT/default used 88.5G - >> zroot/ROOT/default available 720M - >> zroot/ROOT/default referenced 36.5G - >> zroot/ROOT/default compressratio 1.29x - >> zroot/ROOT/default mounted yes - >> zroot/ROOT/default quota none default >> zroot/ROOT/default reservation none default >> zroot/ROOT/default recordsize 128K default >> zroot/ROOT/default mountpoint / local >> zroot/ROOT/default sharenfs off default >> zroot/ROOT/default checksum on default >> zroot/ROOT/default compression lz4 inherited from zroot >> zroot/ROOT/default atime on default >> zroot/ROOT/default devices on default >> zroot/ROOT/default exec on default >> zroot/ROOT/default setuid on default >> zroot/ROOT/default readonly off default >> zroot/ROOT/default jailed off default >> zroot/ROOT/default snapdir hidden default >> zroot/ROOT/default aclmode discard default >> zroot/ROOT/default aclinherit restricted default >> zroot/ROOT/default createtxg 93 - >> zroot/ROOT/default canmount noauto local >> zroot/ROOT/default xattr on default >> zroot/ROOT/default copies 1 default >> zroot/ROOT/default version 5 - >> zroot/ROOT/default utf8only off - >> zroot/ROOT/default normalization none - >> zroot/ROOT/default casesensitivity sensitive - >> zroot/ROOT/default vscan off default >> zroot/ROOT/default nbmand off default >> zroot/ROOT/default sharesmb off default >> zroot/ROOT/default refquota none default >> zroot/ROOT/default refreservation none default >> zroot/ROOT/default guid 15509736917917171623 - >> zroot/ROOT/default primarycache all default >> zroot/ROOT/default secondarycache all default >> zroot/ROOT/default usedbysnapshots 52.0G - >> zroot/ROOT/default usedbydataset 36.5G - >> zroot/ROOT/default usedbychildren 0B - >> zroot/ROOT/default usedbyrefreservation 0B - >> zroot/ROOT/default logbias latency default >> zroot/ROOT/default objsetid 58 - >> zroot/ROOT/default dedup off default >> zroot/ROOT/default mlslabel none default >> zroot/ROOT/default sync standard default >> zroot/ROOT/default dnodesize legacy default >> zroot/ROOT/default refcompressratio 1.32x - >> zroot/ROOT/default written 1.53G - >> zroot/ROOT/default logicalused 108G - >> zroot/ROOT/default logicalreferenced 46.0G - >> zroot/ROOT/default volmode default default >> zroot/ROOT/default filesystem_limit none default >> zroot/ROOT/default snapshot_limit none default >> zroot/ROOT/default filesystem_count none default >> zroot/ROOT/default snapshot_count none default >> zroot/ROOT/default snapdev hidden default >> zroot/ROOT/default acltype nfsv4 default >> zroot/ROOT/default context none default >> zroot/ROOT/default fscontext none default >> zroot/ROOT/default defcontext none default >> zroot/ROOT/default rootcontext none default >> zroot/ROOT/default relatime off default >> zroot/ROOT/default redundant_metadata all default >> zroot/ROOT/default overlay on default >> zroot/ROOT/default encryption off default >> zroot/ROOT/default keylocation none default >> zroot/ROOT/default keyformat none default >> zroot/ROOT/default pbkdf2iters 0 default >> zroot/ROOT/default special_small_blocks 0 default >> >> > That looks consistent with the rest of the data. > > Man page for zfsprops mentions this for snapshots; > The used space of a snapshot (see the Snaps= hots > section of zfsconcepts(7)) is space that is > referenced exclusively by this snapshot. I= f this > snapshot is destroyed, the amount of used s= pace > will be freed. Space that is shared by mul= tiple > snapshots isn't accounted for in this metri= c. When > a snapshot is destroyed, space that was pre= viously > shared with this snapshot can become unique= to > snapshots adjacent to it, thus changing the= used > space of those snapshots. The used space o= f the > latest snapshot can also be affected by cha= nges in > the file system. Note that the used space = of a > snapshot is a subset of the written space o= f the > snapshot. > Note : Space that is shared by multiple snapshots isn't accounted for = in this metric. > > My theory is that the snapshots/boot environments share about 52G of w= ith each other but not with the parent dataset. > As there is no data unique to one snapshot it does not show > What you could do is delete all except one bootenvironment. > The snapshot data used should then be unique to that last snapshot and = appear in the lists. > > Paul > > > I just responded to Olivier above but I have managed to free up the=20 necessary space by deleting all the snapshots: df -h Filesystem=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Size=C2=A0=C2=A0=C2=A0 Used=C2=A0=C2= =A0 Avail Capacity=C2=A0=20 Mounted on zroot/ROOT/default=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= 89G=C2=A0=C2=A0=C2=A0=C2=A0 36G=C2=A0=C2=A0=C2=A0=C2=A0 53G 41%=C2=A0=C2= =A0=C2=A0 / For some reason some of the snapshots were automatically 'cloned' and=20 exactly as you suggested above there was a fair amount of space sharing=20 between snapshots. This of course resulted in data being removed from=20 the main snapshot but still remained on disk due another snapshot=20 referencing that data.... Very confusing for an automated process but anyway, I have space again :-= ) Thank you so much everyone for your help and input! Best Regards, Kaya