Date: Mon, 13 Oct 2025 13:40:49 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 290205] ZFS: file stuck in delete queue Message-ID: <bug-290205-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D290205 Bug ID: 290205 Summary: ZFS: file stuck in delete queue Product: Base System Version: 14.3-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Only Me Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: ml@netfence.it I've got a problem where a file is seemingly in the delete queue and has be= en there for months. This isn't an issue with space (I have plenty of), but "zfs send"ing the incriminated pool produces an overly inflated file (51GB after compression, when it should be around 2GB). The box is FreeBSD 14.3/amd64, fully patched. There are two pools, all of then in good HEALTH and with "No known data errors": # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALT= H=20 ALTROOT vm 446G 329G 117G - - 49% 73% 1.00x ONLIN= E - zroot 3.56T 1.52T 2.04T - - 19% 42% 1.00x ONLIN= E - #zfs list -o name,mountpoint,canmount,refer,used,usedbychildren,usedbydataset,usedbyrefr= eservation,usedbysnapshots zroot /zroot on=20=20= =20=20=20=20=20=20=20 88K 1.52T 1.52T 88K 0B 536K zroot/ROOT none on=20=20= =20=20=20=20=20=20=20 88K 65.4G 65.4G 88K 0B 0B zroot/ROOT/default / noauto=20= =20=20 62.1G 65.4G 0B 62.1G 0B 3.23G zroot/ezjail /usr/jails on=20=20= =20=20=20=20=20=20 136K 1.41T 1.41T 136K 0B 328K ... As you can see zroot/ROOT/default has 62.1G used by dataset, yet only 3.3G = seem to exist when using standard file system tools: # du -d 0 -x -h / 3.3G / With the help of the mailing list, we found out the following: # zdb -ddd zroot/ROOT/default 0:-1:A |sort -h -b -k 5 |tail -5 163861 3 128K 128K 77.1M 512 178M 100.00 ZFS plain file 130966 3 128K 128K 96.4M 512 262M 100.00 ZFS plain file 163976 3 128K 128K 139M 512 335M 100.00 ZFS plain file 130956 3 128K 128K 145M 512 343M 100.00 ZFS plain file 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file # zdb -ddddd zroot/ROOT/default 360 Dataset zroot/ROOT/default [ZPL], ID 89, cr_txg 8, 62.1G, 107125 objects, rootbp DVA[0]=3D<0:103e1032000:1000> DVA[1]=3D<0:4365c65000:1000> [L0 DMU o= bjset] fletcher4 uncompressed unencrypted LE contiguous unique double size=3D800L/= 800P birth=3D3 7011214L/37011214P fill=3D107125 cksum=3D0000000e1765a187:00000fbba696e8af:000aec7540e5bc65:05a9323d03d6cb70 Object lvl iblk dblk dsize dnsize lsize %full type 360 3 128K 128K 58.8G 512 90.0G 100.00 ZFS plain file 168 bonus System attribut= es dnode flags: USED_BYTES USERUSED_ACCOUNTED=20 dnode maxblkid: 737498 path on delete queue uid 0 gid 0 ... So this is a 58.8GB file that has, from the user perspective, been deleted,= but it's still referenced by ZFS. Rebooting (several times) did not help. Deleting all zroot/ROOT/default's snapshot was useless. In case it is needed: # zpool get all zroot NAME PROPERTY VALUE SOURCE zroot size 3.56T - zroot capacity 42% - zroot altroot - default zroot health ONLINE - zroot guid 10368181361075995187 - zroot version - default zroot bootfs zroot/ROOT/default local zroot delegation on default zroot autoreplace off default zroot cachefile - default zroot failmode wait default zroot listsnapshots off default zroot autoexpand off default zroot dedupratio 1.00x - zroot free 2.04T - zroot allocated 1.52T - zroot readonly off - zroot ashift 0 default zroot comment - default zroot expandsize - - zroot freeing 0 - zroot fragmentation 19% - zroot leaked 0 - zroot multihost off default zroot checkpoint - - zroot load_guid 8455461369367935358 - zroot autotrim off default zroot compatibility off default zroot bcloneused 0 - zroot bclonesaved 0 - zroot bcloneratio 1.00x - zroot feature@async_destroy enabled local zroot feature@empty_bpobj active local zroot feature@lz4_compress active local zroot feature@multi_vdev_crash_dump enabled local zroot feature@spacemap_histogram active local zroot feature@enabled_txg active local zroot feature@hole_birth active local zroot feature@extensible_dataset active local zroot feature@embedded_data active local zroot feature@bookmarks enabled local zroot feature@filesystem_limits enabled local zroot feature@large_blocks enabled local zroot feature@large_dnode enabled local zroot feature@sha512 enabled local zroot feature@skein enabled local zroot feature@edonr disabled local zroot feature@userobj_accounting disabled local zroot feature@encryption disabled local zroot feature@project_quota disabled local zroot feature@device_removal enabled local zroot feature@obsolete_counts enabled local zroot feature@zpool_checkpoint enabled local zroot feature@spacemap_v2 active local zroot feature@allocation_classes disabled local zroot feature@resilver_defer disabled local zroot feature@bookmark_v2 disabled local zroot feature@redaction_bookmarks disabled local zroot feature@redacted_datasets disabled local zroot feature@bookmark_written disabled local zroot feature@log_spacemap disabled local zroot feature@livelist disabled local zroot feature@device_rebuild disabled local zroot feature@zstd_compress disabled local zroot feature@draid disabled local zroot feature@zilsaxattr disabled local zroot feature@head_errlog disabled local zroot feature@blake3 disabled local zroot feature@block_cloning disabled local zroot feature@vdev_zaps_v2 disabled local # zfs get all zroot/ROOT/default NAME PROPERTY VALUE SOURCE zroot/ROOT/default type filesystem - zroot/ROOT/default creation Sun Dec 15 16:58 2019 - zroot/ROOT/default used 65.4G - zroot/ROOT/default available 1.93T - zroot/ROOT/default referenced 62.1G - zroot/ROOT/default compressratio 1.58x - zroot/ROOT/default mounted yes - zroot/ROOT/default quota none default zroot/ROOT/default reservation none default zroot/ROOT/default recordsize 128K default zroot/ROOT/default mountpoint / local zroot/ROOT/default sharenfs off default zroot/ROOT/default checksum on default zroot/ROOT/default compression lz4 inheri= ted from zroot zroot/ROOT/default atime off inheri= ted from zroot zroot/ROOT/default devices on default zroot/ROOT/default exec on default zroot/ROOT/default setuid on default zroot/ROOT/default readonly off local zroot/ROOT/default jailed off default zroot/ROOT/default snapdir hidden default zroot/ROOT/default aclmode discard default zroot/ROOT/default aclinherit restricted default zroot/ROOT/default createtxg 8 - zroot/ROOT/default canmount noauto local zroot/ROOT/default xattr on default zroot/ROOT/default copies 1 default zroot/ROOT/default version 5 - zroot/ROOT/default utf8only off - zroot/ROOT/default normalization none - zroot/ROOT/default casesensitivity sensitive - zroot/ROOT/default vscan off default zroot/ROOT/default nbmand off default zroot/ROOT/default sharesmb off default zroot/ROOT/default refquota none default zroot/ROOT/default refreservation none default zroot/ROOT/default guid 13999796839276270198 - zroot/ROOT/default primarycache all default zroot/ROOT/default secondarycache all default zroot/ROOT/default usedbysnapshots 3.23G - zroot/ROOT/default usedbydataset 62.1G - zroot/ROOT/default usedbychildren 0B - zroot/ROOT/default usedbyrefreservation 0B - zroot/ROOT/default logbias latency default zroot/ROOT/default objsetid 89 - zroot/ROOT/default dedup off default zroot/ROOT/default mlslabel none default zroot/ROOT/default sync standard default zroot/ROOT/default dnodesize legacy default zroot/ROOT/default refcompressratio 1.56x - zroot/ROOT/default written 1.16M - zroot/ROOT/default logicalused 103G - zroot/ROOT/default logicalreferenced 96.7G - zroot/ROOT/default volmode default default zroot/ROOT/default filesystem_limit none default zroot/ROOT/default snapshot_limit none default zroot/ROOT/default filesystem_count none default zroot/ROOT/default snapshot_count none default zroot/ROOT/default snapdev hidden default zroot/ROOT/default acltype nfsv4 default zroot/ROOT/default context none default zroot/ROOT/default fscontext none default zroot/ROOT/default defcontext none default zroot/ROOT/default rootcontext none default zroot/ROOT/default relatime on default zroot/ROOT/default redundant_metadata all default zroot/ROOT/default overlay on default zroot/ROOT/default encryption off default zroot/ROOT/default keylocation none default zroot/ROOT/default keyformat none default zroot/ROOT/default pbkdf2iters 0 default zroot/ROOT/default special_small_blocks 0 default zroot/ROOT/default snapshots_changed Sun Oct 5 17:00:01 2025 - zroot/ROOT/default prefetch all default zroot/ROOT/default autobackup:auto_zroot true inheri= ted from zroot --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-290205-227>