Date: Wed, 23 Jan 2013 15:37:29 +0100 From: Romain =?iso-8859-1?Q?Tarti=E8re?= <romain@blogreen.org> To: freebsd-fs@freebsd.org Subject: ZFS deduplication Message-ID: <20130123143728.GA84218@blogreen.org>
next in thread | raw e-mail | index | archive | help
--bp/iNruPH9dso1Pn
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hello
My machine has run on full ZFS for quite a few years now without major
problem, but suddenly started to reports inconsistent "statistics". It
all started by `zfs list` reporting too much available space:
> # zfs list data
> NAME USED AVAIL REFER MOUNTPOINT
> data 117G 5,73E 4,52G legacy
"data" being a mirror of two 500 GB disks, the available space is
obviously wrong. The pool is healthy according to `zpool status`:
> # zpool status data
> pool: data
> state: ONLINE
> scan: scrub repaired 0 in 5h1m with 0 errors on Tue Jan 1 09:52:50 2013
> config:
>=20
> NAME STATE READ WRITE CKSUM
> data ONLINE 0 0 0
> mirror-0 ONLINE 0 0 0
> ada0p3 ONLINE 0 0 0
> ada1p3 ONLINE 0 0 0
>=20
> errors: No known data errors
However, `zpool list` reports an inconsistent deduplication value (it
used to be ~1.4 AFAICR):
> zpool list data
> NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
> data 460G 101G 359G 21% 1386985.39x ONLINE -
At first I though about a process that might have created bazillion of
files with the same contents but df(1) does not report any filesystem
with an abnormal number of used inodes. Likewise, df(1) does not show
a filesystem with a huge amount of data that could come from a single
file which content would be a repeated single block of data (but maybe
df(1) can't tell me that):
> df -i
> Filesystem 1K-blocks =
Used Avail Capacity iused ifree %iused Mounted=
on
> data 645500584230938=
0 4735434 6455005837573946 0% 45457 281474976665199 0% /
> devfs =
1 1 0 100% 0 0 100% /dev
> procfs =
4 4 0 100% 1 0 100% /proc
> fdescfs =
1 1 0 100% 4 22497 0% /dev/fd
> data/tmp 645500583758072=
9 6783 6455005837573946 0% 2087 281474976708569 0% /tmp
> data/usr 645500585324521=
1 15671265 6455005837573946 0% 532871 281474976177785 0% /usr
> tank 192093743=
1 490426844 1430510587 26% 1315413 2861021175 0% /usr/ho=
me
> data/poudriere/data 645500583943267=
1 1858725 6455005837573946 0% 11958 281474976698698 0% /usr/lo=
cal/poudriere/data
> data/poudriere/8_3_RELEASE_amd64 645500583838496=
9 811023 6455005837573946 0% 56480 281474976654176 0% /usr/lo=
cal/poudriere/jails/8_3_RELEASE_amd64
> data/poudriere/9_0_RELEASE_amd64 645500583861545=
4 1041508 6455005837573946 0% 66523 281474976644133 0% /usr/lo=
cal/poudriere/jails/9_0_RELEASE_amd64
> data/poudriere/jails/9_1_RELEASE_amd64 645500583865276=
0 1078814 6455005837573946 0% 67766 281474976642890 0% /usr/lo=
cal/poudriere/jails/9_1_RELEASE_amd64
> data/poudriere/ports/default 645500583806405=
5 490109 6455005837573946 0% 151864 281474976558792 0% /usr/lo=
cal/poudriere/ports/default/ports
> data/poudriere/ports/gnome 645500583806928=
3 495337 6455005837573946 0% 152758 281474976557898 0% /usr/lo=
cal/poudriere/ports/gnome/ports
> data/poudriere/ports/mono 645500583806266=
3 488717 6455005837573946 0% 152411 281474976558245 0% /usr/lo=
cal/poudriere/ports/mono/ports
> data/poudriere/ports/romain 645500583806495=
6 491010 6455005837573946 0% 152030 281474976558626 0% /usr/lo=
cal/poudriere/ports/romain/ports
> data/usr/main 645500583808621=
6 512270 6455005837573946 0% 164164 281474976546492 0% /usr/po=
rts
> data/var 645500585671870=
3 19144757 6455005837573946 0% 158141 281474976552515 0% /var
> data/var/cache 645500583757404=
1 95 6455005837573946 0% 17 281474976710639 0% /var/ca=
che
> data/var/cache/portshaker 645500583758706=
8 13122 6455005837573946 0% 4658 281474976705998 0% /var/ca=
che/portshaker
> data/var/cache/portshaker/bsd_sharp 645500583757854=
9 4603 6455005837573946 0% 1695 281474976708961 0% /var/ca=
che/portshaker/bsd_sharp
> data/var/cache/portshaker/e 645500583757523=
0 1284 6455005837573946 0% 977 281474976709679 0% /var/ca=
che/portshaker/e
> data/var/cache/portshaker/freebsd_texlive 645500583757418=
5 239 6455005837573946 0% 124 281474976710532 0% /var/ca=
che/portshaker/freebsd_texlive
> data/var/cache/portshaker/freebsd_texlive_ports 645500583757853=
2 4586 6455005837573946 0% 289 281474976710367 0% /var/ca=
che/portshaker/freebsd_texlive_ports
> data/var/cache/portshaker/freebsd_texlive_ports_marcuscom 645500583757406=
6 120 6455005837573946 0% 29 281474976710627 0% /var/ca=
che/portshaker/freebsd_texlive_ports_marcuscom
> data/var/cache/portshaker/freebsd_texlive_releng 645500583761182=
6 37880 6455005837573946 0% 29335 281474976681321 0% /var/ca=
che/portshaker/freebsd_texlive_releng
> data/var/cache/portshaker/ports 645500583806211=
5 488169 6455005837573946 0% 152039 281474976558617 0% /var/ca=
che/portshaker/ports
> data/var/cache/portshaker/ports/distfiles 645500584283383=
7 5259891 6455005837573946 0% 2605 281474976708051 0% /var/ca=
che/portshaker/ports/distfiles
> data/var/cache/portshaker/ports/packages 645500585631547=
6 18741530 6455005837573946 0% 16799 281474976693857 0% /var/ca=
che/portshaker/ports/packages
> data/var/cache/portshaker/redports 645500583757448=
8 542 6455005837573946 0% 312 281474976710344 0% /var/ca=
che/portshaker/redports
> data/var/cache/portshaker/redports:fneufneu 645500583757548=
2 1536 6455005837573946 0% 317 281474976710339 0% /var/ca=
che/portshaker/redports:fneufneu
> data/var/cache/portshaker/redports:romain 645500583757429=
5 349 6455005837573946 0% 226 281474976710430 0% /var/ca=
che/portshaker/redports:romain
> data/var/cache/portshaker/redports:virtualbox 645500583757471=
7 771 6455005837573946 0% 444 281474976710212 0% /var/ca=
che/portshaker/redports:virtualbox
> data/var/cache/portshaker/redports:xbmc 645500583757397=
7 31 6455005837573946 0% 7 281474976710649 0% /var/ca=
che/portshaker/redports:xbmc
> data/var/cache/portshaker/romain 645500583757701=
1 3065 6455005837573946 0% 930 281474976709726 0% /var/ca=
che/portshaker/romain
> data/var/cache/portshaker/woot 645500583757406=
5 119 6455005837573946 0% 75 281474976710581 0% /var/ca=
che/portshaker/woot
> data/var/jails 645500584133690=
1 3762955 6455005837573946 0% 517515 281474976193141 0% /var/ja=
ils
> data/var/jails/base 645500583840364=
8 829702 6455005837573946 0% 11659 281474976698997 0% /var/ja=
ils/base
> data/var/jails/r230911 645500583785718=
5 283239 6455005837573946 0% 11581 281474976699075 0% /var/ja=
ils/r230911
> data/var/jails/texlive-test 645500584460139=
8 7027452 6455005837573946 0% 496201 281474976214455 0% /var/ja=
ils/texlive-test
> data/var/jails/texlive.home.sigabrt.org 645500585084185=
3 13267907 6455005837573946 0% 377896 281474976332760 0% /var/ja=
ils/texlive.home.sigabrt.org
> data/var/jails/texlive.home.sigabrt.org/ports 645500583823011=
9 656173 6455005837573946 0% 152331 281474976558325 0% /var/ja=
ils/texlive.home.sigabrt.org/usr/ports
> data/var/jails/xbmc 645500584044239=
6 2868450 6455005837573946 0% 228330 281474976482326 0% /var/ja=
ils/xbmc
> devfs =
1 1 0 100% 0 0 100% /var/na=
med/dev
> devfs =
1 1 0 100% 0 0 100% /var/ja=
ils/texlive.home.sigabrt.org/dev
> devfs =
1 1 0 100% 0 0 100% /var/ja=
ils/texlive/dev
Does anyone has an idea about what is going on? Is there any way I can
know which deduplicated blocks are most used and in which file(s) to
locate the source of the problem? Or is there any way to prove that
this reporting is wrong?
I'm running 9.1-STABLE r245578, with zpool v28 (I guess, no way to check
that?) and ZFS v5 for everything but a few snapshots (v4, v3; according
to `zfs get version`).
Thanks!
Romain
--=20
Romain Tarti=E8re <romain@blogreen.org> http://romain.blogreen.org/
pgp: 8234 9A78 E7C0 B807 0B59 80FF BA4D 1D95 5112 336F (ID: 0x5112336F)
(plain text =3Dnon-HTML=3D PGP/GPG encrypted/signed e-mail much appreciated)
--bp/iNruPH9dso1Pn
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
iQGcBAEBAgAGBQJQ//WoAAoJELpNHZVREjNvC7ML/AsIr8KotegWtEx7ka/0IgeE
jj7nyYtdAVdjRkIlgHaHL2vIKZy/6Csf4tE2V56k/hHAHAWEYMHW2At7nKu9mmUQ
CqCjGAf0Pir8ZNNa5PYIA+Rw9HkE4KjHWz2COuyw99IFoFwNO/eIFs2MHG6x04+M
n+GJ0Ra/RUJnttCfsSS5ZmCA+PzVkqG0oztzRZfGGgkJBTg7B//Y3Klp7VpiE2Sc
YEh5IzWxWNMbOyzRqr/4EnBD0xJF1H+WUMmWo+J+Jgn6VD9t8tb1lohzp2/6BosX
l9kumW9WUHcgMQtPwmU5mNIZimfzR0BYickMY77pzi8Y+UV6YB3+rSgkAk6sQphw
DyBUsAJXm7p8ZzpugpBjGMenOuIo+3mQDBmmX8DqkVQEDGPSMO2C4dTSN5jOWG3X
vNecwwQnfQs1dL0nXvwENYZ/uDoD8n64r4OIuqfBtv+TNT6CzAYn0KZTr/Cnl/Oo
bxGYvDZhnLDCd2vKD7DkgxXL/F4GfS5dz9SK+qecbg==
=Y1Ng
-----END PGP SIGNATURE-----
--bp/iNruPH9dso1Pn--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130123143728.GA84218>