Date: Wed, 12 Apr 2017 23:01:10 +0500 From: "Eugene M. Zheganin" <emz@norma.perm.ru> To: freebsd-stable@FreeBSD.org, FreeBSD FS <freebsd-fs@freebsd.org> Subject: zpool list show nonsense on raidz pools, at least it looks like it for me Message-ID: <71ef8400-94ec-1f59-3b2b-bb576ad65b89@norma.perm.ru>
next in thread | raw e-mail | index | archive | help
Hi,
It's not my first letter where I fail to understand the space usage from
zfs utilities, and in previous ones I was kind of convinced that I just
read it wrong, but not this time I guess. See for yourself:
[emz@san01:~]> zpool list data
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
data 17,4T 7,72T 9,66T - 46% 44% 1.00x ONLINE -
Here' as I understand it, zpool says that less than a half of the pool
is used. As far as I know this is very complicated when it comes to the
radiz pools. Let's see:
[emz@san01:~]> zfs list -t all data
NAME USED AVAIL REFER MOUNTPOINT
data 13,3T 186G 27,2K /data
So, if we won't investigate further, it looks like that only 186G is
free. Spoiling - this is the real free space amount, because I've just
managed to free 160 gigs of data, and I really know I was short on space
when sending 30 Gb dataset, because zfs was saying "Not enough free
space". So, let's investigate further:
[emz@san01:~]> zfs list -t all | more
NAME USED AVAIL REFER MOUNTPOINT
data 13,3T 186G 27,2K /data
data/esx 5,23T 186G 27,2K /data/esx
data/esx/boot-esx01 8,25G 193G 561M -
data/esx/boot-esx02 8,25G 193G 561M -
data/esx/boot-esx03 8,25G 193G 561M -
data/esx/boot-esx04 8,25G 193G 561M -
data/esx/boot-esx05 8,25G 193G 561M -
data/esx/boot-esx06 8,25G 193G 561M -
data/esx/boot-esx07 8,25G 193G 962M -
data/esx/boot-esx08 8,25G 193G 562M -
data/esx/boot-esx09 8,25G 193G 562M -
data/esx/boot-esx10 8,25G 193G 595M -
data/esx/boot-esx11 8,25G 193G 539M -
data/esx/boot-esx12 8,25G 193G 539M -
data/esx/boot-esx13 8,25G 193G 539M -
data/esx/boot-esx14 8,25G 193G 539M -
data/esx/boot-esx15 8,25G 193G 539M -
data/esx/boot-esx16 8,25G 193G 541M -
data/esx/boot-esx17 8,25G 193G 540M -
data/esx/boot-esx18 8,25G 193G 539M -
data/esx/boot-esx19 8,25G 193G 542M -
data/esx/boot-esx20 8,25G 194G 12,8K -
data/esx/boot-esx21 8,25G 194G 12,8K -
data/esx/boot-esx22 8,25G 193G 913M -
data/esx/boot-esx23 8,25G 193G 558M -
data/esx/boot-esx24 8,25G 194G 12,8K -
data/esx/boot-esx25 8,25G 194G 12,8K -
data/esx/boot-esx26 8,25G 194G 12,8K -
data/esx/shared 5,02T 2,59T 2,61T -
data/reference 6,74T 4,17T 2,73T -
data/reference@ver7_214 127M - 2,73T -
data/reference@ver2_739 12,8M - 2,73T -
data/reference@ver2_740 5,80M - 2,73T -
data/reference@ver2_741 4,55M - 2,73T -
data/reference@ver2_742 993K - 2,73T -
data/reference-ver2_739-worker100 1,64G 186G 2,73T -
data/reference-ver2_739-worker101 254M 186G 2,73T -
data/reference-ver2_739-worker102 566K 186G 2,73T -
data/reference-ver2_739-worker103 260M 186G 2,73T -
data/reference-ver2_739-worker104 8,74G 186G 2,73T -
data/reference-ver2_739-worker105 4,19G 186G 2,73T -
data/reference-ver2_739-worker106 1,72G 186G 2,73T -
data/reference-ver2_739-worker107 282M 186G 2,73T -
data/reference-ver2_739-worker108 1,27M 186G 2,73T -
data/reference-ver2_739-worker109 8,74G 186G 2,73T -
data/reference-ver2_739-worker110 8,74G 186G 2,73T -
data/reference-ver2_739-worker111 8,74G 186G 2,73T -
data/reference-ver2_739-worker112 8,74G 186G 2,73T -
data/reference-ver2_739-worker113 838K 186G 2,73T -
data/reference-ver2_739-worker114 8,74G 186G 2,73T -
data/reference-ver2_739-worker115 8,74G 186G 2,73T -
data/reference-ver2_739-worker116 8,74G 186G 2,73T -
data/reference-ver2_739-worker117 8,74G 186G 2,73T -
data/reference-ver2_739-worker118 8,74G 186G 2,73T -
data/reference-ver2_739-worker119 8,74G 186G 2,73T -
data/reference-ver2_739-worker120 8,74G 186G 2,73T -
data/reference-ver2_739-worker121 8,74G 186G 2,73T -
data/reference-ver2_739-worker122 8,74G 186G 2,73T -
data/reference-ver2_739-worker123 8,74G 186G 2,73T -
data/reference-ver2_739-worker124 8,74G 186G 2,73T -
data/reference-ver2_739-worker125 8,74G 186G 2,73T -
data/reference-ver2_739-worker126 8,74G 186G 2,73T -
data/reference-ver2_739-worker127 8,74G 186G 2,73T -
data/reference-ver2_739-worker128 8,74G 186G 2,73T -
data/reference-ver2_739-worker129 8,74G 186G 2,73T -
data/reference-ver2_739-worker130 8,74G 186G 2,73T -
data/reference-ver2_739-worker131 8,74G 186G 2,73T -
data/reference-ver2_739-worker132 8,74G 186G 2,73T -
data/reference-ver2_739-worker133 8,74G 186G 2,73T -
data/reference-ver2_739-worker134 8,74G 186G 2,73T -
data/reference-ver2_739-worker135 8,74G 186G 2,73T -
data/reference-ver2_739-worker136 8,74G 186G 2,73T -
data/reference-ver2_739-worker137 8,74G 186G 2,73T -
data/reference-ver2_739-worker138 8,74G 186G 2,73T -
data/reference-ver2_739-worker139 8,74G 186G 2,73T -
data/reference-ver2_739-worker140 8,74G 186G 2,73T -
data/reference-ver2_739-worker141 8,74G 186G 2,73T -
data/reference-ver2_739-worker142 8,74G 186G 2,73T -
data/reference-ver2_739-worker143 8,73G 186G 2,73T -
data/reference-ver2_739-worker144 9,16G 186G 2,73T -
data/reference-ver2_739-worker145 8,74G 186G 2,73T -
data/reference-ver2_739-worker146 8,74G 186G 2,73T -
data/reference-ver2_739-worker147 8,74G 186G 2,73T -
data/reference-ver2_739-worker148 8,74G 186G 2,73T -
data/reference-ver2_739-worker149 8,74G 186G 2,73T -
data/reference-ver2_739-worker150 8,74G 186G 2,73T -
data/reference-ver2_739-worker151 8,74G 186G 2,73T -
data/reference-ver2_739-worker152 8,74G 186G 2,73T -
data/reference-ver2_739-worker53 385K 186G 2,73T -
data/reference-ver2_739-worker58 9,49M 186G 2,73T -
data/reference-ver2_739-worker81 8,20G 186G 2,73T -
data/reference-ver2_739-worker82 7,64G 186G 2,73T -
data/reference-ver2_739-worker83 6,84G 186G 2,73T -
data/reference-ver2_739-worker84 5,95G 186G 2,73T -
data/reference-ver2_739-worker85 5,09G 186G 2,73T -
data/reference-ver2_739-worker86 3,76G 186G 2,73T -
data/reference-ver2_739-worker87 4,06G 186G 2,73T -
data/reference-ver2_739-worker88 1,92G 186G 2,73T -
data/reference-ver2_739-worker89 6,83G 186G 2,73T -
data/reference-ver2_739-worker90 6,55G 186G 2,73T -
data/reference-ver2_739-worker91 5,25G 186G 2,73T -
data/reference-ver2_739-worker92 4,18G 186G 2,73T -
data/reference-ver2_739-worker93 3,18G 186G 2,73T -
data/reference-ver2_739-worker94 983M 186G 2,73T -
data/reference-ver2_739-worker95 525M 186G 2,73T -
data/reference-ver2_739-worker96 650M 186G 2,73T -
data/reference-ver2_739-worker97 5,78G 186G 2,73T -
data/reference-ver2_739-worker98 4,81G 186G 2,73T -
data/reference-ver2_739-worker99 4,93G 186G 2,73T -
data/reference-ver2_740-worker56 361K 186G 2,73T -
data/reference-ver2_741-worker74 3,98M 186G 2,73T -
data/reference-ver2_742-worker02 934K 186G 2,73T -
data/reference-ver2_742-worker03 935K 186G 2,73T -
data/reference-ver2_742-worker04 935K 186G 2,73T -
data/reference-ver2_742-worker05 935K 186G 2,73T -
data/reference-ver2_742-worker06 935K 186G 2,73T -
data/reference-ver2_742-worker08 934K 186G 2,73T -
data/reference-ver2_742-worker09 15,9M 186G 2,73T -
data/reference-ver2_742-worker10 16,2M 186G 2,73T -
data/reference-ver2_742-worker11 15,4M 186G 2,73T -
data/reference-ver2_742-worker12 935K 186G 2,73T -
data/reference-ver2_742-worker13 15,9M 186G 2,73T -
data/reference-ver2_742-worker14 935K 186G 2,73T -
data/reference-ver2_742-worker15 15,5M 186G 2,73T -
data/reference-ver2_742-worker16 934K 186G 2,73T -
data/reference-ver2_742-worker17 934K 186G 2,73T -
data/reference-ver2_742-worker18 935K 186G 2,73T -
data/reference-ver2_742-worker19 531M 186G 2,73T -
data/reference-ver2_742-worker20 935K 186G 2,73T -
data/reference-ver2_742-worker21 935K 186G 2,73T -
data/reference-ver2_742-worker22 5,27G 186G 2,73T -
data/reference-ver2_742-worker23 934K 186G 2,73T -
data/reference-ver2_742-worker24 5,27G 186G 2,73T -
data/reference-ver2_742-worker25 934K 186G 2,73T -
data/reference-ver2_742-worker26 934K 186G 2,73T -
data/reference-ver2_742-worker27 15,8M 186G 2,73T -
data/reference-ver2_742-worker28 935K 186G 2,73T -
data/reference-ver2_742-worker29 934K 186G 2,73T -
data/reference-ver2_742-worker30 934K 186G 2,73T -
data/reference-ver2_742-worker31 934K 186G 2,73T -
data/reference-ver2_742-worker32 444M 186G 2,73T -
data/reference-ver2_742-worker33 935K 186G 2,73T -
data/reference-ver2_742-worker34 15,9M 186G 2,73T -
data/reference-ver2_742-worker35 1,12M 186G 2,73T -
data/reference-ver2_742-worker36 935K 186G 2,73T -
data/reference-ver2_742-worker37 5,27G 186G 2,73T -
data/reference-ver2_742-worker38 935K 186G 2,73T -
data/reference-ver2_742-worker39 935K 186G 2,73T -
data/reference-ver2_742-worker40 935K 186G 2,73T -
data/reference-ver2_742-worker41 15,5M 186G 2,73T -
data/reference-ver2_742-worker42 934K 186G 2,73T -
data/reference-ver2_742-worker43 15,4M 186G 2,73T -
data/reference-ver2_742-worker44 429M 186G 2,73T -
data/reference-ver2_742-worker45 15,5M 186G 2,73T -
data/reference-ver2_742-worker46 935K 186G 2,73T -
data/reference-ver2_742-worker47 934K 186G 2,73T -
data/reference-ver2_742-worker48 15,5M 186G 2,73T -
data/reference-ver2_742-worker49 935K 186G 2,73T -
data/reference-ver2_742-worker50 934K 186G 2,73T -
data/reference-ver2_742-worker51 935K 186G 2,73T -
data/reference-ver2_742-worker52 935K 186G 2,73T -
data/reference-ver2_742-worker54 935K 186G 2,73T -
data/reference-ver2_742-worker55 934K 186G 2,73T -
data/reference-ver2_742-worker57 935K 186G 2,73T -
data/reference-ver2_742-worker59 934K 186G 2,73T -
data/reference-ver2_742-worker60 15,5M 186G 2,73T -
data/reference-ver2_742-worker61 934K 186G 2,73T -
data/reference-ver2_742-worker62 934K 186G 2,73T -
data/reference-ver2_742-worker63 934K 186G 2,73T -
data/reference-ver2_742-worker64 15,5M 186G 2,73T -
data/reference-ver2_742-worker65 934K 186G 2,73T -
data/reference-ver2_742-worker66 935K 186G 2,73T -
data/reference-ver2_742-worker67 934K 186G 2,73T -
data/reference-ver2_742-worker68 935K 186G 2,73T -
data/reference-ver2_742-worker69 934K 186G 2,73T -
data/reference-ver2_742-worker70 935K 186G 2,73T -
data/reference-ver2_742-worker71 15,5M 186G 2,73T -
data/reference-ver2_742-worker72 935K 186G 2,73T -
data/reference-ver2_742-worker73 935K 186G 2,73T -
data/reference-ver2_742-worker75 934K 186G 2,73T -
data/reference-ver2_742-worker76 935K 186G 2,73T -
data/reference-ver2_742-worker77 934K 186G 2,73T -
data/reference-ver2_742-worker78 15,7M 186G 2,73T -
data/reference-ver2_742-worker79 15,4M 186G 2,73T -
data/reference-ver2_742-worker80 935K 186G 2,73T -
data/reference-ver7_214-worker07 2,19M 186G 2,73T -
data/reference-ver7_214-worker53 2,18M 186G 2,73T -
data/userdata 825G 186G 27,2K /data/userdata
data/userdata/userdata 27,2K 186G 27,2K
/data/userdata/userdata
data/userdata/worker01 8,25G 190G 3,77G -
data/userdata/worker02 8,25G 190G 3,48G -
data/userdata/worker03 8,25G 190G 4,07G -
data/userdata/worker04 8,25G 191G 2,99G -
data/userdata/worker05 8,25G 190G 3,46G -
data/userdata/worker06 8,25G 191G 2,89G -
data/userdata/worker07 8,25G 190G 3,65G -
data/userdata/worker08 8,25G 190G 3,55G -
data/userdata/worker09 8,25G 190G 3,64G -
data/userdata/worker10 8,25G 190G 3,85G -
data/userdata/worker11 8,25G 191G 3,28G -
data/userdata/worker12 8,25G 191G 3,19G -
data/userdata/worker13 8,25G 191G 2,93G -
data/userdata/worker14 8,25G 191G 2,89G -
data/userdata/worker15 8,25G 190G 3,57G -
data/userdata/worker153 8,25G 194G 12,8K -
data/userdata/worker154 8,25G 194G 12,8K -
data/userdata/worker155 8,25G 194G 12,8K -
data/userdata/worker156 8,25G 194G 12,8K -
data/userdata/worker157 8,25G 194G 12,8K -
data/userdata/worker158 8,25G 194G 12,8K -
data/userdata/worker159 8,25G 194G 12,8K -
data/userdata/worker16 8,25G 190G 4,23G -
data/userdata/worker160 8,25G 194G 12,8K -
data/userdata/worker161 8,25G 194G 12,8K -
data/userdata/worker162 8,25G 194G 12,8K -
data/userdata/worker163 8,25G 194G 12,8K -
data/userdata/worker164 8,25G 194G 12,8K -
data/userdata/worker165 8,25G 194G 12,8K -
data/userdata/worker166 8,25G 194G 12,8K -
data/userdata/worker167 8,25G 194G 12,8K -
data/userdata/worker168 8,25G 194G 12,8K -
data/userdata/worker169 8,25G 194G 12,8K -
data/userdata/worker17 8,25G 190G 3,63G -
data/userdata/worker170 8,25G 194G 12,8K -
data/userdata/worker171 8,25G 194G 12,8K -
data/userdata/worker172 8,25G 194G 12,8K -
data/userdata/worker18 8,25G 191G 3,19G -
data/userdata/worker19 8,25G 191G 3,02G -
data/userdata/worker20 8,25G 192G 2,35G -
data/userdata/worker21 8,25G 188G 5,42G -
data/userdata/worker22 8,25G 191G 2,76G -
data/userdata/worker23 8,25G 191G 2,61G -
data/userdata/worker24 8,25G 191G 3,22G -
data/userdata/worker25 8,25G 189G 4,97G -
data/userdata/worker26 8,25G 191G 3,23G -
data/userdata/worker27 8,25G 189G 4,90G -
data/userdata/worker28 8,25G 189G 5,31G -
data/userdata/worker29 8,25G 189G 4,67G -
data/userdata/worker30 8,25G 190G 3,46G -
data/userdata/worker31 8,25G 191G 3,36G -
data/userdata/worker32 8,25G 190G 4,11G -
data/userdata/worker33 8,25G 190G 4,31G -
data/userdata/worker34 8,25G 191G 3,31G -
data/userdata/worker35 8,25G 190G 4,23G -
data/userdata/worker36 8,25G 191G 3,24G -
data/userdata/worker37 8,25G 191G 3,16G -
data/userdata/worker38 8,25G 191G 2,89G -
data/userdata/worker39 8,25G 189G 4,49G -
data/userdata/worker40 8,25G 190G 4,21G -
data/userdata/worker41 8,25G 190G 3,42G -
data/userdata/worker42 8,25G 191G 3,32G -
data/userdata/worker43 8,25G 190G 3,71G -
data/userdata/worker44 8,25G 191G 3,33G -
data/userdata/worker45 8,25G 189G 5,04G -
data/userdata/worker46 8,25G 190G 3,76G -
data/userdata/worker47 8,25G 189G 5,13G -
data/userdata/worker48 8,25G 190G 3,81G -
data/userdata/worker49 8,25G 191G 2,72G -
data/userdata/worker50 8,25G 190G 3,78G -
data/userdata/worker51 8,25G 191G 3,08G -
data/userdata/worker52 8,25G 190G 3,38G -
data/userdata/worker53 8,25G 190G 4,10G -
data/userdata/worker54 8,25G 190G 3,48G -
data/userdata/worker55 8,25G 189G 4,39G -
data/userdata/worker56 8,25G 190G 3,58G -
data/userdata/worker57 8,25G 190G 3,97G -
data/userdata/worker58 8,25G 190G 4,02G -
data/userdata/worker59 8,25G 190G 3,57G -
data/userdata/worker60 8,25G 190G 4,04G -
data/userdata/worker61 8,25G 190G 3,73G -
data/userdata/worker62 8,25G 191G 3,21G -
data/userdata/worker63 8,25G 191G 3,37G -
data/userdata/worker64 8,25G 189G 4,77G -
data/userdata/worker65 8,25G 189G 5,34G -
data/userdata/worker66 8,25G 190G 3,50G -
data/userdata/worker67 8,25G 190G 3,84G -
data/userdata/worker68 8,25G 189G 4,73G -
data/userdata/worker69 8,25G 191G 3,20G -
data/userdata/worker70 8,25G 190G 3,96G -
data/userdata/worker71 8,25G 189G 5,11G -
data/userdata/worker72 8,25G 191G 2,92G -
data/userdata/worker73 8,25G 189G 4,53G -
data/userdata/worker74 8,25G 189G 4,39G -
data/userdata/worker75 8,25G 190G 4,26G -
data/userdata/worker76 8,25G 189G 4,67G -
data/userdata/worker77 8,25G 189G 5,30G -
data/userdata/worker78 8,25G 190G 3,98G -
data/userdata/worker79 8,25G 190G 4,10G -
data/userdata/worker80 8,25G 190G 3,48G -
This are getting really complicated now.
What I don't understand is:
- why the amount of free space changes from dataset to dataset ? I mean
they all share the same free space pool, all have the same
refreservation=none, but the AVAIL differs. When it comes to workerX
datasets, it differs slightly, but when it comes to the large zvols,
like esx/shared or reference, it differs a lot !
- why the esx/shared and reference datasets are shown like they can be
enlarged ? I mean, I really don't have THAT much of free space.
Here are their properties:
[emz@san01:~]> zfs get all data/esx/shared
NAME PROPERTY VALUE SOURCE
data/esx/shared type volume -
data/esx/shared creation вт февр. 21 10:19 2017 -
data/esx/shared used 5,02T -
data/esx/shared available 2,59T -
data/esx/shared referenced 2,61T -
data/esx/shared compressratio 1.81x -
data/esx/shared reservation none default
data/esx/shared volsize 5T local
data/esx/shared volblocksize 64K -
data/esx/shared checksum on default
data/esx/shared compression lzjb inherited
from data
data/esx/shared readonly off default
data/esx/shared copies 1 default
data/esx/shared refreservation 5,02T local
data/esx/shared primarycache all default
data/esx/shared secondarycache all default
data/esx/shared usedbysnapshots 0 -
data/esx/shared usedbydataset 2,61T -
data/esx/shared usedbychildren 0 -
data/esx/shared usedbyrefreservation 2,41T -
data/esx/shared logbias latency default
data/esx/shared dedup off default
data/esx/shared mlslabel -
data/esx/shared sync standard default
data/esx/shared refcompressratio 1.81x -
data/esx/shared written 2,61T -
data/esx/shared logicalused 4,71T -
data/esx/shared logicalreferenced 4,71T -
data/esx/shared volmode default default
data/esx/shared snapshot_limit none default
data/esx/shared snapshot_count none default
data/esx/shared redundant_metadata all default
[emz@san01:~]> zfs get all data/reference
NAME PROPERTY VALUE SOURCE
data/reference type volume -
data/reference creation вт февр. 21 11:12 2017 -
data/reference used 6,74T -
data/reference available 4,17T -
data/reference referenced 2,73T -
data/reference compressratio 1.08x -
data/reference reservation none default
data/reference volsize 3,97T local
data/reference volblocksize 64K -
data/reference checksum on default
data/reference compression lzjb local
data/reference readonly off default
data/reference copies 1 default
data/reference refreservation 3,98T local
data/reference primarycache all default
data/reference secondarycache all default
data/reference usedbysnapshots 21,6G -
data/reference usedbydataset 2,73T -
data/reference usedbychildren 0 -
data/reference usedbyrefreservation 3,98T -
data/reference logbias latency default
data/reference dedup off default
data/reference mlslabel -
data/reference sync standard default
data/reference refcompressratio 1.08x -
data/reference written 1,10M -
data/reference logicalused 2,98T -
data/reference logicalreferenced 2,95T -
data/reference volmode default default
data/reference snapshot_limit none default
data/reference snapshot_count none default
data/reference redundant_metadata all default
Could please someone explain why they show as having like half of the
total pool space as AVAIL ? I thing this is directly related to the fact
that zpool list shows only 44% of the total pool space is used. And I
use this value to monitor the pool space usage, looks like I'm totally
failing with this.
I also don't understand whe the zvol of the size 3.97T really uses 6.74T
of the space. I found an article, explaing that the volblocksize and the
sector size has to do something with this, and this happens when the
device block size is 4k, and volblocksize is default, thus 8k. Mine
disks sector size is 512 native, so this is really not the case. I'm
also having equal number of disks in vdevs, and they are 5:
[root@san01:~]# zpool status data
pool: data
state: ONLINE
scan: scrub repaired 111K in 40h42m with 0 errors on Wed Mar 1
03:34:56 2017
config:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
da0 ONLINE 0 0 0
da1 ONLINE 0 0 0
diskid/DISK-96JS100XTB4V ONLINE 0 0 0
diskid/DISK-96JS101TTB4V ONLINE 0 0 0
diskid/DISK-96JS100MTB4V ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
diskid/DISK-96JS1016TB4V ONLINE 0 0 0
diskid/DISK-96JS1002TB4V ONLINE 0 0 0
diskid/DISK-96LS1066TB4V ONLINE 0 0 0
diskid/DISK-96JS101JTB4V ONLINE 0 0 0
diskid/DISK-96LS106TTB4V ONLINE 0 0 0
errors: No known data errors
So I guess poor vdev alignment is also not the case. I used to resize
the zvol a couple of times, and I suspect this is the main reason,
because I've read in another article that this is probably the case when
"poor alignment" happens.
I will really appreciate if someone will explain this.
Thanks.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?71ef8400-94ec-1f59-3b2b-bb576ad65b89>