From owner-freebsd-stable@freebsd.org Wed Apr 12 18:01:12 2017 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 04C78D3B2B5; Wed, 12 Apr 2017 18:01:12 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [IPv6:2a00:7540:1::5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Vivat-Trade UNIX Root CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4529DB26; Wed, 12 Apr 2017 18:01:11 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [192.168.243.2] ([192.168.243.2]) (authenticated bits=0) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTPSA id v3CI15bD009360 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Wed, 12 Apr 2017 23:01:06 +0500 (YEKT) (envelope-from emz@norma.perm.ru) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=norma.perm.ru; s=key; t=1492020066; bh=W43TYZL3QHw6Gam1ePqA9lwtlPS106SGe3hjGS6htkk=; h=To:From:Subject:Date; b=cR1v64/5F+mQLyJaTjoThqRJlISXGcsZQe0Y8Er9tk2DKSftnlhoiw8o63dUDhiz6 MN3iMvpLveI1hsQ7LarbfFr12LYQOmeYzA8cR2c9bJXLjKZ7zSfQx8NA1JYWxZ4rgP fBP0bd8wQwKM2NhtP02bFSQnGDjVr7mm1F8uA4vM= To: freebsd-stable@FreeBSD.org, FreeBSD FS From: "Eugene M. Zheganin" Subject: zpool list show nonsense on raidz pools, at least it looks like it for me Message-ID: <71ef8400-94ec-1f59-3b2b-bb576ad65b89@norma.perm.ru> Date: Wed, 12 Apr 2017 23:01:10 +0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Apr 2017 18:01:12 -0000 Hi, It's not my first letter where I fail to understand the space usage from zfs utilities, and in previous ones I was kind of convinced that I just read it wrong, but not this time I guess. See for yourself: [emz@san01:~]> zpool list data NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT data 17,4T 7,72T 9,66T - 46% 44% 1.00x ONLINE - Here' as I understand it, zpool says that less than a half of the pool is used. As far as I know this is very complicated when it comes to the radiz pools. Let's see: [emz@san01:~]> zfs list -t all data NAME USED AVAIL REFER MOUNTPOINT data 13,3T 186G 27,2K /data So, if we won't investigate further, it looks like that only 186G is free. Spoiling - this is the real free space amount, because I've just managed to free 160 gigs of data, and I really know I was short on space when sending 30 Gb dataset, because zfs was saying "Not enough free space". So, let's investigate further: [emz@san01:~]> zfs list -t all | more NAME USED AVAIL REFER MOUNTPOINT data 13,3T 186G 27,2K /data data/esx 5,23T 186G 27,2K /data/esx data/esx/boot-esx01 8,25G 193G 561M - data/esx/boot-esx02 8,25G 193G 561M - data/esx/boot-esx03 8,25G 193G 561M - data/esx/boot-esx04 8,25G 193G 561M - data/esx/boot-esx05 8,25G 193G 561M - data/esx/boot-esx06 8,25G 193G 561M - data/esx/boot-esx07 8,25G 193G 962M - data/esx/boot-esx08 8,25G 193G 562M - data/esx/boot-esx09 8,25G 193G 562M - data/esx/boot-esx10 8,25G 193G 595M - data/esx/boot-esx11 8,25G 193G 539M - data/esx/boot-esx12 8,25G 193G 539M - data/esx/boot-esx13 8,25G 193G 539M - data/esx/boot-esx14 8,25G 193G 539M - data/esx/boot-esx15 8,25G 193G 539M - data/esx/boot-esx16 8,25G 193G 541M - data/esx/boot-esx17 8,25G 193G 540M - data/esx/boot-esx18 8,25G 193G 539M - data/esx/boot-esx19 8,25G 193G 542M - data/esx/boot-esx20 8,25G 194G 12,8K - data/esx/boot-esx21 8,25G 194G 12,8K - data/esx/boot-esx22 8,25G 193G 913M - data/esx/boot-esx23 8,25G 193G 558M - data/esx/boot-esx24 8,25G 194G 12,8K - data/esx/boot-esx25 8,25G 194G 12,8K - data/esx/boot-esx26 8,25G 194G 12,8K - data/esx/shared 5,02T 2,59T 2,61T - data/reference 6,74T 4,17T 2,73T - data/reference@ver7_214 127M - 2,73T - data/reference@ver2_739 12,8M - 2,73T - data/reference@ver2_740 5,80M - 2,73T - data/reference@ver2_741 4,55M - 2,73T - data/reference@ver2_742 993K - 2,73T - data/reference-ver2_739-worker100 1,64G 186G 2,73T - data/reference-ver2_739-worker101 254M 186G 2,73T - data/reference-ver2_739-worker102 566K 186G 2,73T - data/reference-ver2_739-worker103 260M 186G 2,73T - data/reference-ver2_739-worker104 8,74G 186G 2,73T - data/reference-ver2_739-worker105 4,19G 186G 2,73T - data/reference-ver2_739-worker106 1,72G 186G 2,73T - data/reference-ver2_739-worker107 282M 186G 2,73T - data/reference-ver2_739-worker108 1,27M 186G 2,73T - data/reference-ver2_739-worker109 8,74G 186G 2,73T - data/reference-ver2_739-worker110 8,74G 186G 2,73T - data/reference-ver2_739-worker111 8,74G 186G 2,73T - data/reference-ver2_739-worker112 8,74G 186G 2,73T - data/reference-ver2_739-worker113 838K 186G 2,73T - data/reference-ver2_739-worker114 8,74G 186G 2,73T - data/reference-ver2_739-worker115 8,74G 186G 2,73T - data/reference-ver2_739-worker116 8,74G 186G 2,73T - data/reference-ver2_739-worker117 8,74G 186G 2,73T - data/reference-ver2_739-worker118 8,74G 186G 2,73T - data/reference-ver2_739-worker119 8,74G 186G 2,73T - data/reference-ver2_739-worker120 8,74G 186G 2,73T - data/reference-ver2_739-worker121 8,74G 186G 2,73T - data/reference-ver2_739-worker122 8,74G 186G 2,73T - data/reference-ver2_739-worker123 8,74G 186G 2,73T - data/reference-ver2_739-worker124 8,74G 186G 2,73T - data/reference-ver2_739-worker125 8,74G 186G 2,73T - data/reference-ver2_739-worker126 8,74G 186G 2,73T - data/reference-ver2_739-worker127 8,74G 186G 2,73T - data/reference-ver2_739-worker128 8,74G 186G 2,73T - data/reference-ver2_739-worker129 8,74G 186G 2,73T - data/reference-ver2_739-worker130 8,74G 186G 2,73T - data/reference-ver2_739-worker131 8,74G 186G 2,73T - data/reference-ver2_739-worker132 8,74G 186G 2,73T - data/reference-ver2_739-worker133 8,74G 186G 2,73T - data/reference-ver2_739-worker134 8,74G 186G 2,73T - data/reference-ver2_739-worker135 8,74G 186G 2,73T - data/reference-ver2_739-worker136 8,74G 186G 2,73T - data/reference-ver2_739-worker137 8,74G 186G 2,73T - data/reference-ver2_739-worker138 8,74G 186G 2,73T - data/reference-ver2_739-worker139 8,74G 186G 2,73T - data/reference-ver2_739-worker140 8,74G 186G 2,73T - data/reference-ver2_739-worker141 8,74G 186G 2,73T - data/reference-ver2_739-worker142 8,74G 186G 2,73T - data/reference-ver2_739-worker143 8,73G 186G 2,73T - data/reference-ver2_739-worker144 9,16G 186G 2,73T - data/reference-ver2_739-worker145 8,74G 186G 2,73T - data/reference-ver2_739-worker146 8,74G 186G 2,73T - data/reference-ver2_739-worker147 8,74G 186G 2,73T - data/reference-ver2_739-worker148 8,74G 186G 2,73T - data/reference-ver2_739-worker149 8,74G 186G 2,73T - data/reference-ver2_739-worker150 8,74G 186G 2,73T - data/reference-ver2_739-worker151 8,74G 186G 2,73T - data/reference-ver2_739-worker152 8,74G 186G 2,73T - data/reference-ver2_739-worker53 385K 186G 2,73T - data/reference-ver2_739-worker58 9,49M 186G 2,73T - data/reference-ver2_739-worker81 8,20G 186G 2,73T - data/reference-ver2_739-worker82 7,64G 186G 2,73T - data/reference-ver2_739-worker83 6,84G 186G 2,73T - data/reference-ver2_739-worker84 5,95G 186G 2,73T - data/reference-ver2_739-worker85 5,09G 186G 2,73T - data/reference-ver2_739-worker86 3,76G 186G 2,73T - data/reference-ver2_739-worker87 4,06G 186G 2,73T - data/reference-ver2_739-worker88 1,92G 186G 2,73T - data/reference-ver2_739-worker89 6,83G 186G 2,73T - data/reference-ver2_739-worker90 6,55G 186G 2,73T - data/reference-ver2_739-worker91 5,25G 186G 2,73T - data/reference-ver2_739-worker92 4,18G 186G 2,73T - data/reference-ver2_739-worker93 3,18G 186G 2,73T - data/reference-ver2_739-worker94 983M 186G 2,73T - data/reference-ver2_739-worker95 525M 186G 2,73T - data/reference-ver2_739-worker96 650M 186G 2,73T - data/reference-ver2_739-worker97 5,78G 186G 2,73T - data/reference-ver2_739-worker98 4,81G 186G 2,73T - data/reference-ver2_739-worker99 4,93G 186G 2,73T - data/reference-ver2_740-worker56 361K 186G 2,73T - data/reference-ver2_741-worker74 3,98M 186G 2,73T - data/reference-ver2_742-worker02 934K 186G 2,73T - data/reference-ver2_742-worker03 935K 186G 2,73T - data/reference-ver2_742-worker04 935K 186G 2,73T - data/reference-ver2_742-worker05 935K 186G 2,73T - data/reference-ver2_742-worker06 935K 186G 2,73T - data/reference-ver2_742-worker08 934K 186G 2,73T - data/reference-ver2_742-worker09 15,9M 186G 2,73T - data/reference-ver2_742-worker10 16,2M 186G 2,73T - data/reference-ver2_742-worker11 15,4M 186G 2,73T - data/reference-ver2_742-worker12 935K 186G 2,73T - data/reference-ver2_742-worker13 15,9M 186G 2,73T - data/reference-ver2_742-worker14 935K 186G 2,73T - data/reference-ver2_742-worker15 15,5M 186G 2,73T - data/reference-ver2_742-worker16 934K 186G 2,73T - data/reference-ver2_742-worker17 934K 186G 2,73T - data/reference-ver2_742-worker18 935K 186G 2,73T - data/reference-ver2_742-worker19 531M 186G 2,73T - data/reference-ver2_742-worker20 935K 186G 2,73T - data/reference-ver2_742-worker21 935K 186G 2,73T - data/reference-ver2_742-worker22 5,27G 186G 2,73T - data/reference-ver2_742-worker23 934K 186G 2,73T - data/reference-ver2_742-worker24 5,27G 186G 2,73T - data/reference-ver2_742-worker25 934K 186G 2,73T - data/reference-ver2_742-worker26 934K 186G 2,73T - data/reference-ver2_742-worker27 15,8M 186G 2,73T - data/reference-ver2_742-worker28 935K 186G 2,73T - data/reference-ver2_742-worker29 934K 186G 2,73T - data/reference-ver2_742-worker30 934K 186G 2,73T - data/reference-ver2_742-worker31 934K 186G 2,73T - data/reference-ver2_742-worker32 444M 186G 2,73T - data/reference-ver2_742-worker33 935K 186G 2,73T - data/reference-ver2_742-worker34 15,9M 186G 2,73T - data/reference-ver2_742-worker35 1,12M 186G 2,73T - data/reference-ver2_742-worker36 935K 186G 2,73T - data/reference-ver2_742-worker37 5,27G 186G 2,73T - data/reference-ver2_742-worker38 935K 186G 2,73T - data/reference-ver2_742-worker39 935K 186G 2,73T - data/reference-ver2_742-worker40 935K 186G 2,73T - data/reference-ver2_742-worker41 15,5M 186G 2,73T - data/reference-ver2_742-worker42 934K 186G 2,73T - data/reference-ver2_742-worker43 15,4M 186G 2,73T - data/reference-ver2_742-worker44 429M 186G 2,73T - data/reference-ver2_742-worker45 15,5M 186G 2,73T - data/reference-ver2_742-worker46 935K 186G 2,73T - data/reference-ver2_742-worker47 934K 186G 2,73T - data/reference-ver2_742-worker48 15,5M 186G 2,73T - data/reference-ver2_742-worker49 935K 186G 2,73T - data/reference-ver2_742-worker50 934K 186G 2,73T - data/reference-ver2_742-worker51 935K 186G 2,73T - data/reference-ver2_742-worker52 935K 186G 2,73T - data/reference-ver2_742-worker54 935K 186G 2,73T - data/reference-ver2_742-worker55 934K 186G 2,73T - data/reference-ver2_742-worker57 935K 186G 2,73T - data/reference-ver2_742-worker59 934K 186G 2,73T - data/reference-ver2_742-worker60 15,5M 186G 2,73T - data/reference-ver2_742-worker61 934K 186G 2,73T - data/reference-ver2_742-worker62 934K 186G 2,73T - data/reference-ver2_742-worker63 934K 186G 2,73T - data/reference-ver2_742-worker64 15,5M 186G 2,73T - data/reference-ver2_742-worker65 934K 186G 2,73T - data/reference-ver2_742-worker66 935K 186G 2,73T - data/reference-ver2_742-worker67 934K 186G 2,73T - data/reference-ver2_742-worker68 935K 186G 2,73T - data/reference-ver2_742-worker69 934K 186G 2,73T - data/reference-ver2_742-worker70 935K 186G 2,73T - data/reference-ver2_742-worker71 15,5M 186G 2,73T - data/reference-ver2_742-worker72 935K 186G 2,73T - data/reference-ver2_742-worker73 935K 186G 2,73T - data/reference-ver2_742-worker75 934K 186G 2,73T - data/reference-ver2_742-worker76 935K 186G 2,73T - data/reference-ver2_742-worker77 934K 186G 2,73T - data/reference-ver2_742-worker78 15,7M 186G 2,73T - data/reference-ver2_742-worker79 15,4M 186G 2,73T - data/reference-ver2_742-worker80 935K 186G 2,73T - data/reference-ver7_214-worker07 2,19M 186G 2,73T - data/reference-ver7_214-worker53 2,18M 186G 2,73T - data/userdata 825G 186G 27,2K /data/userdata data/userdata/userdata 27,2K 186G 27,2K /data/userdata/userdata data/userdata/worker01 8,25G 190G 3,77G - data/userdata/worker02 8,25G 190G 3,48G - data/userdata/worker03 8,25G 190G 4,07G - data/userdata/worker04 8,25G 191G 2,99G - data/userdata/worker05 8,25G 190G 3,46G - data/userdata/worker06 8,25G 191G 2,89G - data/userdata/worker07 8,25G 190G 3,65G - data/userdata/worker08 8,25G 190G 3,55G - data/userdata/worker09 8,25G 190G 3,64G - data/userdata/worker10 8,25G 190G 3,85G - data/userdata/worker11 8,25G 191G 3,28G - data/userdata/worker12 8,25G 191G 3,19G - data/userdata/worker13 8,25G 191G 2,93G - data/userdata/worker14 8,25G 191G 2,89G - data/userdata/worker15 8,25G 190G 3,57G - data/userdata/worker153 8,25G 194G 12,8K - data/userdata/worker154 8,25G 194G 12,8K - data/userdata/worker155 8,25G 194G 12,8K - data/userdata/worker156 8,25G 194G 12,8K - data/userdata/worker157 8,25G 194G 12,8K - data/userdata/worker158 8,25G 194G 12,8K - data/userdata/worker159 8,25G 194G 12,8K - data/userdata/worker16 8,25G 190G 4,23G - data/userdata/worker160 8,25G 194G 12,8K - data/userdata/worker161 8,25G 194G 12,8K - data/userdata/worker162 8,25G 194G 12,8K - data/userdata/worker163 8,25G 194G 12,8K - data/userdata/worker164 8,25G 194G 12,8K - data/userdata/worker165 8,25G 194G 12,8K - data/userdata/worker166 8,25G 194G 12,8K - data/userdata/worker167 8,25G 194G 12,8K - data/userdata/worker168 8,25G 194G 12,8K - data/userdata/worker169 8,25G 194G 12,8K - data/userdata/worker17 8,25G 190G 3,63G - data/userdata/worker170 8,25G 194G 12,8K - data/userdata/worker171 8,25G 194G 12,8K - data/userdata/worker172 8,25G 194G 12,8K - data/userdata/worker18 8,25G 191G 3,19G - data/userdata/worker19 8,25G 191G 3,02G - data/userdata/worker20 8,25G 192G 2,35G - data/userdata/worker21 8,25G 188G 5,42G - data/userdata/worker22 8,25G 191G 2,76G - data/userdata/worker23 8,25G 191G 2,61G - data/userdata/worker24 8,25G 191G 3,22G - data/userdata/worker25 8,25G 189G 4,97G - data/userdata/worker26 8,25G 191G 3,23G - data/userdata/worker27 8,25G 189G 4,90G - data/userdata/worker28 8,25G 189G 5,31G - data/userdata/worker29 8,25G 189G 4,67G - data/userdata/worker30 8,25G 190G 3,46G - data/userdata/worker31 8,25G 191G 3,36G - data/userdata/worker32 8,25G 190G 4,11G - data/userdata/worker33 8,25G 190G 4,31G - data/userdata/worker34 8,25G 191G 3,31G - data/userdata/worker35 8,25G 190G 4,23G - data/userdata/worker36 8,25G 191G 3,24G - data/userdata/worker37 8,25G 191G 3,16G - data/userdata/worker38 8,25G 191G 2,89G - data/userdata/worker39 8,25G 189G 4,49G - data/userdata/worker40 8,25G 190G 4,21G - data/userdata/worker41 8,25G 190G 3,42G - data/userdata/worker42 8,25G 191G 3,32G - data/userdata/worker43 8,25G 190G 3,71G - data/userdata/worker44 8,25G 191G 3,33G - data/userdata/worker45 8,25G 189G 5,04G - data/userdata/worker46 8,25G 190G 3,76G - data/userdata/worker47 8,25G 189G 5,13G - data/userdata/worker48 8,25G 190G 3,81G - data/userdata/worker49 8,25G 191G 2,72G - data/userdata/worker50 8,25G 190G 3,78G - data/userdata/worker51 8,25G 191G 3,08G - data/userdata/worker52 8,25G 190G 3,38G - data/userdata/worker53 8,25G 190G 4,10G - data/userdata/worker54 8,25G 190G 3,48G - data/userdata/worker55 8,25G 189G 4,39G - data/userdata/worker56 8,25G 190G 3,58G - data/userdata/worker57 8,25G 190G 3,97G - data/userdata/worker58 8,25G 190G 4,02G - data/userdata/worker59 8,25G 190G 3,57G - data/userdata/worker60 8,25G 190G 4,04G - data/userdata/worker61 8,25G 190G 3,73G - data/userdata/worker62 8,25G 191G 3,21G - data/userdata/worker63 8,25G 191G 3,37G - data/userdata/worker64 8,25G 189G 4,77G - data/userdata/worker65 8,25G 189G 5,34G - data/userdata/worker66 8,25G 190G 3,50G - data/userdata/worker67 8,25G 190G 3,84G - data/userdata/worker68 8,25G 189G 4,73G - data/userdata/worker69 8,25G 191G 3,20G - data/userdata/worker70 8,25G 190G 3,96G - data/userdata/worker71 8,25G 189G 5,11G - data/userdata/worker72 8,25G 191G 2,92G - data/userdata/worker73 8,25G 189G 4,53G - data/userdata/worker74 8,25G 189G 4,39G - data/userdata/worker75 8,25G 190G 4,26G - data/userdata/worker76 8,25G 189G 4,67G - data/userdata/worker77 8,25G 189G 5,30G - data/userdata/worker78 8,25G 190G 3,98G - data/userdata/worker79 8,25G 190G 4,10G - data/userdata/worker80 8,25G 190G 3,48G - This are getting really complicated now. What I don't understand is: - why the amount of free space changes from dataset to dataset ? I mean they all share the same free space pool, all have the same refreservation=none, but the AVAIL differs. When it comes to workerX datasets, it differs slightly, but when it comes to the large zvols, like esx/shared or reference, it differs a lot ! - why the esx/shared and reference datasets are shown like they can be enlarged ? I mean, I really don't have THAT much of free space. Here are their properties: [emz@san01:~]> zfs get all data/esx/shared NAME PROPERTY VALUE SOURCE data/esx/shared type volume - data/esx/shared creation вт февр. 21 10:19 2017 - data/esx/shared used 5,02T - data/esx/shared available 2,59T - data/esx/shared referenced 2,61T - data/esx/shared compressratio 1.81x - data/esx/shared reservation none default data/esx/shared volsize 5T local data/esx/shared volblocksize 64K - data/esx/shared checksum on default data/esx/shared compression lzjb inherited from data data/esx/shared readonly off default data/esx/shared copies 1 default data/esx/shared refreservation 5,02T local data/esx/shared primarycache all default data/esx/shared secondarycache all default data/esx/shared usedbysnapshots 0 - data/esx/shared usedbydataset 2,61T - data/esx/shared usedbychildren 0 - data/esx/shared usedbyrefreservation 2,41T - data/esx/shared logbias latency default data/esx/shared dedup off default data/esx/shared mlslabel - data/esx/shared sync standard default data/esx/shared refcompressratio 1.81x - data/esx/shared written 2,61T - data/esx/shared logicalused 4,71T - data/esx/shared logicalreferenced 4,71T - data/esx/shared volmode default default data/esx/shared snapshot_limit none default data/esx/shared snapshot_count none default data/esx/shared redundant_metadata all default [emz@san01:~]> zfs get all data/reference NAME PROPERTY VALUE SOURCE data/reference type volume - data/reference creation вт февр. 21 11:12 2017 - data/reference used 6,74T - data/reference available 4,17T - data/reference referenced 2,73T - data/reference compressratio 1.08x - data/reference reservation none default data/reference volsize 3,97T local data/reference volblocksize 64K - data/reference checksum on default data/reference compression lzjb local data/reference readonly off default data/reference copies 1 default data/reference refreservation 3,98T local data/reference primarycache all default data/reference secondarycache all default data/reference usedbysnapshots 21,6G - data/reference usedbydataset 2,73T - data/reference usedbychildren 0 - data/reference usedbyrefreservation 3,98T - data/reference logbias latency default data/reference dedup off default data/reference mlslabel - data/reference sync standard default data/reference refcompressratio 1.08x - data/reference written 1,10M - data/reference logicalused 2,98T - data/reference logicalreferenced 2,95T - data/reference volmode default default data/reference snapshot_limit none default data/reference snapshot_count none default data/reference redundant_metadata all default Could please someone explain why they show as having like half of the total pool space as AVAIL ? I thing this is directly related to the fact that zpool list shows only 44% of the total pool space is used. And I use this value to monitor the pool space usage, looks like I'm totally failing with this. I also don't understand whe the zvol of the size 3.97T really uses 6.74T of the space. I found an article, explaing that the volblocksize and the sector size has to do something with this, and this happens when the device block size is 4k, and volblocksize is default, thus 8k. Mine disks sector size is 512 native, so this is really not the case. I'm also having equal number of disks in vdevs, and they are 5: [root@san01:~]# zpool status data pool: data state: ONLINE scan: scrub repaired 111K in 40h42m with 0 errors on Wed Mar 1 03:34:56 2017 config: NAME STATE READ WRITE CKSUM data ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 diskid/DISK-96JS100XTB4V ONLINE 0 0 0 diskid/DISK-96JS101TTB4V ONLINE 0 0 0 diskid/DISK-96JS100MTB4V ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 diskid/DISK-96JS1016TB4V ONLINE 0 0 0 diskid/DISK-96JS1002TB4V ONLINE 0 0 0 diskid/DISK-96LS1066TB4V ONLINE 0 0 0 diskid/DISK-96JS101JTB4V ONLINE 0 0 0 diskid/DISK-96LS106TTB4V ONLINE 0 0 0 errors: No known data errors So I guess poor vdev alignment is also not the case. I used to resize the zvol a couple of times, and I suspect this is the main reason, because I've read in another article that this is probably the case when "poor alignment" happens. I will really appreciate if someone will explain this. Thanks.