From owner-freebsd-fs@FreeBSD.ORG Thu Nov 1 00:19:45 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E6C67FEA; Thu, 1 Nov 2012 00:19:45 +0000 (UTC) (envelope-from prvs=1652892d21=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 805E88FC08; Thu, 1 Nov 2012 00:19:43 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50000903417.msg; Thu, 01 Nov 2012 00:19:42 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Thu, 01 Nov 2012 00:19:42 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1652892d21=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: From: "Steven Hartland" To: "Peter Jeremy" References: <27087376D1C14132A3CC1B4016912F6D@multiplay.co.uk> <20121031212346.GL3309@server.rulingia.com> Subject: Re: ZFS corruption due to lack of space? Date: Thu, 1 Nov 2012 00:19:38 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 01 Nov 2012 00:19:46 -0000 ----- Original Message ----- From: "Steven Hartland" To: "Peter Jeremy" Cc: ; Sent: Thursday, November 01, 2012 12:09 AM Subject: Re: ZFS corruption due to lack of space? > On 2012-Oct-31 17:25:09 -0000, Steven Hartland wrote: >>>Been running some tests on new hardware here to verify all >>>is good. One of the tests was to fill the zfs array which >>>seems like its totally corrupted the tank. >> >>I've accidently "filled" a pool, and had multiple processes try to >>write to the full pool, without either emptying the free space reserve >>(so I could still delete the offending files) or corrupting the pool. > > Same here but its the first time I've had ZIL in place at the time so > wondering if that may be playing a factor. > >> Had you tried to read/write the raw disks before you tried the >> ZFS testing? > > Yes, didn't see any issues but then it wasn't checksuming so tbh I > wouldn't have noticed if it was silently corrupting data. > >>Do you have compression and/or dedupe enabled on the pool? > > Nope bog standard raidz2 no additional settings > >>>1. Given the information it seems like the multiple writes filling >>>the disk may have caused metadata corruption? >> >> I don't recall seeing this reported before. > > Nore me and we've been using ZFS for years, but never filled a pool > with such known simultanious access + ZIL before > >>>2. Is there anyway to stop the scrub? >> >>Other than freeing up some space, I don't think so. If this is a test >>pool that you don't need, you could try destroying it and re-creating >>it - that may be quicker and easier than recovering the existing pool. > > Artems trick of cat /dev/null > /tank2/ worked and I've now > managed to stop the scrub :) > >>>3. Surely low space should never prevent stopping a scrub? >> >> As Artem noted, ZFS is a copy-on-write filesystem. It is supposed to >> reserve some free space to allow metadata updates (stop scrubs, delete >> files, etc) even when it is "full" but I have seen reports of this not >> working correctly in the past. A truncate-in-place may work. > > Yes it did thanks, but as you said if this metadata update was failing > due to out of space lends credability to the fact that the same lack of > space and hence failure to update metadata could have also caused the > corruption in the first place. > > Its interesting to note that the zpool is reporting pleanty of free space > even when the root zfs volume was showing 0, so you would expect there > to be pleanty of space for it be able to stop the scrub but it appears > not which is definitely interesting and could point to the underlying > cause? > > zpool list tank2 > NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT > tank2 19T 18.7T 304G 98% 1.00x ONLINE - > > zfs list tank2 > NAME USED AVAIL REFER MOUNTPOINT > tank2 13.3T 0 13.3T /tank2 > > Current state is:- > scan: scrub in progress since Wed Oct 31 16:13:53 2012 > 1.64T scanned out of 18.7T at 62.8M/s, 79h12m to go > 280M repaired, 8.76% done > > Something else that was interesting is while the scrub was running > devd was using a good amount of CPU 40% of a 3.3Ghz core, which I've > never seen before. Any ideas why its usage would be so high? In case its useful here's the output from a zdb tank2 so far:- zdb tank2 Cached configuration: version: 28 name: 'tank2' state: 0 txg: 39502 pool_guid: 15779146362913479443 hostid: 1751781486 vdev_children: 3 vdev_tree: type: 'root' id: 0 guid: 15779146362913479443 create_txg: 4 children[0]: type: 'raidz' id: 0 guid: 8518972900227438019 nparity: 2 metaslab_array: 33 metaslab_shift: 37 ashift: 9 asize: 21004116295680 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 15577380450172137060 path: '/dev/mfisyspd0' phys_path: '/dev/mfisyspd0' whole_disk: 1 DTL: 236 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 16940350228793267704 path: '/dev/mfisyspd1' phys_path: '/dev/mfisyspd1' whole_disk: 1 DTL: 235 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 9264743178245473794 path: '/dev/mfisyspd2' phys_path: '/dev/mfisyspd2' whole_disk: 1 DTL: 234 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 432716341673487166 path: '/dev/mfisyspd3' phys_path: '/dev/mfisyspd3' whole_disk: 1 DTL: 233 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 18217760646550913544 path: '/dev/mfisyspd4' phys_path: '/dev/mfisyspd4' whole_disk: 1 DTL: 232 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 6964614355298004256 path: '/dev/mfisyspd5' phys_path: '/dev/mfisyspd5' whole_disk: 1 DTL: 231 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 4397961270160308034 path: '/dev/mfisyspd6' phys_path: '/dev/mfisyspd6' whole_disk: 1 DTL: 230 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 13757670211012452260 path: '/dev/mfisyspd7p3' phys_path: '/dev/mfisyspd7p3' whole_disk: 1 metaslab_array: 32 metaslab_shift: 27 ashift: 9 asize: 14125891584 is_log: 1 DTL: 237 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 7315839509249482920 path: '/dev/mfisyspd8p3' phys_path: '/dev/mfisyspd8p3' whole_disk: 1 metaslab_array: 31 metaslab_shift: 27 ashift: 9 asize: 14125891584 is_log: 1 DTL: 229 create_txg: 4 MOS Configuration: version: 28 name: 'tank2' state: 0 txg: 39502 pool_guid: 15779146362913479443 hostid: 1751781486 vdev_children: 3 vdev_tree: type: 'root' id: 0 guid: 15779146362913479443 create_txg: 4 children[0]: type: 'raidz' id: 0 guid: 8518972900227438019 nparity: 2 metaslab_array: 33 metaslab_shift: 37 ashift: 9 asize: 21004116295680 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 15577380450172137060 path: '/dev/mfisyspd0' phys_path: '/dev/mfisyspd0' whole_disk: 1 DTL: 236 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 16940350228793267704 path: '/dev/mfisyspd1' phys_path: '/dev/mfisyspd1' whole_disk: 1 DTL: 235 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 9264743178245473794 path: '/dev/mfisyspd2' phys_path: '/dev/mfisyspd2' whole_disk: 1 DTL: 234 create_txg: 4 children[3]: type: 'disk' id: 3 guid: 432716341673487166 path: '/dev/mfisyspd3' phys_path: '/dev/mfisyspd3' whole_disk: 1 DTL: 233 create_txg: 4 children[4]: type: 'disk' id: 4 guid: 18217760646550913544 path: '/dev/mfisyspd4' phys_path: '/dev/mfisyspd4' whole_disk: 1 DTL: 232 create_txg: 4 children[5]: type: 'disk' id: 5 guid: 6964614355298004256 path: '/dev/mfisyspd5' phys_path: '/dev/mfisyspd5' whole_disk: 1 DTL: 231 create_txg: 4 children[6]: type: 'disk' id: 6 guid: 4397961270160308034 path: '/dev/mfisyspd6' phys_path: '/dev/mfisyspd6' whole_disk: 1 DTL: 230 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 13757670211012452260 path: '/dev/mfisyspd7p3' phys_path: '/dev/mfisyspd7p3' whole_disk: 1 metaslab_array: 32 metaslab_shift: 27 ashift: 9 asize: 14125891584 is_log: 1 DTL: 237 create_txg: 4 children[2]: type: 'disk' id: 2 guid: 7315839509249482920 path: '/dev/mfisyspd8p3' phys_path: '/dev/mfisyspd8p3' whole_disk: 1 metaslab_array: 31 metaslab_shift: 27 ashift: 9 asize: 14125891584 is_log: 1 DTL: 229 create_txg: 4 Uberblock: magic = 0000000000bab10c version = 28 txg = 39547 guid_sum = 6486691012039134504 timestamp = 1351727718 UTC = Wed Oct 31 23:55:18 2012 All DDTs are empty Metaslabs: vdev 0 metaslabs 152 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 0 offset 0 spacemap 36 free 1.41G segments 4170 maxsize 1.38G freepct 1% metaslab 1 offset 2000000000 spacemap 65 free 1.45G segments 4508 maxsize 1.40G freepct 1% metaslab 2 offset 4000000000 spacemap 70 free 1.44G segments 5723 maxsize 1.39G freepct 1% metaslab 3 offset 6000000000 spacemap 77 free 1.41G segments 6635 maxsize 1.35G freepct 1% metaslab 4 offset 8000000000 spacemap 80 free 1.46G segments 6408 maxsize 1.40G freepct 1% metaslab 5 offset a000000000 spacemap 81 free 1.47G segments 6480 maxsize 1.42G freepct 1% metaslab 6 offset c000000000 spacemap 82 free 1.48G segments 6684 maxsize 1.42G freepct 1% metaslab 7 offset e000000000 spacemap 83 free 1.48G segments 6873 maxsize 1.42G freepct 1% metaslab 8 offset 10000000000 spacemap 84 free 1.48G segments 7298 maxsize 1.42G freepct 1% metaslab 9 offset 12000000000 spacemap 85 free 1.43G segments 6699 maxsize 1.37G freepct 1% metaslab 10 offset 14000000000 spacemap 88 free 1.50G segments 6781 maxsize 1.44G freepct 1% metaslab 11 offset 16000000000 spacemap 89 free 1.50G segments 6434 maxsize 1.44G freepct 1% metaslab 12 offset 18000000000 spacemap 90 free 1.51G segments 7188 maxsize 1.44G freepct 1% metaslab 13 offset 1a000000000 spacemap 91 free 1.49G segments 6712 maxsize 1.42G freepct 1% metaslab 14 offset 1c000000000 spacemap 92 free 1.52G segments 6810 maxsize 1.46G freepct 1% metaslab 15 offset 1e000000000 spacemap 93 free 1.52G segments 8306 maxsize 1.41G freepct 1% metaslab 16 offset 20000000000 spacemap 94 free 1.49G segments 21881 maxsize 660M freepct 1% metaslab 17 offset 22000000000 spacemap 97 free 1.50G segments 9590 maxsize 1.32G freepct 1% metaslab 18 offset 24000000000 spacemap 98 free 1.53G segments 6921 maxsize 1.47G freepct 1% metaslab 19 offset 26000000000 spacemap 99 free 1.53G segments 7982 maxsize 1.46G freepct 1% metaslab 20 offset 28000000000 spacemap 100 free 1.55G segments 7943 maxsize 1.48G freepct 1% metaslab 21 offset 2a000000000 spacemap 101 free 1.54G segments 8049 maxsize 1.47G freepct 1% metaslab 22 offset 2c000000000 spacemap 102 free 1.54G segments 8205 maxsize 1.46G freepct 1% metaslab 23 offset 2e000000000 spacemap 103 free 1.53G segments 11339 maxsize 1.37G freepct 1% metaslab 24 offset 30000000000 spacemap 104 free 1.55G segments 11536 maxsize 1.38G freepct 1% metaslab 25 offset 32000000000 spacemap 105 free 1.58G segments 7281 maxsize 1.50G freepct 1% metaslab 26 offset 34000000000 spacemap 108 free 1.57G segments 7917 maxsize 1.49G freepct 1% metaslab 27 offset 36000000000 spacemap 109 free 1.59G segments 8446 maxsize 1.51G freepct 1% metaslab 28 offset 38000000000 spacemap 110 free 1.59G segments 8437 maxsize 1.51G freepct 1% metaslab 29 offset 3a000000000 spacemap 35 free 1.60G segments 22991 maxsize 1.21G freepct 1% metaslab 30 offset 3c000000000 spacemap 111 free 1.57G segments 8358 maxsize 1.49G freepct 1% metaslab 31 offset 3e000000000 spacemap 112 free 1.62G segments 7724 maxsize 1.53G freepct 1% metaslab 32 offset 40000000000 spacemap 113 free 1.62G segments 8314 maxsize 1.53G freepct 1% metaslab 33 offset 42000000000 spacemap 114 free 1.61G segments 8294 maxsize 1.52G freepct 1% metaslab 34 offset 44000000000 spacemap 115 free 1.63G segments 8422 maxsize 1.54G freepct 1% metaslab 35 offset 46000000000 spacemap 116 free 1.60G segments 8125 maxsize 1.51G freepct 1% metaslab 36 offset 48000000000 spacemap 117 free 1.62G segments 8048 maxsize 1.53G freepct 1% metaslab 37 offset 4a000000000 spacemap 118 free 1.60G segments 8648 maxsize 1.51G freepct 1% metaslab 38 offset 4c000000000 spacemap 119 free 1.66G segments 8316 maxsize 1.57G freepct 1% metaslab 39 offset 4e000000000 spacemap 87 free 1.62G segments 27153 maxsize 1.05G freepct 1% metaslab 40 offset 50000000000 spacemap 121 free 1.66G segments 9314 maxsize 1.56G freepct 1% metaslab 41 offset 52000000000 spacemap 122 free 1.64G segments 9334 maxsize 1.56G freepct 1% metaslab 42 offset 54000000000 spacemap 123 free 1.67G segments 10391 maxsize 1.51G freepct 1% metaslab 43 offset 56000000000 spacemap 126 free 1.65G segments 12514 maxsize 1.49G freepct 1% metaslab 44 offset 58000000000 spacemap 127 free 1.67G segments 13441 maxsize 1.50G freepct 1% metaslab 45 offset 5a000000000 spacemap 129 free 1.70G segments 13288 maxsize 1.54G freepct 1% metaslab 46 offset 5c000000000 spacemap 96 free 1.71G segments 27184 maxsize 1.07G freepct 1% metaslab 47 offset 5e000000000 spacemap 130 free 1.69G segments 10019 maxsize 1.61G freepct 1% metaslab 48 offset 60000000000 spacemap 133 free 1.69G segments 13025 maxsize 1.53G freepct 1% metaslab 49 offset 62000000000 spacemap 135 free 1.71G segments 10562 maxsize 1.63G freepct 1% metaslab 50 offset 64000000000 spacemap 136 free 1.74G segments 9827 maxsize 1.66G freepct 1% metaslab 51 offset 66000000000 spacemap 137 free 1.73G segments 10206 maxsize 1.65G freepct 1% metaslab 52 offset 68000000000 spacemap 138 free 1.75G segments 9747 maxsize 1.67G freepct 1% metaslab 53 offset 6a000000000 spacemap 139 free 1.76G segments 14248 maxsize 1.57G freepct 1% metaslab 54 offset 6c000000000 spacemap 107 free 1.76G segments 29803 maxsize 987M freepct 1% metaslab 55 offset 6e000000000 spacemap 142 free 1.76G segments 9068 maxsize 1.68G freepct 1% metaslab 56 offset 70000000000 spacemap 143 free 1.76G segments 10561 maxsize 1.68G freepct 1% metaslab 57 offset 72000000000 spacemap 144 free 1.78G segments 10234 maxsize 1.70G freepct 1% metaslab 58 offset 74000000000 spacemap 34 free 1.79G segments 12737 maxsize 1.49G freepct 1% metaslab 59 offset 76000000000 spacemap 145 free 1.80G segments 10211 maxsize 1.71G freepct 1% metaslab 60 offset 78000000000 spacemap 146 free 1.77G segments 10696 maxsize 1.68G freepct 1% metaslab 61 offset 7a000000000 spacemap 147 free 1.81G segments 10934 maxsize 1.71G freepct 1% metaslab 62 offset 7c000000000 spacemap 148 free 1.81G segments 8698 maxsize 1.73G freepct 1% metaslab 63 offset 7e000000000 spacemap 149 free 1.82G segments 9165 maxsize 1.74G freepct 1% metaslab 64 offset 80000000000 spacemap 152 free 1.83G segments 9388 maxsize 1.74G freepct 1% metaslab 65 offset 82000000000 spacemap 154 free 1.84G segments 11321 maxsize 1.74G freepct 1% metaslab 66 offset 84000000000 spacemap 155 free 1.85G segments 10040 maxsize 1.76G freepct 1% metaslab 67 offset 86000000000 spacemap 156 free 1.86G segments 10531 maxsize 1.77G freepct 1% metaslab 68 offset 88000000000 spacemap 86 free 1.84G segments 8518 maxsize 1.73G freepct 1% metaslab 69 offset 8a000000000 spacemap 120 free 1.87G segments 18100 maxsize 1.51G freepct 1% metaslab 70 offset 8c000000000 spacemap 157 free 1.88G segments 12773 maxsize 1.70G freepct 1% metaslab 71 offset 8e000000000 spacemap 159 free 1.89G segments 11443 maxsize 1.79G freepct 1% metaslab 72 offset 90000000000 spacemap 125 free 1.90G segments 13633 maxsize 1.72G freepct 1% metaslab 73 offset 92000000000 spacemap 160 free 1.91G segments 10724 maxsize 1.81G freepct 1% metaslab 74 offset 94000000000 spacemap 161 free 1.92G segments 10550 maxsize 1.77G freepct 1% metaslab 75 offset 96000000000 spacemap 162 free 1.92G segments 10027 maxsize 1.83G freepct 1% metaslab 76 offset 98000000000 spacemap 132 free 1.93G segments 16007 maxsize 1.69G freepct 1% metaslab 77 offset 9a000000000 spacemap 164 free 1.94G segments 9721 maxsize 1.84G freepct 1% metaslab 78 offset 9c000000000 spacemap 134 free 1.95G segments 26262 maxsize 1.35G freepct 1% metaslab 79 offset 9e000000000 spacemap 165 free 1.95G segments 7968 maxsize 1.88G freepct 1% metaslab 80 offset a0000000000 spacemap 166 free 1.97G segments 7757 maxsize 1.89G freepct 1% metaslab 81 offset a2000000000 spacemap 167 free 1.97G segments 9206 maxsize 1.89G freepct 1% metaslab 82 offset a4000000000 spacemap 168 free 1.98G segments 9225 maxsize 1.89G freepct 1% metaslab 83 offset a6000000000 spacemap 106 free 1.99G segments 24197 maxsize 1.43G freepct 1% metaslab 84 offset a8000000000 spacemap 169 free 2.00G segments 9637 maxsize 1.91G freepct 1% metaslab 85 offset aa000000000 spacemap 170 free 2.01G segments 10167 maxsize 1.92G freepct 1% metaslab 86 offset ac000000000 spacemap 171 free 2.02G segments 12180 maxsize 1.85G freepct 1% metaslab 87 offset ae000000000 spacemap 174 free 2.02G segments 9716 maxsize 1.93G freepct 1% metaslab 88 offset b0000000000 spacemap 175 free 2.04G segments 10583 maxsize 1.94G freepct 1% metaslab 89 offset b2000000000 spacemap 176 free 2.05G segments 9935 maxsize 1.95G freepct 1% metaslab 90 offset b4000000000 spacemap 177 free 2.06G segments 10459 maxsize 1.96G freepct 1% metaslab 91 offset b6000000000 spacemap 178 free 2.07G segments 9396 maxsize 1.98G freepct 1% metaslab 92 offset b8000000000 spacemap 179 free 2.07G segments 8301 maxsize 1.96G freepct 1% metaslab 93 offset ba000000000 spacemap 151 free 2.08G segments 17800 maxsize 1.52G freepct 1% metaslab 94 offset bc000000000 spacemap 181 free 2.10G segments 10951 maxsize 2.00G freepct 1% metaslab 95 offset be000000000 spacemap 182 free 2.11G segments 11002 maxsize 2.01G freepct 1% metaslab 96 offset c0000000000 spacemap 183 free 2.12G segments 10855 maxsize 2.01G freepct 1% metaslab 97 offset c2000000000 spacemap 95 free 2.13G segments 13168 maxsize 1.96G freepct 1% metaslab 98 offset c4000000000 spacemap 184 free 2.13G segments 9408 maxsize 2.04G freepct 1% metaslab 99 offset c6000000000 spacemap 185 free 2.15G segments 10694 maxsize 2.04G freepct 1% metaslab 100 offset c8000000000 spacemap 186 free 2.16G segments 10563 maxsize 2.05G freepct 1% metaslab 101 offset ca000000000 spacemap 187 free 2.17G segments 11059 maxsize 2.06G freepct 1% metaslab 102 offset cc000000000 spacemap 188 free 2.18G segments 11516 maxsize 2.07G freepct 1% metaslab 103 offset ce000000000 spacemap 189 free 2.19G segments 10700 maxsize 2.08G freepct 1% metaslab 104 offset d0000000000 spacemap 163 free 2.20G segments 7869 maxsize 1.77G freepct 1% metaslab 105 offset d2000000000 spacemap 131 free 2.22G segments 11941 maxsize 2.08G freepct 1% metaslab 106 offset d4000000000 spacemap 190 free 9.24G segments 11407 maxsize 2.11G freepct 7% metaslab 107 offset d6000000000 spacemap 141 free 2.24G segments 13038 maxsize 2.00G freepct 1% metaslab 108 offset d8000000000 spacemap 192 free 9.7G segments 10472 maxsize 2.13G freepct 7% metaslab 109 offset da000000000 spacemap 193 free 8.61G segments 11389 maxsize 2.14G freepct 6% metaslab 110 offset dc000000000 spacemap 194 free 6.34G segments 10513 maxsize 2.16G freepct 4% metaslab 111 offset de000000000 spacemap 195 free 7.58G segments 10168 maxsize 2.17G freepct 5% metaslab 112 offset e0000000000 spacemap 197 free 7.14G segments 6922 maxsize 2.22G freepct 5% metaslab 113 offset e2000000000 spacemap 198 free 2.30G segments 6757 maxsize 2.24G freepct 1% metaslab 114 offset e4000000000 spacemap 199 free 2.32G segments 7008 maxsize 2.26G freepct 1% metaslab 115 offset e6000000000 spacemap 200 free 2.33G segments 6518 maxsize 2.27G freepct 1% metaslab 116 offset e8000000000 spacemap 173 free 2.35G segments 13857 maxsize 1.92G freepct 1% metaslab 117 offset ea000000000 spacemap 201 free 2.36G segments 7124 maxsize 2.30G freepct 1% metaslab 118 offset ec000000000 spacemap 202 free 2.37G segments 6936 maxsize 2.31G freepct 1% metaslab 119 offset ee000000000 spacemap 203 free 2.38G segments 6679 maxsize 2.32G freepct 1% metaslab 120 offset f0000000000 spacemap 204 free 2.39G segments 6818 maxsize 2.34G freepct 1% metaslab 121 offset f2000000000 spacemap 205 free 2.41G segments 7423 maxsize 2.34G freepct 1% metaslab 122 offset f4000000000 spacemap 150 free 2.42G segments 15678 maxsize 2.14G freepct 1% metaslab 123 offset f6000000000 spacemap 158 free 2.43G segments 9980 maxsize 2.28G freepct 1% metaslab 124 offset f8000000000 spacemap 180 free 2.45G segments 11702 maxsize 1.71G freepct 1% metaslab 125 offset fa000000000 spacemap 206 free 2.46G segments 7070 maxsize 2.40G freepct 1% metaslab 126 offset fc000000000 spacemap 124 free 2.47G segments 11485 maxsize 2.31G freepct 1% metaslab 127 offset fe000000000 spacemap 128 free 2.49G segments 2051 maxsize 2.42G freepct 1% metaslab 128 offset 100000000000 spacemap 207 free 2.50G segments 7309 maxsize 2.44G freepct 1% metaslab 129 offset 102000000000 spacemap 208 free 2.52G segments 7151 maxsize 2.46G freepct 1% metaslab 130 offset 104000000000 spacemap 209 free 2.52G segments 6041 maxsize 2.47G freepct 1% metaslab 131 offset 106000000000 spacemap 210 free 2.54G segments 6910 maxsize 2.47G freepct 1% metaslab 132 offset 108000000000 spacemap 211 free 2.56G segments 6816 maxsize 2.50G freepct 1% metaslab 133 offset 10a000000000 spacemap 212 free 2.57G segments 6182 maxsize 2.51G freepct 2% metaslab 134 offset 10c000000000 spacemap 213 free 2.59G segments 7541 maxsize 2.52G freepct 2% metaslab 135 offset 10e000000000 spacemap 214 free 2.61G segments 7810 maxsize 2.53G freepct 2% metaslab 136 offset 110000000000 spacemap 215 free 2.62G segments 6822 maxsize 2.56G freepct 2% metaslab 137 offset 112000000000 spacemap 191 free 9.39G segments 10891 maxsize 2.28G freepct 7% metaslab 138 offset 114000000000 spacemap 216 free 2.65G segments 7295 maxsize 2.58G freepct 2% metaslab 139 offset 116000000000 spacemap 217 free 2.67G segments 7435 maxsize 2.60G freepct 2% metaslab 140 offset 118000000000 spacemap 218 free 2.68G segments 6952 maxsize 2.62G freepct 2% metaslab 141 offset 11a000000000 spacemap 196 free 2.70G segments 5975 maxsize 2.37G freepct 2% metaslab 142 offset 11c000000000 spacemap 219 free 2.72G segments 7547 maxsize 2.65G freepct 2% metaslab 143 offset 11e000000000 spacemap 220 free 2.74G segments 7570 maxsize 2.67G freepct 2% metaslab 144 offset 120000000000 spacemap 221 free 2.75G segments 7321 maxsize 2.69G freepct 2% metaslab 145 offset 122000000000 spacemap 172 free 2.77G segments 3490 maxsize 2.68G freepct 2% metaslab 146 offset 124000000000 spacemap 222 free 2.79G segments 7448 maxsize 2.72G freepct 2% metaslab 147 offset 126000000000 spacemap 223 free 2.81G segments 7309 maxsize 2.74G freepct 2% metaslab 148 offset 128000000000 spacemap 224 free 2.82G segments 7367 maxsize 2.75G freepct 2% metaslab 149 offset 12a000000000 spacemap 225 free 2.84G segments 7249 maxsize 2.77G freepct 2% metaslab 150 offset 12c000000000 spacemap 226 free 2.86G segments 8091 maxsize 2.78G freepct 2% metaslab 151 offset 12e000000000 spacemap 153 free 2.88G segments 2771 maxsize 2.74G freepct 2% vdev 1 metaslabs 105 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 0 offset 0 spacemap 38 free 128M segments 1 maxsize 128M freepct 99% metaslab 1 offset 8000000 spacemap 39 free 128M segments 1 maxsize 128M freepct 100% metaslab 2 offset 10000000 spacemap 42 free 128M segments 1 maxsize 128M freepct 100% metaslab 3 offset 18000000 spacemap 41 free 128M segments 1 maxsize 128M freepct 100% metaslab 4 offset 20000000 spacemap 51 free 128M segments 1 maxsize 128M freepct 100% metaslab 5 offset 28000000 spacemap 50 free 128M segments 1 maxsize 128M freepct 100% metaslab 6 offset 30000000 spacemap 49 free 128M segments 1 maxsize 128M freepct 100% metaslab 7 offset 38000000 spacemap 48 free 128M segments 1 maxsize 128M freepct 100% metaslab 8 offset 40000000 spacemap 47 free 128M segments 1 maxsize 128M freepct 100% metaslab 9 offset 48000000 spacemap 46 free 128M segments 1 maxsize 128M freepct 100% metaslab 10 offset 50000000 spacemap 45 free 128M segments 1 maxsize 128M freepct 100% metaslab 11 offset 58000000 spacemap 59 free 128M segments 1 maxsize 128M freepct 100% metaslab 12 offset 60000000 spacemap 62 free 128M segments 1 maxsize 128M freepct 100% metaslab 13 offset 68000000 spacemap 61 free 128M segments 1 maxsize 128M freepct 100% metaslab 14 offset 70000000 spacemap 67 free 128M segments 1 maxsize 128M freepct 100% metaslab 15 offset 78000000 spacemap 66 free 128M segments 1 maxsize 128M freepct 100% metaslab 16 offset 80000000 spacemap 74 free 128M segments 1 maxsize 128M freepct 100% metaslab 17 offset 88000000 spacemap 73 free 128M segments 1 maxsize 128M freepct 100% metaslab 18 offset 90000000 spacemap 75 free 128M segments 1 maxsize 128M freepct 100% metaslab 19 offset 98000000 spacemap 78 free 128M segments 1 maxsize 128M freepct 100% metaslab 20 offset a0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 21 offset a8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 22 offset b0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 23 offset b8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 24 offset c0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 25 offset c8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 26 offset d0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 27 offset d8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 28 offset e0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 29 offset e8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 30 offset f0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 31 offset f8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 32 offset 100000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 33 offset 108000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 34 offset 110000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 35 offset 118000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 36 offset 120000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 37 offset 128000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 38 offset 130000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 39 offset 138000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 40 offset 140000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 41 offset 148000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 42 offset 150000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 43 offset 158000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 44 offset 160000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 45 offset 168000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 46 offset 170000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 47 offset 178000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 48 offset 180000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 49 offset 188000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 50 offset 190000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 51 offset 198000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 52 offset 1a0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 53 offset 1a8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 54 offset 1b0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 55 offset 1b8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 56 offset 1c0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 57 offset 1c8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 58 offset 1d0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 59 offset 1d8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 60 offset 1e0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 61 offset 1e8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 62 offset 1f0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 63 offset 1f8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 64 offset 200000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 65 offset 208000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 66 offset 210000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 67 offset 218000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 68 offset 220000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 69 offset 228000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 70 offset 230000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 71 offset 238000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 72 offset 240000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 73 offset 248000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 74 offset 250000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 75 offset 258000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 76 offset 260000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 77 offset 268000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 78 offset 270000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 79 offset 278000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 80 offset 280000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 81 offset 288000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 82 offset 290000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 83 offset 298000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 84 offset 2a0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 85 offset 2a8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 86 offset 2b0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 87 offset 2b8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 88 offset 2c0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 89 offset 2c8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 90 offset 2d0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 91 offset 2d8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 92 offset 2e0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 93 offset 2e8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 94 offset 2f0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 95 offset 2f8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 96 offset 300000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 97 offset 308000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 98 offset 310000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 99 offset 318000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 100 offset 320000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 101 offset 328000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 102 offset 330000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 103 offset 338000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 104 offset 340000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% vdev 2 metaslabs 105 offset spacemap free --------------- ------------------- --------------- ------------- metaslab 0 offset 0 spacemap 37 free 128M segments 1 maxsize 128M freepct 100% metaslab 1 offset 8000000 spacemap 40 free 128M segments 1 maxsize 128M freepct 100% metaslab 2 offset 10000000 spacemap 44 free 128M segments 1 maxsize 128M freepct 100% metaslab 3 offset 18000000 spacemap 43 free 128M segments 1 maxsize 128M freepct 100% metaslab 4 offset 20000000 spacemap 58 free 128M segments 1 maxsize 128M freepct 100% metaslab 5 offset 28000000 spacemap 57 free 128M segments 1 maxsize 128M freepct 100% metaslab 6 offset 30000000 spacemap 56 free 128M segments 1 maxsize 128M freepct 100% metaslab 7 offset 38000000 spacemap 55 free 128M segments 1 maxsize 128M freepct 100% metaslab 8 offset 40000000 spacemap 54 free 128M segments 1 maxsize 128M freepct 100% metaslab 9 offset 48000000 spacemap 53 free 128M segments 1 maxsize 128M freepct 100% metaslab 10 offset 50000000 spacemap 52 free 128M segments 1 maxsize 128M freepct 100% metaslab 11 offset 58000000 spacemap 60 free 128M segments 1 maxsize 128M freepct 100% metaslab 12 offset 60000000 spacemap 64 free 128M segments 1 maxsize 128M freepct 100% metaslab 13 offset 68000000 spacemap 63 free 128M segments 1 maxsize 128M freepct 100% metaslab 14 offset 70000000 spacemap 69 free 128M segments 1 maxsize 128M freepct 100% metaslab 15 offset 78000000 spacemap 68 free 128M segments 1 maxsize 128M freepct 100% metaslab 16 offset 80000000 spacemap 72 free 128M segments 1 maxsize 128M freepct 100% metaslab 17 offset 88000000 spacemap 71 free 128M segments 1 maxsize 128M freepct 100% metaslab 18 offset 90000000 spacemap 76 free 128M segments 1 maxsize 128M freepct 100% metaslab 19 offset 98000000 spacemap 79 free 128M segments 1 maxsize 128M freepct 100% metaslab 20 offset a0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 21 offset a8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 22 offset b0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 23 offset b8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 24 offset c0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 25 offset c8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 26 offset d0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 27 offset d8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 28 offset e0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 29 offset e8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 30 offset f0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 31 offset f8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 32 offset 100000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 33 offset 108000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 34 offset 110000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 35 offset 118000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 36 offset 120000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 37 offset 128000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 38 offset 130000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 39 offset 138000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 40 offset 140000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 41 offset 148000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 42 offset 150000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 43 offset 158000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 44 offset 160000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 45 offset 168000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 46 offset 170000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 47 offset 178000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 48 offset 180000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 49 offset 188000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 50 offset 190000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 51 offset 198000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 52 offset 1a0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 53 offset 1a8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 54 offset 1b0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 55 offset 1b8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 56 offset 1c0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 57 offset 1c8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 58 offset 1d0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 59 offset 1d8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 60 offset 1e0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 61 offset 1e8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 62 offset 1f0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 63 offset 1f8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 64 offset 200000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 65 offset 208000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 66 offset 210000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 67 offset 218000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 68 offset 220000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 69 offset 228000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 70 offset 230000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 71 offset 238000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 72 offset 240000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 73 offset 248000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 74 offset 250000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 75 offset 258000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 76 offset 260000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 77 offset 268000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 78 offset 270000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 79 offset 278000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 80 offset 280000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 81 offset 288000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 82 offset 290000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 83 offset 298000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 84 offset 2a0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 85 offset 2a8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 86 offset 2b0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 87 offset 2b8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 88 offset 2c0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 89 offset 2c8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 90 offset 2d0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 91 offset 2d8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 92 offset 2e0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 93 offset 2e8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 94 offset 2f0000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 95 offset 2f8000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 96 offset 300000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 97 offset 308000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 98 offset 310000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 99 offset 318000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 100 offset 320000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 101 offset 328000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 102 offset 330000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 103 offset 338000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% metaslab 104 offset 340000000 spacemap 0 free 128M segments 1 maxsize 128M freepct 100% Dataset mos [META], ID 0, cr_txg 4, 459M, 235 objects Object lvl iblk dblk dsize lsize %full type 0 2 16K 16K 256K 128K 91.80 DMU dnode 1 1 16K 16K 19.0K 32K 100.00 object directory 2 1 16K 512 0 512 0.00 DSL directory 3 1 16K 512 3.00K 512 100.00 DSL props 4 1 16K 512 3.00K 512 100.00 DSL directory child map 5 1 16K 512 0 512 0.00 DSL directory 6 1 16K 512 3.00K 512 100.00 DSL props 7 1 16K 512 3.00K 512 100.00 DSL directory child map 8 1 16K 512 0 512 0.00 DSL directory 9 1 16K 512 3.00K 512 100.00 DSL props 10 1 16K 512 3.00K 512 100.00 DSL directory child map 11 1 16K 128K 0 128K 0.00 bpobj 12 1 16K 512 0 512 0.00 DSL directory 13 1 16K 512 3.00K 512 100.00 DSL props 14 1 16K 512 3.00K 512 100.00 DSL directory child map 15 1 16K 512 0 512 0.00 DSL dataset 16 1 16K 512 3.00K 512 100.00 DSL dataset snap map 17 1 16K 512 3.00K 512 100.00 DSL deadlist map 18 1 16K 512 0 512 0.00 DSL dataset 19 1 16K 512 3.00K 512 100.00 DSL deadlist map 20 1 16K 128K 0 128K 0.00 bpobj 21 1 16K 512 0 512 0.00 DSL dataset 22 1 16K 512 3.00K 512 100.00 DSL dataset snap map 23 1 16K 512 3.00K 512 100.00 DSL deadlist map 24 1 16K 128K 0 128K 0.00 bpobj 25 1 16K 512 3.00K 512 100.00 DSL dataset next clones 26 1 16K 512 3.00K 512 100.00 DSL dir clones 27 1 16K 16K 51.0K 16K 100.00 packed nvlist 28 1 16K 16K 51.0K 16K 100.00 bpobj (Z=uncompressed) 29 1 16K 128K 16K 128K 100.00 SPA history 30 1 16K 16K 6.50K 16K 100.00 packed nvlist 31 1 16K 512 3.00K 512 100.00 object array 32 1 16K 512 3.00K 512 100.00 object array 33 1 16K 512 9.5K 1.50K 100.00 object array 34 2 16K 4K 467K 176K 100.00 SPA space map 35 2 16K 4K 851K 296K 100.00 SPA space map 36 2 16K 4K 275K 88.0K 100.00 SPA space map 37 1 16K 4K 0 4K 0.00 SPA space map 38 1 16K 4K 3.00K 4K 100.00 SPA space map 39 1 16K 4K 0 4K 0.00 SPA space map 40 1 16K 4K 0 4K 0.00 SPA space map 41 1 16K 4K 0 4K 0.00 SPA space map 42 1 16K 4K 0 4K 0.00 SPA space map 43 1 16K 4K 0 4K 0.00 SPA space map 44 1 16K 4K 0 4K 0.00 SPA space map 45 1 16K 4K 0 4K 0.00 SPA space map 46 1 16K 4K 0 4K 0.00 SPA space map 47 1 16K 4K 3.00K 4K 100.00 SPA space map 48 1 16K 4K 0 4K 0.00 SPA space map 49 1 16K 4K 0 4K 0.00 SPA space map 50 1 16K 4K 0 4K 0.00 SPA space map 51 1 16K 4K 0 4K 0.00 SPA space map 52 1 16K 4K 0 4K 0.00 SPA space map 53 1 16K 4K 0 4K 0.00 SPA space map 54 1 16K 4K 3.00K 4K 100.00 SPA space map 55 1 16K 4K 0 4K 0.00 SPA space map 56 1 16K 4K 0 4K 0.00 SPA space map 57 1 16K 4K 0 4K 0.00 SPA space map 58 1 16K 4K 0 4K 0.00 SPA space map 59 1 16K 4K 0 4K 0.00 SPA space map 60 1 16K 4K 0 4K 0.00 SPA space map 61 1 16K 4K 0 4K 0.00 SPA space map 62 1 16K 4K 0 4K 0.00 SPA space map 63 1 16K 4K 0 4K 0.00 SPA space map 64 1 16K 4K 0 4K 0.00 SPA space map 65 2 16K 4K 291K 92.0K 100.00 SPA space map 66 1 16K 4K 0 4K 0.00 SPA space map 67 1 16K 4K 0 4K 0.00 SPA space map 68 1 16K 4K 0 4K 0.00 SPA space map 69 1 16K 4K 0 4K 0.00 SPA space map 70 2 16K 4K 311K 100K 100.00 SPA space map 71 1 16K 4K 0 4K 0.00 SPA space map 72 1 16K 4K 0 4K 0.00 SPA space map 73 1 16K 4K 0 4K 0.00 SPA space map 74 1 16K 4K 0 4K 0.00 SPA space map 75 1 16K 4K 0 4K 0.00 SPA space map 76 1 16K 4K 0 4K 0.00 SPA space map 77 2 16K 4K 336K 104K 100.00 SPA space map 78 1 16K 4K 0 4K 0.00 SPA space map 79 1 16K 4K 0 4K 0.00 SPA space map 80 2 16K 4K 330K 104K 100.00 SPA space map 81 2 16K 4K 333K 104K 100.00 SPA space map 82 2 16K 4K 339K 108K 100.00 SPA space map 83 2 16K 4K 343K 108K 100.00 SPA space map 84 2 16K 4K 368K 116K 100.00 SPA space map 85 2 16K 4K 339K 108K 100.00 SPA space map 86 2 16K 4K 384K 120K 100.00 SPA space map 87 2 16K 4K 1.15M 368K 100.00 SPA space map 88 2 16K 4K 346K 108K 100.00 SPA space map 89 2 16K 4K 339K 108K 100.00 SPA space map 90 2 16K 4K 349K 108K 100.00 SPA space map 91 2 16K 4K 339K 108K 100.00 SPA space map 92 2 16K 4K 365K 116K 100.00 SPA space map 93 2 16K 4K 391K 120K 100.00 SPA space map 94 2 16K 4K 867K 268K 100.00 SPA space map 95 2 16K 4K 506K 156K 100.00 SPA space map 96 2 16K 4K 858K 268K 100.00 SPA space map 97 2 16K 4K 490K 152K 100.00 SPA space map 98 2 16K 4K 346K 112K 100.00 SPA space map 99 2 16K 4K 371K 116K 100.00 SPA space map 100 2 16K 4K 362K 112K 100.00 SPA space map 101 2 16K 4K 384K 120K 100.00 SPA space map 102 2 16K 4K 400K 124K 100.00 SPA space map 103 2 16K 4K 522K 160K 100.00 SPA space map 104 2 16K 4K 477K 148K 100.00 SPA space map 105 2 16K 4K 346K 112K 100.00 SPA space map 106 2 16K 4K 774K 240K 100.00 SPA space map 107 2 16K 4K 906K 312K 100.00 SPA space map 108 2 16K 4K 359K 112K 100.00 SPA space map 109 2 16K 4K 387K 120K 100.00 SPA space map 110 2 16K 4K 387K 120K 100.00 SPA space map 111 2 16K 4K 413K 128K 100.00 SPA space map 112 2 16K 4K 375K 120K 100.00 SPA space map 113 2 16K 4K 378K 116K 100.00 SPA space map 114 2 16K 4K 410K 128K 100.00 SPA space map 115 2 16K 4K 400K 124K 100.00 SPA space map 116 2 16K 4K 381K 116K 100.00 SPA space map 117 2 16K 4K 387K 120K 100.00 SPA space map 118 2 16K 4K 397K 124K 100.00 SPA space map 119 2 16K 4K 394K 120K 100.00 SPA space map 120 2 16K 4K 826K 256K 100.00 SPA space map 121 2 16K 4K 407K 124K 100.00 SPA space map 122 2 16K 4K 429K 132K 100.00 SPA space map 123 2 16K 4K 451K 140K 100.00 SPA space map 124 2 16K 4K 461K 144K 100.00 SPA space map 125 2 16K 4K 519K 160K 100.00 SPA space map 126 2 16K 4K 477K 148K 100.00 SPA space map 127 2 16K 4K 605K 188K 100.00 SPA space map 128 2 16K 4K 227K 100K 100.00 SPA space map 129 2 16K 4K 528K 164K 100.00 SPA space map 130 2 16K 4K 426K 132K 100.00 SPA space map 131 2 16K 4K 528K 164K 100.00 SPA space map 132 2 16K 4K 589K 184K 100.00 SPA space map 133 2 16K 4K 605K 188K 100.00 SPA space map 134 2 16K 4K 790K 276K 100.00 SPA space map 135 2 16K 4K 439K 136K 100.00 SPA space map 136 2 16K 4K 423K 132K 100.00 SPA space map 137 2 16K 4K 455K 140K 100.00 SPA space map 138 2 16K 4K 410K 128K 100.00 SPA space map 139 2 16K 4K 528K 164K 100.00 SPA space map 141 2 16K 4K 531K 164K 100.00 SPA space map 142 2 16K 4K 397K 124K 100.00 SPA space map 143 2 16K 4K 439K 136K 100.00 SPA space map 144 2 16K 4K 423K 132K 100.00 SPA space map 145 2 16K 4K 419K 132K 100.00 SPA space map 146 2 16K 4K 477K 148K 100.00 SPA space map 147 2 16K 4K 461K 144K 100.00 SPA space map 148 2 16K 4K 384K 120K 100.00 SPA space map 149 2 16K 4K 403K 124K 100.00 SPA space map 150 2 16K 4K 560K 172K 100.00 SPA space map 151 2 16K 4K 586K 212K 100.00 SPA space map 152 2 16K 4K 407K 128K 100.00 SPA space map 153 2 16K 4K 183K 88.0K 100.00 SPA space map 154 2 16K 4K 506K 156K 100.00 SPA space map 155 2 16K 4K 506K 156K 100.00 SPA space map 156 2 16K 4K 432K 136K 100.00 SPA space map 157 2 16K 4K 499K 156K 100.00 SPA space map 158 2 16K 4K 442K 140K 100.00 SPA space map 159 2 16K 4K 448K 140K 100.00 SPA space map 160 2 16K 4K 435K 136K 100.00 SPA space map 161 2 16K 4K 471K 148K 100.00 SPA space map 162 2 16K 4K 435K 140K 100.00 SPA space map 163 2 16K 4K 343K 136K 100.00 SPA space map 164 2 16K 4K 426K 132K 100.00 SPA space map 165 2 16K 4K 365K 124K 100.00 SPA space map 166 2 16K 4K 371K 120K 100.00 SPA space map 167 2 16K 4K 416K 136K 100.00 SPA space map 168 2 16K 4K 426K 132K 100.00 SPA space map 169 2 16K 4K 419K 132K 100.00 SPA space map 170 2 16K 4K 461K 144K 100.00 SPA space map 171 2 16K 4K 541K 156K 100.00 SPA space map 172 2 16K 4K 237K 100K 100.00 SPA space map 173 2 16K 4K 531K 176K 100.00 SPA space map 174 2 16K 4K 426K 136K 100.00 SPA space map 175 2 16K 4K 445K 140K 100.00 SPA space map 176 2 16K 4K 426K 136K 100.00 SPA space map 177 2 16K 4K 451K 140K 100.00 SPA space map 178 2 16K 4K 423K 136K 100.00 SPA space map 179 2 16K 4K 413K 136K 100.00 SPA space map 180 2 16K 4K 439K 164K 100.00 SPA space map 181 2 16K 4K 471K 148K 100.00 SPA space map 182 2 16K 4K 458K 144K 100.00 SPA space map 183 2 16K 4K 471K 148K 100.00 SPA space map 184 2 16K 4K 419K 132K 100.00 SPA space map 185 2 16K 4K 490K 152K 100.00 SPA space map 186 2 16K 4K 445K 140K 100.00 SPA space map 187 2 16K 4K 464K 144K 100.00 SPA space map 188 2 16K 4K 467K 148K 100.00 SPA space map 189 2 16K 4K 451K 140K 100.00 SPA space map 190 2 16K 4K 522K 164K 100.00 SPA space map 191 2 16K 4K 503K 144K 100.00 SPA space map 192 2 16K 4K 544K 156K 100.00 SPA space map 193 2 16K 4K 535K 168K 100.00 SPA space map 194 2 16K 4K 487K 152K 100.00 SPA space map 195 2 16K 4K 429K 136K 100.00 SPA space map 196 2 16K 4K 291K 120K 100.00 SPA space map 197 2 16K 4K 339K 112K 100.00 SPA space map 198 2 16K 4K 359K 124K 100.00 SPA space map 199 2 16K 4K 311K 120K 100.00 SPA space map 200 2 16K 4K 314K 116K 100.00 SPA space map 201 2 16K 4K 336K 128K 100.00 SPA space map 202 2 16K 4K 317K 120K 100.00 SPA space map 203 2 16K 4K 327K 120K 100.00 SPA space map 204 2 16K 4K 346K 124K 100.00 SPA space map 205 2 16K 4K 323K 124K 100.00 SPA space map 206 2 16K 4K 339K 116K 100.00 SPA space map 207 2 16K 4K 349K 124K 100.00 SPA space map 208 2 16K 4K 333K 124K 100.00 SPA space map 209 2 16K 4K 291K 112K 100.00 SPA space map 210 2 16K 4K 336K 120K 100.00 SPA space map 211 2 16K 4K 320K 120K 100.00 SPA space map 212 2 16K 4K 291K 112K 100.00 SPA space map 213 2 16K 4K 349K 128K 100.00 SPA space map 214 2 16K 4K 397K 132K 100.00 SPA space map 215 2 16K 4K 330K 116K 100.00 SPA space map 216 2 16K 4K 320K 120K 100.00 SPA space map 217 2 16K 4K 394K 132K 100.00 SPA space map 218 2 16K 4K 336K 120K 100.00 SPA space map 219 2 16K 4K 327K 124K 100.00 SPA space map 220 2 16K 4K 349K 128K 100.00 SPA space map 221 2 16K 4K 391K 128K 100.00 SPA space map 222 2 16K 4K 339K 124K 100.00 SPA space map 223 2 16K 4K 339K 124K 100.00 SPA space map 224 2 16K 4K 365K 124K 100.00 SPA space map 225 2 16K 4K 346K 124K 100.00 SPA space map 226 2 16K 4K 400K 136K 100.00 SPA space map 228 3 16K 16K 395M 433M 99.95 persistent error log 229 1 16K 4K 0 4K 0.00 SPA space map 230 1 16K 4K 0 4K 0.00 SPA space map 231 1 16K 4K 0 4K 0.00 SPA space map 232 1 16K 4K 0 4K 0.00 SPA space map 233 1 16K 4K 0 4K 0.00 SPA space map 234 1 16K 4K 0 4K 0.00 SPA space map 235 1 16K 4K 0 4K 0.00 SPA space map 236 1 16K 4K 0 4K 0.00 SPA space map 237 1 16K 4K 0 4K 0.00 SPA space map Dataset tank2 [ZPL], ID 21, cr_txg 1, 13.3T, 37 objects ZIL header: claim_txg 0, claim_blk_seq 0, claim_lr_seq 0 replay_seq 0, flags 0x0 Object lvl iblk dblk dsize lsize %full type 0 7 16K 16K 40.5K 32K 57.81 DMU dnode -1 1 16K 512 2K 512 100.00 ZFS user/group used -2 1 16K 512 2K 512 100.00 ZFS user/group used 1 1 16K 512 2K 512 100.00 ZFS master node 2 1 16K 512 2K 512 100.00 SA master node 3 1 16K 512 2K 512 100.00 ZFS delete queue 4 1 16K 1.50K 2K 1.50K 100.00 ZFS directory 5 1 16K 1.50K 2K 1.50K 100.00 SA attr registration 6 1 16K 16K 10.5K 32K 100.00 SA attr layouts 7 1 16K 512 2K 512 100.00 ZFS directory 8 5 16K 128K 510G 510G 100.00 ZFS plain file 9 5 16K 128K 476G 476G 100.00 ZFS plain file 10 5 16K 128K 473G 473G 100.00 ZFS plain file 11 5 16K 128K 467G 467G 100.00 ZFS plain file 12 5 16K 128K 428G 428G 100.00 ZFS plain file 13 5 16K 128K 455G 455G 100.00 ZFS plain file 14 5 16K 128K 478G 478G 100.00 ZFS plain file 15 5 16K 128K 517G 517G 100.00 ZFS plain file 16 5 16K 128K 487G 487G 100.00 ZFS plain file 17 5 16K 128K 513G 513G 100.00 ZFS plain file 18 5 16K 128K 489G 489G 100.00 ZFS plain file 19 5 16K 128K 494G 493G 100.00 ZFS plain file 20 5 16K 128K 492G 492G 100.00 ZFS plain file 21 5 16K 128K 488G 487G 100.00 ZFS plain file 22 1 16K 1K 2K 1K 100.00 ZFS directory 23 4 16K 128K 107G 107G 100.00 ZFS plain file 24 4 16K 128K 92.4G 92.4G 100.00 ZFS plain file 25 4 16K 128K 97.2G 97.2G 100.00 ZFS plain file 26 4 16K 128K 0 128K 0.00 ZFS plain file 27 4 16K 128K 149G 149G 100.00 ZFS plain file 28 4 16K 128K 221G 221G 100.00 ZFS plain file 29 4 16K 128K 93.8G 93.8G 100.00 ZFS plain file 30 4 16K 128K 66.0G 66.0G 100.00 ZFS plain file 31 5 16K 128K 5.74T 5.74T 100.00 ZFS plain file 32 4 16K 128K 48.0G 48.0G 100.00 ZFS plain file 33 4 16K 128K 12.0G 12.0G 100.00 ZFS plain file 34 4 16K 128K 11.5G 11.5G 100.00 ZFS plain file 35 4 16K 128K 11.6G 11.5G 100.00 ZFS plain file 36 4 16K 128K 29.9G 29.8G 100.00 ZFS plain file 37 1 16K 512 0 512 0.00 ZFS plain file ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.