From owner-freebsd-stable@FreeBSD.ORG Mon Aug 2 23:11:55 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 601B0106566B for ; Mon, 2 Aug 2010 23:11:55 +0000 (UTC) (envelope-from dan@langille.org) Received: from nyi.unixathome.org (nyi.unixathome.org [64.147.113.42]) by mx1.freebsd.org (Postfix) with ESMTP id 23ECC8FC16 for ; Mon, 2 Aug 2010 23:11:54 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by nyi.unixathome.org (Postfix) with ESMTP id 1A99950B8A for ; Tue, 3 Aug 2010 00:11:54 +0100 (BST) X-Virus-Scanned: amavisd-new at unixathome.org Received: from nyi.unixathome.org ([127.0.0.1]) by localhost (nyi.unixathome.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id mqQ8GYBW3q7g for ; Tue, 3 Aug 2010 00:11:46 +0100 (BST) Received: from smtp-auth.unixathome.org (smtp-auth.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) by nyi.unixathome.org (Postfix) with ESMTPSA id A482C50B88 for ; Tue, 3 Aug 2010 00:11:46 +0100 (BST) Message-ID: <4C5750A4.7050104@langille.org> Date: Mon, 02 Aug 2010 19:11:32 -0400 From: Dan Langille Organization: The FreeBSD Diary User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.7) Gecko/20100713 Thunderbird/3.1.1 MIME-Version: 1.0 To: freebsd-stable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Where's the space? raidz2 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Aug 2010 23:11:55 -0000 I recently altered an existing raidz2 pool from using 7 vdevs of about 931G to 1.81TB. In fact, the existing pool used half of each HDD. I then wanted to go to using [almost] all of each HDD. I offline'd each vdev, adjusted the HDD paritions using gpart, then replaced the vdev. After letting the resilver occur, I did the next vdev. The space available after this process did not go up as I expected. I have about 4TB in the pool, not the 8 or 9TB I expected. At present, I have this: $ zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT storage 6.31T 5.51T 819G 87% ONLINE - Which is: $ zpool status pool: storage state: ONLINE scrub: scrub in progress for 3h26m, 63.33% done, 1h59m to go config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz2 ONLINE 0 0 0 gpt/disk01-live ONLINE 0 0 0 gpt/disk02-live ONLINE 0 0 0 gpt/disk03-live ONLINE 0 0 0 gpt/disk04-live ONLINE 0 0 0 gpt/disk05-live ONLINE 0 0 0 gpt/disk06-live ONLINE 0 0 0 gpt/disk07-live ONLINE 0 0 0 errors: No known data errors The disks involved are: $ gpart show ada0 ada1 ada2 ada3 ada4 ada5 ada6 => 34 3907029101 ada0 GPT (1.8T) 34 2014 - free - (1.0M) 2048 3906617453 1 freebsd-zfs (1.8T) 3906619501 409634 - free - (200M) => 34 3907029101 ada1 GPT (1.8T) 34 2014 - free - (1.0M) 2048 3906617453 1 freebsd-zfs (1.8T) 3906619501 409634 - free - (200M) => 34 3907029101 ada2 GPT (1.8T) 34 2014 - free - (1.0M) 2048 3906617453 1 freebsd-zfs (1.8T) 3906619501 409634 - free - (200M) => 34 3907029101 ada3 GPT (1.8T) 34 2014 - free - (1.0M) 2048 3906617453 1 freebsd-zfs (1.8T) 3906619501 409634 - free - (200M) => 34 3907029101 ada4 GPT (1.8T) 34 2014 - free - (1.0M) 2048 3906617453 1 freebsd-zfs (1.8T) 3906619501 409634 - free - (200M) => 34 3907029101 ada5 GPT (1.8T) 34 2014 - free - (1.0M) 2048 3906617453 1 freebsd-zfs (1.8T) 3906619501 409634 - free - (200M) => 34 3907029101 ada6 GPT (1.8T) 34 2014 - free - (1.0M) 2048 3906617453 1 freebsd-zfs (1.8T) 3906619501 409634 - free - (200M) Checking the sizes are all equal: $ gpart show ada0 ada1 ada2 ada3 ada4 ada5 ada6 | grep freebsd-zfs 2048 3906617453 1 freebsd-zfs (1.8T) 2048 3906617453 1 freebsd-zfs (1.8T) 2048 3906617453 1 freebsd-zfs (1.8T) 2048 3906617453 1 freebsd-zfs (1.8T) 2048 3906617453 1 freebsd-zfs (1.8T) 2048 3906617453 1 freebsd-zfs (1.8T) 2048 3906617453 1 freebsd-zfs (1.8T) The labels on these are: $ gpart list ada0 ada1 ada2 ada3 ada4 ada5 ada6 | grep label label: disk06-live label: disk01-live label: disk02-live label: disk03-live label: disk04-live label: disk05-live label: disk07-live So, I'm not sure where I went wrong yet... far more information than you ever wanted to know follows: $ gpart list Geom name: ada2 fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada2p1 Mediasize: 2000188135936 (1.8T) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: disk02-live length: 2000188135936 offset: 1048576 type: freebsd-zfs index: 1 end: 3906619500 start: 2048 Consumers: 1. Name: ada2 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Mode: r1w1e3 Geom name: mirror/gm0 fwheads: 255 fwsectors: 63 last: 156301424 first: 63 entries: 4 scheme: MBR Providers: 1. Name: mirror/gm0s1 Mediasize: 80026329600 (75G) Sectorsize: 512 Mode: r5w5e9 attrib: active rawtype: 165 length: 80026329600 offset: 32256 type: freebsd index: 1 end: 156301487 start: 63 Consumers: 1. Name: mirror/gm0 Mediasize: 80026361344 (75G) Sectorsize: 512 Mode: r5w5e14 Geom name: mirror/gm0s1 fwheads: 255 fwsectors: 63 last: 156301424 first: 0 entries: 8 scheme: BSD Providers: 1. Name: mirror/gm0s1a Mediasize: 1073741824 (1.0G) Sectorsize: 512 Mode: r1w1e1 rawtype: 7 length: 1073741824 offset: 0 type: freebsd-ufs index: 1 end: 2097151 start: 0 2. Name: mirror/gm0s1b Mediasize: 6442450944 (6.0G) Sectorsize: 512 Mode: r1w1e0 rawtype: 1 length: 6442450944 offset: 1073741824 type: freebsd-swap index: 2 end: 14680063 start: 2097152 3. Name: mirror/gm0s1d Mediasize: 4294967296 (4.0G) Sectorsize: 512 Mode: r1w1e1 rawtype: 7 length: 4294967296 offset: 7516192768 type: freebsd-ufs index: 4 end: 23068671 start: 14680064 4. Name: mirror/gm0s1e Mediasize: 4294967296 (4.0G) Sectorsize: 512 Mode: r1w1e1 rawtype: 7 length: 4294967296 offset: 11811160064 type: freebsd-ufs index: 5 end: 31457279 start: 23068672 5. Name: mirror/gm0s1f Mediasize: 63920202240 (60G) Sectorsize: 512 Mode: r1w1e1 rawtype: 7 length: 63920202240 offset: 16106127360 type: freebsd-ufs index: 6 end: 156301424 start: 31457280 Consumers: 1. Name: mirror/gm0s1 Mediasize: 80026329600 (75G) Sectorsize: 512 Mode: r5w5e9 Geom name: ada0 fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada0p1 Mediasize: 2000188135936 (1.8T) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: disk06-live length: 2000188135936 offset: 1048576 type: freebsd-zfs index: 1 end: 3906619500 start: 2048 Consumers: 1. Name: ada0 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Mode: r1w1e3 Geom name: ada6 fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada6p1 Mediasize: 2000188135936 (1.8T) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: disk07-live length: 2000188135936 offset: 1048576 type: freebsd-zfs index: 1 end: 3906619500 start: 2048 Consumers: 1. Name: ada6 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Mode: r1w1e3 Geom name: ada1 fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada1p1 Mediasize: 2000188135936 (1.8T) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: disk01-live length: 2000188135936 offset: 1048576 type: freebsd-zfs index: 1 end: 3906619500 start: 2048 Consumers: 1. Name: ada1 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Mode: r1w1e3 Geom name: ada3 fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada3p1 Mediasize: 2000188135936 (1.8T) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: disk03-live length: 2000188135936 offset: 1048576 type: freebsd-zfs index: 1 end: 3906619500 start: 2048 Consumers: 1. Name: ada3 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Mode: r1w1e3 Geom name: ada4 fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada4p1 Mediasize: 2000188135936 (1.8T) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: disk04-live length: 2000188135936 offset: 1048576 type: freebsd-zfs index: 1 end: 3906619500 start: 2048 Consumers: 1. Name: ada4 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Mode: r1w1e3 Geom name: ada5 fwheads: 16 fwsectors: 63 last: 3907029134 first: 34 entries: 128 scheme: GPT Providers: 1. Name: ada5p1 Mediasize: 2000188135936 (1.8T) Sectorsize: 512 Mode: r1w1e2 rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b label: disk05-live length: 2000188135936 offset: 1048576 type: freebsd-zfs index: 1 end: 3906619500 start: 2048 Consumers: 1. Name: ada5 Mediasize: 2000398934016 (1.8T) Sectorsize: 512 Mode: r1w1e3 -- Dan Langille - http://langille.org/