Date: Mon, 02 Aug 2010 19:11:32 -0400 From: Dan Langille <dan@langille.org> To: freebsd-stable <freebsd-stable@freebsd.org> Subject: Where's the space? raidz2 Message-ID: <4C5750A4.7050104@langille.org>
next in thread | raw e-mail | index | archive | help
I recently altered an existing raidz2 pool from using 7 vdevs of about
931G to 1.81TB. In fact, the existing pool used half of each HDD. I
then wanted to go to using [almost] all of each HDD.
I offline'd each vdev, adjusted the HDD paritions using gpart, then
replaced the vdev. After letting the resilver occur, I did the next vdev.
The space available after this process did not go up as I expected. I
have about 4TB in the pool, not the 8 or 9TB I expected.
At present, I have this:
$ zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
storage 6.31T 5.51T 819G 87% ONLINE -
Which is:
$ zpool status
pool: storage
state: ONLINE
scrub: scrub in progress for 3h26m, 63.33% done, 1h59m to go
config:
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz2 ONLINE 0 0 0
gpt/disk01-live ONLINE 0 0 0
gpt/disk02-live ONLINE 0 0 0
gpt/disk03-live ONLINE 0 0 0
gpt/disk04-live ONLINE 0 0 0
gpt/disk05-live ONLINE 0 0 0
gpt/disk06-live ONLINE 0 0 0
gpt/disk07-live ONLINE 0 0 0
errors: No known data errors
The disks involved are:
$ gpart show ada0 ada1 ada2 ada3 ada4 ada5 ada6
=> 34 3907029101 ada0 GPT (1.8T)
34 2014 - free - (1.0M)
2048 3906617453 1 freebsd-zfs (1.8T)
3906619501 409634 - free - (200M)
=> 34 3907029101 ada1 GPT (1.8T)
34 2014 - free - (1.0M)
2048 3906617453 1 freebsd-zfs (1.8T)
3906619501 409634 - free - (200M)
=> 34 3907029101 ada2 GPT (1.8T)
34 2014 - free - (1.0M)
2048 3906617453 1 freebsd-zfs (1.8T)
3906619501 409634 - free - (200M)
=> 34 3907029101 ada3 GPT (1.8T)
34 2014 - free - (1.0M)
2048 3906617453 1 freebsd-zfs (1.8T)
3906619501 409634 - free - (200M)
=> 34 3907029101 ada4 GPT (1.8T)
34 2014 - free - (1.0M)
2048 3906617453 1 freebsd-zfs (1.8T)
3906619501 409634 - free - (200M)
=> 34 3907029101 ada5 GPT (1.8T)
34 2014 - free - (1.0M)
2048 3906617453 1 freebsd-zfs (1.8T)
3906619501 409634 - free - (200M)
=> 34 3907029101 ada6 GPT (1.8T)
34 2014 - free - (1.0M)
2048 3906617453 1 freebsd-zfs (1.8T)
3906619501 409634 - free - (200M)
Checking the sizes are all equal:
$ gpart show ada0 ada1 ada2 ada3 ada4 ada5 ada6 | grep freebsd-zfs
2048 3906617453 1 freebsd-zfs (1.8T)
2048 3906617453 1 freebsd-zfs (1.8T)
2048 3906617453 1 freebsd-zfs (1.8T)
2048 3906617453 1 freebsd-zfs (1.8T)
2048 3906617453 1 freebsd-zfs (1.8T)
2048 3906617453 1 freebsd-zfs (1.8T)
2048 3906617453 1 freebsd-zfs (1.8T)
The labels on these are:
$ gpart list ada0 ada1 ada2 ada3 ada4 ada5 ada6 | grep label
label: disk06-live
label: disk01-live
label: disk02-live
label: disk03-live
label: disk04-live
label: disk05-live
label: disk07-live
So, I'm not sure where I went wrong yet... far more information than
you ever wanted to know follows:
$ gpart list
Geom name: ada2
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
Mediasize: 2000188135936 (1.8T)
Sectorsize: 512
Mode: r1w1e2
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: disk02-live
length: 2000188135936
offset: 1048576
type: freebsd-zfs
index: 1
end: 3906619500
start: 2048
Consumers:
1. Name: ada2
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3
Geom name: mirror/gm0
fwheads: 255
fwsectors: 63
last: 156301424
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: mirror/gm0s1
Mediasize: 80026329600 (75G)
Sectorsize: 512
Mode: r5w5e9
attrib: active
rawtype: 165
length: 80026329600
offset: 32256
type: freebsd
index: 1
end: 156301487
start: 63
Consumers:
1. Name: mirror/gm0
Mediasize: 80026361344 (75G)
Sectorsize: 512
Mode: r5w5e14
Geom name: mirror/gm0s1
fwheads: 255
fwsectors: 63
last: 156301424
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: mirror/gm0s1a
Mediasize: 1073741824 (1.0G)
Sectorsize: 512
Mode: r1w1e1
rawtype: 7
length: 1073741824
offset: 0
type: freebsd-ufs
index: 1
end: 2097151
start: 0
2. Name: mirror/gm0s1b
Mediasize: 6442450944 (6.0G)
Sectorsize: 512
Mode: r1w1e0
rawtype: 1
length: 6442450944
offset: 1073741824
type: freebsd-swap
index: 2
end: 14680063
start: 2097152
3. Name: mirror/gm0s1d
Mediasize: 4294967296 (4.0G)
Sectorsize: 512
Mode: r1w1e1
rawtype: 7
length: 4294967296
offset: 7516192768
type: freebsd-ufs
index: 4
end: 23068671
start: 14680064
4. Name: mirror/gm0s1e
Mediasize: 4294967296 (4.0G)
Sectorsize: 512
Mode: r1w1e1
rawtype: 7
length: 4294967296
offset: 11811160064
type: freebsd-ufs
index: 5
end: 31457279
start: 23068672
5. Name: mirror/gm0s1f
Mediasize: 63920202240 (60G)
Sectorsize: 512
Mode: r1w1e1
rawtype: 7
length: 63920202240
offset: 16106127360
type: freebsd-ufs
index: 6
end: 156301424
start: 31457280
Consumers:
1. Name: mirror/gm0s1
Mediasize: 80026329600 (75G)
Sectorsize: 512
Mode: r5w5e9
Geom name: ada0
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
Mediasize: 2000188135936 (1.8T)
Sectorsize: 512
Mode: r1w1e2
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: disk06-live
length: 2000188135936
offset: 1048576
type: freebsd-zfs
index: 1
end: 3906619500
start: 2048
Consumers:
1. Name: ada0
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3
Geom name: ada6
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada6p1
Mediasize: 2000188135936 (1.8T)
Sectorsize: 512
Mode: r1w1e2
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: disk07-live
length: 2000188135936
offset: 1048576
type: freebsd-zfs
index: 1
end: 3906619500
start: 2048
Consumers:
1. Name: ada6
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3
Geom name: ada1
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
Mediasize: 2000188135936 (1.8T)
Sectorsize: 512
Mode: r1w1e2
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: disk01-live
length: 2000188135936
offset: 1048576
type: freebsd-zfs
index: 1
end: 3906619500
start: 2048
Consumers:
1. Name: ada1
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3
Geom name: ada3
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
Mediasize: 2000188135936 (1.8T)
Sectorsize: 512
Mode: r1w1e2
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: disk03-live
length: 2000188135936
offset: 1048576
type: freebsd-zfs
index: 1
end: 3906619500
start: 2048
Consumers:
1. Name: ada3
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3
Geom name: ada4
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
Mediasize: 2000188135936 (1.8T)
Sectorsize: 512
Mode: r1w1e2
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: disk04-live
length: 2000188135936
offset: 1048576
type: freebsd-zfs
index: 1
end: 3906619500
start: 2048
Consumers:
1. Name: ada4
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3
Geom name: ada5
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada5p1
Mediasize: 2000188135936 (1.8T)
Sectorsize: 512
Mode: r1w1e2
rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
label: disk05-live
length: 2000188135936
offset: 1048576
type: freebsd-zfs
index: 1
end: 3906619500
start: 2048
Consumers:
1. Name: ada5
Mediasize: 2000398934016 (1.8T)
Sectorsize: 512
Mode: r1w1e3
--
Dan Langille - http://langille.org/
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C5750A4.7050104>