Date: Wed, 03 Nov 2010 16:21:38 +0000 From: GM <gildenman@gmail.com> To: freebsd-stable@freebsd.org Subject: zfs zpool replace with partition fails only works with full disk Message-ID: <4CD18C12.3030509@gmail.com>
next in thread | raw e-mail | index | archive | help
Hi,
It seems geom_gpt keeps partitions open which prevents zfs working?
FreeBSD 8.1-STABLE #18 r213245
I'm attempting to replace a 6x1TB raidz2 with 2TB drives, one by one.
Currently the pool is simply using the full drive on each of the 1TB.
I wanted to create a GPT partition on the new 2TB drives with a small freebsd-boot, freebsd-zfs for each part of the raidz2 and freebsd-swap.
If I shutdown, pull a 1TB (ada6), replace with a 2TB (ada6), do the gpart magic and try to replace
zpool replace pool2 ada6 gpt/pool.0
it works. Shows the new device as online. A hd of gpt/pool.0 shows it's written zfs stuff in there. All seems ok.
However, as soon as I try a scrub, I get:
ZFS: vdev failure, zpool=pool2 type=vdev.open_failed
and then zpool status informs me it cannot open the device.
I tried all sorts of combinations to get it working. Export/import pool, reboot, use ada6p2 instead of the label.
Nothing worked.
However, using the full disk does work:
zpool replace pool2 ada6 ada6
It would seem that something geom_gpt? is keeping the partition open somehow?
That's what some posts alluded to, but I saw no solution.
Any ideas why it only works with the full drive and not a partition?
Thanks.
PS Bit more info if above is not clear. I've been messing about a while so currently after
blanking 10K blocks from start & end of drive and recreating GPT:
# gpart show ada6
=> 34 3907029101 ada6 GPT (1.8T)
34 128 1 freebsd-boot (64K)
162 3902834669 2 freebsd-zfs (1.8T)
3902834831 4194304 3 freebsd-swap (2.0G)
# hd -v /dev/gpt/pool.0 | head # -v flag cos partition is completely empty
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
# zpool status pool2
pool: pool2
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: scrub stopped after 0h5m with 0 errors on Wed Nov 3 15:41:08 2010
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
# zpool replace pool2 17439832775304853620 ada6p2
# zpool status pool2
pool: pool2
state: DEGRADED
scrub: scrub in progress for 0h0m, 0.00% done, 574h31m to go
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
replacing DEGRADED 0 1.57K 0
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6
ada6p2 UNAVAIL 0 0 0 cannot open
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
# tail -1 /var/log/messages
Nov 3 15:57:03 echo root: ZFS: vdev failure, zpool=pool2 type=vdev.open_failed
# hd /dev/ada6p2 | head
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00003fd0 00 00 00 00 00 00 00 00 11 7a 0c b1 7a da 10 02 |.........z.±zÚ..|
00003fe0 3f 2a 6e 7f 80 8f f4 97 fc ce aa 58 16 9f 90 af |?*n...ô.üΪX...¯|
00003ff0 8b b4 6d ff 57 ea d1 cb ab 5f 46 0d db 92 c6 6e |.´mÿWêÑË«_F.Û.Æn|
00004000 01 01 00 00 00 00 00 00 00 00 00 01 00 00 00 24 |...............$|
00004010 00 00 00 20 00 00 00 07 76 65 72 73 69 6f 6e 00 |... ....version.|
00004020 00 00 00 08 00 00 00 01 00 00 00 00 00 00 00 0f |................|
00004030 00 00 00 24 00 00 00 20 00 00 00 04 6e 61 6d 65 |...$... ....name|
00004040 00 00 00 09 00 00 00 01 00 00 00 05 70 6f 6f 6c |............pool|
# zpool export pool2
# zpool import pool2
# zpool status pool2
pool: pool2
state: DEGRADED
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
replacing DEGRADED 0 0 11
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6
ada6p2 ONLINE 0 0 0
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
*** drive shows ONLINE so zfs can see it, until a scrub...
# zpool scrub pool2
# tail /var/log/messages
Nov 3 16:09:15 echo root: ZFS: vdev failure, zpool=pool2 type=vdev.open_failed
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733264352768 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=600505936896 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876391536128 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733279049728 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=599389077504 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876399264256 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733279050240 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=599389078016 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876399264768 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733279050752 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=599389078528 size=512 error=6
Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876399265280 size=512 error=6
# zpool status pool2
pool: pool2
state: DEGRADED
scrub: scrub in progress for 0h0m, 0.00% done, 769h25m to go
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
replacing DEGRADED 12 3.25K 11
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6
ada6p2 UNAVAIL 0 0 0 cannot open
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
# zpool status pool2
pool: pool2
state: DEGRADED
scrub: scrub in progress for 0h0m, 0.00% done, 830h58m to go
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
replacing DEGRADED 12 4.90K 11
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6
ada6p2 UNAVAIL 0 0 0 cannot open
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
*** At any time this can be made to work by simply using the full drive:
# zpool detach pool2 ada6p2
# zpool status pool2
pool: pool2
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://www.sun.com/msg/ZFS-8000-4J
scrub: scrub stopped after 0h3m with 0 errors on Wed Nov 3 16:12:48 2010
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
# zpool replace pool2 17439832775304853620 ada6
# zpool status pool2
pool: pool2
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h0m, 0.00% done, 603h22m to go
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6/old
ada6 ONLINE 0 0 0 12.1M resilvered
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
# zpool status pool2
pool: pool2
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scrub: resilver in progress for 0h6m, 1.48% done, 7h27m to go
config:
NAME STATE READ WRITE CKSUM
pool2 DEGRADED 0 0 0
raidz2 DEGRADED 0 0 0
replacing DEGRADED 0 0 0
17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6/old
ada6 ONLINE 0 0 0 13.2G resilvered
ada9 ONLINE 0 0 0
ada8 ONLINE 0 0 0
ada10 ONLINE 0 0 0
ada7 ONLINE 0 0 0
ada11 ONLINE 0 0 0
errors: No known data errors
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4CD18C12.3030509>