From owner-freebsd-stable@FreeBSD.ORG Wed Nov 3 16:50:40 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0BE0C106566C for ; Wed, 3 Nov 2010 16:50:40 +0000 (UTC) (envelope-from gildenman@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 8DC3D8FC17 for ; Wed, 3 Nov 2010 16:50:36 +0000 (UTC) Received: by wyb42 with SMTP id 42so822164wyb.13 for ; Wed, 03 Nov 2010 09:50:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:content-type :content-transfer-encoding; bh=Zp6E9ZDmE11Tr0V6jPC+Maspjp/X7oQQdAlnVxH0Rmc=; b=m+X+Nmp5A9fWZuWyZMuV2OHrEKDXzYq3vGEwQsoQloAzMk29yGQMapNF4yQhWR8WaR wzWSFj43xnCqF+8Bc16g2b+PQtd8klZC+pEn0tbkEpuu330jQQpwQ2JDJoG2qdM5GzEg vObqJxBOT7u2hOdNZseJ6vd7XyHhFmA7WwkmM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; b=iWJ6rGfyMJL3Rvy19nt98BKDBWqz0n8YvGeRYFBaSvzpKx5RrsoHpM0DCfufvca2ev TodE4rwzbqTh3Ttau3MpWEqMwrAR1Rfie1ZQdBs8j18sA49UpwrF0Uk3YevfrJciH6fC BsziN0V8awwf9tZpCur59QK9FmibwAAQ1QHeU= Received: by 10.216.165.77 with SMTP id d55mr12919233wel.23.1288801289180; Wed, 03 Nov 2010 09:21:29 -0700 (PDT) Received: from [192.168.1.4] (94-193-80-220.zone7.bethere.co.uk [94.193.80.220]) by mx.google.com with ESMTPS id w8sm5701953wei.45.2010.11.03.09.21.26 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 03 Nov 2010 09:21:27 -0700 (PDT) Message-ID: <4CD18C12.3030509@gmail.com> Date: Wed, 03 Nov 2010 16:21:38 +0000 From: GM User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.12) Gecko/20101027 Thunderbird/3.1.6 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Subject: zfs zpool replace with partition fails only works with full disk X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Nov 2010 16:50:40 -0000 Hi, It seems geom_gpt keeps partitions open which prevents zfs working? FreeBSD 8.1-STABLE #18 r213245 I'm attempting to replace a 6x1TB raidz2 with 2TB drives, one by one. Currently the pool is simply using the full drive on each of the 1TB. I wanted to create a GPT partition on the new 2TB drives with a small freebsd-boot, freebsd-zfs for each part of the raidz2 and freebsd-swap. If I shutdown, pull a 1TB (ada6), replace with a 2TB (ada6), do the gpart magic and try to replace zpool replace pool2 ada6 gpt/pool.0 it works. Shows the new device as online. A hd of gpt/pool.0 shows it's written zfs stuff in there. All seems ok. However, as soon as I try a scrub, I get: ZFS: vdev failure, zpool=pool2 type=vdev.open_failed and then zpool status informs me it cannot open the device. I tried all sorts of combinations to get it working. Export/import pool, reboot, use ada6p2 instead of the label. Nothing worked. However, using the full disk does work: zpool replace pool2 ada6 ada6 It would seem that something geom_gpt? is keeping the partition open somehow? That's what some posts alluded to, but I saw no solution. Any ideas why it only works with the full drive and not a partition? Thanks. PS Bit more info if above is not clear. I've been messing about a while so currently after blanking 10K blocks from start & end of drive and recreating GPT: # gpart show ada6 => 34 3907029101 ada6 GPT (1.8T) 34 128 1 freebsd-boot (64K) 162 3902834669 2 freebsd-zfs (1.8T) 3902834831 4194304 3 freebsd-swap (2.0G) # hd -v /dev/gpt/pool.0 | head # -v flag cos partition is completely empty 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| # zpool status pool2 pool: pool2 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-2Q scrub: scrub stopped after 0h5m with 0 errors on Wed Nov 3 15:41:08 2010 config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6 ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors # zpool replace pool2 17439832775304853620 ada6p2 # zpool status pool2 pool: pool2 state: DEGRADED scrub: scrub in progress for 0h0m, 0.00% done, 574h31m to go config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 replacing DEGRADED 0 1.57K 0 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6 ada6p2 UNAVAIL 0 0 0 cannot open ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors # tail -1 /var/log/messages Nov 3 15:57:03 echo root: ZFS: vdev failure, zpool=pool2 type=vdev.open_failed # hd /dev/ada6p2 | head 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00003fd0 00 00 00 00 00 00 00 00 11 7a 0c b1 7a da 10 02 |.........z.±zÚ..| 00003fe0 3f 2a 6e 7f 80 8f f4 97 fc ce aa 58 16 9f 90 af |?*n...ô.üΪX...¯| 00003ff0 8b b4 6d ff 57 ea d1 cb ab 5f 46 0d db 92 c6 6e |.´mÿWêÑË«_F.Û.Æn| 00004000 01 01 00 00 00 00 00 00 00 00 00 01 00 00 00 24 |...............$| 00004010 00 00 00 20 00 00 00 07 76 65 72 73 69 6f 6e 00 |... ....version.| 00004020 00 00 00 08 00 00 00 01 00 00 00 00 00 00 00 0f |................| 00004030 00 00 00 24 00 00 00 20 00 00 00 04 6e 61 6d 65 |...$... ....name| 00004040 00 00 00 09 00 00 00 01 00 00 00 05 70 6f 6f 6c |............pool| # zpool export pool2 # zpool import pool2 # zpool status pool2 pool: pool2 state: DEGRADED scrub: none requested config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 replacing DEGRADED 0 0 11 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6 ada6p2 ONLINE 0 0 0 ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors *** drive shows ONLINE so zfs can see it, until a scrub... # zpool scrub pool2 # tail /var/log/messages Nov 3 16:09:15 echo root: ZFS: vdev failure, zpool=pool2 type=vdev.open_failed Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733264352768 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=600505936896 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876391536128 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733279049728 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=599389077504 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876399264256 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733279050240 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=599389078016 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876399264768 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=733279050752 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=599389078528 size=512 error=6 Nov 3 16:09:17 echo root: ZFS: vdev I/O failure, zpool=pool2 path= offset=876399265280 size=512 error=6 # zpool status pool2 pool: pool2 state: DEGRADED scrub: scrub in progress for 0h0m, 0.00% done, 769h25m to go config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 replacing DEGRADED 12 3.25K 11 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6 ada6p2 UNAVAIL 0 0 0 cannot open ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors # zpool status pool2 pool: pool2 state: DEGRADED scrub: scrub in progress for 0h0m, 0.00% done, 830h58m to go config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 replacing DEGRADED 12 4.90K 11 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6 ada6p2 UNAVAIL 0 0 0 cannot open ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors *** At any time this can be made to work by simply using the full drive: # zpool detach pool2 ada6p2 # zpool status pool2 pool: pool2 state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: scrub stopped after 0h3m with 0 errors on Wed Nov 3 16:12:48 2010 config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6 ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors # zpool replace pool2 17439832775304853620 ada6 # zpool status pool2 pool: pool2 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h0m, 0.00% done, 603h22m to go config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6/old ada6 ONLINE 0 0 0 12.1M resilvered ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors # zpool status pool2 pool: pool2 state: DEGRADED status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scrub: resilver in progress for 0h6m, 1.48% done, 7h27m to go config: NAME STATE READ WRITE CKSUM pool2 DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 replacing DEGRADED 0 0 0 17439832775304853620 UNAVAIL 0 0 0 was /dev/ada6/old ada6 ONLINE 0 0 0 13.2G resilvered ada9 ONLINE 0 0 0 ada8 ONLINE 0 0 0 ada10 ONLINE 0 0 0 ada7 ONLINE 0 0 0 ada11 ONLINE 0 0 0 errors: No known data errors