From owner-freebsd-fs@FreeBSD.ORG Sun Aug 21 12:50:06 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 253EC1065672; Sun, 21 Aug 2011 12:50:06 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id AC5728FC08; Sun, 21 Aug 2011 12:50:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.4/8.14.4) with ESMTP id p7LCo43l069183; Sun, 21 Aug 2011 16:50:04 +0400 (MSD) (envelope-from marck@rinet.ru) Date: Sun, 21 Aug 2011 16:50:04 +0400 (MSD) From: Dmitry Morozovsky To: freebsd-fs@freebsd.org In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (woozle.rinet.ru [0.0.0.0]); Sun, 21 Aug 2011 16:50:04 +0400 (MSD) Cc: Pawel Jakub Dawidek Subject: Re: strange ZFS v28 states after disk upgrades/rebuilds X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Aug 2011 12:50:06 -0000 On Sun, 21 Aug 2011, Dmitry Morozovsky wrote: > Dear colleagues, > > I'm not sure how did I make this, but try to explain: > > my home file server was fresh 8-stable/amd64, booted from CF, and ZFS-root with > 5x1.5T raidz + ssd as cache. raidz was built on raw disks ad4..ad12. ZFS v28 > > I'm starting to upgrade disks to Hitachi 3T, now with GPT on them. First change > (ad12) worked seamlessly. Next two were not, some hangs, some reboots, some > unables to import pool on boot (latter each time disappeard after booting > single into CF /bootdisk, mount -u -w /, zpool import, reboot) > > after last reboot from single user I've got (BTW, resilvering seems to survive > reboot, but can't report proper resilvering speed) [snip] > > then disk activity stops, and zpool is locked: > > -- 8< -- > root@hamster:~# zpool status -v > load: 0.00 cmd: zpool 6300 [spa_namespace_lock] 6.02r 0.00u 0.00s 0% 2032k > -- 8< -- after hard reset, boot proceeds normally, but now I have similarly strange config: marck@hamster:~> zpool status pool: hm state: DEGRADED scan: resilvered 1.80G in 2h26m with 0 errors on Sun Aug 21 15:39:15 2011 config: NAME STATE READ WRITE CKSUM hm DEGRADED 0 0 0 raidz1-0 DEGRADED 0 0 0 ad4 ONLINE 0 0 0 ad6 ONLINE 0 0 0 replacing-2 DEGRADED 0 0 0 13001111841528871597 UNAVAIL 0 0 0 was /dev/ad8 gptid/3962b8a3-cb6d-11e0-a2b4-0007e90d0cbb ONLINE 0 0 0 replacing-3 DEGRADED 0 0 0 4143382663317400064 UNAVAIL 0 0 0 was /dev/ad10 6273508279307911610 UNAVAIL 0 0 0 was /dev/ad10 13164605370838846626 UNAVAIL 0 0 0 was /dev/ad10 gptid/fabf95d4-cb4a-11e0-bdbd-0007e90d0cbb ONLINE 0 0 0 gptid/9faf12fa-ca5b-11e0-b59d-0007e90d0cbb ONLINE 0 0 0 cache ad14h ONLINE 0 0 0 errors: No known data errors Relevant part of zpool history: 2011-08-16.22:13:58 zpool set autoexpand=on hm 2011-08-17.01:20:14 zpool scrub hm 2011-08-17.11:20:53 zpool clear hm 2011-08-19.16:07:16 zpool replace hm ad12 /dev/gpt/hm4 2011-08-20.00:35:51 zpool replace hm ad4 gpt/hm0 2011-08-20.03:38:47 zpool clear hm 2011-08-20.17:29:31 zpool import hm 2011-08-20.17:35:12 zpool offline hm ad10 2011-08-20.17:35:23 zpool online hm ad10 2011-08-20.17:41:15 zpool scrub -s hm 2011-08-20.17:42:40 zpool offline hm 6273508279307911610 2011-08-20.17:46:40 zpool scrub -s hm 2011-08-20.17:46:59 zpool replace hm ad10 /dev/gpt/hm3 2011-08-20.20:25:51 zpool offline hm ad10 2011-08-20.20:26:44 zpool online hm ad10 2011-08-20.20:26:57 zpool online hm 6273508279307911610 2011-08-20.20:27:42 zpool online hm gpt/hm3 2011-08-20.20:28:02 zpool offline hm gpt/hm3 2011-08-20.20:28:25 zpool online hm gpt/hm3 2011-08-20.20:30:27 zpool scrub -s hm 2011-08-20.20:34:19 zpool export hm 2011-08-20.20:34:51 zpool import hm 2011-08-20.20:40:02 zpool replace hm /dev/ad10 /dev/gpt/hm3 2011-08-20.23:47:14 zpool import hm 2011-08-21.02:02:03 zpool replace hm /dev/ad8 /dev/gpt/hm2 2011-08-21.13:12:38 zpool import hm -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------