From owner-freebsd-fs@FreeBSD.ORG Fri Apr 30 20:51:30 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 197931065670 for ; Fri, 30 Apr 2010 20:51:30 +0000 (UTC) (envelope-from morganw@chemikals.org) Received: from warped.bluecherry.net (warped.bluecherry.net [66.138.159.247]) by mx1.freebsd.org (Postfix) with ESMTP id D2CED8FC18 for ; Fri, 30 Apr 2010 20:51:29 +0000 (UTC) Received: from volatile.chemikals.org (adsl-67-123-77.shv.bellsouth.net [98.67.123.77]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by warped.bluecherry.net (Postfix) with ESMTPSA id E597487DCB3A; Fri, 30 Apr 2010 15:51:26 -0500 (CDT) Received: from localhost (morganw@localhost [127.0.0.1]) by volatile.chemikals.org (8.14.4/8.14.4) with ESMTP id o3UKpAtq086568; Fri, 30 Apr 2010 15:51:10 -0500 (CDT) (envelope-from morganw@chemikals.org) Date: Fri, 30 Apr 2010 15:51:10 -0500 (CDT) From: Wes Morgan X-X-Sender: morganw@volatile To: Freddie Cash In-Reply-To: Message-ID: References: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Scanned: clamav-milter 0.95.3 at warped X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: "Cannot replace a replacing drive" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2010 20:51:30 -0000 On Thu, 29 Apr 2010, Freddie Cash wrote: > On Thu, Apr 29, 2010 at 6:06 PM, Wes Morgan wrote: > > > On Wed, 28 Apr 2010, Freddie Cash wrote: > > > > > Going through the archives, I see that others have run into this issue, > > and > > > managed to solve it via "zpool detach". However, looking closely at the > > > archived messages, all the successful tests had one thing in common: 1 > > > drive ONLINE, 1 drive FAULTED. If a drive is online, obviously it can be > > > detached. In all the cases where people have been unsuccessful at fixing > > > this situation, 1 drive is OFFLINE, and 1 drive is FAULTED. As is our > > case: > > > > > > > What happened to the drive to fault it? > > > > Am in the process of replacing 500 GB drives with 1.5 TB drives, to > increase the available storage space in the pool (process went flawlessly on > the other storage server). First 3 disks in the vdev replaced without > issues. > > 4th disk turned out to be a dud. Nothing but timeouts and read/write errors > during the replace. So I popped it out, put in a different 1.5 TB drive, > glabel'd it with the same name ... and the pool went "boom". > > Now I'm stuck with a "label/disk04" device that can't be replaced, can't be > offlined, can't be detached. > > Tried exporting the pool, importing the pool, with and without the disk in > the system. All kinds of variations on detach, online, offline, replace on > the old device, the new device, the UUIDs. Can you send the output of zpool history? > I'm really hoping there's a way to recover from this, but it doesn't look > like it. Will probably have to destroy/recreate the pool next week, using > the 1.5 TB drives from the get-go. I'm sure you can still recover it. Just have some patience.