From owner-freebsd-fs@FreeBSD.ORG  Fri Apr 30 21:00:09 2010
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52])
	by hub.freebsd.org (Postfix) with ESMTP id A164B106564A
	for <freebsd-fs@freebsd.org>; Fri, 30 Apr 2010 21:00:09 +0000 (UTC)
	(envelope-from morganw@chemikals.org)
Received: from warped.bluecherry.net (warped.bluecherry.net [66.138.159.247])
	by mx1.freebsd.org (Postfix) with ESMTP id 6D2598FC13
	for <freebsd-fs@freebsd.org>; Fri, 30 Apr 2010 21:00:09 +0000 (UTC)
Received: from volatile.chemikals.org (adsl-67-123-77.shv.bellsouth.net
	[98.67.123.77])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(Client did not present a certificate)
	by warped.bluecherry.net (Postfix) with ESMTPSA id 10E1187DC608;
	Fri, 30 Apr 2010 16:00:07 -0500 (CDT)
Received: from localhost (morganw@localhost [127.0.0.1])
	by volatile.chemikals.org (8.14.4/8.14.4) with ESMTP id o3UKxwL3089114; 
	Fri, 30 Apr 2010 15:59:58 -0500 (CDT)
	(envelope-from morganw@chemikals.org)
Date: Fri, 30 Apr 2010 15:59:58 -0500 (CDT)
From: Wes Morgan <morganw@chemikals.org>
X-X-Sender: morganw@volatile
To: Freddie Cash <fjwcash@gmail.com>
In-Reply-To: <alpine.BSF.2.00.1004301550160.32311@ibyngvyr>
Message-ID: <alpine.BSF.2.00.1004301558010.32311@ibyngvyr>
References: <y2pb269bc571004280847od80c334cmc3a073cd7d20a927@mail.gmail.com>
	<alpine.BSF.2.00.1004292005410.58938@ibyngvyr>
	<p2ib269bc571004291840m85acbbafna617db0825766fd3@mail.gmail.com>
	<alpine.BSF.2.00.1004301550160.32311@ibyngvyr>
User-Agent: Alpine 2.00 (BSF 1167 2008-08-23)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
X-Virus-Scanned: clamav-milter 0.95.3 at warped
X-Virus-Status: Clean
Cc: freebsd-fs@freebsd.org
Subject: Re: ZFS: "Cannot replace a replacing drive"
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 30 Apr 2010 21:00:09 -0000

On Fri, 30 Apr 2010, Wes Morgan wrote:

> On Thu, 29 Apr 2010, Freddie Cash wrote:
>
> > On Thu, Apr 29, 2010 at 6:06 PM, Wes Morgan <morganw@chemikals.org> wrote:
> >
> > > On Wed, 28 Apr 2010, Freddie Cash wrote:
> > >
> > > > Going through the archives, I see that others have run into this issue,
> > > and
> > > > managed to solve it via "zpool detach".  However, looking closely at the
> > > > archived messages, all the successful tests had one thing in common:  1
> > > > drive ONLINE, 1 drive FAULTED.  If a drive is online, obviously it can be
> > > > detached.  In all the cases where people have been unsuccessful at fixing
> > > > this situation, 1 drive is OFFLINE, and 1 drive is FAULTED.  As is our
> > > case:
> > > >
> > >
> > > What happened to the drive to fault it?
> > >
> > > Am in the process of replacing 500 GB drives with 1.5 TB drives, to
> > increase the available storage space in the pool (process went flawlessly on
> > the other storage server).  First 3 disks in the vdev replaced without
> > issues.

Also, where is the original 500gb drive (#4) that you replaced? You should
be able to put that drive back in and let it resilver itself to catch back
up to the other devices, then cancel the replacement.