From owner-freebsd-fs@FreeBSD.ORG Fri Apr 30 01:40:44 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ECF6D106564A for ; Fri, 30 Apr 2010 01:40:44 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-iw0-f204.google.com (mail-iw0-f204.google.com [209.85.223.204]) by mx1.freebsd.org (Postfix) with ESMTP id B0D438FC19 for ; Fri, 30 Apr 2010 01:40:43 +0000 (UTC) Received: by iwn42 with SMTP id 42so11981425iwn.14 for ; Thu, 29 Apr 2010 18:40:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=gfN/4mUyjSk9jDmo/f4YW3Rb7Ky+MVsN3Hi85efBBsQ=; b=mU5KjmSbWyf9efrGmJU8qZsWXn5NkYt4sxi68yOPwPXRZ4O5BYoMpxREUzOZVZuE+2 w7ZdeVYmnVsyeuKMZcSdOUw0u9v9HzL9MWvyVyMFW7y3OE/77kSCDKuskBLjuWz3Nyuu yhP9RcrVewkqtkyJScy/dc6cZ8Kr/VqlmSqEo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=B4JUOA+nCqtN9CRrB/NwwgVpDW/9q6h5dHzS4+ttx38nFHdmcK6TqHnULc45kXKjKS ieTtkKSm4j1eV6lnIt0OhlxqHPqyXG3JWrdXxWQtCVXQdx1m+xe/Inwt4oKaVdeHeXRW pNBH7i7pK+St9Z+0uxxQEksuq+rBI8+XJ/nUI= MIME-Version: 1.0 Received: by 10.231.176.7 with SMTP id bc7mr2066341ibb.2.1272591636176; Thu, 29 Apr 2010 18:40:36 -0700 (PDT) Received: by 10.231.18.74 with HTTP; Thu, 29 Apr 2010 18:40:36 -0700 (PDT) In-Reply-To: References: Date: Thu, 29 Apr 2010 18:40:36 -0700 Message-ID: From: Freddie Cash To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFS: "Cannot replace a replacing drive" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2010 01:40:45 -0000 On Thu, Apr 29, 2010 at 6:06 PM, Wes Morgan wrote: > On Wed, 28 Apr 2010, Freddie Cash wrote: > > > Going through the archives, I see that others have run into this issue, > and > > managed to solve it via "zpool detach". However, looking closely at the > > archived messages, all the successful tests had one thing in common: 1 > > drive ONLINE, 1 drive FAULTED. If a drive is online, obviously it can be > > detached. In all the cases where people have been unsuccessful at fixing > > this situation, 1 drive is OFFLINE, and 1 drive is FAULTED. As is our > case: > > > > What happened to the drive to fault it? > > Am in the process of replacing 500 GB drives with 1.5 TB drives, to increase the available storage space in the pool (process went flawlessly on the other storage server). First 3 disks in the vdev replaced without issues. 4th disk turned out to be a dud. Nothing but timeouts and read/write errors during the replace. So I popped it out, put in a different 1.5 TB drive, glabel'd it with the same name ... and the pool went "boom". Now I'm stuck with a "label/disk04" device that can't be replaced, can't be offlined, can't be detached. Tried exporting the pool, importing the pool, with and without the disk in the system. All kinds of variations on detach, online, offline, replace on the old device, the new device, the UUIDs. Nothing. [Now I know, for the future, to stress-test a drive before putting it into the pool.] I'm really hoping there's a way to recover from this, but it doesn't look like it. Will probably have to destroy/recreate the pool next week, using the 1.5 TB drives from the get-go. -- Freddie Cash fjwcash@gmail.com