From owner-freebsd-fs@FreeBSD.ORG Wed Apr 28 15:47:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7C27D106564A for ; Wed, 28 Apr 2010 15:47:18 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-pw0-f54.google.com (mail-pw0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 538C88FC08 for ; Wed, 28 Apr 2010 15:47:18 +0000 (UTC) Received: by pwi9 with SMTP id 9so10603754pwi.13 for ; Wed, 28 Apr 2010 08:47:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=Nb/QNTs4LFzG/bXJ+BeLHankxFZM+UB9zMc/PYejpns=; b=iqUhWuZhJLrAXou3P8TEZTgb6W6H1feO9FWah3KlBhYSxRznPY3kCheyV9YYHnd3nU jA4YmNjZ5g3dsJgrW3IHJzenNnOGRy61/BH3X9OqF+6F6tQBXb90ugG7Hr3k1LEzJuuG 4AC5guz54EZMSudhrFa3j+DQYzX80lvrU21Zc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=nnMjrGcsR+Jo9200xHAbdN2GNMnYXSjHb2xdLUCfxlszT83JbOYo8iNZMfArLAGFHn XQnQknSFS7k6xUb5SljNpc1fW2mudgrOkbdneBU+p/TCtA/QcSuA1JR0KWzDAAfAU7d8 qSF/bFueHjyKXeSPGaZE1T4mKL9X8xGP4uQ4Y= MIME-Version: 1.0 Received: by 10.114.23.15 with SMTP id 15mr9205918waw.45.1272469621111; Wed, 28 Apr 2010 08:47:01 -0700 (PDT) Received: by 10.231.18.74 with HTTP; Wed, 28 Apr 2010 08:47:00 -0700 (PDT) Date: Wed, 28 Apr 2010 08:47:00 -0700 Message-ID: From: Freddie Cash To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: ZFS: "Cannot replace a replacing drive" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Apr 2010 15:47:18 -0000 Going through the archives, I see that others have run into this issue, and managed to solve it via "zpool detach". However, looking closely at the archived messages, all the successful tests had one thing in common: 1 drive ONLINE, 1 drive FAULTED. If a drive is online, obviously it can be detached. In all the cases where people have been unsuccessful at fixing this situation, 1 drive is OFFLINE, and 1 drive is FAULTED. As is our case: [fcash@thehive ~]$ zpool status -v pool: storage state: DEGRADED status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: none requested config: NAME STATE READ WRITE CKSUM storage DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 label/disk01 ONLINE 0 0 0 label/disk02 ONLINE 0 0 0 label/disk03 ONLINE 0 0 0 replacing UNAVAIL 0 534 0 insufficient replicas label/disk04/old OFFLINE 0 544 0 label/disk04 FAULTED 0 544 0 corrupted data label/disk13 ONLINE 0 0 0 label/disk14 ONLINE 0 0 0 label/disk15 ONLINE 0 0 0 label/disk16 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 label/disk05 ONLINE 0 0 0 label/disk06 ONLINE 0 0 0 label/disk07 ONLINE 0 0 0 label/disk08 ONLINE 0 0 0 label/disk17 ONLINE 0 0 0 label/disk18 ONLINE 0 0 0 label/disk19 ONLINE 0 0 0 label/disk20 ONLINE 0 0 0 raidz2 ONLINE 0 0 0 label/disk09 ONLINE 0 0 0 label/disk10 ONLINE 0 0 0 label/disk11 ONLINE 0 0 0 label/disk12 ONLINE 0 0 0 label/disk21 ONLINE 0 0 0 label/disk22 ONLINE 0 0 0 label/disk23 ONLINE 0 0 0 label/disk24 ONLINE 0 0 0 cache label/cache ONLINE 0 0 0 errors: No known data errors [fcash@thehive ~]$ sudo zpool replace storage label/disk04 cannot replace label/disk04 with label/disk04: cannot replace a replacing device Note the OFFLINE status for label/disk04/old. I cannot get either drive to detach, or to replace, or to online, or to offline. "zpool online" on the old device changes the status to UNAVAIL. "zpool detach" and "zpool offline" give the same error: no valid replicas. I've tried removing the underlying device, booting with the drive in the system and without the drive in the system, all kinds of zpool commands, all without success. Is there any way to recover from this error? Or am I doomed to destroy a 10 TB pool? FreeBSD thehive.sd73.bc.ca 8.0-STABLE FreeBSD 8.0-STABLE #3: Fri Jan 15 11:08:47 PST 2010 root@thehive.sd73.bc.ca:/usr/obj/usr/src-8/sys/ZFSHOST amd64 ZFSv13 -- Freddie Cash fjwcash@gmail.com