From owner-freebsd-fs@FreeBSD.ORG Tue Dec 22 22:45:56 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23B8D106566C for ; Tue, 22 Dec 2009 22:45:56 +0000 (UTC) (envelope-from stevenschlansker@gmail.com) Received: from mail-yw0-f172.google.com (mail-yw0-f172.google.com [209.85.211.172]) by mx1.freebsd.org (Postfix) with ESMTP id D13DD8FC19 for ; Tue, 22 Dec 2009 22:45:55 +0000 (UTC) Received: by ywh2 with SMTP id 2so6924489ywh.27 for ; Tue, 22 Dec 2009 14:45:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:content-type :content-transfer-encoding:subject:date:message-id:to:mime-version :x-mailer; bh=r7sTZSER3Lo07PMSPAZRMFhEKAn0nF/qG7dpYt0YuZY=; b=HrwgwYxtFt0j/M89v44fG2Bgl+jjrOybDj8uNiU7vL+9YaqNITsQjdYJqC5KjYcp2b LnwdkWf720A/aN42vlub2gugjDoMEiQxqDxa6h60Ej3bcpMl2pSa+7JWimg4SKA7PZqF omxA+GnVJ/A0i5CGwNTzBcCHoVcnRe6Z7uozI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:content-type:content-transfer-encoding:subject:date:message-id :to:mime-version:x-mailer; b=RBnqATI9cBTqmVeen78oRIMru7PwHbz/NblyHqKAJOm4kd1RWuaEZhz1zEUc4OepSv kzNyC861EY0M9khJR1gD+ImcxO3MHOltzmVAG4ZHXGMHLBO+V9LLZ7BCa5Jx0flDW2kZ CheG2UtLxXL/Vl47uNCcRwKAsPWYIqJymNwM0= Received: by 10.150.106.15 with SMTP id e15mr2886735ybc.300.1261520334281; Tue, 22 Dec 2009 14:18:54 -0800 (PST) Received: from ?192.168.42.92? (70-36-134-162.dsl.dynamic.sonic.net [70.36.134.162]) by mx.google.com with ESMTPS id 4sm2998966yxd.34.2009.12.22.14.18.52 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 22 Dec 2009 14:18:53 -0800 (PST) From: Steven Schlansker Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 22 Dec 2009 14:18:20 -0800 Message-Id: <048AF210-8B9A-40EF-B970-E8794EC66B2F@gmail.com> To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1077) X-Mailer: Apple Mail (2.1077) Subject: ZFS: Can't repair raidz2 (Cannot replace a replacing device) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2009 22:45:56 -0000 Hello fellow FreeBSDers, I've got a nice shiny ZFS raidz2 array set up, but it's gotten stuck in a DEGRADED state and I can't figure out how to recover! Here's the array as it stands now: [steven@universe:~]% sudo zpool status pool: universe state: DEGRADED scrub: scrub in progress for 0h9m, 2.19% done, 6h41m to go config: NAME STATE READ WRITE CKSUM universe DEGRADED 0 0 0 raidz2 DEGRADED 0 0 0 ad16 ONLINE 0 0 0 replacing UNAVAIL 0 5.93K 0 = insufficient replicas 3961920099899285277 UNAVAIL 0 7.11K 0 was = /dev/concat/back0/old 6170688083648327969 UNAVAIL 0 7.11K 0 was = /dev/ad12 ad8 ONLINE 0 0 0 concat/back2 ONLINE 0 0 0 ad10 ONLINE 0 0 0 concat/ad4ex ONLINE 0 0 0 ad24 ONLINE 0 0 0 concat/ad6ex ONLINE 0 0 0 errors: No known data errors One of my drives failed. I replaced it, and in the process of replacing accidentally pulled the other drive. Now, I can't seem to fix it in any = way - [steven@universe:~]% sudo zpool replace universe 3961920099899285277 = ad26 cannot replace 3961920099899285277 with ad26: cannot replace a replacing = device [steven@universe:~]% sudo zpool replace universe 6170688083648327969 = ad26 cannot replace 6170688083648327969 with ad26: cannot replace a replacing = device [steven@universe:~]% sudo zpool detach universe 6170688083648327969 cannot detach 6170688083648327969: no valid replicas [steven@universe:~]% sudo zpool detach universe 3961920099899285277 cannot detach 3961920099899285277: no valid replicas Any thoughts? As a corollary, you may notice some funky concat business going on. This is because I have drives which are very slightly different in size = (< 1MB) and whenever one of them goes down and I bring the pool up, it helpfully = (?) expands the pool by a whole megabyte then won't let the drive back in. This is extremely frustrating... is there any way to fix that? I'm eventually going to keep expanding each of my drives one megabyte at a = time using gconcat and space on another drive! Very frustrating... Thank you for any suggestions, Steven=