From owner-freebsd-fs@FreeBSD.ORG Thu Dec 24 00:44:09 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 752AC1065697 for ; Thu, 24 Dec 2009 00:44:09 +0000 (UTC) (envelope-from rincebrain@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.153]) by mx1.freebsd.org (Postfix) with ESMTP id 01A938FC20 for ; Thu, 24 Dec 2009 00:44:08 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id e21so452472fga.13 for ; Wed, 23 Dec 2009 16:44:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=vCs2TihcBUA7Qse0ljhZGqS1WA8CMopXyEqumdLdjhU=; b=Hpnm9swW20d5m9v2hXaL2gIUFUVHNZ1egar8+rwbbdBlukDzjPPA0AH+MJrYsHpc20 AerWMpMH+XmJ3SIi0RAWlU1wtWi7pFF2RMEp0ibmH1RSMzrIDwTLzWBfJVYorquWg9Jw 9Ck17fsadz5P5hT+5JlVbuZmO/AJPxfxuBasE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=oAad6qN7p8Ba3hjWiuRDaS9oe/KZvd7e+6mievQm2eikG6ztnDpaA1N5Fxss7kGGey hlY7MQSovG/i3dbEnFMa3zcalhn6OqfyMpQKJDhXIMXi+vw6Wd/Af5hUemNx9myNNUQQ R8caQvjdAi5ONv+czxTJ+YZriGnXe5paU3CvU= MIME-Version: 1.0 Received: by 10.239.190.69 with SMTP id w5mr1276859hbh.143.1261615447865; Wed, 23 Dec 2009 16:44:07 -0800 (PST) In-Reply-To: <9CEE3EE5-2CF7-440E-B5F4-D2BD796EA55C@gmail.com> References: <048AF210-8B9A-40EF-B970-E8794EC66B2F@gmail.com> <4B315320.5050504@quip.cz> <5da0588e0912221741r48395defnd11e34728d2b7b97@mail.gmail.com> <9CEE3EE5-2CF7-440E-B5F4-D2BD796EA55C@gmail.com> Date: Wed, 23 Dec 2009 19:44:07 -0500 Message-ID: <5da0588e0912231644w2a7afb9dg41ceffbafc8c2df6@mail.gmail.com> From: Rich To: Steven Schlansker Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: Can't repair raidz2 (Cannot replace a replacing device) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Dec 2009 00:44:09 -0000 Export then import, perhaps? I don't honestly know what to suggest - there are horrid workarounds you could do involving manually diddling the metadata state, but I feel like the correct solution is to open up a bug report and get a fix put in. - Rich On Wed, Dec 23, 2009 at 7:36 PM, Steven Schlansker wrote: > > On Dec 22, 2009, at 5:41 PM, Rich wrote: > >> http://kerneltrap.org/mailarchive/freebsd-fs/2009/9/30/6457763 may be >> useful to you - it's what we did when we got stuck in a resilver loop. >> I recall being in the same state you're in right now at one point, and >> getting out of it from there. >> >> I think if you apply that patch, you'll be able to cancel the >> resilver, and then resilver again with the device you'd like to >> resilver with. >> > > Thanks for the suggestion, but the problem isn't that it's stuck > in a resilver loop (which is what the patch seems to try to avoid) > but that I can't detach a drive. > > Now I got clever and fudged a label onto the new drive (copied the first > 50MB of one of the dying drives), ran a scrub, and have this layout - > > =A0pool: universe > =A0state: DEGRADED > status: One or more devices has experienced an unrecoverable error. =A0An > =A0 =A0 =A0 =A0attempt was made to correct the error. =A0Applications are= unaffected. > action: Determine if the device needs to be replaced, and clear the error= s > =A0 =A0 =A0 =A0using 'zpool clear' or replace the device with 'zpool repl= ace'. > =A0 see: http://www.sun.com/msg/ZFS-8000-9P > =A0scrub: scrub completed after 20h58m with 0 errors on Wed Dec 23 11:36:= 43 2009 > config: > > =A0 =A0 =A0 =A0NAME =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 STATE =A0= =A0 READ WRITE CKSUM > =A0 =A0 =A0 =A0universe =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 DEGRADED =A0 = =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0raidz2 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 DEGRADED = =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0ad16 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ONLINE = =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0replacing =A0 =A0 =A0 =A0 =A0 =A0 =A0DEGRADED =A0 = =A0 0 =A0 =A0 0 40.7M > =A0 =A0 =A0 =A0 =A0 =A0 =A0ad26 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ONLINE = =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 =A0506G repaired > =A0 =A0 =A0 =A0 =A0 =A0 =A06170688083648327969 =A0UNAVAIL =A0 =A0 =A00 88= .7M =A0 =A0 0 =A0was /dev/ad12 > =A0 =A0 =A0 =A0 =A0 =A0ad8 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0ONLINE = =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0concat/back2 =A0 =A0 =A0 =A0 =A0 ONLINE =A0 =A0 = =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0ad10 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ONLINE = =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0concat/ad4ex =A0 =A0 =A0 =A0 =A0 ONLINE =A0 =A0 = =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0ad24 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 ONLINE = =A0 =A0 =A0 0 =A0 =A0 0 =A0 =A0 0 > =A0 =A0 =A0 =A0 =A0 =A0concat/ad6ex =A0 =A0 =A0 =A0 =A0 ONLINE =A0 =A0 = =A048 =A0 =A0 0 =A0 =A0 0 =A028.5K repaired > > Why has the replacing vdev not gone away? =A0I still can't detach - > [steven@universe:~]% sudo zpool detach universe 6170688083648327969 > cannot detach 6170688083648327969: no valid replicas > even though now there actually is a valid replica (ad26) > > Additionally, running zpool clear hangs permanently and in fact freezes a= ll IO > to the pool. =A0Since I've mounted /usr from the pool, this is effectivel= y > death to the system. =A0Any other zfs commands seem to work okay > (zpool scrub, zfs mount, etc.). =A0Just clear is insta-death. =A0I can't > help but suspect that this is caused by the now non-sensical vdev configu= ration > (replacing with one good drive and one nonexistent one)... > > Any further thoughts? =A0Thanks, > Steven > > >> - Rich >> >> On Tue, Dec 22, 2009 at 6:15 PM, Miroslav Lachman <000.fbsd@quip.cz> wro= te: >>> Steven Schlansker wrote: >>>> >>>> As a corollary, you may notice some funky concat business going on. >>>> This is because I have drives which are very slightly different in siz= e (< >>>> =A01MB) >>>> and whenever one of them goes down and I bring the pool up, it helpful= ly >>>> (?) >>>> expands the pool by a whole megabyte then won't let the drive back in. >>>> This is extremely frustrating... is there any way to fix that? =A0I'm >>>> eventually going to keep expanding each of my drives one megabyte at a >>>> time >>>> using gconcat and space on another drive! =A0Very frustrating... >>> >>> You can avoid it by partitioning the drives to the well known 'minimal'= size >>> (size of smallest disk) and use the partition instead of raw disk. >>> For example ad12s1 instead of ad12 (if you creat slices by fdisk) >>> of ad12p1 (if you creat partitions by gpart) >>> >>> You can also use labels instead of device name. >>> >>> Miroslav Lachman >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" >>> >> >> >> >> -- >> >> If you are over 80 years old and accompanied by your parents, we will >> cash your check. > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > --=20 Forest fires cause Smokey Bears.