From owner-freebsd-fs@FreeBSD.ORG Thu Aug 11 16:51:44 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A7351065670 for ; Thu, 11 Aug 2011 16:51:44 +0000 (UTC) (envelope-from jdc@koitsu.dyndns.org) Received: from qmta09.westchester.pa.mail.comcast.net (qmta09.westchester.pa.mail.comcast.net [76.96.62.96]) by mx1.freebsd.org (Postfix) with ESMTP id DD1838FC19 for ; Thu, 11 Aug 2011 16:51:43 +0000 (UTC) Received: from omta24.westchester.pa.mail.comcast.net ([76.96.62.76]) by qmta09.westchester.pa.mail.comcast.net with comcast id K4ja1h0021ei1Bg594rkbz; Thu, 11 Aug 2011 16:51:44 +0000 Received: from koitsu.dyndns.org ([67.180.84.87]) by omta24.westchester.pa.mail.comcast.net with comcast id K4rh1h01x1t3BNj3k4riLZ; Thu, 11 Aug 2011 16:51:43 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 3E1B4102C1A; Thu, 11 Aug 2011 09:51:40 -0700 (PDT) Date: Thu, 11 Aug 2011 09:51:40 -0700 From: Jeremy Chadwick To: Martin Cracauer Message-ID: <20110811165140.GA1421@icarus.home.lan> References: <20110811160314.GA25076@cons.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110811160314.GA25076@cons.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS zpool mirror drive replacement confusion X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Aug 2011 16:51:44 -0000 On Thu, Aug 11, 2011 at 12:03:14PM -0400, Martin Cracauer wrote: > My computer keeps doing what I say, not what I want :-) First point: OP is using ZFSv15 or earlier. I can tell from the mirror vdev (not pool) being called "mirror", not "mirror-0". Readers should take note of that fact. To the OP: it's very important you disclose exactly what FreeBSD version you're using when submitting help requests. "I use FreeBSD 8" is insufficient; providing uname -a is always best. > This zpool with a 2-drive mirror had a dead drive (ad10) which I > removed. After I tried to put ad2 as a replacement drive I ended up > with the drive not in the mirror, it seems to have extended the zpool > (it's size?). > > NAME STATE READ WRITE CKSUM > cbackup3 DEGRADED 0 0 0 > mirror DEGRADED 0 0 0 > ad6 ONLINE 0 0 0 > ad10 REMOVED 0 0 0 > ad2 ONLINE 0 0 11 The pool "grew" because your pool is now, effectively, a stripe (RAID-0) of two disks (ad6 and ad2). Keep reading. > Unfortunately I didn't keep note of the specific command that I used. > > Two questions: > > - Does somebody know offhand what I did and what I should have done > instead to get ad2 into the mirror? I'm pretty sure I know what happened, but the problem is that I'm having severe difficulty phrasing the explanation. Adding to the annoyance is the fact that you did not bother to keep track of the commands you issued. The problem is something I have run into myself many times. It's a syntax-specific "quirk" with the zpool command, and ONLY applies when working with mirrors. Furthermore, the "gotcha" may be gone with ZFSv28 (which you aren't running); I'm not sure. The "quirk" is that you have to specify the name of an existing (working) mirror device *before* the new device you're attempting to integrate. E.g. "zpool {command} pool {existingdev} {newdev}", rather than "zpool {command} pool {newdev}". Failure to provide the name of the device of the existing mirror member results in the pool becoming a stripe, which you have no way to recover from without recreating the entire pool. I would show you an example but all my systems run ZFSv28 and I can't seem to find any online documentation which helps back up my statement. I hope someone else here can phrase what I'm trying to say in a more eloquent manner and provide documentation/references confirming it. > - Would there have been a way to remove ad2 from this set? It didn't > let me, saying it can only remove drives from mirror or raid sets. > I think it should have been possible before I mounted the filesystem > in there read-write for the first time after the drive add. This > seems like an easy mistake to make. What you have right now is a pool that consists of both a mirror (ad6+ad10), as well as a stripe (the mirror + ad2). As you know, stripe means the equivalent of RAID-0; lose the disk and you lose your pool. You will need to do this, given that you said "the array doesn't contain important data": # zpool destroy cbackup3 # zpool create cbackup3 mirror ad2 ad10 In the future please be sure to take note of all the commands you do, as well as any physical actions you take (yanking disks, etc.) between commands. ZFS is amazing and awesome, and the power of the zpool command is unmatched. However, I will agree that the likelihood of a mistake being made is quite high given that its commands use words which could mean the same thing as one another; "add" vs. "attach", "remove" vs. "detach", etc.. Yet they differ severely, and issuing the wrong one can be fatal in some situations. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |