From owner-freebsd-fs@FreeBSD.ORG Sun Dec 7 01:29:18 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F1BAB80B for ; Sun, 7 Dec 2014 01:29:18 +0000 (UTC) Received: from mail-qc0-f172.google.com (mail-qc0-f172.google.com [209.85.216.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4F32B21 for ; Sun, 7 Dec 2014 01:29:18 +0000 (UTC) Received: by mail-qc0-f172.google.com with SMTP id m20so2162773qcx.17 for ; Sat, 06 Dec 2014 17:29:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=VYLeDsVapRdsko+T1Cocy14vJoOd/HbDkkzfPtXA/BQ=; b=EoOAyDm1k0mBNF6yi3rmkdetSZH7dEECB8XczwpLKp2q3H5lOeSMoSuhcRBKulV+x0 DYwXzOpo2EMrDoR/hWybeWJQhLVA07kQMNxcc+xjFCouz0a8kz/CPijhs33NpVKtagZJ D/moKKRWeh0e9/vJA1WSmWM5WqMl9GdpEFB/b13vfSkQC+ZwHvXFvovr5w6JsZq84eUS kKOpxPF8GNgdWj36oCd0noPlVSr/ptNs9/m4gp5YKDT655RnuXzn6ehf21Uo9B068v4T Yt4vBY+zFD9pDXo77BZ+F3Azj8jm1I51OYpWKbOS+1IwlzramxVks/uZtbpzpuHzr5vj 9q9Q== X-Gm-Message-State: ALoCoQk96Oj8wJSYC1fDWwEowoSCvjViLh8ZQEnvjUSbzXNDYnGsPR509L6xVUlTMoU88fwm8lpu MIME-Version: 1.0 X-Received: by 10.224.121.142 with SMTP id h14mr39668810qar.80.1417915352252; Sat, 06 Dec 2014 17:22:32 -0800 (PST) Received: by 10.140.39.48 with HTTP; Sat, 6 Dec 2014 17:22:32 -0800 (PST) In-Reply-To: <54825E70.20900@sorbs.net> References: <54825E70.20900@sorbs.net> Date: Sat, 6 Dec 2014 18:22:32 -0700 Message-ID: Subject: Re: ZFS weird issue... From: Will Andrews To: Michelle Sullivan Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Dec 2014 01:29:19 -0000 On Fri, Dec 5, 2014 at 6:40 PM, Michelle Sullivan wrote: > Days later new drive to replace the dead drive arrived and was > inserted. System refused to re-add as there was data in the cache, so > rebooted and cleared the cache (as per many on web faq's) Reconfigured > it to match the others. Can't do a zpool replace mfid8 because that's > already in the pool... (was mfid9) can't use mfid15 because zpool > reports it's not part of the config... can't use the uniq-id it received > (can't find vdev) ... HELP!! :) [...] > root@colossus:~ # zpool status -v [...] > pool: sorbs > state: DEGRADED > status: One or more devices could not be opened. Sufficient replicas > exist for > the pool to continue functioning in a degraded state. > action: Attach the missing device and online it using 'zpool online'. > see: http://illumos.org/msg/ZFS-8000-2Q > scan: scrub in progress since Fri Dec 5 17:11:29 2014 > 2.51T scanned out of 29.9T at 89.4M/s, 89h7m to go > 0 repaired, 8.40% done > config: > > NAME STATE READ WRITE CKSUM > sorbs DEGRADED 0 0 0 > raidz2-0 DEGRADED 0 0 0 > mfid0 ONLINE 0 0 0 > mfid1 ONLINE 0 0 0 > mfid2 ONLINE 0 0 0 > mfid3 ONLINE 0 0 0 > mfid4 ONLINE 0 0 0 > mfid5 ONLINE 0 0 0 > mfid6 ONLINE 0 0 0 > mfid7 ONLINE 0 0 0 > spare-8 DEGRADED 0 0 0 > 1702922605 UNAVAIL 0 0 0 was /dev/mfid8 > mfid14 ONLINE 0 0 0 > mfid8 ONLINE 0 0 0 > mfid9 ONLINE 0 0 0 > mfid10 ONLINE 0 0 0 > mfid11 ONLINE 0 0 0 > mfid12 ONLINE 0 0 0 > mfid13 ONLINE 0 0 0 > spares > 933862663 INUSE was /dev/mfid14 > > errors: No known data errors > root@colossus:~ # uname -a > FreeBSD colossus.sorbs.net 9.2-RELEASE FreeBSD 9.2-RELEASE #0 r255898: > Thu Sep 26 22:50:31 UTC 2013 > root@bake.isc.freebsd.org:/usr/obj/usr/src/sys/GENERIC amd64 [...] > root@colossus:~ # ls -l /dev/mfi* > crw-r----- 1 root operator 0x22 Dec 5 17:18 /dev/mfi0 > crw-r----- 1 root operator 0x68 Dec 5 17:18 /dev/mfid0 > crw-r----- 1 root operator 0x69 Dec 5 17:18 /dev/mfid1 > crw-r----- 1 root operator 0x78 Dec 5 17:18 /dev/mfid10 > crw-r----- 1 root operator 0x79 Dec 5 17:18 /dev/mfid11 > crw-r----- 1 root operator 0x7a Dec 5 17:18 /dev/mfid12 > crw-r----- 1 root operator 0x82 Dec 5 17:18 /dev/mfid13 > crw-r----- 1 root operator 0x83 Dec 5 17:18 /dev/mfid14 > crw-r----- 1 root operator 0x84 Dec 5 17:18 /dev/mfid15 > crw-r----- 1 root operator 0x6a Dec 5 17:18 /dev/mfid2 > crw-r----- 1 root operator 0x6b Dec 5 17:18 /dev/mfid3 > crw-r----- 1 root operator 0x6c Dec 5 17:18 /dev/mfid4 > crw-r----- 1 root operator 0x6d Dec 5 17:18 /dev/mfid5 > crw-r----- 1 root operator 0x6e Dec 5 17:18 /dev/mfid6 > crw-r----- 1 root operator 0x75 Dec 5 17:18 /dev/mfid7 > crw-r----- 1 root operator 0x76 Dec 5 17:18 /dev/mfid8 > crw-r----- 1 root operator 0x77 Dec 5 17:18 /dev/mfid9 > root@colossus:~ # Hi, >From the above it appears your replacement drive's current name is mfid15, and the spare is now mfid14. What commands did you run that failed? Can you provide a copy of the first label from 'zdb -l /dev/mfid0'? The label will provide you with the full vdev guid that you need to replace the original drive with a new one. Another thing you could do is wait for the spare to finish resilvering, then promote it to replace the original drive, and make your new one a spare. Considering the time required to resilver this pool configuration, that may be preferable for you. --Will.