From owner-freebsd-fs@FreeBSD.ORG Thu Oct 4 19:34:09 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 90A2D1065673 for ; Thu, 4 Oct 2012 19:34:09 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 156198FC18 for ; Thu, 4 Oct 2012 19:34:08 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id x43so698372wey.13 for ; Thu, 04 Oct 2012 12:34:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:date:subject:to:message-id:mime-version:x-mailer; bh=HohGR9Xk6iohlM6MXJUMZ/6uIhWbIEEkN7eI6AWCV6U=; b=SrA6D7iK0qQeWdKl/EyeTsmFWanvBBdDvVeyKZ4XYywBTNjhZPtZEcrAmvaGXmEOdh DvIxkmd306AdZzynTnCmwoFi0xEwe6HHHF/9PoO2VZPlddQWc6P6J93lrTx2YylmZcEI i3McqwJUCKc6kSugMliid98SwkJ6mREu7WLHMWm86AlE/YccmIGTBGVB3nF1XBuOvePx 3XCTecQ9UNn+WC5Pm86uw9r+ZtwSjUEE1tnFBcmW56j6MNu5WafsjhkpHrs5C+hWDIND mQYQgdtADVGNAKi+JN7Di7waezQ+3vM35LdEUoEmX6JZUL9fZxqUAV9d8uJGeK1VKnU0 BnKw== Received: by 10.180.94.226 with SMTP id df2mr40302378wib.11.1349379241797; Thu, 04 Oct 2012 12:34:01 -0700 (PDT) Received: from [10.0.0.86] ([93.152.184.10]) by mx.google.com with ESMTPS id k2sm16383706wiz.7.2012.10.04.12.34.00 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 04 Oct 2012 12:34:00 -0700 (PDT) From: Nikolay Denev Date: Thu, 4 Oct 2012 22:33:59 +0300 To: "" Message-Id: <5A5FE35F-7D68-4E83-A88D-3002B51F2E00@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) X-Mailer: Apple Mail (2.1498) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: zpool scrub on pool from geli devices offlines the pool? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Oct 2012 19:34:09 -0000 Hi, I have a zfs pool from 24 disks encrypted with geli. I just did a zpool scrub tank, and that probably reopened all of the = devices, but this caused geli "detach on last close" to kick in=20 which resulted in offline pool from UNAVAILABLE devices.=20 pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool = clear'. see: http://illumos.org/msg/ZFS-8000-HC scan: scrub in progress since Thu Oct 4 21:19:15 2012 1 scanned out of 8.29T at 1/s, (scan is slow, no estimated time) 0 repaired, 0.00% done config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 raidz2-0 UNAVAIL 0 0 0 4340223731536330140 UNAVAIL 0 0 0 was = /dev/mfid1.eli 5260313034754791769 UNAVAIL 0 0 0 was = /dev/mfid2.eli 3388275563832205054 UNAVAIL 0 0 0 was = /dev/mfid3.eli 4279885200356306835 UNAVAIL 0 0 0 was = /dev/mfid4.eli 17520568003934998783 UNAVAIL 0 0 0 was = /dev/mfid5.eli 14683427064986614232 UNAVAIL 0 0 0 was = /dev/mfid6.eli 5604251825626821 UNAVAIL 0 0 0 was = /dev/mfid7.eli 2878395114688866721 UNAVAIL 0 0 0 was = /dev/mfid8.eli raidz2-1 UNAVAIL 0 0 0 1560240233906009318 UNAVAIL 0 0 0 was = /dev/mfid9.eli 17390515268955717943 UNAVAIL 0 0 0 was = /dev/mfid10.eli 16346219034888442254 UNAVAIL 0 0 0 was = /dev/mfid11.eli 16181936453927970171 UNAVAIL 0 0 0 was = /dev/mfid12.eli 13672668419715232053 UNAVAIL 0 0 0 was = /dev/mfid13.eli 8576569675278017750 UNAVAIL 0 0 0 was = /dev/mfid14.eli 7122599902867613575 UNAVAIL 0 0 0 was = /dev/mfid15.eli 6165832151020850637 UNAVAIL 0 0 0 was = /dev/mfid16.eli raidz2-2 UNAVAIL 0 0 0 2529143736541278973 UNAVAIL 0 0 0 was = /dev/mfid17.eli 5815783978070201610 UNAVAIL 0 0 0 was = /dev/mfid18.eli 10521963168174464672 UNAVAIL 0 0 0 was = /dev/mfid19.eli 17880694802593963336 UNAVAIL 0 0 0 was = /dev/mfid20.eli 2868521416175385324 UNAVAIL 0 0 0 was = /dev/mfid21.eli 16369604825508697024 UNAVAIL 0 0 0 was = /dev/mfid22.eli 10849928960759331453 UNAVAIL 0 0 0 was = /dev/mfid23.eli 7128010358193490217 UNAVAIL 0 0 0 was = /dev/mfid24.eli errors: 1 data errors, use '-v' for a list Dmesg shows : GEOM_ELI: Detached mfid1.eli on last close. =85 GEOM_ELI: Detached mfid24.eli on last close. I then did /etc/rc.d/geli restart and zpool clear tank, and it is back = online, but shows permanent metadata errors=85 Any ideas why this happned from a simple zpool scrub, and how it can be = prevented? Just disable "detach on last close" for the geli devices?