From owner-freebsd-fs@FreeBSD.ORG Wed Mar 13 14:12:35 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4F97ACBB for ; Wed, 13 Mar 2013 14:12:35 +0000 (UTC) (envelope-from darksoul@darkbsd.org) Received: from denrei.darkbsd.org (denrei.darkbsd.org [91.121.179.66]) by mx1.freebsd.org (Postfix) with ESMTP id 01F9D8AB for ; Wed, 13 Mar 2013 14:12:34 +0000 (UTC) Received: from denrei.darkbsd.org (localhost [127.0.0.1]) by denrei.darkbsd.org (Postfix) with ESMTP id E65A4C81 for ; Wed, 13 Mar 2013 15:12:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; s=selector1; bh=Jol/lfuPfs0++ZatAgIlehTA/wU=; b=q 7mKZoze8PyB2g3wrLbL8JiZFvoh0V4Rle1QQ9qYoxISyrszKBDdjOR/0dbF12fge s46LkEVHpzflLryTeORZueBwcRegVeCe+1C2e4Y8GP/3OHbOPb2LU8KLfHVLZQS4 kvGrhfcuBoFwPd8/kKzrxs3P9cWYo3o2I+mXknMrM8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; q=dns; s=selector1; b=HqbJAvYKTRkBCFu/P0aKA2WoY7l 5YpZsO5WIq9xiCeGT9v7vgwjYW35eWg07RXRLmH1UpVyRRzgKvvgUWfjp13shu+b wGhdcZS14ha67K3QoQrqneun6+owtptaTf+9r7ifpQGAhTMDkzAiwgJJobyUmH5F kURYbb2Fai3gYEzs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darkbsd.org; h= content-type:content-type:in-reply-to:references:subject:subject :mime-version:user-agent:from:from:date:date:message-id:received :received; s=selector1; t=1363183944; bh=4Nk4GeV3hp9KFCJNPlTRPK/ ZEUAJrPISFLxKtkwNUo0=; b=mBEe01WWLPWEVEUw69EED8gDHckEvwozyswywpA vHpr9KseP0EfZWjDyMprw15VK5aM14xgH7ZtyHxRlVoFYXRTTCcIB0ESBxX/9MHo 9S1CFqP0LOUbqyOpcR5kM9EJuMm4XG98+tSuEEhAe3VEz/bQSFOKExRU5W3/bHrb 1Y3Q= X-Virus-Scanned: amavisd-new at darkbsd.org Received: from denrei.darkbsd.org ([127.0.0.1]) by denrei.darkbsd.org (denrei.darkbsd.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 36xaddxBmxIn for ; Wed, 13 Mar 2013 15:12:24 +0100 (CET) Received: from [IPv6:2001:470:24:42d::42] (archer.yomi.darkbsd.org [IPv6:2001:470:24:42d::42]) (Authenticated sender: darksoul@darkbsd.org) by denrei.darkbsd.org (Postfix) with ESMTPSA id 566A7C80 for ; Wed, 13 Mar 2013 15:12:22 +0100 (CET) Message-ID: <51408940.9000609@darkbsd.org> Date: Wed, 13 Mar 2013 23:12:16 +0900 From: DarkSoul User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Panic loop on ZFS with 9.1-RELEASE References: <513B58B6.2090903@darkbsd.org> <513B6E1E.6080805@darkbsd.org> <513B7555.1010701@darkbsd.org> In-Reply-To: <513B7555.1010701@darkbsd.org> X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig2AE1FCAA1F9E474431CFBC3A" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Mar 2013 14:12:35 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig2AE1FCAA1F9E474431CFBC3A Content-Type: text/plain; charset=ISO-2022-JP Content-Transfer-Encoding: 7bit Just a very quick heads up. I still haven't succeeded in importing the pool readwrite, but I have succeeded in importing it readonly. This has been confirmed as a bug by the ZFS illumos ML people. Description : You can't import readonly a pool that has cache devices, because the import will try to send write IOs to auxiliary vdevs, and hit an assert() call, thus provoking a panic. Workaround : Destroy cache devices before zpool import -o readonly=on -f . Cheers, On 03/10/2013 02:45 AM, Stephane LAPIE wrote: > Pinpoint analysis of the zpool on the broken vdev gives the following > information : > > # zdb -AAA -e -mm prana 1 33 > > Metaslabs: > vdev 1 > metaslabs 145 offset spacemap free > --------------- ------------------- --------------- ------------- > metaslab 33 offset 21000000000 spacemap 303 free 11.9G > WARNING: zfs: allocating allocated segment(offset=2335563722752 size=1024) > > Assertion failed: sm->sm_space == space (0x2f927f400 == 0x2f927f800), > file > /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c, > line 353. > pid 51 (zdb), uid 0: exited on signal 6 (core dumped) > Abort trap (core dumped) > > Just in case, root vdev 1 is made of the following devices : > children[1]: > type: 'raidz' > id: 1 > guid: 1078755695237588414 > nparity: 1 > metaslab_array: 175 > metaslab_shift: 36 > ashift: 9 > asize: 10001970626560 > is_log: 0 > children[0]: > type: 'disk' > id: 0 > guid: 12900041001921590764 > path: '/dev/da10' > phys_path: '/dev/da10' > whole_disk: 0 > DTL: 4127 > children[1]: > type: 'disk' > id: 1 > guid: 7211789756938666186 > path: '/dev/da3' > phys_path: '/dev/da3' > whole_disk: 1 > DTL: 4119 > children[2]: > type: 'disk' > id: 2 > guid: 12094368820342087236 > path: '/dev/da5' > phys_path: '/dev/da5' > whole_disk: 1 > DTL: 212 > children[3]: > type: 'disk' > id: 3 > guid: 6868867539761908697 > path: '/dev/da4' > phys_path: '/dev/da4' > whole_disk: 0 > DTL: 4173 > children[4]: > type: 'disk' > id: 4 > guid: 3091570768700552191 > path: '/dev/da6' > phys_path: '/dev/da6' > whole_disk: 0 > DTL: 4182 > > At this point I am nearly considering ripping these out and zpool > importing while ignoring missing devices... :/ > > On 03/10/2013 02:15 AM, Stephane LAPIE wrote: >> Posting a quick update. >> >> I ran a "zdb -emm" command to figure out what was going on, and it blew >> up in my face with an abort trap here : >> - vdev 0 has 145 metaslabs, which are cleared without any problems. >> - vdev 1 has 145 metaslabs, but fails in the middle : >> metaslab 32 offset 20000000000 spacemap 289 free 1.64G >> segments 19509 maxsize 41.7M freepct 2% >> metaslab 33 offset 21000000000 spacemap 303 free 11.9G >> error: zfs: allocating allocated segment(offset=2335563722752 size=1024) >> Abort trap(core dumped) >> >> Converting offset 2335563722752 from earlier kernel panic messages gives >> : 21fca723000, which matches the broken metaslab found by zdb. >> >> Is there anything I can do at this point, using zdb? >> It just sounds surrealistic I have ONE broken metaslab (seemingly?) and >> that I can't recover anything... >> >> Cheers, >> >> On 03/10/2013 12:43 AM, Stephane LAPIE wrote: >>> Hello list, >>> >>> I currently am faced with a sudden death case I can't understand at all, >>> and I would be very appreciating of any explanation or assistance :( >>> >>> Here is my current kernel version : >>> FreeBSD 9.1-STABLE FreeBSD 9.1-STABLE #5 r245055: Thu Jan 17 13:12:59 >>> JST 2013 >>> darksoul@eirei-no-za.yomi.darkbsd.org:/usr/obj/usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/sys/DARK-2012KERN >>> amd64 >>> (Kernel is basically a lightened GENERIC kernel without VESA options and >>> unneeded controllers removed) >>> >>> The pool is a set of 3x raidz1 (5 drives), + 2 cache devices + mirrored >>> transaction log >>> >>> Booting and trying to import the pool is met with : >>> Solaris(panic): zfs: panic: allocating allocated >>> segment(offset=2335563722752 size=1024) >>> >>> Booting single mode on my emergency flash card with a base OS and zpool >>> import -o readonly=on is met with : >>> panic: solaris assert: zio->io_type != ZIO_TYPE_WRITE || >>> spa_writeable(spa), file: >>> /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, >>> line: 2461 >>> >>> I tried zpool import -F -f, zpool import -F -f -m after removing the >>> mirrored transaction log devices, but after 40s of trying to import, it >>> just blows up. >>> >>> I am currently running "zdb -emm" as per the procedure suggested here : >>> http://simplex.swordsaint.net/?p=199 if only to get some debug information. >>> >>> Thanks in advance for your time. >>> >>> Cheers, >>> >>> >>> -- >>> Stephane LAPIE, EPITA SRS, Promo 2005 >>> "Even when they have digital readouts, I can't understand them." >>> --MegaTokyo >> -- >> Stephane LAPIE, EPITA SRS, Promo 2005 >> "Even when they have digital readouts, I can't understand them." >> --MegaTokyo -- Stephane LAPIE, EPITA SRS, Promo 2005 "Even when they have digital readouts, I can't understand them." --MegaTokyo --------------enig2AE1FCAA1F9E474431CFBC3A Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iF4EAREIAAYFAlFAiUEACgkQDJ4OK7D3FWQcogEAs/t505xibhP4EWsRGiAF8+qO NDV/kBSdgU7Cd/UB118BALK603KdwiW4fxn/NnGBZa4T0k5NhUWUrwQ/YgjgUZWO =6itv -----END PGP SIGNATURE----- --------------enig2AE1FCAA1F9E474431CFBC3A--