From owner-freebsd-fs@FreeBSD.ORG Sat Mar 9 17:46:15 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C2F17ED for ; Sat, 9 Mar 2013 17:46:15 +0000 (UTC) (envelope-from stephane.lapie@darkbsd.org) Received: from denrei.darkbsd.org (denrei.darkbsd.org [IPv6:2001:41d0:1:f442::1]) by mx1.freebsd.org (Postfix) with ESMTP id 8EF101FD for ; Sat, 9 Mar 2013 17:46:14 +0000 (UTC) Received: from denrei.darkbsd.org (localhost [127.0.0.1]) by denrei.darkbsd.org (Postfix) with ESMTP id 81550E99 for ; Sat, 9 Mar 2013 18:46:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; s=selector1; bh=/Xw46G3uBK1ZlYyx3tukTYCqCqM=; b=Q OaJntrjRHoZUYa/C+Vaslahc2LKC9mMVD/WQQRGd2jMOaF5uoguW+xcLlMy9/2yL Y2qqGF+teVX4ToH7D6HSkwKqExn6EN12iv/AlrDQS24/xwD1KBHoRhVehGQw/4Ew ig7VVJiq7G012RGfqpqAxgNnHfLDGcqX9eej2h5N4w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=darkbsd.org; h=message-id :date:from:mime-version:to:subject:references:in-reply-to :content-type; q=dns; s=selector1; b=RxD3PDkIUKDJroZjZVymV1JJNSX TWRIDApk7kokjBxUg7iCm1njBUvXPywWjQyNkBIG+PM7HWMe/9KfQTxYGXoXLQpz JcUF3slxnFDiS03BBz3mhkWeSFogFaIzH6Js8a7en+yByg+aoRWFcBOrXNZornwP T7aed+Juf4mwO+oE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=darkbsd.org; h= content-type:content-type:in-reply-to:references:subject:subject :mime-version:user-agent:from:from:date:date:message-id:received :received; s=selector1; t=1362851164; bh=1+68QKk4ORGPwm10mM/fNOL BcMmtH3Sn4M51X8jCmNQ=; b=QsH7PxwWev0IZ4CZ9Tft04EE9LGo2D9LEFiE205 nSMS+h/cvXeFLzP9Ac5Rd/A6UcE1PsKazJRG0bs61VjoxyojbN0feXUvED72077T 9pCDAN8YNLI8YHca6mS8OzU9qaUlMfX9J/rlp5DpwtrEjrB8o3hJ8DLmarLivKDd IfC0= X-Virus-Scanned: amavisd-new at darkbsd.org Received: from denrei.darkbsd.org ([127.0.0.1]) by denrei.darkbsd.org (denrei.darkbsd.org [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 2mnBmjRnI4Yr for ; Sat, 9 Mar 2013 18:46:04 +0100 (CET) Received: from [IPv6:2001:470:24:42d::42] (archer.yomi.darkbsd.org [IPv6:2001:470:24:42d::42]) (Authenticated sender: darksoul@darkbsd.org) by denrei.darkbsd.org (Postfix) with ESMTPSA id 4E3EEE98 for ; Sat, 9 Mar 2013 18:46:02 +0100 (CET) Message-ID: <513B7555.1010701@darkbsd.org> Date: Sun, 10 Mar 2013 02:45:57 +0900 From: Stephane LAPIE User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: Re: Panic loop on ZFS with 9.1-RELEASE References: <513B58B6.2090903@darkbsd.org> <513B6E1E.6080805@darkbsd.org> In-Reply-To: <513B6E1E.6080805@darkbsd.org> X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enigBBCFCDBBE6DBB23636CFC8BA" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Mar 2013 17:46:15 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBBCFCDBBE6DBB23636CFC8BA Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Pinpoint analysis of the zpool on the broken vdev gives the following information : # zdb -AAA -e -mm prana 1 33 Metaslabs: vdev 1 metaslabs 145 offset spacemap free =20 --------------- ------------------- --------------- -----------= -- metaslab 33 offset 21000000000 spacemap 303 free 11.= 9G WARNING: zfs: allocating allocated segment(offset=3D2335563722752 size=3D= 1024) Assertion failed: sm->sm_space =3D=3D space (0x2f927f400 =3D=3D 0x2f927f8= 00), file /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/cddl/lib/libzpool/= =2E./../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c, line 353. pid 51 (zdb), uid 0: exited on signal 6 (core dumped) Abort trap (core dumped) Just in case, root vdev 1 is made of the following devices : children[1]: type: 'raidz' id: 1 guid: 1078755695237588414 nparity: 1 metaslab_array: 175 metaslab_shift: 36 ashift: 9 asize: 10001970626560 is_log: 0 children[0]: type: 'disk' id: 0 guid: 12900041001921590764 path: '/dev/da10' phys_path: '/dev/da10' whole_disk: 0 DTL: 4127 children[1]: type: 'disk' id: 1 guid: 7211789756938666186 path: '/dev/da3' phys_path: '/dev/da3' whole_disk: 1 DTL: 4119 children[2]: type: 'disk' id: 2 guid: 12094368820342087236 path: '/dev/da5' phys_path: '/dev/da5' whole_disk: 1 DTL: 212 children[3]: type: 'disk' id: 3 guid: 6868867539761908697 path: '/dev/da4' phys_path: '/dev/da4' whole_disk: 0 DTL: 4173 children[4]: type: 'disk' id: 4 guid: 3091570768700552191 path: '/dev/da6' phys_path: '/dev/da6' whole_disk: 0 DTL: 4182 At this point I am nearly considering ripping these out and zpool importing while ignoring missing devices... :/ On 03/10/2013 02:15 AM, Stephane LAPIE wrote: > Posting a quick update. > > I ran a "zdb -emm" command to figure out what was going on, and it blew= > up in my face with an abort trap here : > - vdev 0 has 145 metaslabs, which are cleared without any problems. > - vdev 1 has 145 metaslabs, but fails in the middle : > metaslab 32 offset 20000000000 spacemap 289 free 1.64G= > segments 19509 maxsize 41.7M freepct 2%= > metaslab 33 offset 21000000000 spacemap 303 free 11.9G= > error: zfs: allocating allocated segment(offset=3D2335563722752 size=3D= 1024) > Abort trap(core dumped) > > Converting offset 2335563722752 from earlier kernel panic messages give= s > : 21fca723000, which matches the broken metaslab found by zdb. > > Is there anything I can do at this point, using zdb? > It just sounds surrealistic I have ONE broken metaslab (seemingly?) and= > that I can't recover anything... > > Cheers, > > On 03/10/2013 12:43 AM, Stephane LAPIE wrote: >> Hello list, >> >> I currently am faced with a sudden death case I can't understand at al= l, >> and I would be very appreciating of any explanation or assistance :( >> >> Here is my current kernel version : >> FreeBSD 9.1-STABLE FreeBSD 9.1-STABLE #5 r245055: Thu Jan 17 13:12:59= >> JST 2013 >> darksoul@eirei-no-za.yomi.darkbsd.org:/usr/obj/usr/storage/tech/eirei-= no-za.yomi.darkbsd.org/usr/src/sys/DARK-2012KERN=20 >> amd64 >> (Kernel is basically a lightened GENERIC kernel without VESA options a= nd >> unneeded controllers removed) >> >> The pool is a set of 3x raidz1 (5 drives), + 2 cache devices + mirrore= d >> transaction log >> >> Booting and trying to import the pool is met with : >> Solaris(panic): zfs: panic: allocating allocated >> segment(offset=3D2335563722752 size=3D1024) >> >> Booting single mode on my emergency flash card with a base OS and zpoo= l >> import -o readonly=3Don is met with : >> panic: solaris assert: zio->io_type !=3D ZIO_TYPE_WRITE || >> spa_writeable(spa), file: >> /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/sys/modules/zfs= /../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, >> line: 2461 >> >> I tried zpool import -F -f, zpool import -F -f -m after removing the >> mirrored transaction log devices, but after 40s of trying to import, i= t >> just blows up. >> >> I am currently running "zdb -emm" as per the procedure suggested here = : >> http://simplex.swordsaint.net/?p=3D199 if only to get some debug infor= mation. >> >> Thanks in advance for your time. >> >> Cheers, >> >> >> --=20 >> Stephane LAPIE, EPITA SRS, Promo 2005 >> "Even when they have digital readouts, I can't understand them." >> --MegaTokyo > > --=20 > Stephane LAPIE, EPITA SRS, Promo 2005 > "Even when they have digital readouts, I can't understand them." > --MegaTokyo --------------enigBBCFCDBBE6DBB23636CFC8BA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iF4EAREIAAYFAlE7dVUACgkQDJ4OK7D3FWQVbgD/ajAVkm/QkciZuNmMT11EkSDV 3semGzseAYsYopdpI40A/0u9MYMeUjVK1r8nUaNQXBkCQhxE760uIAgJMn+Hp3w4 =Z06k -----END PGP SIGNATURE----- --------------enigBBCFCDBBE6DBB23636CFC8BA--