Date: Sun, 10 Mar 2013 02:45:57 +0900 From: Stephane LAPIE <stephane.lapie@darkbsd.org> To: freebsd-fs@freebsd.org Subject: Re: Panic loop on ZFS with 9.1-RELEASE Message-ID: <513B7555.1010701@darkbsd.org> In-Reply-To: <513B6E1E.6080805@darkbsd.org> References: <513B58B6.2090903@darkbsd.org> <513B6E1E.6080805@darkbsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enigBBCFCDBBE6DBB23636CFC8BA
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Pinpoint analysis of the zpool on the broken vdev gives the following
information :
# zdb -AAA -e -mm prana 1 33
Metaslabs:
vdev 1
metaslabs 145 offset spacemap free =20
--------------- ------------------- --------------- -----------=
--
metaslab 33 offset 21000000000 spacemap 303 free 11.=
9G
WARNING: zfs: allocating allocated segment(offset=3D2335563722752 size=3D=
1024)
Assertion failed: sm->sm_space =3D=3D space (0x2f927f400 =3D=3D 0x2f927f8=
00),
file
/usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/cddl/lib/libzpool/=
=2E./../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/space_map.c,
line 353.
pid 51 (zdb), uid 0: exited on signal 6 (core dumped)
Abort trap (core dumped)
Just in case, root vdev 1 is made of the following devices :
children[1]:
type: 'raidz'
id: 1
guid: 1078755695237588414
nparity: 1
metaslab_array: 175
metaslab_shift: 36
ashift: 9
asize: 10001970626560
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 12900041001921590764
path: '/dev/da10'
phys_path: '/dev/da10'
whole_disk: 0
DTL: 4127
children[1]:
type: 'disk'
id: 1
guid: 7211789756938666186
path: '/dev/da3'
phys_path: '/dev/da3'
whole_disk: 1
DTL: 4119
children[2]:
type: 'disk'
id: 2
guid: 12094368820342087236
path: '/dev/da5'
phys_path: '/dev/da5'
whole_disk: 1
DTL: 212
children[3]:
type: 'disk'
id: 3
guid: 6868867539761908697
path: '/dev/da4'
phys_path: '/dev/da4'
whole_disk: 0
DTL: 4173
children[4]:
type: 'disk'
id: 4
guid: 3091570768700552191
path: '/dev/da6'
phys_path: '/dev/da6'
whole_disk: 0
DTL: 4182
At this point I am nearly considering ripping these out and zpool
importing while ignoring missing devices... :/
On 03/10/2013 02:15 AM, Stephane LAPIE wrote:
> Posting a quick update.
>
> I ran a "zdb -emm" command to figure out what was going on, and it blew=
> up in my face with an abort trap here :
> - vdev 0 has 145 metaslabs, which are cleared without any problems.
> - vdev 1 has 145 metaslabs, but fails in the middle :
> metaslab 32 offset 20000000000 spacemap 289 free 1.64G=
> segments 19509 maxsize 41.7M freepct 2%=
> metaslab 33 offset 21000000000 spacemap 303 free 11.9G=
> error: zfs: allocating allocated segment(offset=3D2335563722752 size=3D=
1024)
> Abort trap(core dumped)
>
> Converting offset 2335563722752 from earlier kernel panic messages give=
s
> : 21fca723000, which matches the broken metaslab found by zdb.
>
> Is there anything I can do at this point, using zdb?
> It just sounds surrealistic I have ONE broken metaslab (seemingly?) and=
> that I can't recover anything...
>
> Cheers,
>
> On 03/10/2013 12:43 AM, Stephane LAPIE wrote:
>> Hello list,
>>
>> I currently am faced with a sudden death case I can't understand at al=
l,
>> and I would be very appreciating of any explanation or assistance :(
>>
>> Here is my current kernel version :
>> FreeBSD 9.1-STABLE FreeBSD 9.1-STABLE #5 r245055: Thu Jan 17 13:12:59=
>> JST 2013
>> darksoul@eirei-no-za.yomi.darkbsd.org:/usr/obj/usr/storage/tech/eirei-=
no-za.yomi.darkbsd.org/usr/src/sys/DARK-2012KERN=20
>> amd64
>> (Kernel is basically a lightened GENERIC kernel without VESA options a=
nd
>> unneeded controllers removed)
>>
>> The pool is a set of 3x raidz1 (5 drives), + 2 cache devices + mirrore=
d
>> transaction log
>>
>> Booting and trying to import the pool is met with :
>> Solaris(panic): zfs: panic: allocating allocated
>> segment(offset=3D2335563722752 size=3D1024)
>>
>> Booting single mode on my emergency flash card with a base OS and zpoo=
l
>> import -o readonly=3Don is met with :
>> panic: solaris assert: zio->io_type !=3D ZIO_TYPE_WRITE ||
>> spa_writeable(spa), file:
>> /usr/storage/tech/eirei-no-za.yomi.darkbsd.org/usr/src/sys/modules/zfs=
/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c,
>> line: 2461
>>
>> I tried zpool import -F -f, zpool import -F -f -m after removing the
>> mirrored transaction log devices, but after 40s of trying to import, i=
t
>> just blows up.
>>
>> I am currently running "zdb -emm" as per the procedure suggested here =
:
>> http://simplex.swordsaint.net/?p=3D199 if only to get some debug infor=
mation.
>>
>> Thanks in advance for your time.
>>
>> Cheers,
>>
>>
>> --=20
>> Stephane LAPIE, EPITA SRS, Promo 2005
>> "Even when they have digital readouts, I can't understand them."
>> --MegaTokyo
>
> --=20
> Stephane LAPIE, EPITA SRS, Promo 2005
> "Even when they have digital readouts, I can't understand them."
> --MegaTokyo
--------------enigBBCFCDBBE6DBB23636CFC8BA
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iF4EAREIAAYFAlE7dVUACgkQDJ4OK7D3FWQVbgD/ajAVkm/QkciZuNmMT11EkSDV
3semGzseAYsYopdpI40A/0u9MYMeUjVK1r8nUaNQXBkCQhxE760uIAgJMn+Hp3w4
=Z06k
-----END PGP SIGNATURE-----
--------------enigBBCFCDBBE6DBB23636CFC8BA--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?513B7555.1010701>