From owner-freebsd-stable@FreeBSD.ORG Sun Mar 2 11:49:28 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09E4F106566C; Sun, 2 Mar 2008 11:49:28 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.delphij.net (delphij-pt.tunnel.tserv2.fmt.ipv6.he.net [IPv6:2001:470:1f03:2c9::2]) by mx1.freebsd.org (Postfix) with ESMTP id 847E68FC1D; Sun, 2 Mar 2008 11:49:26 +0000 (UTC) (envelope-from delphij@delphij.net) Received: from tarsier.geekcn.org (tarsier.geekcn.org [202.108.54.204]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by tarsier.delphij.net (Postfix) with ESMTPS id 6D73628449; Sun, 2 Mar 2008 19:49:25 +0800 (CST) Received: from localhost (tarsier.geekcn.org [202.108.54.204]) by tarsier.geekcn.org (Postfix) with ESMTP id E2003EBACA9; Sun, 2 Mar 2008 19:49:24 +0800 (CST) X-Virus-Scanned: amavisd-new at geekcn.org Received: from tarsier.geekcn.org ([202.108.54.204]) by localhost (mail.geekcn.org [202.108.54.204]) (amavisd-new, port 10024) with ESMTP id O89-Yn8QlgYU; Sun, 2 Mar 2008 19:49:19 +0800 (CST) Received: from li-xins-macbook.lan (c-67-161-39-180.hsd1.ca.comcast.net [67.161.39.180]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by tarsier.geekcn.org (Postfix) with ESMTPSA id 4B4F5EB6DF2; Sun, 2 Mar 2008 19:49:17 +0800 (CST) DomainKey-Signature: a=rsa-sha1; s=default; d=delphij.net; c=nofws; q=dns; h=message-id:date:from:reply-to:organization:user-agent: mime-version:to:cc:subject:x-enigmail-version:openpgp:content-type; b=TOI2yfZ7rkXIJGjyk5hn+VXmfzeAZhahfQm3lpKV3ebvX43nZA88RQHS/xsR0HMeA 4JxU/mUfKTQUfJdjZkRyg== Message-ID: <47CA942F.2080901@delphij.net> Date: Sun, 02 Mar 2008 03:49:03 -0800 From: LI Xin Organization: The FreeBSD Project User-Agent: Thunderbird 2.0.0.12 (Macintosh/20080213) MIME-Version: 1.0 To: FreeBSD Stable X-Enigmail-Version: 0.95.6 OpenPGP: url=http://www.delphij.net/delphij.asc Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="------------enigD32EDA9C0DB3DBBF8BAA1DD9" Cc: Pawel Jakub Dawidek , Quake Lee Subject: Reliably trigger-able ZFS panic X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: d@delphij.net List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Mar 2008 11:49:28 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigD32EDA9C0DB3DBBF8BAA1DD9 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable Hi, The following iozone test case on ZFS would reliably trigger panic: /usr/local/bin/iozone -M -e -+u -T -t 128 -S 4096 -L 64 -R -r 4k -s 30g=20 -i 0 -i 1 -i 2 -i 8 -+p 70 -C Unfortunately the kgdb can not reveal useful backtrace. I have tried=20 KDB_TRACE, but have not yet be able to further investigate it. fs12# kgdb /boot/kernel/kernel.symbols vmcore.0 [GDB will not be able to debug user-mode threads:=20 /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you = are welcome to change it and/or distribute copies of it under certain=20 conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for detail= s. This GDB was configured as "amd64-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid =3D 5; apic id =3D 05 fault virtual address =3D 0x18 fault code =3D supervisor read data, page not present instruction pointer =3D 0x8:0xffffffff80763d16 stack pointer =3D 0x10:0xffffffffd94798f0 frame pointer =3D 0x10:0xffffffffd9479920 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 340 (txg_thread_enter) trap number =3D 12 panic: page fault cpuid =3D 5 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x17a trap_fatal() at trap_fatal+0x29f trap_pfault() at trap_pfault+0x294 trap() at trap+0x2ea calltrap() at calltrap+0x8 --- trap 0xc, rip =3D 0xffffffff80763d16, rsp =3D 0xffffffffd94798f0, rbp= =3D=20 0xffffffffd9479920 --- dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x26 dmu_objset_sync() at dmu_objset_sync+0x12d dsl_pool_sync() at dsl_pool_sync+0x72 spa_sync() at spa_sync+0x390 txg_sync_thread() at txg_sync_thread+0x12f fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffffffd9479d30, rbp =3D 0 --- Uptime: 25m7s Physical memory: 4081 MB Dumping 1139 MB: 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964 948 = 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676 660=20 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388 372=20 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100 84=20 68 52 36 20 4 #0 doadump () at pcpu.h:194 194 pcpu.h: No such file or directory. in pcpu.h (kgdb) add-symbol-file /boot/kernel/zfs.ko.symbols add symbol table from file "/boot/kernel/zfs.ko.symbols" at (y or n) y Reading symbols from /boot/kernel/zfs.ko.symbols...done. (kgdb) where #0 doadump () at pcpu.h:194 #1 0xffffffff80277aa8 in boot (howto=3D260) at=20 /usr/src/sys/kern/kern_shutdown.c:409 #2 0xffffffff80277f07 in panic (fmt=3DVariable "fmt" is not available. ) at /usr/src/sys/kern/kern_shutdown.c:563 #3 0xffffffff80465a1f in trap_fatal (frame=3D0xc, eva=3DVariable "eva" i= s=20 not available. ) at /usr/src/sys/amd64/amd64/trap.c:724 #4 0xffffffff80465e04 in trap_pfault (frame=3D0xffffffffd9479840, usermo= de=3D0) at /usr/src/sys/amd64/amd64/trap.c:641 #5 0xffffffff8046677a in trap (frame=3D0xffffffffd9479840) at=20 /usr/src/sys/amd64/amd64/trap.c:410 #6 0xffffffff8044babe in calltrap () at=20 /usr/src/sys/amd64/amd64/exception.S:169 #7 0xffffffff80763d16 in ?? () #8 0x0000000000000004 in adjust_ace_pair () #9 0x0000000000000004 in adjust_ace_pair () #10 0xffffffffd94799e0 in ?? () #11 0xffffffff80763e7d in ?? () #12 0xffffff0004275a80 in ?? () #13 0xffffff00045a1190 in ?? () #14 0xffffffff807639b0 in ?? () #15 0xffffffff80763f20 in ?? () #16 0xffffff00042dc800 in ?? () #17 0x0000000000000004 in adjust_ace_pair () #18 0xffffffffd9479990 in ?? () #19 0x000000000000b55d in z_deflateInit2_ (strm=3D0xffffff00042dc8e0,=20 level=3D70109184, method=3D68351768, windowBits=3D68351600, memLevel=3D76231808, strategy=3D76231808,=20 version=3DCannot access memory at address 0xffffffff00040010 ) at=20 /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/zmod/deflat= e.c:318 Previous frame inner to this frame (corrupt stack?) --=20 Xin LI http://www.delphij.net/ FreeBSD - The Power to Serve! --------------enigD32EDA9C0DB3DBBF8BAA1DD9 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFHypQvOfuToMruuMARCvWnAJ4/TF2S8Q0BlMXGVvFeCLH0NPj3bACeMGZN 7C4iVCxjFyIFwtIZaL/WOtg= =JcN0 -----END PGP SIGNATURE----- --------------enigD32EDA9C0DB3DBBF8BAA1DD9--