From owner-freebsd-fs@FreeBSD.ORG Mon Jun 2 20:06:27 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4CE89C9F for ; Mon, 2 Jun 2014 20:06:27 +0000 (UTC) Received: from smtp1.multiplay.co.uk (smtp1.multiplay.co.uk [85.236.96.35]) by mx1.freebsd.org (Postfix) with ESMTP id F0FE02EED for ; Mon, 2 Jun 2014 20:06:26 +0000 (UTC) Received: by smtp1.multiplay.co.uk (Postfix, from userid 65534) id E55FB20E7088C; Mon, 2 Jun 2014 20:06:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.multiplay.co.uk X-Spam-Level: ** X-Spam-Status: No, score=2.9 required=8.0 tests=AWL,BAYES_00,DOS_OE_TO_MX, FSL_HELO_NON_FQDN_1, HELO_NO_DOMAIN, RDNS_DYNAMIC, STOX_REPLY_TYPE, TVD_FINGER_02 autolearn=no version=3.3.1 Received: from r2d2 (82-69-141-170.dsl.in-addr.zen.co.uk [82.69.141.170]) by smtp1.multiplay.co.uk (Postfix) with ESMTPS id 82BF620E70886; Mon, 2 Jun 2014 20:06:20 +0000 (UTC) Message-ID: From: "Steven Hartland" To: , References: <5388D64D.4030400@bayphoto.com> <5388E5B4.3030002@bayphoto.com> <538BBEB7.4070008@bayphoto.com> <782C34792E95484DBA631A96FE3BEF20@multiplay.co.uk> <538C9CF3.6070208@bayphoto.com> <16ADD4D9DC73403C9669D8F34FDBD316@multiplay.co.uk> <538CB3EA.9010807@bayphoto.com> <6C6FB182781541CEBF627998B73B1DB4@multiplay.co.uk> <538CC16A.6060207@bayphoto.com> Subject: Re: ZFS Kernel Panic on 10.0-RELEASE Date: Mon, 2 Jun 2014 21:06:25 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="Windows-1252"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jun 2014 20:06:27 -0000 Could you check what vd is at frame 7? ----- Original Message ----- From: "Mike Carlson" On 6/2/2014 10:35 AM, Steven Hartland wrote: > ----- Original Message ----- From: "Mike Carlson" >> I have a crash dump, its 3.2GB in size. >> >> How can I get this to the right people? I can put it up on S3, but >> I'd prefer to limit its access since it may contain sensitive >> information (we dont have local accounts aside from root, but better >> to be safe) > > For starters can you put the top contents of core.X.txt somewhere. > > If there's sensitive info in there just the top kgdb section with > the backtrace will be good initially. > > Regards > Steve > > I don't have a core.0.txt, I only have: ~/p/z/dump> ls -al total 347690 drwxr-xr-x 3 mikec wheel 8 Jun 2 03:25 . drwxr-xr-x 4 mikec wheel 5 Jun 2 10:44 .. drwxrwxr-x 2 mikec operator 2 Jun 2 03:07 .snap -rw-r--r-- 1 mikec wheel 2 Jun 2 03:24 bounds -rw------- 1 mikec wheel 446 Jun 2 03:24 info.0 lrwxr-xr-x 1 mikec wheel 6 Jun 2 03:25 info.last -> info.0 -rw------- 1 mikec wheel 3469885440 Jun 2 03:25 vmcore.0 lrwxr-xr-x 1 mikec wheel 8 Jun 2 03:25 vmcore.last -> vmcore.0 But, here is the kgdb output (with backtrace): ~/p/z/dump> cat ../kgdb_backtrace.txt <118>root@:/ # zfs set canmount=on zroot/data/working <118>root@:/ # zfs mount zroot/data/working Fatal trap 12: page fault while in kernel mode cpuid = 14; apic id = 22 fault virtual address = 0x4a0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8185a39f stack pointer = 0x28:0xfffffe1834608570 frame pointer = 0x28:0xfffffe18346085b0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 2 (txg_thread_enter) trap number = 12 panic: page fault cpuid = 14 KDB: stack backtrace: #0 0xffffffff808e7ee0 at kdb_backtrace+0x60 #1 0xffffffff808af9c5 at panic+0x155 #2 0xffffffff80c8e7b2 at trap_fatal+0x3a2 #3 0xffffffff80c8ea89 at trap_pfault+0x2c9 #4 0xffffffff80c8e216 at trap+0x5e6 #5 0xffffffff80c754b2 at calltrap+0x8 #6 0xffffffff8182eb5a at dsl_dataset_block_kill+0x3a #7 0xffffffff8182b967 at dnode_sync+0x237 #8 0xffffffff81823fcb at dmu_objset_sync_dnodes+0x2b #9 0xffffffff81823e4d at dmu_objset_sync+0x1ed #10 0xffffffff8183829a at dsl_pool_sync+0xca #11 0xffffffff81853a4e at spa_sync+0x52e #12 0xffffffff8185c925 at txg_sync_thread+0x375 #13 0xffffffff80881a9a at fork_exit+0x9a #14 0xffffffff80c759ee at fork_trampoline+0xe Uptime: 26m15s Dumping 3309 out of 98234 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/zfs.ko.symbols...done. Loaded symbols for /boot/kernel/zfs.ko.symbols Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. Loaded symbols for /boot/kernel/opensolaris.ko.symbols #0 doadump (textdump=) at pcpu.h:219 219 __asm("movq %%gs:%1,%0" : "=r" (td) (kgdb) backtrace #0 doadump (textdump=) at pcpu.h:219 #1 0xffffffff808af640 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:447 #2 0xffffffff808afa04 in panic (fmt=) at /usr/src/sys/kern/kern_shutdown.c:754 #3 0xffffffff80c8e7b2 in trap_fatal (frame=, eva=) at /usr/src/sys/amd64/amd64/trap.c:882 #4 0xffffffff80c8ea89 in trap_pfault (frame=0xfffffe18346084c0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:699 #5 0xffffffff80c8e216 in trap (frame=0xfffffe18346084c0) at /usr/src/sys/amd64/amd64/trap.c:463 #6 0xffffffff80c754b2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232 #7 0xffffffff8185a39f in bp_get_dsize_sync (spa=0xfffff80041835000, bp=0xfffffe001b8a1780) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa_misc.c:1635 #8 0xffffffff8182eb5a in dsl_dataset_block_kill (ds=0xfffff800410fec00, bp=0xfffffe001b8a1780, tx=0xfffff8004faa0600, async=0) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:129 #9 0xffffffff8182b967 in dnode_sync (dn=0xfffff8004fe626c0, tx=0xfffff8004faa0600) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:128 #10 0xffffffff81823fcb in dmu_objset_sync_dnodes (list=0xfffff80041956b10, newlist=, tx=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:945 #11 0xffffffff81823e4d in dmu_objset_sync (os=0xfffff80041956800, pio=0xfffff800418c43b0, tx=0xfffff8004faa0600) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1062 #12 0xffffffff8183829a in dsl_pool_sync (dp=0xfffff8004183c000, txg=) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:413 #13 0xffffffff81853a4e in spa_sync (spa=0xfffff80041835000, txg=3373534) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6410 #14 0xffffffff8185c925 in txg_sync_thread (arg=0xfffff8004183c000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:515 #15 0xffffffff80881a9a in fork_exit (callout=0xffffffff8185c5b0 , arg=0xfffff8004183c000, frame=0xfffffe1834608ac0) at /usr/src/sys/kern/kern_fork.c:995 #16 0xffffffff80c759ee in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:606 #17 0x0000000000000000 in ?? () Current language: auto; currently minimal If anyone wants to help out and check out the vmcore file, email me off the list and I'll provide a S3 url of the tar'd + xz file. Thanks, Mike C