From owner-freebsd-current@FreeBSD.ORG Thu Oct 3 06:54:39 2013 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 18C96745 for ; Thu, 3 Oct 2013 06:54:39 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 52110279B for ; Thu, 3 Oct 2013 06:54:37 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA25482; Thu, 03 Oct 2013 09:51:14 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1VRckj-000Njm-Va; Thu, 03 Oct 2013 09:51:14 +0300 Message-ID: <524D13AB.2020800@FreeBSD.org> Date: Thu, 03 Oct 2013 09:50:19 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:24.0) Gecko/20100101 Thunderbird/24.0 MIME-Version: 1.0 To: Keith White Subject: Re: ZFS panic with r255937 References: <60850.74.51.61.7.1380496284.squirrel@courriel.site.uottawa.ca> <524C0DB2.5040202@FreeBSD.org> In-Reply-To: X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-current@FreeBSD.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Oct 2013 06:54:39 -0000 on 02/10/2013 20:59 Keith White said the following: > On Wed, 2 Oct 2013, Andriy Gapon wrote: > >> on 30/09/2013 02:11 kwhite@site.uottawa.ca said the following: >>> Sorry, debugging this is *way* beyond me. Any hints, patches to try? >> >> Please share the stack trace. >> >> -- >> Andriy Gapon > > There's now a pr for this panic: kern/182570 > > Here's the stack trace: > > root@freebsd10:/usr/src # kgdb /boot/kernel/kernel /var/crash/vmcore.last > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > panic: solaris assert: dn->dn_maxblkid == 0 && > (BP_IS_HOLE(&dn->dn_phys->dn_blkptr[0]) || dnode_block_freed(dn, 0)), file: > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c, > line: 598 > cpuid = 1 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00992b3280 > kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe00992b3330 > vpanic() at vpanic+0x126/frame 0xfffffe00992b3370 > panic() at panic+0x43/frame 0xfffffe00992b33d0 > assfail() at assfail+0x22/frame 0xfffffe00992b33e0 > dnode_reallocate() at dnode_reallocate+0x225/frame 0xfffffe00992b3430 > dmu_object_reclaim() at dmu_object_reclaim+0x123/frame 0xfffffe00992b3480 > dmu_recv_stream() at dmu_recv_stream+0xd79/frame 0xfffffe00992b36b0 > zfs_ioc_recv() at zfs_ioc_recv+0x96c/frame 0xfffffe00992b3920 > zfsdev_ioctl() at zfsdev_ioctl+0x54a/frame 0xfffffe00992b39c0 > devfs_ioctl_f() at devfs_ioctl_f+0xf0/frame 0xfffffe00992b3a20 > kern_ioctl() at kern_ioctl+0x2ca/frame 0xfffffe00992b3a90 > sys_ioctl() at sys_ioctl+0x11f/frame 0xfffffe00992b3ae0 > amd64_syscall() at amd64_syscall+0x265/frame 0xfffffe00992b3bf0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe00992b3bf0 Thank you very much. To me this looks very similar to a problem discovered and fixed in illumos some time ago. Please check if the following change improves the situation for you. https://github.com/avg-I/freebsd/commit/a7e7dece215bc5d00077e9c7f4db34d9e5c30659 Raw: https://github.com/avg-I/freebsd/commit/a7e7dece215bc5d00077e9c7f4db34d9e5c30659.patch > --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x8019dc9ca, rsp = > 0x7fffffff5ad8, rbp = 0x7fffffff5b60 --- > KDB: enter: panic > Uptime: 37m10s > Dumping 874 out of 2023 MB: > > ... > > (kgdb) where > #0 doadump (textdump=1) at pcpu.h:218 > #1 0xffffffff808b7217 in kern_reboot (howto=260) at > /usr/src/sys/kern/kern_shutdown.c:447 > #2 0xffffffff808b7725 in vpanic (fmt=, ap= out>) at /usr/src/sys/kern/kern_shutdown.c:754 > #3 0xffffffff808b7773 in panic (fmt=) at > /usr/src/sys/kern/kern_shutdown.c:683 > #4 0xffffffff81e52222 in assfail (a=, f= out>, l=) at > /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 > > #5 0xffffffff81d09735 in dnode_reallocate (dn=0xfffff8006dde3000, > ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, bonustype=DMU_OT_SA, > bonuslen=168, tx=0xfffff8006d7a2600) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:596 > > #6 0xffffffff81cff463 in dmu_object_reclaim (os=, > object=, ot=DMU_OT_PLAIN_FILE_CONTENTS, blocksize=1024, > bonustype=DMU_OT_SA, bonuslen=168) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_object.c:155 > > #7 0xffffffff81cfd849 in dmu_recv_stream (drc=0xfffffe00992b3728, fp= optimized out>, voffp=0xfffffe00992b3718, cleanup_fd=8, action_handlep= optimized out>) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_send.c:1231 > > #8 0xffffffff81d8b1fc in zfs_ioc_recv (zc=0xfffffe00372e1000) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:4102 > > #9 0xffffffff81d864ea in zfsdev_ioctl (dev=, zcmd= optimized out>, arg=, flag=, td= optimized out>) > at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:5960 > > #10 0xffffffff807af840 in devfs_ioctl_f (fp=0xfffff800052e1460, com=3222821403, > data=0xfffff8006ff1faa0, cred=, td=0xfffff8003f7ba920) at > /usr/src/sys/fs/devfs/devfs_vnops.c:757 > #11 0xffffffff8090e94a in kern_ioctl (td=0xfffff8003f7ba920, fd= out>, com=0) at file.h:319 > #12 0xffffffff8090e62f in sys_ioctl (td=0xfffff8003f7ba920, > uap=0xfffffe00992b3b80) at /usr/src/sys/kern/sys_generic.c:698 > #13 0xffffffff80caee35 in amd64_syscall (td=0xfffff8003f7ba920, traced=0) at > subr_syscall.c:134 > #14 0xffffffff80c961ab in Xfast_syscall () at > /usr/src/sys/amd64/amd64/exception.S:391 > #15 0x00000008019dc9ca in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) > > > Here's a how to repeat: > > Assuming pool "tank" and non-existent tank/nobj tank/xobj > > === cut here === > #!/bin/sh > > zfs create tank/nobj > zfs snapshot tank/nobj@0 > __MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 > buildkernel > zfs snapshot tank/nobj@1 > #find /tank/nobj -name '.h' -print | xargs rm > zfs snapshot tank/nobj@2 > #rm -rf /tank/nobj/* > zfs snapshot tank/nobj@3 > __MAKECONF=/dev/null SRCCONF=/dev/null MAKEOBJDIRPREFIX=/tank/nobj make -j6 > buildkernel > zfs snapshot tank/nobj@4 > #rm -rf /tank/nobj/* > zfs snapshot tank/nobj@5 > > zfs send -vR tank/nobj@5 | zfs recv -vF tank/xobj > > === cut here === > > The commented out lines are historical attempts. > > ...keith -- Andriy Gapon