From owner-freebsd-fs@FreeBSD.ORG Sun Jul 20 14:04:05 2014 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EBA20B36; Sun, 20 Jul 2014 14:04:04 +0000 (UTC) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C2A5A2630; Sun, 20 Jul 2014 14:04:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Content-Type:MIME-Version:Message-ID:Subject:To:From:Date; bh=900LGaysOB2Y78iPGwks4OhbRxCvIzb8vQzUlwyExnY=; b=mSw4lN8GRSG6L+0WPBH6gu+bAHzhZAPQ2/KJvZpcYQlUgqnLQdY7IpW1m2Sus0wyG0sLtnsLFBDeKMZsr60v8R9RIu9UDBbxrOyTrfJxklS3qO3si9TbS9L2sxG5qjCw0C1aJFKWV0eYgyigjxEh0PZb5eKSJN1UEJe8E2WuTJE=; Received: from 107-128-180-255.lightspeed.austtx.sbcglobal.net ([107.128.180.255]:39578 helo=borg.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1.2:DHE-RSA-AES256-GCM-SHA384:256) (Exim 4.82_1-5b7a7c0-XX (FreeBSD)) (envelope-from ) id 1X8ric-000ANz-35; Sun, 20 Jul 2014 09:04:04 -0500 Date: Sun, 20 Jul 2014 09:03:50 -0500 From: Larry Rosenman To: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: [ZFS][PANIC] Solaris Assert/zio.c:2548 Message-ID: <20140720140350.GA8498@borg.lerctr.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Score: -2.9 (--) X-LERCTR-Spam-Score: -2.9 (--) X-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, TVD_RCVD_IP=0.001 X-LERCTR-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, TVD_RCVD_IP=0.001 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 14:04:05 -0000 Got the following panic overnight (I think while a nightly rsync was running): Dump header from device /dev/gpt/swap0 Architecture: amd64 Architecture Version: 2 Dump Length: 8122101760B (7745 MB) Blocksize: 512 Dumptime: Sun Jul 20 03:22:18 2014 Hostname: borg.lerctr.org Magic: FreeBSD Kernel Dump Version String: FreeBSD 11.0-CURRENT #50 r268894M: Sat Jul 19 18:06:08 CDT 2014 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER Panic String: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2874 Dump Parity: 763150733 Bounds: 5 Dump Status: good borg.lerctr.org dumped core - see /var/crash/vmcore.5 Sun Jul 20 03:28:12 CDT 2014 FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #50 r268894M: Sat Jul 19 18:06:08 CDT 2014 root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2874 GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: 2874 cpuid = 7 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe100c49f930 kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe100c49f9e0 vpanic() at vpanic+0x126/frame 0xfffffe100c49fa20 panic() at panic+0x43/frame 0xfffffe100c49fa80 assfail() at assfail+0x1d/frame 0xfffffe100c49fa90 zio_vdev_io_assess() at zio_vdev_io_assess+0x2ed/frame 0xfffffe100c49fac0 zio_execute() at zio_execute+0x1e9/frame 0xfffffe100c49fb20 taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame 0xfffffe100c49fb80 taskqueue_thread_loop() at taskqueue_thread_loop+0x9b/frame 0xfffffe100c49fbb0 fork_exit() at fork_exit+0x84/frame 0xfffffe100c49fbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe100c49fbf0 --- trap 0, rip = 0, rsp = 0xfffffe100c49fcb0, rbp = 0 --- Uptime: 8h57m17s (ada2:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00 (ada2:ahcich2:0:0:0): CAM status: Command timeout (ada2:ahcich2:0:0:0): Error 5, Retries exhausted (ada2:ahcich2:0:0:0): Synchronize cache failed Dumping 7745 out of 64463 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% Reading symbols from /boot/kernel/linux.ko.symbols...done. Loaded symbols for /boot/kernel/linux.ko.symbols Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. Loaded symbols for /boot/kernel/if_lagg.ko.symbols Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. Loaded symbols for /boot/kernel/snd_spicds.ko.symbols Reading symbols from /boot/kernel/coretemp.ko.symbols...done. Loaded symbols for /boot/kernel/coretemp.ko.symbols Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. Loaded symbols for /boot/kernel/ichsmb.ko.symbols Reading symbols from /boot/kernel/smbus.ko.symbols...done. Loaded symbols for /boot/kernel/smbus.ko.symbols Reading symbols from /boot/kernel/ichwd.ko.symbols...done. Loaded symbols for /boot/kernel/ichwd.ko.symbols Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. Loaded symbols for /boot/kernel/cpuctl.ko.symbols Reading symbols from /boot/kernel/crypto.ko.symbols...done. Loaded symbols for /boot/kernel/crypto.ko.symbols Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. Loaded symbols for /boot/kernel/cryptodev.ko.symbols Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. Loaded symbols for /boot/kernel/dtraceall.ko.symbols Reading symbols from /boot/kernel/profile.ko.symbols...done. Loaded symbols for /boot/kernel/profile.ko.symbols Reading symbols from /boot/kernel/cyclic.ko.symbols...done. Loaded symbols for /boot/kernel/cyclic.ko.symbols Reading symbols from /boot/kernel/dtrace.ko.symbols...done. Loaded symbols for /boot/kernel/dtrace.ko.symbols Reading symbols from /boot/kernel/systrace_freebsd32.ko.symbols...done. Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols Reading symbols from /boot/kernel/systrace.ko.symbols...done. Loaded symbols for /boot/kernel/systrace.ko.symbols Reading symbols from /boot/kernel/sdt.ko.symbols...done. Loaded symbols for /boot/kernel/sdt.ko.symbols Reading symbols from /boot/kernel/lockstat.ko.symbols...done. Loaded symbols for /boot/kernel/lockstat.ko.symbols Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. Loaded symbols for /boot/kernel/fasttrap.ko.symbols Reading symbols from /boot/kernel/fbt.ko.symbols...done. Loaded symbols for /boot/kernel/fbt.ko.symbols Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. Loaded symbols for /boot/kernel/dtnfscl.ko.symbols Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. Loaded symbols for /boot/kernel/dtmalloc.ko.symbols Reading symbols from /boot/modules/vboxdrv.ko...done. Loaded symbols for /boot/modules/vboxdrv.ko Reading symbols from /boot/kernel/ipmi.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi.ko.symbols Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. Loaded symbols for /boot/kernel/radeonkms.ko.symbols Reading symbols from /boot/kernel/iicbb.ko.symbols...done. Loaded symbols for /boot/kernel/iicbb.ko.symbols Reading symbols from /boot/kernel/iicbus.ko.symbols...done. Loaded symbols for /boot/kernel/iicbus.ko.symbols Reading symbols from /boot/kernel/iic.ko.symbols...done. Loaded symbols for /boot/kernel/iic.ko.symbols Reading symbols from /boot/kernel/drm2.ko.symbols...done. Loaded symbols for /boot/kernel/drm2.ko.symbols Reading symbols from /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. Loaded symbols for /boot/kernel/fdescfs.ko.symbols Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. Loaded symbols for /boot/kernel/linprocfs.ko.symbols Reading symbols from /boot/kernel/uhid.ko.symbols...done. Loaded symbols for /boot/kernel/uhid.ko.symbols Reading symbols from /boot/modules/vboxnetflt.ko...done. Loaded symbols for /boot/modules/vboxnetflt.ko Reading symbols from /boot/kernel/netgraph.ko.symbols...done. Loaded symbols for /boot/kernel/netgraph.ko.symbols Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. Loaded symbols for /boot/kernel/ng_ether.ko.symbols Reading symbols from /boot/modules/vboxnetadp.ko...done. Loaded symbols for /boot/modules/vboxnetadp.ko #0 doadump (textdump=1) at pcpu.h:219 219 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=1) at pcpu.h:219 #1 0xffffffff80a055f7 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:445 #2 0xffffffff80a05b35 in vpanic (fmt=, ap=) at /usr/src/sys/kern/kern_shutdown.c:744 #3 0xffffffff80a05b83 in panic (fmt=0x0) at /usr/src/sys/kern/kern_shutdown.c:673 #4 0xffffffff8032d05d in assfail (a=, f=, l=) at /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 #5 0xffffffff8040ad6d in zio_vdev_io_assess (ziop=) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2874 #6 0xffffffff80405dd9 in zio_execute (zio=0xfffff809e449b730) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1416 #7 0xffffffff80a4de60 in taskqueue_run_locked (queue=0xfffff8002255e800) at /usr/src/sys/kern/subr_taskqueue.c:356 #8 0xffffffff80a4e8db in taskqueue_thread_loop (arg=) at /usr/src/sys/kern/subr_taskqueue.c:623 #9 0xffffffff809d3cc4 in fork_exit ( callout=0xffffffff80a4e840 , arg=0xfffff80022611470, frame=0xfffffe100c49fc00) at /usr/src/sys/kern/kern_fork.c:977 #10 0xffffffff80df5afe in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:605 #11 0x0000000000000000 in ?? () Current language: auto; currently minimal (kgdb) vmcore is available. I re-ran the rsync, and no panic. Ideas? -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688