From owner-freebsd-current@FreeBSD.ORG Sun Jul 20 19:20:48 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 57826E31; Sun, 20 Jul 2014 19:20:48 +0000 (UTC) Received: from thebighonker.lerctr.org (thebighonker.lerctr.org [IPv6:2001:470:1f0f:3ad:223:7dff:fe9e:6e8a]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "thebighonker.lerctr.org", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 26093229E; Sun, 20 Jul 2014 19:20:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lerctr.org; s=lerami; h=Message-ID:References:In-Reply-To:Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version; bh=j+F8ia0fTQnOBTnG4StOjA8CP63GfrH5fyR/MZ123Lc=; b=Kvp29JuEuw8tBqnk5IX7UXxQ+KVUxh7+yr904Rw0zLgkG6OgdM1uFtedMuO/6JNZXCa8L2iE2glM+nHZCkR3Efhr8xEN74ih8HReExcuWmEg/rfFRb18ALfxX6R8wR+EBmSUt+GKj+afQwtRcCQUoiTITx4+oBPYDnHKojOCoOo=; Received: from localhost.lerctr.org ([127.0.0.1]:27329 helo=webmail.lerctr.org) by thebighonker.lerctr.org with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.82_1-5b7a7c0-XX (FreeBSD)) (envelope-from ) id 1X8wf6-000Exd-AJ; Sun, 20 Jul 2014 14:20:46 -0500 Received: from 107-128-180-255.lightspeed.austtx.sbcglobal.net ([107.128.180.255]) by webmail.lerctr.org with HTTP (HTTP/1.1 POST); Sun, 20 Jul 2014 14:20:44 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sun, 20 Jul 2014 14:20:44 -0500 From: Larry Rosenman To: Steven Hartland Subject: Re: [ZFS][PANIC] Solaris Assert/zio.c:2548 In-Reply-To: <8D84B82C674B495DBF4951E0EE0D7117@multiplay.co.uk> References: <20140720140350.GA8498@borg.lerctr.org> <8D84B82C674B495DBF4951E0EE0D7117@multiplay.co.uk> Message-ID: <21dbecad27074fe34610bc587e6d0764@thebighonker.lerctr.org> X-Sender: ler@lerctr.org User-Agent: Roundcube Webmail/1.0.1 X-Spam-Score: -2.9 (--) X-LERCTR-Spam-Score: -2.9 (--) X-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.001 X-LERCTR-Spam-Report: SpamScore (-2.9/5.0) ALL_TRUSTED=-1, BAYES_00=-1.9, RP_MATCHES_RCVD=-0.001 Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jul 2014 19:20:48 -0000 On 2014-07-20 14:18, Steven Hartland wrote: > Can you provide the details of the zio which caused the panic? > > Also does any of your pools support trim? No, on the trim. Can you walk me through getting the zio you need? > > Regards > Steve > > ----- Original Message ----- From: "Larry Rosenman" > To: ; > Sent: Sunday, July 20, 2014 3:03 PM > Subject: [ZFS][PANIC] Solaris Assert/zio.c:2548 > > >> Got the following panic overnight (I think while a nightly rsync was >> running): >> >> Dump header from device /dev/gpt/swap0 >> Architecture: amd64 >> Architecture Version: 2 >> Dump Length: 8122101760B (7745 MB) >> Blocksize: 512 >> Dumptime: Sun Jul 20 03:22:18 2014 >> Hostname: borg.lerctr.org >> Magic: FreeBSD Kernel Dump >> Version String: FreeBSD 11.0-CURRENT #50 r268894M: Sat Jul 19 >> 18:06:08 CDT 2014 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER >> Panic String: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), >> file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, >> line: 2874 >> Dump Parity: 763150733 >> Bounds: 5 >> Dump Status: good >> >> >> borg.lerctr.org dumped core - see /var/crash/vmcore.5 >> >> Sun Jul 20 03:28:12 CDT 2014 >> >> FreeBSD borg.lerctr.org 11.0-CURRENT FreeBSD 11.0-CURRENT #50 >> r268894M: Sat Jul 19 18:06:08 CDT 2014 >> root@borg.lerctr.org:/usr/obj/usr/src/sys/VT-LER amd64 >> >> panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: >> 2874 >> >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and >> you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for >> details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> panic: solaris assert: !(zio->io_flags & ZIO_FLAG_DELEGATED), file: >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c, line: >> 2874 >> cpuid = 7 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame >> 0xfffffe100c49f930 >> kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe100c49f9e0 >> vpanic() at vpanic+0x126/frame 0xfffffe100c49fa20 >> panic() at panic+0x43/frame 0xfffffe100c49fa80 >> assfail() at assfail+0x1d/frame 0xfffffe100c49fa90 >> zio_vdev_io_assess() at zio_vdev_io_assess+0x2ed/frame >> 0xfffffe100c49fac0 >> zio_execute() at zio_execute+0x1e9/frame 0xfffffe100c49fb20 >> taskqueue_run_locked() at taskqueue_run_locked+0xf0/frame >> 0xfffffe100c49fb80 >> taskqueue_thread_loop() at taskqueue_thread_loop+0x9b/frame >> 0xfffffe100c49fbb0 >> fork_exit() at fork_exit+0x84/frame 0xfffffe100c49fbf0 >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe100c49fbf0 >> --- trap 0, rip = 0, rsp = 0xfffffe100c49fcb0, rbp = 0 --- >> Uptime: 8h57m17s >> (ada2:ahcich2:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 >> 00 00 >> (ada2:ahcich2:0:0:0): CAM status: Command timeout >> (ada2:ahcich2:0:0:0): Error 5, Retries exhausted >> (ada2:ahcich2:0:0:0): Synchronize cache failed >> Dumping 7745 out of 64463 >> MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91% >> >> Reading symbols from /boot/kernel/linux.ko.symbols...done. >> Loaded symbols for /boot/kernel/linux.ko.symbols >> Reading symbols from /boot/kernel/if_lagg.ko.symbols...done. >> Loaded symbols for /boot/kernel/if_lagg.ko.symbols >> Reading symbols from /boot/kernel/snd_envy24ht.ko.symbols...done. >> Loaded symbols for /boot/kernel/snd_envy24ht.ko.symbols >> Reading symbols from /boot/kernel/snd_spicds.ko.symbols...done. >> Loaded symbols for /boot/kernel/snd_spicds.ko.symbols >> Reading symbols from /boot/kernel/coretemp.ko.symbols...done. >> Loaded symbols for /boot/kernel/coretemp.ko.symbols >> Reading symbols from /boot/kernel/ichsmb.ko.symbols...done. >> Loaded symbols for /boot/kernel/ichsmb.ko.symbols >> Reading symbols from /boot/kernel/smbus.ko.symbols...done. >> Loaded symbols for /boot/kernel/smbus.ko.symbols >> Reading symbols from /boot/kernel/ichwd.ko.symbols...done. >> Loaded symbols for /boot/kernel/ichwd.ko.symbols >> Reading symbols from /boot/kernel/cpuctl.ko.symbols...done. >> Loaded symbols for /boot/kernel/cpuctl.ko.symbols >> Reading symbols from /boot/kernel/crypto.ko.symbols...done. >> Loaded symbols for /boot/kernel/crypto.ko.symbols >> Reading symbols from /boot/kernel/cryptodev.ko.symbols...done. >> Loaded symbols for /boot/kernel/cryptodev.ko.symbols >> Reading symbols from /boot/kernel/dtraceall.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtraceall.ko.symbols >> Reading symbols from /boot/kernel/profile.ko.symbols...done. >> Loaded symbols for /boot/kernel/profile.ko.symbols >> Reading symbols from /boot/kernel/cyclic.ko.symbols...done. >> Loaded symbols for /boot/kernel/cyclic.ko.symbols >> Reading symbols from /boot/kernel/dtrace.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtrace.ko.symbols >> Reading symbols from >> /boot/kernel/systrace_freebsd32.ko.symbols...done. >> Loaded symbols for /boot/kernel/systrace_freebsd32.ko.symbols >> Reading symbols from /boot/kernel/systrace.ko.symbols...done. >> Loaded symbols for /boot/kernel/systrace.ko.symbols >> Reading symbols from /boot/kernel/sdt.ko.symbols...done. >> Loaded symbols for /boot/kernel/sdt.ko.symbols >> Reading symbols from /boot/kernel/lockstat.ko.symbols...done. >> Loaded symbols for /boot/kernel/lockstat.ko.symbols >> Reading symbols from /boot/kernel/fasttrap.ko.symbols...done. >> Loaded symbols for /boot/kernel/fasttrap.ko.symbols >> Reading symbols from /boot/kernel/fbt.ko.symbols...done. >> Loaded symbols for /boot/kernel/fbt.ko.symbols >> Reading symbols from /boot/kernel/dtnfscl.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtnfscl.ko.symbols >> Reading symbols from /boot/kernel/dtmalloc.ko.symbols...done. >> Loaded symbols for /boot/kernel/dtmalloc.ko.symbols >> Reading symbols from /boot/modules/vboxdrv.ko...done. >> Loaded symbols for /boot/modules/vboxdrv.ko >> Reading symbols from /boot/kernel/ipmi.ko.symbols...done. >> Loaded symbols for /boot/kernel/ipmi.ko.symbols >> Reading symbols from /boot/kernel/ipmi_linux.ko.symbols...done. >> Loaded symbols for /boot/kernel/ipmi_linux.ko.symbols >> Reading symbols from /boot/kernel/radeonkms.ko.symbols...done. >> Loaded symbols for /boot/kernel/radeonkms.ko.symbols >> Reading symbols from /boot/kernel/iicbb.ko.symbols...done. >> Loaded symbols for /boot/kernel/iicbb.ko.symbols >> Reading symbols from /boot/kernel/iicbus.ko.symbols...done. >> Loaded symbols for /boot/kernel/iicbus.ko.symbols >> Reading symbols from /boot/kernel/iic.ko.symbols...done. >> Loaded symbols for /boot/kernel/iic.ko.symbols >> Reading symbols from /boot/kernel/drm2.ko.symbols...done. >> Loaded symbols for /boot/kernel/drm2.ko.symbols >> Reading symbols from >> /boot/kernel/radeonkmsfw_R100_cp.ko.symbols...done. >> Loaded symbols for /boot/kernel/radeonkmsfw_R100_cp.ko.symbols >> Reading symbols from /boot/kernel/fdescfs.ko.symbols...done. >> Loaded symbols for /boot/kernel/fdescfs.ko.symbols >> Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. >> Loaded symbols for /boot/kernel/linprocfs.ko.symbols >> Reading symbols from /boot/kernel/uhid.ko.symbols...done. >> Loaded symbols for /boot/kernel/uhid.ko.symbols >> Reading symbols from /boot/modules/vboxnetflt.ko...done. >> Loaded symbols for /boot/modules/vboxnetflt.ko >> Reading symbols from /boot/kernel/netgraph.ko.symbols...done. >> Loaded symbols for /boot/kernel/netgraph.ko.symbols >> Reading symbols from /boot/kernel/ng_ether.ko.symbols...done. >> Loaded symbols for /boot/kernel/ng_ether.ko.symbols >> Reading symbols from /boot/modules/vboxnetadp.ko...done. >> Loaded symbols for /boot/modules/vboxnetadp.ko >> #0 doadump (textdump=1) at pcpu.h:219 >> 219 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) #0 doadump (textdump=1) at pcpu.h:219 >> #1 0xffffffff80a055f7 in kern_reboot (howto=260) >> at /usr/src/sys/kern/kern_shutdown.c:445 >> #2 0xffffffff80a05b35 in vpanic (fmt=, >> ap=) at /usr/src/sys/kern/kern_shutdown.c:744 >> #3 0xffffffff80a05b83 in panic (fmt=0x0) >> at /usr/src/sys/kern/kern_shutdown.c:673 >> #4 0xffffffff8032d05d in assfail (a=, >> f=, l=) >> at >> /usr/src/sys/cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:81 >> #5 0xffffffff8040ad6d in zio_vdev_io_assess (ziop=> out>) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:2874 >> #6 0xffffffff80405dd9 in zio_execute (zio=0xfffff809e449b730) >> at >> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1416 >> #7 0xffffffff80a4de60 in taskqueue_run_locked >> (queue=0xfffff8002255e800) >> at /usr/src/sys/kern/subr_taskqueue.c:356 >> #8 0xffffffff80a4e8db in taskqueue_thread_loop (arg=> out>) >> at /usr/src/sys/kern/subr_taskqueue.c:623 >> #9 0xffffffff809d3cc4 in fork_exit ( >> callout=0xffffffff80a4e840 , >> arg=0xfffff80022611470, frame=0xfffffe100c49fc00) >> at /usr/src/sys/kern/kern_fork.c:977 >> #10 0xffffffff80df5afe in fork_trampoline () >> at /usr/src/sys/amd64/amd64/exception.S:605 >> #11 0x0000000000000000 in ?? () >> Current language: auto; currently minimal >> (kgdb) >> >> >> vmcore is available. >> >> I re-ran the rsync, and no panic. >> >> Ideas? >> >> >> -- Larry Rosenman http://www.lerctr.org/~ler >> Phone: +1 214-642-9640 E-Mail: ler@lerctr.org >> US Mail: 108 Turvey Cove, Hutto, TX 78634-5688 >> _______________________________________________ >> freebsd-current@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to >> "freebsd-current-unsubscribe@freebsd.org" >> -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler@lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688