From owner-freebsd-fs@FreeBSD.ORG Wed Apr 21 19:50:03 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 87AAC1065679 for ; Wed, 21 Apr 2010 19:50:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 753AE8FC08 for ; Wed, 21 Apr 2010 19:50:03 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o3LJo3r6088990 for ; Wed, 21 Apr 2010 19:50:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o3LJo3Dp088989; Wed, 21 Apr 2010 19:50:03 GMT (envelope-from gnats) Date: Wed, 21 Apr 2010 19:50:03 GMT Message-Id: <201004211950.o3LJo3Dp088989@freefall.freebsd.org> To: freebsd-fs@FreeBSD.org From: Alex Bakhtin Cc: Subject: Re: kern/145339: [zfs] deadlock after detaching block device from raidz pool X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Alex Bakhtin List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Apr 2010 19:50:03 -0000 The following reply was made to PR kern/145339; it has been noted by GNATS. From: Alex Bakhtin To: Andriy Gapon Cc: bug-followup@freebsd.org, Pawel Jakub Dawidek Subject: Re: kern/145339: [zfs] deadlock after detaching block device from raidz pool Date: Wed, 21 Apr 2010 23:42:20 +0400 Andriy, Sorry for delay, gmail put your mail into spam folder. > Are you sure that this is a deadlock? Sorry, the problem description seems to be not 100 percent clear. > If yes, could you please describe what you see in more details. On GENERIC I discovered a deadlock when I detach device from raidz pool if there is intensive writing to the pool. The box responds to pings but doesn't respond to power button (ACPI request ignored). I built kernel with the following config: > cat /sys/amd64/conf/DEBUG include GENERIC ident DEBUG options ALT_BREAK_TO_DEBUGGER options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC options KDB options DDB options INCLUDE_CONFIG_FILE and got this crash. After looking into crashinfo I assumed that it crashes in _mtx_lock_flags because of debugging options (as I can see - there are many asserts in this function) but probably I'm wrong. I checked /mnt/crash directory and discovered that there is a full crash info gathered by sysutils/bsdcrashtar. Probably, this info could help to find the root cause? tar tvzf crash.10.tar.gz drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/ lrwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/machine -> usr/src.old/sys/amd64/include drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/mnt/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/boot/ -rwxr-xr-x 0 root wheel 211 Apr 3 06:13 crash.10/debug.sh -rw-r--r-- 0 root wheel 50 Apr 3 06:13 crash.10/README drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/boot/kernel/ -r-xr-xr-x 0 root wheel 12581947 Apr 3 06:13 crash.10/boot/kernel/kernel -r-xr-xr-x 0 root wheel 44335787 Apr 3 06:13 crash.10/boot/kernel/kernel.symbols -r-xr-xr-x 0 root wheel 1532664 Apr 3 06:13 crash.10/boot/kernel/zfs.ko -r-xr-xr-x 0 root wheel 12693960 Apr 3 06:13 crash.10/boot/kernel/zfs.ko.symbols -r-xr-xr-x 0 root wheel 9832 Apr 3 06:13 crash.10/boot/kernel/opensolaris.ko -r-xr-xr-x 0 root wheel 145808 Apr 3 06:13 crash.10/boot/kernel/opensolaris.ko.symbols -r-xr-xr-x 0 root wheel 146048 Apr 3 06:13 crash.10/boot/kernel/geom_mirror.ko -r-xr-xr-x 0 root wheel 314512 Apr 3 06:13 crash.10/boot/kernel/geom_mirror.ko.symbols drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cam/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/ddb/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/dev/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/fs/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/geom/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/kern/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/modules/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/net/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/nfs/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/nfsserver/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/rpc/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/security/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/sys/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/ufs/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/vm/ -rw-r--r-- 0 root wheel 83274 Apr 3 06:13 crash.10/usr/src.old/sys/vm/uma_core.c -rw-r--r-- 0 root wheel 26799 Apr 3 06:13 crash.10/usr/src.old/sys/vm/vm_glue.c -rw-r--r-- 0 root wheel 104178 Apr 3 06:13 crash.10/usr/src.old/sys/vm/vm_map.c -rw-r--r-- 0 root wheel 45074 Apr 3 06:13 crash.10/usr/src.old/sys/vm/vm_pageout.c -rw-r--r-- 0 root wheel 4978 Apr 3 06:13 crash.10/usr/src.old/sys/vm/vm_zeroidle.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/ufs/ffs/ -rw-r--r-- 0 root wheel 189645 Apr 3 06:13 crash.10/usr/src.old/sys/ufs/ffs/ffs_softdep.c -rw-r--r-- 0 root wheel 18465 Apr 3 06:13 crash.10/usr/src.old/sys/sys/buf.h -rw-r--r-- 0 root wheel 8941 Apr 3 06:13 crash.10/usr/src.old/sys/sys/file.h -rw-r--r-- 0 root wheel 32408 Apr 3 06:13 crash.10/usr/src.old/sys/sys/mbuf.h drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/security/audit/ -rw-r--r-- 0 root wheel 15538 Apr 3 06:13 crash.10/usr/src.old/sys/security/audit/audit_worker.c -rw-r--r-- 0 root wheel 30942 Apr 3 06:13 crash.10/usr/src.old/sys/rpc/svc.c -rw-r--r-- 0 root wheel 13440 Apr 3 06:13 crash.10/usr/src.old/sys/nfsserver/nfs_srvkrpc.c -rw-r--r-- 0 root wheel 4928 Apr 3 06:13 crash.10/usr/src.old/sys/nfs/nfs_nfssvc.c -rw-r--r-- 0 root wheel 45178 Apr 3 06:13 crash.10/usr/src.old/sys/net/flowtable.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/compat/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/ -rw-r--r-- 0 root wheel 129959 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c -rw-r--r-- 0 root wheel 17453 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c -rw-r--r-- 0 root wheel 112698 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c -rw-r--r-- 0 root wheel 15102 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c -rw-r--r-- 0 root wheel 15178 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_geom.c -rw-r--r-- 0 root wheel 120600 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c -rw-r--r-- 0 root wheel 63052 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/compat/opensolaris/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/compat/opensolaris/kern/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/compat/opensolaris/sys/ -rw-r--r-- 0 root wheel 3843 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/compat/opensolaris/sys/atomic.h -rw-r--r-- 0 root wheel 3673 Apr 3 06:13 crash.10/usr/src.old/sys/cddl/compat/opensolaris/kern/opensolaris_taskq.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/modules/zfs/ -rw-r--r-- 0 root wheel 21756 Apr 3 06:13 crash.10/usr/src.old/sys/kern/init_main.c -rw-r--r-- 0 root wheel 11706 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_condvar.c -rw-r--r-- 0 root wheel 24056 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_exit.c -rw-r--r-- 0 root wheel 22111 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_fork.c -rw-r--r-- 0 root wheel 46996 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_intr.c -rw-r--r-- 0 root wheel 25236 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_malloc.c -rw-r--r-- 0 root wheel 23736 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_mutex.c -rw-r--r-- 0 root wheel 79669 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_sig.c -rw-r--r-- 0 root wheel 15720 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_synch.c -rw-r--r-- 0 root wheel 36031 Apr 3 06:13 crash.10/usr/src.old/sys/kern/kern_time.c -rw-r--r-- 0 root wheel 71672 Apr 3 06:13 crash.10/usr/src.old/sys/kern/sched_ule.c -rw-r--r-- 0 root wheel 12187 Apr 3 06:13 crash.10/usr/src.old/sys/kern/subr_kdb.c -rw-r--r-- 0 root wheel 33114 Apr 3 06:13 crash.10/usr/src.old/sys/kern/subr_sleepqueue.c -rw-r--r-- 0 root wheel 10565 Apr 3 06:13 crash.10/usr/src.old/sys/kern/subr_taskqueue.c -rw-r--r-- 0 root wheel 34998 Apr 3 06:13 crash.10/usr/src.old/sys/kern/sys_generic.c -rw-r--r-- 0 root wheel 48706 Apr 3 06:13 crash.10/usr/src.old/sys/kern/tty.c -rw-r--r-- 0 root wheel 28015 Apr 3 06:13 crash.10/usr/src.old/sys/kern/tty_ttydisc.c -rw-r--r-- 0 root wheel 93141 Apr 3 06:13 crash.10/usr/src.old/sys/kern/uipc_socket.c -rw-r--r-- 0 root wheel 109887 Apr 3 06:13 crash.10/usr/src.old/sys/kern/vfs_bio.c -rw-r--r-- 0 root wheel 110274 Apr 3 06:13 crash.10/usr/src.old/sys/kern/vfs_subr.c -rw-r--r-- 0 root wheel 19944 Apr 3 06:13 crash.10/usr/src.old/sys/geom/geom_io.c -rw-r--r-- 0 root wheel 6915 Apr 3 06:13 crash.10/usr/src.old/sys/geom/geom_kern.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/fs/devfs/ -rw-r--r-- 0 root wheel 36833 Apr 3 06:13 crash.10/usr/src.old/sys/fs/devfs/devfs_vnops.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/dev/fdc/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/dev/md/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/dev/random/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/dev/usb/ -rw-r--r-- 0 root wheel 13618 Apr 3 06:13 crash.10/usr/src.old/sys/dev/usb/usb_process.c -rw-r--r-- 0 root wheel 11623 Apr 3 06:13 crash.10/usr/src.old/sys/dev/random/randomdev_soft.c -rw-r--r-- 0 root wheel 31483 Apr 3 06:13 crash.10/usr/src.old/sys/dev/md/md.c -rw-r--r-- 0 root wheel 49685 Apr 3 06:13 crash.10/usr/src.old/sys/dev/fdc/fdc.c -rw-r--r-- 0 root wheel 17269 Apr 3 06:13 crash.10/usr/src.old/sys/ddb/db_command.c -rw-r--r-- 0 root wheel 6005 Apr 3 06:13 crash.10/usr/src.old/sys/ddb/db_main.c -rw-r--r-- 0 root wheel 15802 Apr 3 06:13 crash.10/usr/src.old/sys/ddb/db_script.c -rw-r--r-- 0 root wheel 125529 Apr 3 06:13 crash.10/usr/src.old/sys/cam/cam_xpt.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/amd64/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pc/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/xen/ -rw-r--r-- 0 root wheel 1848 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/_bus.h -rw-r--r-- 0 root wheel 8622 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/_inttypes.h -rw-r--r-- 0 root wheel 4152 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/_limits.h -rw-r--r-- 0 root wheel 5605 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/_stdint.h -rw-r--r-- 0 root wheel 4437 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/_types.h -rw-r--r-- 0 root wheel 3188 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/acpica_machdep.h -rw-r--r-- 0 root wheel 14393 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/apicreg.h -rw-r--r-- 0 root wheel 9118 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/apicvar.h -rw-r--r-- 0 root wheel 3183 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/asm.h -rw-r--r-- 0 root wheel 7814 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/asmacros.h -rw-r--r-- 0 root wheel 16060 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/atomic.h -rw-r--r-- 0 root wheel 33225 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/bus.h -rw-r--r-- 0 root wheel 1558 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/bus_dma.h -rw-r--r-- 0 root wheel 1036 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/clock.h -rw-r--r-- 0 root wheel 2850 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/cpu.h -rw-r--r-- 0 root wheel 14534 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/cpufunc.h -rw-r--r-- 0 root wheel 2218 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/cputypes.h -rw-r--r-- 0 root wheel 3175 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/db_machdep.h -rw-r--r-- 0 root wheel 3938 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/elf.h -rw-r--r-- 0 root wheel 4572 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/endian.h -rw-r--r-- 0 root wheel 1830 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/exec.h -rw-r--r-- 0 root wheel 3135 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/float.h -rw-r--r-- 0 root wheel 2099 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/floatingpoint.h -rw-r--r-- 0 root wheel 3912 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/fpu.h -rw-r--r-- 0 root wheel 2808 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/frame.h -rw-r--r-- 0 root wheel 1867 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/gdb_machdep.h -rw-r--r-- 0 root wheel 8880 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/ieeefp.h -rw-r--r-- 0 root wheel 2951 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/in_cksum.h -rw-r--r-- 0 root wheel 6089 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/intr_machdep.h -rw-r--r-- 0 root wheel 1503 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/iodev.h -rw-r--r-- 0 root wheel 1914 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/kdb.h -rw-r--r-- 0 root wheel 2462 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/legacyvar.h -rw-r--r-- 0 root wheel 1976 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/limits.h -rw-r--r-- 0 root wheel 1898 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/mca.h -rw-r--r-- 0 root wheel 3753 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/md_var.h -rw-r--r-- 0 root wheel 1605 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/memdev.h -rw-r--r-- 0 root wheel 1629 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/metadata.h -rw-r--r-- 0 root wheel 1769 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/minidump.h -rw-r--r-- 0 root wheel 1595 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/mp_watchdog.h -rw-r--r-- 0 root wheel 3994 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/mptable.h -rw-r--r-- 0 root wheel 1787 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/mutex.h -rw-r--r-- 0 root wheel 1879 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/nexusvar.h -rw-r--r-- 0 root wheel 5770 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/param.h -rw-r--r-- 0 root wheel 3397 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pcb.h -rw-r--r-- 0 root wheel 2023 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pci_cfgreg.h -rw-r--r-- 0 root wheel 7786 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pcpu.h -rw-r--r-- 0 root wheel 10731 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pmap.h -rw-r--r-- 0 root wheel 4236 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pmc_mdep.h -rw-r--r-- 0 root wheel 1949 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/ppireg.h -rw-r--r-- 0 root wheel 2939 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/proc.h -rw-r--r-- 0 root wheel 3710 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/psl.h -rw-r--r-- 0 root wheel 6071 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/profile.h -rw-r--r-- 0 root wheel 1791 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/ptrace.h -rw-r--r-- 0 root wheel 4453 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/reg.h -rw-r--r-- 0 root wheel 2342 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/reloc.h -rw-r--r-- 0 root wheel 1991 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/resource.h -rw-r--r-- 0 root wheel 1918 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/runq.h -rw-r--r-- 0 root wheel 10148 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/segments.h -rw-r--r-- 0 root wheel 2237 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/setjmp.h -rw-r--r-- 0 root wheel 2166 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/sf_buf.h -rw-r--r-- 0 root wheel 1979 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/sigframe.h -rw-r--r-- 0 root wheel 3377 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/signal.h -rw-r--r-- 0 root wheel 2314 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/smp.h -rw-r--r-- 0 root wheel 18029 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/specialreg.h -rw-r--r-- 0 root wheel 1454 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/stack.h -rw-r--r-- 0 root wheel 2647 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/stdarg.h -rw-r--r-- 0 root wheel 3068 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/sysarch.h -rw-r--r-- 0 root wheel 2125 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/timerreg.h -rw-r--r-- 0 root wheel 4008 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/trap.h -rw-r--r-- 0 root wheel 3001 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/tss.h -rw-r--r-- 0 root wheel 3326 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/ucontext.h -rw-r--r-- 0 root wheel 3451 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/varargs.h -rw-r--r-- 0 root wheel 2094 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/vm.h -rw-r--r-- 0 root wheel 7181 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/vmparam.h -rw-r--r-- 0 root wheel 10175 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/xen/hypercall.h -rw-r--r-- 0 root wheel 3418 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/xen/synch_bitops.h -rw-r--r-- 0 root wheel 9309 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/xen/xen-os.h -rw-r--r-- 0 root wheel 2728 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/xen/xenfunc.h -rw-r--r-- 0 root wheel 7859 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/xen/xenpmap.h -rw-r--r-- 0 root wheel 3537 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/xen/xenvar.h -rw-r--r-- 0 root wheel 2791 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pc/bios.h -rw-r--r-- 0 root wheel 1013 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/include/pc/display.h -rw-r--r-- 0 root wheel 21900 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/amd64/exception.S -rw-r--r-- 0 root wheel 3094 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/amd64/locore.S -rw-r--r-- 0 root wheel 35899 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/amd64/mp_machdep.c -rw-r--r-- 0 root wheel 127991 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/amd64/pmap.c -rw-r--r-- 0 root wheel 28692 Apr 3 06:13 crash.10/usr/src.old/sys/amd64/amd64/trap.c drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/mnt/obj/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/mnt/crash/ -rw------- 0 root wheel 3331166208 Apr 3 06:13 crash.10/mnt/crash/vmcore.10 -rw------- 0 root wheel 470 Apr 3 06:13 crash.10/mnt/crash/info.10 drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/mnt/obj/usr/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/mnt/obj/usr/src.old/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/mnt/obj/usr/src.old/sys/ drwxr-xr-x 0 root wheel 0 Apr 3 06:13 crash.10/mnt/obj/usr/src.old/sys/DEBUG/ -rw-r--r-- 0 root wheel 92901 Apr 3 06:13 crash.10/mnt/obj/usr/src.old/sys/DEBUG/vnode_if.c > I am asking because to me it seems like a NULL pointer crash: >> Fatal trap 12: page fault while in kernel mode >> cpuid = 1; apic id = 01 >> fault virtual address = 0x48 > > It looks like perhaps zio->io_vd became NULL while an I/O response was traveling > up and vdev_geom_io_intr was not prepared to handle that. > >> _mtx_lock_flags() at _mtx_lock_flags+0x39 >> vdev_geom_io_intr() at vdev_geom_io_intr+0x62 >> g_io_schedule_up() at g_io_schedule_up+0xed >> g_up_procbody() at g_up_procbody+0x6f >> fork_exit() at fork_exit+0x12a >> fork_trampoline() at fork_trampoline+0xe If there is any info I can gather on GENERIC - please let me know. The only crashinfo I have is on debug kernel. Alex Bakhtin