From owner-freebsd-stable@FreeBSD.ORG Thu Dec 8 11:55:13 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E7D06106564A for ; Thu, 8 Dec 2011 11:55:12 +0000 (UTC) (envelope-from marck@rinet.ru) Received: from woozle.rinet.ru (woozle.rinet.ru [195.54.192.68]) by mx1.freebsd.org (Postfix) with ESMTP id 4B7898FC0A for ; Thu, 8 Dec 2011 11:55:11 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by woozle.rinet.ru (8.14.4/8.14.4) with ESMTP id pB8BtAWM064061; Thu, 8 Dec 2011 15:55:10 +0400 (MSK) (envelope-from marck@rinet.ru) Date: Thu, 8 Dec 2011 15:55:10 +0400 (MSK) From: Dmitry Morozovsky To: Kostik Belousov In-Reply-To: Message-ID: References: <20111207133035.GF50300@deviant.kiev.zoral.com.ua> <20111207182030.GI50300@deviant.kiev.zoral.com.ua> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) X-NCC-RegID: ru.rinet X-OpenPGP-Key-ID: 6B691B03 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (woozle.rinet.ru [0.0.0.0]); Thu, 08 Dec 2011 15:55:10 +0400 (MSK) Cc: freebsd-stable@freebsd.org Subject: Re: tmpfs deadlock on stable/9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2011 11:55:13 -0000 On Thu, 8 Dec 2011, Dmitry Morozovsky wrote: > > > > It is available at http://bsd.woozle.net/tmpfs-lock-20111207.txt (~260k) > > > > > > > > BTW, at least some of the debugger commands referenced (show locks, show > > > > alllocks) are no longer exist > > > This means that you do not have witness in your kernel. > > > Look at the reference I pointed you once more. > > > > Ah I see, my bad. I'll test this with WITNESS-enabled kernel tomorrow and > > return with updated results. > > Hmm, I'd rebuilt kernel with WITNESS/WITNESS_SKIPSPIN and can't reproduce this > deadlock. Will try further. On the other hand, after boot I see several LORs: Dec 8 02:34:18 beaver kernel: lock order reversal: Dec 8 02:34:18 beaver kernel: 1st 0xfffffe005163b818 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:829 Dec 8 02:34:18 beaver kernel: 2nd 0xfffffe005163cbd8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2166 Dec 8 02:34:18 beaver kernel: KDB: stack backtrace: Dec 8 02:34:18 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 02:34:18 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 02:34:18 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 02:34:18 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 02:34:18 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 02:34:18 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 02:34:18 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:34:18 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 02:34:18 beaver kernel: vget() at vget+0x56 Dec 8 02:34:18 beaver kernel: devfs_allocv() at devfs_allocv+0x13f Dec 8 02:34:18 beaver kernel: devfs_root() at devfs_root+0x4d Dec 8 02:34:18 beaver kernel: vfs_donmount() at vfs_donmount+0xdcb Dec 8 02:34:18 beaver kernel: sys_nmount() at sys_nmount+0x63 Dec 8 02:34:18 beaver kernel: amd64_syscall() at amd64_syscall+0x25a Dec 8 02:34:18 beaver kernel: Xfast_syscall() at Xfast_syscall+0xf7 Dec 8 02:34:18 beaver kernel: --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800ab5ecc, rsp = 0x7fffffffccc8, rbp = 0x801009048 --- Dec 8 02:34:29 beaver kernel: 4-i386.lab.rinet.ru Dec 8 02:34:32 beaver kernel: 6-i386.lab.rinet.ru. Dec 8 02:41:15 beaver kernel: lock order reversal: Dec 8 02:41:15 beaver kernel: 1st 0xfffffe004ea069f8 tmpfs (tmpfs) @ /usr/src/sys/fs/nullfs/null_vnops.c:597 Dec 8 02:41:15 beaver kernel: 2nd 0xfffffe019db46bd8 zfs (zfs) @ /usr/src/sys/fs/nullfs/null_vnops.c:597 Dec 8 02:41:15 beaver kernel: KDB: stack backtrace: Dec 8 02:41:15 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 02:41:15 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 02:41:15 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 02:41:15 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 02:41:15 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 02:41:15 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 02:41:15 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:41:15 beaver kernel: null_lock() at null_lock+0xbb Dec 8 02:41:15 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:41:15 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 02:41:15 beaver kernel: nullfs_root() at nullfs_root+0x45 Dec 8 02:41:15 beaver kernel: vfs_donmount() at vfs_donmount+0xdcb Dec 8 02:41:15 beaver kernel: sys_nmount() at sys_nmount+0x63 Dec 8 02:41:15 beaver kernel: amd64_syscall() at amd64_syscall+0x25a Dec 8 02:41:15 beaver kernel: Xfast_syscall() at Xfast_syscall+0xf7 Dec 8 02:41:15 beaver kernel: --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x8008a1ecc, rsp = 0x7fffffffceb8, rbp = 0x7fffffffcf30 --- Dec 8 02:41:16 beaver kernel: lock order reversal: Dec 8 02:41:16 beaver kernel: 1st 0xfffffe0171868638 tmpfs (tmpfs) @ /usr/src/sys/fs/nullfs/null_vnops.c:597 Dec 8 02:41:16 beaver kernel: 2nd 0xfffffe004ea0ebd8 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:2166 Dec 8 02:41:16 beaver kernel: KDB: stack backtrace: Dec 8 02:41:16 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 02:41:16 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 02:41:16 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 02:41:16 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 02:41:16 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 02:41:16 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 02:41:16 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:41:16 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 02:41:16 beaver kernel: vget() at vget+0x56 Dec 8 02:41:16 beaver kernel: devfs_allocv() at devfs_allocv+0x13f Dec 8 02:41:16 beaver kernel: devfs_root() at devfs_root+0x4d Dec 8 02:41:16 beaver kernel: vfs_donmount() at vfs_donmount+0xdcb Dec 8 02:41:16 beaver kernel: sys_nmount() at sys_nmount+0x63 Dec 8 02:41:16 beaver kernel: amd64_syscall() at amd64_syscall+0x25a Dec 8 02:41:16 beaver kernel: Xfast_syscall() at Xfast_syscall+0xf7 Dec 8 02:41:16 beaver kernel: --- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x800ab5ecc, rsp = 0x7fffffffc738, rbp = 0x801009048 --- Dec 8 02:41:34 beaver kernel: procfs registered Dec 8 02:41:34 beaver kernel: linprocfs registered Dec 8 02:47:22 beaver kernel: lock order reversal: Dec 8 02:47:22 beaver kernel: 1st 0xfffffe000f17e458 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:1766 Dec 8 02:47:22 beaver kernel: 2nd 0xfffffe01aa73d458 zfs (zfs) @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:138 Dec 8 02:47:22 beaver kernel: KDB: stack backtrace: Dec 8 02:47:22 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 02:47:22 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 02:47:22 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 02:47:22 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 02:47:22 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 02:47:22 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 02:47:22 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 02:47:22 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 02:47:22 beaver kernel: zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x57 Dec 8 02:47:22 beaver kernel: zfs_znode_alloc() at zfs_znode_alloc+0x4c Dec 8 02:47:22 beaver kernel: zfs_zget() at zfs_zget+0x2af Dec 8 02:47:22 beaver kernel: zfs_get_data() at zfs_get_data+0x4a Dec 8 02:47:22 beaver kernel: zil_commit() at zil_commit+0x541 Dec 8 02:47:22 beaver kernel: zfs_sync() at zfs_sync+0xb3 Dec 8 02:47:22 beaver kernel: sync_fsync() at sync_fsync+0x13a Dec 8 02:47:22 beaver kernel: sync_vnode() at sync_vnode+0x157 Dec 8 02:47:22 beaver kernel: sched_sync() at sched_sync+0x1b9 Dec 8 02:47:22 beaver kernel: fork_exit() at fork_exit+0x11d Dec 8 02:47:22 beaver kernel: fork_trampoline() at fork_trampoline+0xe Dec 8 02:47:22 beaver kernel: --- trap 0, rip = 0, rsp = 0xffffff8205430d00, rbp = 0 --- Dec 8 02:56:30 beaver kernel: pid 75253 (conftest), uid 0: exited on signal 11 (core dumped) Dec 8 07:30:01 beaver kernel: lock order reversal: Dec 8 07:30:01 beaver kernel: 1st 0xfffffe0009f90818 syncer (syncer) @ /usr/src/sys/kern/vfs_subr.c:1766 Dec 8 07:30:01 beaver kernel: 2nd 0xfffffe0208f249f8 tmpfs (tmpfs) @ /usr/src/sys/kern/vfs_subr.c:2166 Dec 8 07:30:01 beaver kernel: KDB: stack backtrace: Dec 8 07:30:01 beaver kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Dec 8 07:30:01 beaver kernel: kdb_backtrace() at kdb_backtrace+0x37 Dec 8 07:30:01 beaver kernel: _witness_debugger() at _witness_debugger+0x2c Dec 8 07:30:01 beaver kernel: witness_checkorder() at witness_checkorder+0x651 Dec 8 07:30:01 beaver kernel: __lockmgr_args() at __lockmgr_args+0xb98 Dec 8 07:30:01 beaver kernel: vop_stdlock() at vop_stdlock+0x39 Dec 8 07:30:01 beaver kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 Dec 8 07:30:01 beaver kernel: _vn_lock() at _vn_lock+0x47 Dec 8 07:30:01 beaver kernel: vget() at vget+0x56 Dec 8 07:30:01 beaver kernel: vfs_msync() at vfs_msync+0xa5 Dec 8 07:30:01 beaver kernel: sync_fsync() at sync_fsync+0x12a Dec 8 07:30:01 beaver kernel: sync_vnode() at sync_vnode+0x157 Dec 8 07:30:01 beaver kernel: sched_sync() at sched_sync+0x1b9 Dec 8 07:30:01 beaver kernel: fork_exit() at fork_exit+0x11d Dec 8 07:30:01 beaver kernel: fork_trampoline() at fork_trampoline+0xe Dec 8 07:30:01 beaver kernel: --- trap 0, rip = 0, rsp = 0xffffff8205430d00, rbp = 0 --- -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------