From owner-freebsd-fs@FreeBSD.ORG Mon Nov 2 11:02:04 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 784ED1065676 for ; Mon, 2 Nov 2009 11:02:04 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop2.sarenet.es (proxypop2.sarenet.es [194.30.0.95]) by mx1.freebsd.org (Postfix) with ESMTP id 4F1388FC24 for ; Mon, 2 Nov 2009 11:02:02 +0000 (UTC) Received: from [172.16.1.204] (izaro.sarenet.es [192.148.167.11]) by proxypop2.sarenet.es (Postfix) with ESMTP id E99A473124 for ; Mon, 2 Nov 2009 12:02:01 +0100 (CET) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Mime-Version: 1.0 (Apple Message framework v1076) From: Borja Marcos In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl> Date: Mon, 2 Nov 2009 12:02:01 +0100 Content-Transfer-Encoding: 7bit Message-Id: References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es> <4AEA0EAD.1050302@memberwebs.com> <20091030232805.GA2996@garage.freebsd.pl> To: freebsd-fs@freebsd.org X-Mailer: Apple Mail (2.1076) Subject: Re: 8.0-RC2: ZFS deadlock with zfs receive X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 11:02:04 -0000 (Resending to the list, I replied only to Pawel) On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote: > On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote: >> Borja Marcos wrote: >>> I've been sending some alltraces to pjd about this easy to reproduce >>> problem. I am using zfs send/zfs receive to replicate a dataset >>> from one >>> server to another. At 1 minute intervals, an incremental snapshot is >>> sent to update the dataset copy. If there is reading activity on the >>> dataset copy, a deadlock can happen rendering ZFS and all the FS >>> subsystem unusable. I've tried with 8.0RC2 and it still happens. > > I was able to reproduce it, but I don't have fix yet. > >> FWIW, another (or the same) zfs recv deadlock I've been trying to >> get to >> the bottom of: >> >> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/ >> 006999.html > > Could you guys recompile your kernel after uncommenting line: > > #CFLAGS+=-DDEBUG=1 > > in sys/modules/zfs/Makefile? > > You should see panic on assertion instead of deadlock, I believe. No panic, it just deadlocked. Last trace of LORs Nov 2 13:45:44 kernel: lock order reversal: Nov 2 13:45:44 kernel: 1st 0xffffff00025d4c38 ds- >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ bplist.c:189 Nov 2 13:45:44 kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi- >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dnode.c:705 Nov 2 13:45:44 kernel: KDB: stack backtrace: Nov 2 13:45:44 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:45:44 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:45:44 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:45:44 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:45:44 kernel: dnode_setdirty() at dnode_setdirty+0xbc Nov 2 13:45:44 kernel: dbuf_dirty() at dbuf_dirty+0x516 Nov 2 13:45:44 kernel: bplist_enqueue() at bplist_enqueue+0xbd Nov 2 13:45:44 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 Nov 2 13:45:44 kernel: dmu_objset_sync() at dmu_objset_sync+0x1fe Nov 2 13:45:44 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:45:44 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:45:44 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:45:44 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:45:44 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:45:44 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:45:44 kernel: lock order reversal: Nov 2 13:45:44 kernel: 1st 0xffffff0002eb9d38 dr->dt.di.dr_mtx (dr- >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1905 Nov 2 13:45:44 kernel: 2nd 0xffffff0002ed3000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dbuf.c:543 Nov 2 13:45:44 kernel: KDB: stack backtrace: Nov 2 13:45:44 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:45:44 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:45:44 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:45:44 kernel: _sx_slock() at _sx_slock+0x55 Nov 2 13:45:44 kernel: dbuf_read() at dbuf_read+0x2ad Nov 2 13:45:44 kernel: dbuf_will_dirty() at dbuf_will_dirty+0x53 Nov 2 13:45:44 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0xe9 Nov 2 13:45:44 kernel: dbuf_write() at dbuf_write+0x24c Nov 2 13:45:44 kernel: dbuf_sync_list() at dbuf_sync_list+0x159 Nov 2 13:45:44 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f Nov 2 13:45:44 kernel: dnode_sync() at dnode_sync+0xc12 Nov 2 13:45:44 kernel: dmu_objset_sync() at dmu_objset_sync+0x134 Nov 2 13:45:44 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:45:44 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:45:44 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:45:44 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:45:44 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:45:44 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:03 kernel: lock order reversal: Nov 2 13:46:03 kernel: 1st 0xffffff00025d4c38 ds- >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ bplist.c:152 Nov 2 13:46:03 kernel: 2nd 0xffffff0002f01ae0 dn->dn_dbufs_mtx (dn- >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1518 Nov 2 13:46:03 kernel: KDB: stack backtrace: Nov 2 13:46:03 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:03 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:03 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:03 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:03 kernel: dbuf_destroy() at dbuf_destroy+0x58 Nov 2 13:46:03 kernel: bplist_cache() at bplist_cache+0x2e Nov 2 13:46:03 kernel: bplist_iterate() at bplist_iterate+0xb3 Nov 2 13:46:03 kernel: bplist_space_birthrange() at bplist_space_birthrange+0x60 Nov 2 13:46:03 kernel: dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee Nov 2 13:46:03 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:03 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:03 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:03 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:03 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:03 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:03 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:03 kernel: lock order reversal: Nov 2 13:46:03 kernel: 1st 0xffffff00025d4c38 ds- >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ bplist.c:152 Nov 2 13:46:03 kernel: 2nd 0xffffffff81121d50 h->hash_mutexes[i] (h- >hash_mutexes[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dbuf.c:191 Nov 2 13:46:03 kernel: KDB: stack backtrace: Nov 2 13:46:03 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:03 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:03 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:03 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:03 kernel: dbuf_destroy() at dbuf_destroy+0x111 Nov 2 13:46:03 kernel: bplist_cache() at bplist_cache+0x2e Nov 2 13:46:03 kernel: bplist_iterate() at bplist_iterate+0xb3 Nov 2 13:46:03 kernel: bplist_space_birthrange() at bplist_space_birthrange+0x60 Nov 2 13:46:03 kernel: dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee Nov 2 13:46:03 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:03 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:03 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:03 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:03 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:03 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:03 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:35 kernel: lock order reversal: Nov 2 13:46:35 kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock (dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171 Nov 2 13:46:35 kernel: 2nd 0xffffff0002784938 ds->ds_opening_lock (ds->ds_opening_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:483 Nov 2 13:46:35 kernel: KDB: stack backtrace: Nov 2 13:46:35 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:35 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:35 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:35 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:35 kernel: dmu_objset_create_impl() at dmu_objset_create_impl+0x50 Nov 2 13:46:35 kernel: dmu_objset_create_sync() at dmu_objset_create_sync+0xfc Nov 2 13:46:35 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:35 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:35 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:35 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:35 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:35 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:35 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:35 kernel: lock order reversal: Nov 2 13:46:35 kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock (dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171 Nov 2 13:46:35 kernel: 2nd 0xffffff0002fb7578 zfs (zfs) @ /pool/ newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ fs/zfs/zfs_znode.c:152 Nov 2 13:46:35 kernel: KDB: stack backtrace: Nov 2 13:46:35 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:35 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:35 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:35 kernel: __lockmgr_args() at __lockmgr_args+0xd03 Nov 2 13:46:35 kernel: vop_stdlock() at vop_stdlock+0x39 Nov 2 13:46:35 kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b Nov 2 13:46:35 kernel: _vn_lock() at _vn_lock+0x57 Nov 2 13:46:35 kernel: zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x65 Nov 2 13:46:35 kernel: zfs_create_fs() at zfs_create_fs+0x2ab Nov 2 13:46:35 kernel: dmu_objset_create_sync() at dmu_objset_create_sync+0x116 Nov 2 13:46:35 kernel: dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 Nov 2 13:46:35 kernel: dsl_pool_sync() at dsl_pool_sync+0x122 Nov 2 13:46:35 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:35 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:35 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:35 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:35 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:36 kernel: lock order reversal: Nov 2 13:46:36 kernel: 1st 0xffffff001a0a7430 db->db_mtx (db- >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1724 Nov 2 13:46:36 kernel: 2nd 0xffffff001a4c0708 dn->dn_dbufs_mtx (dn- >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dnode_sync.c:373 Nov 2 13:46:36 kernel: KDB: stack backtrace: Nov 2 13:46:36 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:36 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:36 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:36 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:36 kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x57 Nov 2 13:46:36 kernel: dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0xd4 Nov 2 13:46:36 kernel: dmu_objset_evict() at dmu_objset_evict+0xbf Nov 2 13:46:36 kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54 Nov 2 13:46:36 kernel: dbuf_evict_user() at dbuf_evict_user+0x55 Nov 2 13:46:36 kernel: dbuf_rele() at dbuf_rele+0x173 Nov 2 13:46:36 kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c Nov 2 13:46:36 kernel: spa_sync() at spa_sync+0x618 Nov 2 13:46:36 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:36 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:36 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:36 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:36 kernel: lock order reversal: Nov 2 13:46:36 kernel: 1st 0xffffff001a0a7430 db->db_mtx (db- >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1724 Nov 2 13:46:36 kernel: 2nd 0xffffff001a4c03d8 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:409 Nov 2 13:46:36 kernel: KDB: stack backtrace: Nov 2 13:46:36 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:36 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:36 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:36 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:36 kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x1b8 Nov 2 13:46:36 kernel: dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0xd4 Nov 2 13:46:36 kernel: dmu_objset_evict() at dmu_objset_evict+0xbf Nov 2 13:46:36 kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54 Nov 2 13:46:36 kernel: dbuf_evict_user() at dbuf_evict_user+0x55 Nov 2 13:46:36 kernel: dbuf_rele() at dbuf_rele+0x173 Nov 2 13:46:36 kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c Nov 2 13:46:36 kernel: spa_sync() at spa_sync+0x618 Nov 2 13:46:36 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:36 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:36 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:36 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:57 kernel: lock order reversal: Nov 2 13:46:57 kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr- >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1905 Nov 2 13:46:57 kernel: 2nd 0xffffff0002fdb4b0 dn->dn_mtx (dn- >dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1066 Nov 2 13:46:57 kernel: KDB: stack backtrace: Nov 2 13:46:57 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:57 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:57 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:57 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:57 kernel: dbuf_dirty() at dbuf_dirty+0xa03 Nov 2 13:46:57 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0x38b Nov 2 13:46:57 kernel: dbuf_write() at dbuf_write+0x24c Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f Nov 2 13:46:57 last message repeated 5 times Nov 2 13:46:57 kernel: dnode_sync() at dnode_sync+0xc12 Nov 2 13:46:57 kernel: dmu_objset_sync() at dmu_objset_sync+0x134 Nov 2 13:46:57 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:46:57 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:57 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:57 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:57 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:57 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:46:57 kernel: lock order reversal: Nov 2 13:46:57 kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr- >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dbuf.c:1905 Nov 2 13:46:57 kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi- >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/dnode.c:705 Nov 2 13:46:57 kernel: KDB: stack backtrace: Nov 2 13:46:57 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:46:57 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:46:57 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:46:57 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:46:57 kernel: dnode_setdirty() at dnode_setdirty+0xbc Nov 2 13:46:57 kernel: dbuf_dirty() at dbuf_dirty+0xa53 Nov 2 13:46:57 kernel: dsl_dataset_block_kill() at dsl_dataset_block_kill+0x38b Nov 2 13:46:57 kernel: dbuf_write() at dbuf_write+0x24c Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb Nov 2 13:46:57 kernel: dbuf_sync_list() at dbuf_sync_list+0x17f Nov 2 13:46:57 last message repeated 5 times Nov 2 13:46:57 kernel: dnode_sync() at dnode_sync+0xc12 Nov 2 13:46:57 kernel: dmu_objset_sync() at dmu_objset_sync+0x134 Nov 2 13:46:57 kernel: dsl_pool_sync() at dsl_pool_sync+0x88 Nov 2 13:46:57 kernel: spa_sync() at spa_sync+0x35e Nov 2 13:46:57 kernel: txg_sync_thread() at txg_sync_thread+0x2d7 Nov 2 13:46:57 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:46:57 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:46:57 kernel: --- trap 0, rip = 0, rsp = 0xffffff80001ffd30, rbp = 0 --- Nov 2 13:47:28 kernel: lock order reversal: Nov 2 13:47:28 kernel: 1st 0xffffff0002fb6a58 syncer (syncer) @ / pool/newsrc/src/sys/kern/vfs_subr.c:1693 Nov 2 13:47:28 kernel: 2nd 0xffffff001a868350 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863 Nov 2 13:47:28 kernel: KDB: stack backtrace: Nov 2 13:47:28 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:47:28 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:47:28 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:47:28 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:47:28 kernel: zfs_zget() at zfs_zget+0x23a Nov 2 13:47:28 kernel: zfs_get_data() at zfs_get_data+0x5e Nov 2 13:47:28 kernel: zil_commit() at zil_commit+0x5aa Nov 2 13:47:28 kernel: zfs_sync() at zfs_sync+0xa6 Nov 2 13:47:28 kernel: sync_fsync() at sync_fsync+0x13a Nov 2 13:47:28 kernel: VOP_F Nov 2 13:47:28 kernel: SYNC_APV() at VOP_FSYNC_APV+0xb5 Nov 2 13:47:28 kernel: sync_vnode() at sync_vnode+0x157 Nov 2 13:47:28 kernel: sched_sync() at sched_sync+0x1d2 Nov 2 13:47:28 kernel: fork_exit() at fork_exit+0x12a Nov 2 13:47:28 kernel: fork_trampoline() at fork_trampoline+0xe Nov 2 13:47:28 kernel: --- trap 0, rip = 0, rsp = 0xffffff8017aebd30, rbp = 0 --- Nov 2 13:47:44 kernel: lock order reversal: Nov 2 13:47:44 kernel: 1st 0xffffff002325d4c0 zp->z_parent_lock (zp- >z_parent_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/zfs_dir.c:379 Nov 2 13:47:44 kernel: 2nd 0xffffff0002914370 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863 Nov 2 13:47:44 kernel: KDB: stack backtrace: Nov 2 13:47:44 kernel: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a Nov 2 13:47:44 kernel: _witness_debugger() at _witness_debugger+0x2e Nov 2 13:47:44 kernel: witness_checkorder() at witness_checkorder +0x81e Nov 2 13:47:44 kernel: _sx_xlock() at _sx_xlock+0x55 Nov 2 13:47:44 kernel: zfs_zget() at zfs_zget+0x23a Nov 2 13:47:44 kernel: zfs_dirlook() at zfs_dirlook+0x1fc Nov 2 13:47:44 kernel: zfs_lookup() at zfs_lookup+0x257 Nov 2 13:47:44 kernel: zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d Nov 2 13:47:44 kernel: VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV +0xaf Nov 2 13:47:44 kernel: vfs_cache_lookup() at vfs_cache_lookup+0xf0 Nov 2 13:47:44 kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 Nov 2 13:47:44 kernel: lookup() at lookup+0x2eb Nov 2 13:47:44 kernel: namei() at namei+0x4a9 Nov 2 13:47:44 kernel: kern_chdir() at kern_chdir+0x78 Nov 2 13:47:44 kernel: syscall() at syscall+0x1d0 Nov 2 13:47:44 kernel: Xfast_syscall() at Xfast_syscall+0xe1 Nov 2 13:47:44 kernel: --- syscall (12, FreeBSD ELF64, chdir), rip = 0x800da8a9c, rsp = 0x7fffffffe8b8, rbp = 0x8010135e0 ---