Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 2 Nov 2009 12:02:01 +0100
From:      Borja Marcos <borjam@sarenet.es>
To:        freebsd-fs@freebsd.org
Subject:   Re: 8.0-RC2: ZFS deadlock with zfs receive
Message-ID:  <B0CAA891-0A22-4455-9577-EB1AE9E029E9@sarenet.es>
In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl>
References:  <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es> <4AEA0EAD.1050302@memberwebs.com> <20091030232805.GA2996@garage.freebsd.pl>

next in thread | previous in thread | raw e-mail | index | archive | help

(Resending to the list, I replied only to Pawel)


On Oct 31, 2009, at 12:28 AM, Pawel Jakub Dawidek wrote:

> On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote:
>> Borja Marcos wrote:
>>> I've been sending some alltraces to pjd about this easy to reproduce
>>> problem. I am using zfs send/zfs receive to replicate a dataset  
>>> from one
>>> server to another. At 1 minute intervals, an incremental snapshot is
>>> sent to update the dataset copy. If there is reading activity on the
>>> dataset copy, a deadlock can happen rendering ZFS and all the FS
>>> subsystem unusable. I've tried with 8.0RC2 and it still happens.
>
> I was able to reproduce it, but I don't have fix yet.
>
>> FWIW, another (or the same) zfs recv deadlock I've been trying to  
>> get to
>> the bottom of:
>>
>> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/ 
>> 006999.html
>
> Could you guys recompile your kernel after uncommenting line:
>
> #CFLAGS+=-DDEBUG=1
>
> in sys/modules/zfs/Makefile?
>
> You should see panic on assertion instead of deadlock, I believe.

No panic, it just deadlocked.

Last trace of LORs

Nov  2 13:45:44  kernel: lock order reversal:
Nov  2 13:45:44  kernel: 1st 0xffffff00025d4c38 ds- 
 >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ 
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ 
bplist.c:189
Nov  2 13:45:44  kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi- 
 >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dnode.c:705
Nov  2 13:45:44  kernel: KDB: stack backtrace:
Nov  2 13:45:44  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:45:44  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:45:44  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:45:44  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:45:44  kernel: dnode_setdirty() at dnode_setdirty+0xbc
Nov  2 13:45:44  kernel: dbuf_dirty() at dbuf_dirty+0x516
Nov  2 13:45:44  kernel: bplist_enqueue() at bplist_enqueue+0xbd
Nov  2 13:45:44  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0x119
Nov  2 13:45:44  kernel: dmu_objset_sync() at dmu_objset_sync+0x1fe
Nov  2 13:45:44  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:45:44  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:45:44  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:45:44  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:45:44  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:45:44  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:45:44  kernel: lock order reversal:
Nov  2 13:45:44  kernel: 1st 0xffffff0002eb9d38 dr->dt.di.dr_mtx (dr- 
 >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov  2 13:45:44  kernel: 2nd 0xffffff0002ed3000 dn->dn_struct_rwlock  
(dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/dbuf.c:543
Nov  2 13:45:44  kernel: KDB: stack backtrace:
Nov  2 13:45:44  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:45:44  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:45:44  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:45:44  kernel: _sx_slock() at _sx_slock+0x55
Nov  2 13:45:44  kernel: dbuf_read() at dbuf_read+0x2ad
Nov  2 13:45:44  kernel: dbuf_will_dirty() at dbuf_will_dirty+0x53
Nov  2 13:45:44  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0xe9
Nov  2 13:45:44  kernel: dbuf_write() at dbuf_write+0x24c
Nov  2 13:45:44  kernel: dbuf_sync_list() at dbuf_sync_list+0x159
Nov  2 13:45:44  kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov  2 13:45:44  kernel: dnode_sync() at dnode_sync+0xc12
Nov  2 13:45:44  kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov  2 13:45:44  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:45:44  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:45:44  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:45:44  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:45:44  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:45:44  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:03  kernel: lock order reversal:
Nov  2 13:46:03  kernel: 1st 0xffffff00025d4c38 ds- 
 >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ 
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ 
bplist.c:152
Nov  2 13:46:03  kernel: 2nd 0xffffff0002f01ae0 dn->dn_dbufs_mtx (dn- 
 >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dbuf.c:1518
Nov  2 13:46:03  kernel: KDB: stack backtrace:
Nov  2 13:46:03  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:03  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:03  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:03  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:03  kernel: dbuf_destroy() at dbuf_destroy+0x58
Nov  2 13:46:03  kernel: bplist_cache() at bplist_cache+0x2e
Nov  2 13:46:03  kernel: bplist_iterate() at bplist_iterate+0xb3
Nov  2 13:46:03  kernel: bplist_space_birthrange() at  
bplist_space_birthrange+0x60
Nov  2 13:46:03  kernel: dsl_dataset_clone_swap_sync() at  
dsl_dataset_clone_swap_sync+0xee
Nov  2 13:46:03  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:03  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:03  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:03  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:03  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:03  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:03  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:03  kernel: lock order reversal:
Nov  2 13:46:03  kernel: 1st 0xffffff00025d4c38 ds- 
 >ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/ 
sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/ 
bplist.c:152
Nov  2 13:46:03  kernel: 2nd 0xffffffff81121d50 h->hash_mutexes[i] (h- 
 >hash_mutexes[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/dbuf.c:191
Nov  2 13:46:03  kernel: KDB: stack backtrace:
Nov  2 13:46:03  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:03  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:03  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:03  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:03  kernel: dbuf_destroy() at dbuf_destroy+0x111
Nov  2 13:46:03  kernel: bplist_cache() at bplist_cache+0x2e
Nov  2 13:46:03  kernel: bplist_iterate() at bplist_iterate+0xb3
Nov  2 13:46:03  kernel: bplist_space_birthrange() at  
bplist_space_birthrange+0x60
Nov  2 13:46:03  kernel: dsl_dataset_clone_swap_sync() at  
dsl_dataset_clone_swap_sync+0xee
Nov  2 13:46:03  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:03  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:03  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:03  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:03  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:03  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:03  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:35  kernel: lock order reversal:
Nov  2 13:46:35  kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock  
(dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171
Nov  2 13:46:35  kernel: 2nd 0xffffff0002784938 ds->ds_opening_lock  
(ds->ds_opening_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:483
Nov  2 13:46:35  kernel: KDB: stack backtrace:
Nov  2 13:46:35  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:35  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:35  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:35  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:35  kernel: dmu_objset_create_impl() at  
dmu_objset_create_impl+0x50
Nov  2 13:46:35  kernel: dmu_objset_create_sync() at  
dmu_objset_create_sync+0xfc
Nov  2 13:46:35  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:35  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:35  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:35  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:35  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:35  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:35  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:35  kernel: lock order reversal:
Nov  2 13:46:35  kernel: 1st 0xffffff000279c398 dp->dp_config_rwlock  
(dp->dp_config_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/dsl_synctask.c:171
Nov  2 13:46:35  kernel: 2nd 0xffffff0002fb7578 zfs (zfs) @ /pool/ 
newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ 
fs/zfs/zfs_znode.c:152
Nov  2 13:46:35  kernel: KDB: stack backtrace:
Nov  2 13:46:35  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:35  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:35  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:35  kernel: __lockmgr_args() at __lockmgr_args+0xd03
Nov  2 13:46:35  kernel: vop_stdlock() at vop_stdlock+0x39
Nov  2 13:46:35  kernel: VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
Nov  2 13:46:35  kernel: _vn_lock() at _vn_lock+0x57
Nov  2 13:46:35  kernel: zfs_znode_cache_constructor() at  
zfs_znode_cache_constructor+0x65
Nov  2 13:46:35  kernel: zfs_create_fs() at zfs_create_fs+0x2ab
Nov  2 13:46:35  kernel: dmu_objset_create_sync() at  
dmu_objset_create_sync+0x116
Nov  2 13:46:35  kernel: dsl_sync_task_group_sync() at  
dsl_sync_task_group_sync+0x173
Nov  2 13:46:35  kernel: dsl_pool_sync() at dsl_pool_sync+0x122
Nov  2 13:46:35  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:35  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:35  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:35  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:35  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:36  kernel: lock order reversal:
Nov  2 13:46:36  kernel: 1st 0xffffff001a0a7430 db->db_mtx (db- 
 >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dbuf.c:1724
Nov  2 13:46:36  kernel: 2nd 0xffffff001a4c0708 dn->dn_dbufs_mtx (dn- 
 >dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dnode_sync.c:373
Nov  2 13:46:36  kernel: KDB: stack backtrace:
Nov  2 13:46:36  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:36  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:36  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:36  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:36  kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x57
Nov  2 13:46:36  kernel: dmu_objset_evict_dbufs() at  
dmu_objset_evict_dbufs+0xd4
Nov  2 13:46:36  kernel: dmu_objset_evict() at dmu_objset_evict+0xbf
Nov  2 13:46:36  kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54
Nov  2 13:46:36  kernel: dbuf_evict_user() at dbuf_evict_user+0x55
Nov  2 13:46:36  kernel: dbuf_rele() at dbuf_rele+0x173
Nov  2 13:46:36  kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c
Nov  2 13:46:36  kernel: spa_sync() at spa_sync+0x618
Nov  2 13:46:36  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:36  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:36  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:36  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:36  kernel: lock order reversal:
Nov  2 13:46:36  kernel: 1st 0xffffff001a0a7430 db->db_mtx (db- 
 >db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dbuf.c:1724
Nov  2 13:46:36  kernel: 2nd 0xffffff001a4c03d8 dn->dn_struct_rwlock  
(dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:409
Nov  2 13:46:36  kernel: KDB: stack backtrace:
Nov  2 13:46:36  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:36  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:36  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:36  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:36  kernel: dnode_evict_dbufs() at dnode_evict_dbufs+0x1b8
Nov  2 13:46:36  kernel: dmu_objset_evict_dbufs() at  
dmu_objset_evict_dbufs+0xd4
Nov  2 13:46:36  kernel: dmu_objset_evict() at dmu_objset_evict+0xbf
Nov  2 13:46:36  kernel: dsl_dataset_evict() at dsl_dataset_evict+0x54
Nov  2 13:46:36  kernel: dbuf_evict_user() at dbuf_evict_user+0x55
Nov  2 13:46:36  kernel: dbuf_rele() at dbuf_rele+0x173
Nov  2 13:46:36  kernel: dsl_pool_zil_clean() at dsl_pool_zil_clean+0x3c
Nov  2 13:46:36  kernel: spa_sync() at spa_sync+0x618
Nov  2 13:46:36  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:36  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:36  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:36  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:57  kernel: lock order reversal:
Nov  2 13:46:57  kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr- 
 >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov  2 13:46:57  kernel: 2nd 0xffffff0002fdb4b0 dn->dn_mtx (dn- 
 >dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dbuf.c:1066
Nov  2 13:46:57  kernel: KDB: stack backtrace:
Nov  2 13:46:57  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:57  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:57  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:57  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:57  kernel: dbuf_dirty() at dbuf_dirty+0xa03
Nov  2 13:46:57  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0x38b
Nov  2 13:46:57  kernel: dbuf_write() at dbuf_write+0x24c
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov  2 13:46:57  last message repeated 5 times
Nov  2 13:46:57  kernel: dnode_sync() at dnode_sync+0xc12
Nov  2 13:46:57  kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov  2 13:46:57  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:46:57  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:57  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:57  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:57  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:57  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:46:57  kernel: lock order reversal:
Nov  2 13:46:57  kernel: 1st 0xffffff001a81ec38 dr->dt.di.dr_mtx (dr- 
 >dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dbuf.c:1905
Nov  2 13:46:57  kernel: 2nd 0xffffff000279bd40 osi->os_lock (osi- 
 >os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/dnode.c:705
Nov  2 13:46:57  kernel: KDB: stack backtrace:
Nov  2 13:46:57  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:46:57  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:46:57  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:46:57  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:46:57  kernel: dnode_setdirty() at dnode_setdirty+0xbc
Nov  2 13:46:57  kernel: dbuf_dirty() at dbuf_dirty+0xa53
Nov  2 13:46:57  kernel: dsl_dataset_block_kill() at  
dsl_dataset_block_kill+0x38b
Nov  2 13:46:57  kernel: dbuf_write() at dbuf_write+0x24c
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x3eb
Nov  2 13:46:57  kernel: dbuf_sync_list() at dbuf_sync_list+0x17f
Nov  2 13:46:57  last message repeated 5 times
Nov  2 13:46:57  kernel: dnode_sync() at dnode_sync+0xc12
Nov  2 13:46:57  kernel: dmu_objset_sync() at dmu_objset_sync+0x134
Nov  2 13:46:57  kernel: dsl_pool_sync() at dsl_pool_sync+0x88
Nov  2 13:46:57  kernel: spa_sync() at spa_sync+0x35e
Nov  2 13:46:57  kernel: txg_sync_thread() at txg_sync_thread+0x2d7
Nov  2 13:46:57  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:46:57  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:46:57  kernel: --- trap 0, rip = 0, rsp =  
0xffffff80001ffd30, rbp = 0 ---
Nov  2 13:47:28  kernel: lock order reversal:
Nov  2 13:47:28  kernel: 1st 0xffffff0002fb6a58 syncer (syncer) @ / 
pool/newsrc/src/sys/kern/vfs_subr.c:1693
Nov  2 13:47:28  kernel: 2nd 0xffffff001a868350 zfsvfs->z_hold_mtx[i]  
(zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863
Nov  2 13:47:28  kernel: KDB: stack backtrace:
Nov  2 13:47:28  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:47:28  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:47:28  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:47:28  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:47:28  kernel: zfs_zget() at zfs_zget+0x23a
Nov  2 13:47:28  kernel: zfs_get_data() at zfs_get_data+0x5e
Nov  2 13:47:28  kernel: zil_commit() at zil_commit+0x5aa
Nov  2 13:47:28  kernel: zfs_sync() at zfs_sync+0xa6
Nov  2 13:47:28  kernel: sync_fsync() at sync_fsync+0x13a
Nov  2 13:47:28  kernel: VOP_F
Nov  2 13:47:28  kernel: SYNC_APV() at VOP_FSYNC_APV+0xb5
Nov  2 13:47:28  kernel: sync_vnode() at sync_vnode+0x157
Nov  2 13:47:28  kernel: sched_sync() at sched_sync+0x1d2
Nov  2 13:47:28  kernel: fork_exit() at fork_exit+0x12a
Nov  2 13:47:28  kernel: fork_trampoline() at fork_trampoline+0xe
Nov  2 13:47:28  kernel: --- trap 0, rip = 0, rsp =  
0xffffff8017aebd30, rbp = 0 ---
Nov  2 13:47:44  kernel: lock order reversal:
Nov  2 13:47:44  kernel: 1st 0xffffff002325d4c0 zp->z_parent_lock (zp- 
 >z_parent_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/ 
opensolaris/uts/common/fs/zfs/zfs_dir.c:379
Nov  2 13:47:44  kernel: 2nd 0xffffff0002914370 zfsvfs->z_hold_mtx[i]  
(zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/ 
contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:863
Nov  2 13:47:44  kernel: KDB: stack backtrace:
Nov  2 13:47:44  kernel: db_trace_self_wrapper() at  
db_trace_self_wrapper+0x2a
Nov  2 13:47:44  kernel: _witness_debugger() at _witness_debugger+0x2e
Nov  2 13:47:44  kernel: witness_checkorder() at witness_checkorder 
+0x81e
Nov  2 13:47:44  kernel: _sx_xlock() at _sx_xlock+0x55
Nov  2 13:47:44  kernel: zfs_zget() at zfs_zget+0x23a
Nov  2 13:47:44  kernel: zfs_dirlook() at zfs_dirlook+0x1fc
Nov  2 13:47:44  kernel: zfs_lookup() at zfs_lookup+0x257
Nov  2 13:47:44  kernel: zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d
Nov  2 13:47:44  kernel: VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV 
+0xaf
Nov  2 13:47:44  kernel: vfs_cache_lookup() at vfs_cache_lookup+0xf0
Nov  2 13:47:44  kernel: VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7
Nov  2 13:47:44  kernel: lookup() at lookup+0x2eb
Nov  2 13:47:44  kernel: namei() at namei+0x4a9
Nov  2 13:47:44  kernel: kern_chdir() at kern_chdir+0x78
Nov  2 13:47:44  kernel: syscall() at syscall+0x1d0
Nov  2 13:47:44  kernel: Xfast_syscall() at Xfast_syscall+0xe1
Nov  2 13:47:44  kernel: --- syscall (12, FreeBSD ELF64, chdir), rip =  
0x800da8a9c, rsp = 0x7fffffffe8b8, rbp = 0x8010135e0 ---








Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B0CAA891-0A22-4455-9577-EB1AE9E029E9>