Date: Wed, 10 Mar 2010 13:32:02 +0100 From: Borja Marcos <borjam@sarenet.es> To: Pawel Jakub Dawidek <pjd@FreeBSD.org> Cc: freebsd-fs@freebsd.org, FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: Many processes stuck in zfs Message-ID: <E31E46B1-0FDC-4C6A-AD47-9B78D54790EC@sarenet.es> In-Reply-To: <20100310110202.GA1715@garage.freebsd.pl> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <EC9BC6B4-8D0E-4FE3-852F-0E3A24569D33@sarenet.es> <20100309125815.GF3155@garage.freebsd.pl> <CB854F58-03AF-46DD-8153-85FA96037C21@sarenet.es> <BFF1E2D6-B48A-4A5E-ACEE-8577FDB07820@sarenet.es> <20100310110202.GA1715@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mar 10, 2010, at 12:02 PM, Pawel Jakub Dawidek wrote: > On Wed, Mar 10, 2010 at 10:24:49AM +0100, Borja Marcos wrote: >> Tested. Same deadlock remains. > > Ok, to track this down I need the following: > > Uncomment 'CFLAGS+=-DDEBUG=1' line in sys/modules/zfs/Makefile. > > Add the following lines to your kernel config: > > options WITNESS > options WITNESS_SKIPSPIN > options INVARIANTS > options INVARIANT_SUPPORT > options DEBUG_VFS_LOCKS > options DEBUG_LOCKS > options KDB > options DDB > > Recompile your kernel and modules and reboot. > > Once the deadlock occur, enter DDB and send me the output of: > > ps > show alllocks > show lockedvnods > show allchains > alltrace Trying. I started my typical test: Machine 1 doing a make buildworld on a dataset with src and obj on it. Machine 1 replicating incremental snapshots of the dataset to machine 2. Machine 2 running some "tar cf - . | ( cd /pool/anotherdataset && tar xf - )" from the dataset being replicated, ie, doing read operations on the target dataset. This time, with all those debug options, there was no deadlock, but almost an instant trap entering DDB. Unfortunately, I tried to capture the output of "alltrace", etc using the capture option. But couldn't come back to the system to read it. Any ideas? I'm using VMWare Fusion to run FreeBSD for these tests and seems I'm out of luck, I don't see any console output mechanism. When rebooting I was greeted by some LORs lock order reversal: 1st 0xffffff000286c2e8 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:549 2nd 0xffffff000286b0d8 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:1173 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffffd808, rbp = 0x801224140 --- lock order reversal: 1st 0xffffff0002864e70 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:381 2nd 0xffffff00026e5140 osi->os_lock (osi->os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffffd808, rbp = 0x801224140 --- lock order reversal: 1st 0xffffff000286c058 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1116 2nd 0xffffff0002591a38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff0002905638 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1905 2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock (spa->spa_sync_bplist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff000293bcc8 zfs (zfs) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/gfs.c:437 2nd 0xffffff0002916310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x800f48f1c, rsp = 0x7fffffffced8, rbp = 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff00025e4078 zp->z_name_lock (zp->z_name_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:212 2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd8acc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002916210 zfsvfs->z_teardown_inactive_lock (zfsvfs->z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:3724 2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:1027 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x95 zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2fc kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip = 0x800fd8acc, rsp = 0x7fffffffcf38, rbp = 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002861958 buf->b_lock (buf->b_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2509 2nd 0xffffff0002956430 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:421 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_read_done() at dbuf_read_done+0x3b arc_read_done() at arc_read_done+0x1d2 zio_done() at zio_done+0x308 zio_execute() at zio_execute+0xb1 arc_read_nolock() at arc_read_nolock+0x3d0 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dmu_buf_hold() at dmu_buf_hold+0xcc zap_lockdir() at zap_lockdir+0x6e zap_cursor_retrieve() at zap_cursor_retrieve+0x1bc zfs_unlinked_drain() at zfs_unlinked_drain+0xd8 zfsvfs_setup() at zfsvfs_setup+0xfa zfs_mount() at zfs_mount+0x7df vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x800f48f1c, rsp = 0x7fffffffced8, rbp = 0x7fffffffcef8 --- Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000284000) 0.012686807 s ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based forwarding disabled, default to deny, logging disabled lock order reversal: 1st 0xffffff800a2f5d08 bufwait (bufwait) @ /pool/newsrc/src/sys/kern/vfs_bio.c:2559 2nd 0xffffff0002c69800 dirhash (dirhash) @ /pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x88b ufs_mkdir() at ufs_mkdir+0x623 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x80072bb0c, rsp = 0x7fffffffec88, rbp = 0x7fffffffef66 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189 2nd 0xffffff000286ab88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189 2nd 0xffffff0002f6b0d8 dn->dn_mtx (dn->dn_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:606 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_hold_impl() at dnode_hold_impl+0x184 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189 2nd 0xffffff00028842e8 db->db_mtx (db->db_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1724 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_rele() at dbuf_rele+0x2d dnode_hold_impl() at dnode_hold_impl+0x20f dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:189 2nd 0xffffff00027d5d40 osi->os_lock (osi->os_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode.c:687 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_setdirty() at dnode_setdirty+0xbc dbuf_dirty() at dbuf_dirty+0x516 bplist_enqueue() at bplist_enqueue+0xbd dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff0002c84938 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1905 2nd 0xffffff0002f6b7b0 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:543 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dbuf_read() at dbuf_read+0x2ad dbuf_will_dirty() at dbuf_will_dirty+0x53 dsl_dataset_block_kill() at dsl_dataset_block_kill+0xe9 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x159 dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff000291a210 zfsvfs->z_teardown_inactive_lock (zfsvfs->z_teardown_inactive_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:917 2nd 0xffffff00027450f8 ds->ds_rwlock (ds->ds_rwlock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:2864 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a dmu_recv_end() at dmu_recv_end+0x94 zfs_ioc_recv() at zfs_ioc_recv+0x29d zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffff8e98, rbp = 0x7fffffff9bd0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:94 2nd 0xffffff0002f6b330 dn->dn_dbufs_mtx (dn->dn_dbufs_mtx) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1518 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x58 bplist_close() at bplist_close+0x37 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506 dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/bplist.c:94 2nd 0xffffffff81152650 h->hash_mutexes[i] (h->hash_mutexes[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:191 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x111 bplist_close() at bplist_close+0x37 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506 dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8012417d30, rbp = 0 --- lock order reversal: 1st 0xffffff000291a250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1317 2nd 0xffffff000291a310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:966 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_rezget() at zfs_rezget+0x4a zfs_resume_fs() at zfs_resume_fs+0x158 zfs_ioc_recv() at zfs_ioc_recv+0x2b4 zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe7d1c, rsp = 0x7fffffff8e98, rbp = 0x7fffffff9bd0 --- # Borja.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E31E46B1-0FDC-4C6A-AD47-9B78D54790EC>