Date: Tue, 23 Sep 2014 16:12:44 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Bryan Drewery <bdrewery@FreeBSD.org> Cc: FreeBSD FS <freebsd-fs@FreeBSD.org> Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140923131244.GC8870@kib.kiev.ua> In-Reply-To: <5420D5FC.4030600@FreeBSD.org> References: <5420D5FC.4030600@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Sep 22, 2014 at 09:07:56PM -0500, Bryan Drewery wrote:
> This is slightly different than my last report (involving
> zfs_lookup/zfs_dirent). I still need to gather information from that
> previous one.
>
> Here is what I have on this one. Quoted so Thunderbird doesn't trash it:
>
> > 1. Poudriere build startup, run for N=01,...,12 in parallel:
> > a. mount tmpfs on /poudriere/data/.m/exp-10amd64-commit-test/N
> > /poudriere/data/.m is a ZFS mount.
> > b. cpdup /poudriere/data/.m/exp-10amd64-commit-test/ref to /poudriere/data/.m/exp-10amd64-commit-test/N
> > [1] c. mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/N/.p
> > 2. SIGINT during this phase:
> > [2] a. umount -f all tmpfs
> > b. cpdup processes are *not* killed. They continue running. Most of them fail due to their dst tmpfs being gone.
> > [3] c. A cpdup process managed to continue running. Since the tmpfs was gone it started trying to write to the ZFS mount under it. (/poudriere/data/.m)
> >
> > # procstat -kka|egrep "(lockmgr|tmpfs)"
> > [3] 76137 100919 cpdup - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0x862 vop_stdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa vget+0x67 cache_lookup+0x5b2 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xf1 lookup+0x5ad namei+0x4e4 kern_linkat+0x13d sys_link+0x28 filemon_wrapper_link+0x19 amd64_syscall+0x25a
> > [1] 76346 102346 mkdir - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0xea1 vop_stdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa lookup+0x496 namei+0x4e4 kern_mkdirat+0xcb amd64_syscall+0x25a Xfast_syscall+0xfb
> > [2] 76286 102085 umount - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 _sleep+0x366 vfs_write_suspend+0xf0 vfs_write_suspend_umnt+0x38 tmpfs_unmount+0x6d dounmount+0x424 sys_unmount+0x2ec amd64_syscall+0x25a Xfast_syscall+0xfb
> >
> > [3] root 76137 0.7 0.0 14772 2844 3 D+ 3:12PM 0:05.28 cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref /poudriere/data/.m/exp-10amd64-commit-test/02
> > [1] root 76346 0.0 0.0 8200 1916 3 D+ 3:12PM 0:00.00 mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../03/.p
> > [2] root 76286 [command taken from gdb] umount -f /poudriere/data/.m/exp-10amd64-commit-test/02
> >
> > Locked vnodes:
> >
> > 0xfffff80fda0bcce8: tag zfs, type VDIR
> > usecount 2, writecount 0, refcount 3 mountedhere 0x0
> > flags ()
> > v_object 0xfffff805fc81a300 ref 0 pages 0
> > lock type zfs: SHARED (count 2) with exclusive waiters pending
> > mount /poudriere/data/.m from zroot/poudriere/data/.m
> > path /poudriere/data/.m/exp-10am64-commit-test/02
> > 0xfffff8065de471d8: tag zfs, type VDIR
> > usecount 1, writecount 0, refcount 3 mountedhere 0xfffff8058f582330
> > flags ()
> > lock type zfs: EXCL by thread 0xfffff8047c0eb920 (pid 76286) with exclusive and shared waiters pending
> > mount /poudriere/data/.m from zroot/poudriere/data/.m
> > path /poudriere/data/.m
> >
> >
> > Processes:
> >
> > PID 76286 (umount -f /poudriere/data/.m/exp-10amd64-commit-test/02):
> > Waiting in vfs_write_suspend(mp=0xfffff8058f582330) for mn_writeopcount==0
> >
> >
> > (kgdb) thread 521
> > [Switching to thread 521 (Thread 102085)]#0 sched_switch (td=0xfffff8047c0eb920, newtd=<value optimized out>, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932
> > 1932 cpuid = PCPU_GET(cpuid);
> > (kgdb) backtrace
> > #0 sched_switch (td=0xfffff8047c0eb920, newtd=<value optimized out>, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932
> > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493
> > #2 0xffffffff80966af2 in sleepq_switch (wchan=<value optimized out>, pri=<value optimized out>) at /usr/src/sys/kern/subr_sleepqueue.c:552
> > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff8058f5823bc, pri=119) at /usr/src/sys/kern/subr_sleepqueue.c:631
> > #4 0xffffffff80928c86 in _sleep (ident=<value optimized out>, lock=<value optimized out>, priority=<value optimized out>, wmesg=<value optimized out>, sbt=<value optimized out>,
> > pr=<value optimized out>) at /usr/src/sys/kern/kern_synch.c:255
> > #5 0xffffffff809df0b0 in vfs_write_suspend (mp=0xfffff8058f582330, flags=<value optimized out>) at /usr/src/sys/kern/vfs_vnops.c:1793
> > #6 0xffffffff809df308 in vfs_write_suspend_umnt (mp=0xfffff8058f582330) at /usr/src/sys/kern/vfs_vnops.c:1848
> > #7 0xffffffff822ad9ed in tmpfs_unmount (mp=0xfffff8058f582330, mntflags=<value optimized out>) at /usr/src/sys/modules/tmpfs/../../fs/tmpfs/tmpfs_vfsops.c:280
> > #8 0xffffffff809c7664 in dounmount (mp=0xfffff8058f582330, flags=134742016, td=0xfffff8047c0eb920) at /usr/src/sys/kern/vfs_mount.c:1310
> > #9 0xffffffff809c721c in sys_unmount (td=0xfffff8047c0eb920, uap=0xfffffe1248712b80) at /usr/src/sys/kern/vfs_mount.c:1201
> > #10 0xffffffff80d38b6a in amd64_syscall (td=0xfffff8047c0eb920, traced=0) at subr_syscall.c:133
> > #11 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390
> >
> > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_writeopcount
> > $12 = 1
> > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_activevnodelistsize
> > $13 = 1
> > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_vnodecovered
> > $25 = (struct vnode *) 0xfffff8065de471d8
> > (kgdb) p *(struct mount*)0xfffff8058f582330
> > $5 = {mnt_mtx = {lock_object = {lo_name = 0xffffffff80fc4c20 "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1ea00}, mtx_lock = 4}, mnt_gen = 23,
> > mnt_list = {tqe_next = 0x0, tqe_prev = 0xfffff8117c2abce8}, mnt_op = 0xffffffff822b3e40, mnt_vfc = 0xffffffff822b3f60, mnt_vnodecovered = 0xfffff8065de471d8, mnt_syncer = 0x0,
> > mnt_ref = 7357, mnt_nvnodelist = {tqh_first = 0xfffff803f6a691d8, tqh_last = 0xfffff8065dc871f8}, mnt_nvnodelistsize = 7356, mnt_activevnodelist = {tqh_first = 0xfffff806dff493b0,
> > tqh_last = 0xfffff806dff49470}, mnt_activevnodelistsize = 1, mnt_writeopcount = 1, mnt_kern_flag = 150994953, mnt_flag = 4096, mnt_opt = 0xfffff804e377eaf0, mnt_optnew = 0x0,
> > mnt_maxsymlinklen = 0, mnt_stat = {f_version = 537068824, f_type = 135, f_flags = 4096, f_bsize = 4096, f_iosize = 4096, f_blocks = 1835008, f_bfree = 1798116, f_bavail = 1798116,
> > f_files = 25690112, f_ffree = 25682842, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0,
> > f_fsid = {val = {-2027880658, 135}}, f_charspare = '\0' <repeats 79 times>, f_fstypename = "tmpfs\000\000\000\000\000\000\000\000\000\000",
> > f_mntfromname = "tmpfs", '\0' <repeats 82 times>, f_mntonname = "/poudriere/data/.m/exp-10amd64-commit-test/02", '\0' <repeats 42 times>}, mnt_cred = 0xfffff8047c8a4a00,
> > mnt_data = 0xfffff8000bd2b800, mnt_time = 0, mnt_iosize_max = 65536, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 3766194835, mnt_lockref = 0, mnt_secondary_writes = 0,
> > mnt_secondary_accwrites = 0, mnt_susp_owner = 0xfffff8047c0eb920, mnt_gjprovider = 0x0, mnt_explock = {lock_object = {lo_name = 0xffffffff80fa4b29 "explock", lo_flags = 108199936,
> > lo_data = 0, lo_witness = 0xfffffe0000b26c80}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 0, lk_pri = 96}, mnt_upper_link = {tqe_next = 0x0, tqe_prev = 0xfffff8005c842ca0},
> > mnt_uppers = {tqh_first = 0x0, tqh_last = 0xfffff8058f582650}}
> > (kgdb) p (((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tqh_first)
> > $30 = (struct vnode *) 0xfffff806dff493b0
> > (kgdb) p *(((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tqh_first)
> > $24 = {v_tag = 0xffffffff822b2f76 "tmpfs", v_op = 0xffffffff822b3918, v_data = 0xfffff80ae07caae0, v_mount = 0xfffff8058f582330, v_nmntvnodes = {tqe_next = 0xfffff806071d11d8,
> > tqe_prev = 0xfffff804d63963d0}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = {
> > lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffff806dff49400}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff822b2f76 "tmpfs", lo_flags = 116588544,
> > lo_data = 0, lo_witness = 0xfffffe0000b29a00}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {
> > lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, mtx_lock = 4}, v_vnlock = 0xfffff806dff49418, v_actfreelist = {
> > tqe_next = 0x0, tqe_prev = 0xfffff8058f5823a8}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xffffffff80fcd144 "bufobj interlock", lo_flags = 86179840, lo_data = 0,
> > lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0xfffff809dc520a00, bo_synclist = {le_next = 0x0, le_prev = 0x0},
> > bo_private = 0xfffff806dff493b0, __bo_vnode = 0xfffff806dff493b0, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806dff494d0}, bv_root = {pt_root = 0}, bv_cnt = 0},
> > bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806dff494f0}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 4096}, v_pollinfo = 0x0,
> > v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff806dff49538}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0,
> > v_holdcnt = 1, v_usecount = 1, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 115340435, v_type = VREG}
> >
> >
> > PID 76137 (cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref /poudriere/data/.m/exp-10amd64-commit-test/02):
> > Waiting for vget(vp=0xfffff8065de471d8) [which is mnt_vnodecovered from umount -f]
> >
> > (kgdb) thread 520
> > [Switching to thread 520 (Thread 100919)]#0 sched_switch (td=0xfffff8005cd2a490, newtd=<value optimized out>, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932
> > 1932 cpuid = PCPU_GET(cpuid);
> > (kgdb) backtrace
> > #0 sched_switch (td=0xfffff8005cd2a490, newtd=<value optimized out>, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932
> > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493
> > #2 0xffffffff80966af2 in sleepq_switch (wchan=<value optimized out>, pri=<value optimized out>) at /usr/src/sys/kern/subr_sleepqueue.c:552
> > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff8065de47240, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631
> > #4 0xffffffff808fdf3a in sleeplk (lk=<value optimized out>, flags=<value optimized out>, ilk=<value optimized out>, wmesg=<value optimized out>, pri=<value optimized out>,
> > timo=<value optimized out>) at /usr/src/sys/kern/kern_lock.c:225
> > #5 0xffffffff808fcfc2 in __lockmgr_args (lk=0xfffff8065de47240, flags=<value optimized out>, ilk=0xfffff8065de47270, wmesg=<value optimized out>, pri=<value optimized out>,
> > timo=<value optimized out>, file=0xffffffff80fcd0e8 "/usr/src/sys/kern/vfs_subr.c", line=2137) at /usr/src/sys/kern/kern_lock.c:680
> > #6 0xffffffff809be27c in vop_stdlock (ap=<value optimized out>) at lockmgr.h:98
> > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:2082
> > #8 0xffffffff809ddbca in _vn_lock (vp=0xfffff8065de471d8, flags=<value optimized out>, file=0xffffffff80fcd0e8 "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859
> > #9 0xffffffff809cde87 in vget (vp=0xfffff8065de471d8, flags=2097408, td=0xfffff8005cd2a490) at /usr/src/sys/kern/vfs_subr.c:2137
> > #10 0xffffffff809ba132 in cache_lookup (dvp=0xfffff80fda0bcce8, vpp=0xfffffe1247de78b8, cnp=0xfffffe1247de78e0, tsp=0x0, ticksp=0x0) at /usr/src/sys/kern/vfs_cache.c:673
> > #11 0xffffffff809bb2dc in vfs_cache_lookup (ap=<value optimized out>) at /usr/src/sys/kern/vfs_cache.c:1038
> > #12 0xffffffff80e58e41 in VOP_LOOKUP_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:127
> > #13 0xffffffff809c364d in lookup (ndp=0xfffffe1247de7858) at vnode_if.h:54
> > #14 0xffffffff809c2d84 in namei (ndp=0xfffffe1247de7858) at /usr/src/sys/kern/vfs_lookup.c:300
> > #15 0xffffffff809d76dd in kern_linkat (td=0xfffff8005cd2a490, fd1=<value optimized out>, fd2=-100, path1=<value optimized out>, path2=0x80141c600 <Address 0x80141c600 out of bounds>,
> > segflg=UIO_USERSPACE, follow=64) at /usr/src/sys/kern/vfs_syscalls.c:1572
> > #16 0xffffffff809d7518 in sys_link (td=0x0, uap=<value optimized out>) at /usr/src/sys/kern/vfs_syscalls.c:1541
> > #17 0xffffffff820c64e9 in filemon_wrapper_link (td=0x0, uap=0xfffffe1247de7b80) at filemon_wrapper.c:364
> > #18 0xffffffff80d38b6a in amd64_syscall (td=0xfffff8005cd2a490, traced=0) at subr_syscall.c:133
> > #19 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390
> >
> > (kgdb) p *(struct vnode*)0xfffff8065de471d8
> > $34 = {v_tag = 0xffffffff81f0f8ff "zfs", v_op = 0xffffffff81f24b50, v_data = 0xfffff80cb12f22e0, v_mount = 0xfffff8005c8a8660, v_nmntvnodes = {tqe_next = 0xfffff80528f4d938,
> > tqe_prev = 0xfffff804d659cb30}, v_un = {vu_mount = 0xfffff8058f582330, vu_socket = 0xfffff8058f582330, vu_cdev = 0xfffff8058f582330, vu_fifoinfo = 0xfffff8058f582330}, v_hashlist = {
> > le_next = 0x0, le_prev = 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0xfffff805d7724a80, tqh_last = 0xfffff805d7724aa0}, v_cache_dd = 0xfffff805d7724a80,
> > v_lock = {lock_object = {lo_name = 0xffffffff81f0f8ff "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0xfffffe0000b28780}, lk_lock = 18446735296877738278, lk_exslpfail = 0,
> > lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500},
> > mtx_lock = 4}, v_vnlock = 0xfffff8065de47240, v_actfreelist = {tqe_next = 0xfffff80528f4d938, tqe_prev = 0xfffff804d659cbd0}, v_bufobj = {bo_lock = {lock_object = {
> > lo_name = 0xffffffff80fcd144 "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0x0,
> > bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff8065de471d8, __bo_vnode = 0xfffff8065de471d8, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff8065de472f8},
> > bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff8065de47318}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0,
> > bo_bsize = 131072}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff8065de47360}, rl_currdep = 0x0}, v_cstart = 0,
> > v_lasta = 0, v_lastw = 0, v_clen = 0, v_holdcnt = 3, v_usecount = 1, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 106816625, v_type = VDIR}
> >
> > (kgdb) frame 15
> > #15 0xffffffff809d76dd in kern_linkat (td=0xfffff8005cd2a490, fd1=<value optimized out>, fd2=-100, path1=<value optimized out>, path2=0x80141c600 <Address 0x80141c600 out of bounds>,
> > segflg=UIO_USERSPACE, follow=64) at /usr/src/sys/kern/vfs_syscalls.c:1572
> > 1572 if ((error = namei(&nd)) == 0) {
> > (kgdb) info locals
> > mp = (struct mount *) 0xfffff8058f582330
> > nd = {ni_dirp = 0x80141c600 <Address 0x80141c600 out of bounds>, ni_segflg = UIO_USERSPACE, ni_rightsneeded = {cr_rights = {144115188080051200, 288230376151711744}}, ni_startdir = 0x0,
> > ni_rootdir = 0xfffff8005c0ea1d8, ni_topdir = 0x0, ni_dirfd = -100, ni_strictrelative = 0, ni_filecaps = {fc_rights = {cr_rights = {0, 0}}, fc_ioctls = 0x0, fc_nioctls = -1,
> > fc_fcntls = 0}, ni_vp = 0xfffff8065de471d8, ni_dvp = 0xfffff80fda0bcce8, ni_pathlen = 63,
> > ni_next = 0xfffff80e918bb02d "/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", ni_loopcnt = 0, ni_cnd = {cn_nameiop = 1, cn_flags = 134236168,
> > cn_thread = 0xfffff8005cd2a490, cn_cred = 0xfffff8005c22a700, cn_lkflags = 2097152,
> > cn_pnbuf = 0xfffff80e918bb000 "/poudriere/data/.m/exp-10amd64-commit-test/02/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz",
> > cn_nameptr = 0xfffff80e918bb02b "02/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", cn_namelen = 2, cn_consume = 0}}
> > rights = {cr_rights = {144115188080051200, 288230376151711744}}
> > mp = (struct mount *) 0xfffff8058f582330
> > vp = (struct vnode *) 0xfffff806dff493b0
> > error = <value optimized out>
> > kern_linkat
> > mp = (struct mount *) 0xfffff8058f582330
> > vp = (struct vnode *) 0xfffff806dff493b0
> >
> >
> > PID 76346 (mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../03/.p)
> > Waiting on vn_lock(0xfffff80fda0bcce8)
> >
> > (kgdb) thread 522
> > [Switching to thread 522 (Thread 102346)]#0 sched_switch (td=0xfffff80419f86920, newtd=<value optimized out>, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932
> > 1932 cpuid = PCPU_GET(cpuid);
> > (kgdb) backtrace
> > #0 sched_switch (td=0xfffff80419f86920, newtd=<value optimized out>, flags=<value optimized out>) at /usr/src/sys/kern/sched_ule.c:1932
> > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493
> > #2 0xffffffff80966af2 in sleepq_switch (wchan=<value optimized out>, pri=<value optimized out>) at /usr/src/sys/kern/subr_sleepqueue.c:552
> > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff80fda0bcd50, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631
> > #4 0xffffffff808fdf3a in sleeplk (lk=<value optimized out>, flags=<value optimized out>, ilk=<value optimized out>, wmesg=<value optimized out>, pri=<value optimized out>,
> > timo=<value optimized out>) at /usr/src/sys/kern/kern_lock.c:225
> > #5 0xffffffff808fd601 in __lockmgr_args (lk=0xfffff80fda0bcd50, flags=<value optimized out>, ilk=0xfffff80fda0bcd80, wmesg=<value optimized out>, pri=<value optimized out>,
> > timo=<value optimized out>, file=0xffffffff80fcc4f0 "/usr/src/sys/kern/vfs_lookup.c", line=691) at /usr/src/sys/kern/kern_lock.c:939
> > #6 0xffffffff809be27c in vop_stdlock (ap=<value optimized out>) at lockmgr.h:98
> > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=<value optimized out>, a=<value optimized out>) at vnode_if.c:2082
> > #8 0xffffffff809ddbca in _vn_lock (vp=0xfffff80fda0bcce8, flags=<value optimized out>, file=0xffffffff80fcc4f0 "/usr/src/sys/kern/vfs_lookup.c", line=691) at vnode_if.h:859
> > #9 0xffffffff809c3536 in lookup (ndp=0xfffffe1248b7b910) at /usr/src/sys/kern/vfs_lookup.c:691
> > #10 0xffffffff809c2d84 in namei (ndp=0xfffffe1248b7b910) at /usr/src/sys/kern/vfs_lookup.c:300
> > #11 0xffffffff809daf2b in kern_mkdirat (td=0xfffff80419f86920, fd=-100, path=0x7fffffffeca1 <Address 0x7fffffffeca1 out of bounds>, segflg=UIO_USERSPACE, mode=511)
> > at /usr/src/sys/kern/vfs_syscalls.c:3687
> > #12 0xffffffff80d38b6a in amd64_syscall (td=0xfffff80419f86920, traced=0) at subr_syscall.c:133
> > #13 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390
> > #14 0x0000000800953faa in ?? ()
> >
> > (kgdb) p *(struct vnode*) 0xfffff80fda0bcce8
> > $35 = {v_tag = 0xffffffff81f0f8ff "zfs", v_op = 0xffffffff81f24b50, v_data = 0xfffff80c56bf8b80, v_mount = 0xfffff8005c8a8660, v_nmntvnodes = {tqe_next = 0xfffff806d8e1d1d8,
> > tqe_prev = 0xfffff80df75d51f8}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = {
> > lh_first = 0xfffff805d7724a80}, v_cache_dst = {tqh_first = 0xfffff805df597690, tqh_last = 0xfffff805df5976b0}, v_cache_dd = 0xfffff805df597690, v_lock = {lock_object = {
> > lo_name = 0xffffffff81f0f8ff "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0xfffffe0000b28780}, lk_lock = 21, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {
> > lock_object = {lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, mtx_lock = 4}, v_vnlock = 0xfffff80fda0bcd50,
> > v_actfreelist = {tqe_next = 0xfffff806d8e1d1d8, tqe_prev = 0xfffff80528f4d9f8}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xffffffff80fcd144 "bufobj interlock",
> > lo_flags = 86179840, lo_data = 0, lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0xfffff805fc81a300, bo_synclist = {le_next = 0x0,
> > le_prev = 0x0}, bo_private = 0xfffff80fda0bcce8, __bo_vnode = 0xfffff80fda0bcce8, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce08}, bv_root = {pt_root = 0},
> > bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce28}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072},
> > v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce70}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0,
> > v_clen = 0, v_holdcnt = 3, v_usecount = 2, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 18, v_type = VDIR}
> >
>
> Let me know if I can provide any further information.
There is indeed some write op in flight on the tmpfs mount, which
blocks unmount suspension. The 'bt all' from ddb would give some
overview of the threads states.
Follow
https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
to gather required information.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140923131244.GC8870>