From owner-freebsd-fs@FreeBSD.ORG Tue Sep 23 13:12:58 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E803C69; Tue, 23 Sep 2014 13:12:58 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CF5D4BB7; Tue, 23 Sep 2014 13:12:57 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id s8NDCj41010238 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 23 Sep 2014 16:12:45 +0300 (EEST) (envelope-from kostik@tom.home) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua s8NDCj41010238 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id s8NDCiRf010237; Tue, 23 Sep 2014 16:12:44 +0300 (EEST) (envelope-from kostik) Date: Tue, 23 Sep 2014 16:12:44 +0300 From: Konstantin Belousov To: Bryan Drewery Subject: Re: Deadlock with umount -f involving tmpfs on top of ZFS on r271170 Message-ID: <20140923131244.GC8870@kib.kiev.ua> References: <5420D5FC.4030600@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5420D5FC.4030600@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=0.2 required=5.0 tests=ALL_TRUSTED, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: FreeBSD FS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Sep 2014 13:12:58 -0000 On Mon, Sep 22, 2014 at 09:07:56PM -0500, Bryan Drewery wrote: > This is slightly different than my last report (involving > zfs_lookup/zfs_dirent). I still need to gather information from that > previous one. > > Here is what I have on this one. Quoted so Thunderbird doesn't trash it: > > > 1. Poudriere build startup, run for N=01,...,12 in parallel: > > a. mount tmpfs on /poudriere/data/.m/exp-10amd64-commit-test/N > > /poudriere/data/.m is a ZFS mount. > > b. cpdup /poudriere/data/.m/exp-10amd64-commit-test/ref to /poudriere/data/.m/exp-10amd64-commit-test/N > > [1] c. mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/N/.p > > 2. SIGINT during this phase: > > [2] a. umount -f all tmpfs > > b. cpdup processes are *not* killed. They continue running. Most of them fail due to their dst tmpfs being gone. > > [3] c. A cpdup process managed to continue running. Since the tmpfs was gone it started trying to write to the ZFS mount under it. (/poudriere/data/.m) > > > > # procstat -kka|egrep "(lockmgr|tmpfs)" > > [3] 76137 100919 cpdup - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0x862 vop_stdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa vget+0x67 cache_lookup+0x5b2 vfs_cache_lookup+0xac VOP_LOOKUP_APV+0xf1 lookup+0x5ad namei+0x4e4 kern_linkat+0x13d sys_link+0x28 filemon_wrapper_link+0x19 amd64_syscall+0x25a > > [1] 76346 102346 mkdir - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 sleeplk+0x14a __lockmgr_args+0xea1 vop_stdlock+0x3c VOP_LOCK1_APV+0xfc _vn_lock+0xaa lookup+0x496 namei+0x4e4 kern_mkdirat+0xcb amd64_syscall+0x25a Xfast_syscall+0xfb > > [2] 76286 102085 umount - mi_switch+0x179 sleepq_switch+0x152 sleepq_wait+0x43 _sleep+0x366 vfs_write_suspend+0xf0 vfs_write_suspend_umnt+0x38 tmpfs_unmount+0x6d dounmount+0x424 sys_unmount+0x2ec amd64_syscall+0x25a Xfast_syscall+0xfb > > > > [3] root 76137 0.7 0.0 14772 2844 3 D+ 3:12PM 0:05.28 cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref /poudriere/data/.m/exp-10amd64-commit-test/02 > > [1] root 76346 0.0 0.0 8200 1916 3 D+ 3:12PM 0:00.00 mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../03/.p > > [2] root 76286 [command taken from gdb] umount -f /poudriere/data/.m/exp-10amd64-commit-test/02 > > > > Locked vnodes: > > > > 0xfffff80fda0bcce8: tag zfs, type VDIR > > usecount 2, writecount 0, refcount 3 mountedhere 0x0 > > flags () > > v_object 0xfffff805fc81a300 ref 0 pages 0 > > lock type zfs: SHARED (count 2) with exclusive waiters pending > > mount /poudriere/data/.m from zroot/poudriere/data/.m > > path /poudriere/data/.m/exp-10am64-commit-test/02 > > 0xfffff8065de471d8: tag zfs, type VDIR > > usecount 1, writecount 0, refcount 3 mountedhere 0xfffff8058f582330 > > flags () > > lock type zfs: EXCL by thread 0xfffff8047c0eb920 (pid 76286) with exclusive and shared waiters pending > > mount /poudriere/data/.m from zroot/poudriere/data/.m > > path /poudriere/data/.m > > > > > > Processes: > > > > PID 76286 (umount -f /poudriere/data/.m/exp-10amd64-commit-test/02): > > Waiting in vfs_write_suspend(mp=0xfffff8058f582330) for mn_writeopcount==0 > > > > > > (kgdb) thread 521 > > [Switching to thread 521 (Thread 102085)]#0 sched_switch (td=0xfffff8047c0eb920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > 1932 cpuid = PCPU_GET(cpuid); > > (kgdb) backtrace > > #0 sched_switch (td=0xfffff8047c0eb920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 > > #2 0xffffffff80966af2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 > > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff8058f5823bc, pri=119) at /usr/src/sys/kern/subr_sleepqueue.c:631 > > #4 0xffffffff80928c86 in _sleep (ident=, lock=, priority=, wmesg=, sbt=, > > pr=) at /usr/src/sys/kern/kern_synch.c:255 > > #5 0xffffffff809df0b0 in vfs_write_suspend (mp=0xfffff8058f582330, flags=) at /usr/src/sys/kern/vfs_vnops.c:1793 > > #6 0xffffffff809df308 in vfs_write_suspend_umnt (mp=0xfffff8058f582330) at /usr/src/sys/kern/vfs_vnops.c:1848 > > #7 0xffffffff822ad9ed in tmpfs_unmount (mp=0xfffff8058f582330, mntflags=) at /usr/src/sys/modules/tmpfs/../../fs/tmpfs/tmpfs_vfsops.c:280 > > #8 0xffffffff809c7664 in dounmount (mp=0xfffff8058f582330, flags=134742016, td=0xfffff8047c0eb920) at /usr/src/sys/kern/vfs_mount.c:1310 > > #9 0xffffffff809c721c in sys_unmount (td=0xfffff8047c0eb920, uap=0xfffffe1248712b80) at /usr/src/sys/kern/vfs_mount.c:1201 > > #10 0xffffffff80d38b6a in amd64_syscall (td=0xfffff8047c0eb920, traced=0) at subr_syscall.c:133 > > #11 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 > > > > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_writeopcount > > $12 = 1 > > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_activevnodelistsize > > $13 = 1 > > (kgdb) p ((struct mount*)0xfffff8058f582330)->mnt_vnodecovered > > $25 = (struct vnode *) 0xfffff8065de471d8 > > (kgdb) p *(struct mount*)0xfffff8058f582330 > > $5 = {mnt_mtx = {lock_object = {lo_name = 0xffffffff80fc4c20 "struct mount mtx", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1ea00}, mtx_lock = 4}, mnt_gen = 23, > > mnt_list = {tqe_next = 0x0, tqe_prev = 0xfffff8117c2abce8}, mnt_op = 0xffffffff822b3e40, mnt_vfc = 0xffffffff822b3f60, mnt_vnodecovered = 0xfffff8065de471d8, mnt_syncer = 0x0, > > mnt_ref = 7357, mnt_nvnodelist = {tqh_first = 0xfffff803f6a691d8, tqh_last = 0xfffff8065dc871f8}, mnt_nvnodelistsize = 7356, mnt_activevnodelist = {tqh_first = 0xfffff806dff493b0, > > tqh_last = 0xfffff806dff49470}, mnt_activevnodelistsize = 1, mnt_writeopcount = 1, mnt_kern_flag = 150994953, mnt_flag = 4096, mnt_opt = 0xfffff804e377eaf0, mnt_optnew = 0x0, > > mnt_maxsymlinklen = 0, mnt_stat = {f_version = 537068824, f_type = 135, f_flags = 4096, f_bsize = 4096, f_iosize = 4096, f_blocks = 1835008, f_bfree = 1798116, f_bavail = 1798116, > > f_files = 25690112, f_ffree = 25682842, f_syncwrites = 0, f_asyncwrites = 0, f_syncreads = 0, f_asyncreads = 0, f_spare = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, f_namemax = 255, f_owner = 0, > > f_fsid = {val = {-2027880658, 135}}, f_charspare = '\0' , f_fstypename = "tmpfs\000\000\000\000\000\000\000\000\000\000", > > f_mntfromname = "tmpfs", '\0' , f_mntonname = "/poudriere/data/.m/exp-10amd64-commit-test/02", '\0' }, mnt_cred = 0xfffff8047c8a4a00, > > mnt_data = 0xfffff8000bd2b800, mnt_time = 0, mnt_iosize_max = 65536, mnt_export = 0x0, mnt_label = 0x0, mnt_hashseed = 3766194835, mnt_lockref = 0, mnt_secondary_writes = 0, > > mnt_secondary_accwrites = 0, mnt_susp_owner = 0xfffff8047c0eb920, mnt_gjprovider = 0x0, mnt_explock = {lock_object = {lo_name = 0xffffffff80fa4b29 "explock", lo_flags = 108199936, > > lo_data = 0, lo_witness = 0xfffffe0000b26c80}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 0, lk_pri = 96}, mnt_upper_link = {tqe_next = 0x0, tqe_prev = 0xfffff8005c842ca0}, > > mnt_uppers = {tqh_first = 0x0, tqh_last = 0xfffff8058f582650}} > > (kgdb) p (((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tqh_first) > > $30 = (struct vnode *) 0xfffff806dff493b0 > > (kgdb) p *(((struct mount*)0xfffff8058f582330)->mnt_activevnodelist->tqh_first) > > $24 = {v_tag = 0xffffffff822b2f76 "tmpfs", v_op = 0xffffffff822b3918, v_data = 0xfffff80ae07caae0, v_mount = 0xfffff8058f582330, v_nmntvnodes = {tqe_next = 0xfffff806071d11d8, > > tqe_prev = 0xfffff804d63963d0}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = { > > lh_first = 0x0}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffff806dff49400}, v_cache_dd = 0x0, v_lock = {lock_object = {lo_name = 0xffffffff822b2f76 "tmpfs", lo_flags = 116588544, > > lo_data = 0, lo_witness = 0xfffffe0000b29a00}, lk_lock = 1, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = { > > lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, mtx_lock = 4}, v_vnlock = 0xfffff806dff49418, v_actfreelist = { > > tqe_next = 0x0, tqe_prev = 0xfffff8058f5823a8}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xffffffff80fcd144 "bufobj interlock", lo_flags = 86179840, lo_data = 0, > > lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0xfffff809dc520a00, bo_synclist = {le_next = 0x0, le_prev = 0x0}, > > bo_private = 0xfffff806dff493b0, __bo_vnode = 0xfffff806dff493b0, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806dff494d0}, bv_root = {pt_root = 0}, bv_cnt = 0}, > > bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff806dff494f0}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 4096}, v_pollinfo = 0x0, > > v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff806dff49538}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, v_clen = 0, > > v_holdcnt = 1, v_usecount = 1, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 115340435, v_type = VREG} > > > > > > PID 76137 (cpdup -i0 -x /poudriere/data/.m/exp-10amd64-commit-test/ref /poudriere/data/.m/exp-10amd64-commit-test/02): > > Waiting for vget(vp=0xfffff8065de471d8) [which is mnt_vnodecovered from umount -f] > > > > (kgdb) thread 520 > > [Switching to thread 520 (Thread 100919)]#0 sched_switch (td=0xfffff8005cd2a490, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > 1932 cpuid = PCPU_GET(cpuid); > > (kgdb) backtrace > > #0 sched_switch (td=0xfffff8005cd2a490, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 > > #2 0xffffffff80966af2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 > > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff8065de47240, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631 > > #4 0xffffffff808fdf3a in sleeplk (lk=, flags=, ilk=, wmesg=, pri=, > > timo=) at /usr/src/sys/kern/kern_lock.c:225 > > #5 0xffffffff808fcfc2 in __lockmgr_args (lk=0xfffff8065de47240, flags=, ilk=0xfffff8065de47270, wmesg=, pri=, > > timo=, file=0xffffffff80fcd0e8 "/usr/src/sys/kern/vfs_subr.c", line=2137) at /usr/src/sys/kern/kern_lock.c:680 > > #6 0xffffffff809be27c in vop_stdlock (ap=) at lockmgr.h:98 > > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2082 > > #8 0xffffffff809ddbca in _vn_lock (vp=0xfffff8065de471d8, flags=, file=0xffffffff80fcd0e8 "/usr/src/sys/kern/vfs_subr.c", line=2137) at vnode_if.h:859 > > #9 0xffffffff809cde87 in vget (vp=0xfffff8065de471d8, flags=2097408, td=0xfffff8005cd2a490) at /usr/src/sys/kern/vfs_subr.c:2137 > > #10 0xffffffff809ba132 in cache_lookup (dvp=0xfffff80fda0bcce8, vpp=0xfffffe1247de78b8, cnp=0xfffffe1247de78e0, tsp=0x0, ticksp=0x0) at /usr/src/sys/kern/vfs_cache.c:673 > > #11 0xffffffff809bb2dc in vfs_cache_lookup (ap=) at /usr/src/sys/kern/vfs_cache.c:1038 > > #12 0xffffffff80e58e41 in VOP_LOOKUP_APV (vop=, a=) at vnode_if.c:127 > > #13 0xffffffff809c364d in lookup (ndp=0xfffffe1247de7858) at vnode_if.h:54 > > #14 0xffffffff809c2d84 in namei (ndp=0xfffffe1247de7858) at /usr/src/sys/kern/vfs_lookup.c:300 > > #15 0xffffffff809d76dd in kern_linkat (td=0xfffff8005cd2a490, fd1=, fd2=-100, path1=, path2=0x80141c600
, > > segflg=UIO_USERSPACE, follow=64) at /usr/src/sys/kern/vfs_syscalls.c:1572 > > #16 0xffffffff809d7518 in sys_link (td=0x0, uap=) at /usr/src/sys/kern/vfs_syscalls.c:1541 > > #17 0xffffffff820c64e9 in filemon_wrapper_link (td=0x0, uap=0xfffffe1247de7b80) at filemon_wrapper.c:364 > > #18 0xffffffff80d38b6a in amd64_syscall (td=0xfffff8005cd2a490, traced=0) at subr_syscall.c:133 > > #19 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 > > > > (kgdb) p *(struct vnode*)0xfffff8065de471d8 > > $34 = {v_tag = 0xffffffff81f0f8ff "zfs", v_op = 0xffffffff81f24b50, v_data = 0xfffff80cb12f22e0, v_mount = 0xfffff8005c8a8660, v_nmntvnodes = {tqe_next = 0xfffff80528f4d938, > > tqe_prev = 0xfffff804d659cb30}, v_un = {vu_mount = 0xfffff8058f582330, vu_socket = 0xfffff8058f582330, vu_cdev = 0xfffff8058f582330, vu_fifoinfo = 0xfffff8058f582330}, v_hashlist = { > > le_next = 0x0, le_prev = 0x0}, v_cache_src = {lh_first = 0x0}, v_cache_dst = {tqh_first = 0xfffff805d7724a80, tqh_last = 0xfffff805d7724aa0}, v_cache_dd = 0xfffff805d7724a80, > > v_lock = {lock_object = {lo_name = 0xffffffff81f0f8ff "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0xfffffe0000b28780}, lk_lock = 18446735296877738278, lk_exslpfail = 0, > > lk_timo = 51, lk_pri = 96}, v_interlock = {lock_object = {lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, > > mtx_lock = 4}, v_vnlock = 0xfffff8065de47240, v_actfreelist = {tqe_next = 0xfffff80528f4d938, tqe_prev = 0xfffff804d659cbd0}, v_bufobj = {bo_lock = {lock_object = { > > lo_name = 0xffffffff80fcd144 "bufobj interlock", lo_flags = 86179840, lo_data = 0, lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0x0, > > bo_synclist = {le_next = 0x0, le_prev = 0x0}, bo_private = 0xfffff8065de471d8, __bo_vnode = 0xfffff8065de471d8, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff8065de472f8}, > > bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff8065de47318}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, > > bo_bsize = 131072}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff8065de47360}, rl_currdep = 0x0}, v_cstart = 0, > > v_lasta = 0, v_lastw = 0, v_clen = 0, v_holdcnt = 3, v_usecount = 1, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 106816625, v_type = VDIR} > > > > (kgdb) frame 15 > > #15 0xffffffff809d76dd in kern_linkat (td=0xfffff8005cd2a490, fd1=, fd2=-100, path1=, path2=0x80141c600
, > > segflg=UIO_USERSPACE, follow=64) at /usr/src/sys/kern/vfs_syscalls.c:1572 > > 1572 if ((error = namei(&nd)) == 0) { > > (kgdb) info locals > > mp = (struct mount *) 0xfffff8058f582330 > > nd = {ni_dirp = 0x80141c600
, ni_segflg = UIO_USERSPACE, ni_rightsneeded = {cr_rights = {144115188080051200, 288230376151711744}}, ni_startdir = 0x0, > > ni_rootdir = 0xfffff8005c0ea1d8, ni_topdir = 0x0, ni_dirfd = -100, ni_strictrelative = 0, ni_filecaps = {fc_rights = {cr_rights = {0, 0}}, fc_ioctls = 0x0, fc_nioctls = -1, > > fc_fcntls = 0}, ni_vp = 0xfffff8065de471d8, ni_dvp = 0xfffff80fda0bcce8, ni_pathlen = 63, > > ni_next = 0xfffff80e918bb02d "/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", ni_loopcnt = 0, ni_cnd = {cn_nameiop = 1, cn_flags = 134236168, > > cn_thread = 0xfffff8005cd2a490, cn_cred = 0xfffff8005c22a700, cn_lkflags = 2097152, > > cn_pnbuf = 0xfffff80e918bb000 "/poudriere/data/.m/exp-10amd64-commit-test/02/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", > > cn_nameptr = 0xfffff80e918bb02b "02/usr/share/openssl/man/man3/CMS_SignerInfo_get0_signer_id.3.gz", cn_namelen = 2, cn_consume = 0}} > > rights = {cr_rights = {144115188080051200, 288230376151711744}} > > mp = (struct mount *) 0xfffff8058f582330 > > vp = (struct vnode *) 0xfffff806dff493b0 > > error = > > kern_linkat > > mp = (struct mount *) 0xfffff8058f582330 > > vp = (struct vnode *) 0xfffff806dff493b0 > > > > > > PID 76346 (mkdir -p /poudriere/data/.m/exp-10amd64-commit-test/ref/../03/.p) > > Waiting on vn_lock(0xfffff80fda0bcce8) > > > > (kgdb) thread 522 > > [Switching to thread 522 (Thread 102346)]#0 sched_switch (td=0xfffff80419f86920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > 1932 cpuid = PCPU_GET(cpuid); > > (kgdb) backtrace > > #0 sched_switch (td=0xfffff80419f86920, newtd=, flags=) at /usr/src/sys/kern/sched_ule.c:1932 > > #1 0xffffffff80929379 in mi_switch (flags=260, newtd=0x0) at /usr/src/sys/kern/kern_synch.c:493 > > #2 0xffffffff80966af2 in sleepq_switch (wchan=, pri=) at /usr/src/sys/kern/subr_sleepqueue.c:552 > > #3 0xffffffff80966953 in sleepq_wait (wchan=0xfffff80fda0bcd50, pri=96) at /usr/src/sys/kern/subr_sleepqueue.c:631 > > #4 0xffffffff808fdf3a in sleeplk (lk=, flags=, ilk=, wmesg=, pri=, > > timo=) at /usr/src/sys/kern/kern_lock.c:225 > > #5 0xffffffff808fd601 in __lockmgr_args (lk=0xfffff80fda0bcd50, flags=, ilk=0xfffff80fda0bcd80, wmesg=, pri=, > > timo=, file=0xffffffff80fcc4f0 "/usr/src/sys/kern/vfs_lookup.c", line=691) at /usr/src/sys/kern/kern_lock.c:939 > > #6 0xffffffff809be27c in vop_stdlock (ap=) at lockmgr.h:98 > > #7 0xffffffff80e5b75c in VOP_LOCK1_APV (vop=, a=) at vnode_if.c:2082 > > #8 0xffffffff809ddbca in _vn_lock (vp=0xfffff80fda0bcce8, flags=, file=0xffffffff80fcc4f0 "/usr/src/sys/kern/vfs_lookup.c", line=691) at vnode_if.h:859 > > #9 0xffffffff809c3536 in lookup (ndp=0xfffffe1248b7b910) at /usr/src/sys/kern/vfs_lookup.c:691 > > #10 0xffffffff809c2d84 in namei (ndp=0xfffffe1248b7b910) at /usr/src/sys/kern/vfs_lookup.c:300 > > #11 0xffffffff809daf2b in kern_mkdirat (td=0xfffff80419f86920, fd=-100, path=0x7fffffffeca1
, segflg=UIO_USERSPACE, mode=511) > > at /usr/src/sys/kern/vfs_syscalls.c:3687 > > #12 0xffffffff80d38b6a in amd64_syscall (td=0xfffff80419f86920, traced=0) at subr_syscall.c:133 > > #13 0xffffffff80d1a44b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:390 > > #14 0x0000000800953faa in ?? () > > > > (kgdb) p *(struct vnode*) 0xfffff80fda0bcce8 > > $35 = {v_tag = 0xffffffff81f0f8ff "zfs", v_op = 0xffffffff81f24b50, v_data = 0xfffff80c56bf8b80, v_mount = 0xfffff8005c8a8660, v_nmntvnodes = {tqe_next = 0xfffff806d8e1d1d8, > > tqe_prev = 0xfffff80df75d51f8}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0x0, le_prev = 0x0}, v_cache_src = { > > lh_first = 0xfffff805d7724a80}, v_cache_dst = {tqh_first = 0xfffff805df597690, tqh_last = 0xfffff805df5976b0}, v_cache_dd = 0xfffff805df597690, v_lock = {lock_object = { > > lo_name = 0xffffffff81f0f8ff "zfs", lo_flags = 117112832, lo_data = 0, lo_witness = 0xfffffe0000b28780}, lk_lock = 21, lk_exslpfail = 0, lk_timo = 51, lk_pri = 96}, v_interlock = { > > lock_object = {lo_name = 0xffffffff80fc4bc6 "vnode interlock", lo_flags = 16973824, lo_data = 0, lo_witness = 0xfffffe0000b1e500}, mtx_lock = 4}, v_vnlock = 0xfffff80fda0bcd50, > > v_actfreelist = {tqe_next = 0xfffff806d8e1d1d8, tqe_prev = 0xfffff80528f4d9f8}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xffffffff80fcd144 "bufobj interlock", > > lo_flags = 86179840, lo_data = 0, lo_witness = 0xfffffe0000b26580}, rw_lock = 1}, bo_ops = 0xffffffff814ab900, bo_object = 0xfffff805fc81a300, bo_synclist = {le_next = 0x0, > > le_prev = 0x0}, bo_private = 0xfffff80fda0bcce8, __bo_vnode = 0xfffff80fda0bcce8, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce08}, bv_root = {pt_root = 0}, > > bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce28}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_numoutput = 0, bo_flag = 0, bo_bsize = 131072}, > > v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0x0, tqh_last = 0xfffff80fda0bce70}, rl_currdep = 0x0}, v_cstart = 0, v_lasta = 0, v_lastw = 0, > > v_clen = 0, v_holdcnt = 3, v_usecount = 2, v_iflag = 512, v_vflag = 0, v_writecount = 0, v_hash = 18, v_type = VDIR} > > > > Let me know if I can provide any further information. There is indeed some write op in flight on the tmpfs mount, which blocks unmount suspension. The 'bt all' from ddb would give some overview of the threads states. Follow https://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html to gather required information.