From owner-freebsd-current@freebsd.org Tue May 12 22:14:13 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 172572DDE0A for ; Tue, 12 May 2020 22:14:13 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49MBsw6p5rz43Jq for ; Tue, 12 May 2020 22:14:12 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id 56A285D0D for ; Tue, 12 May 2020 22:14:12 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 777011AF72 for ; Tue, 12 May 2020 22:14:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id UaqklSpjPevP for ; Tue, 12 May 2020 22:14:01 +0000 (UTC) To: freebsd-current@FreeBSD.org DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com 42F111AF68 From: Bryan Drewery Autocrypt: addr=bdrewery@FreeBSD.org; prefer-encrypt=mutual; keydata= mQENBFJphmsBCADiFgmS4bIzwZijrS31SjEMzg+n5zNellgM+HkShwehpqCiyhXdWrvH6dTZ a6u50pbUIX7doTR7W7PQHCjCTqtpwvcj0eulZva+iHFp+XrbgSFHn+VVXgkYP2MFySyZRFab D2qqzJBEJofhpv4HvY6uQI5K99pMqKr1Z/lHqsijYYu4RH2OfwB5PinId7xeldzWEonVoCr+ rfxzO/UrgA6v/3layGZcKNHFjmc3NqoN1DXtdaEHqtjIozzbndVkH6lkFvIpIrI6i5ox8pwp VxsxLCr/4Musd5CWgHiet5kSw2SzNeA8FbxdLYCpXNVu+uBACEbCUP+CSNy3NVfEUxsBABEB AAG0JEJyeWFuIERyZXdlcnkgPGJkcmV3ZXJ5QEZyZWVCU0Qub3JnPokBVwQTAQoAQQIbAwUL CQgHAwUVCgkICwUWAwIBAAIeAQIXgAIZARYhBPkXPLLDqup6XIofCTXXcbtuRpfPBQJb5hLu BQkNPvODAAoJEDXXcbtuRpfP9rMH/3f7cfX5rzyEV5QNfV/wS4jFukLoPZ4+nCM/TKxH3pEX 2bLbeQbkk6La8cueQ5Lpoht5XFZ18Y5TbMittngltrlNzoDD0h9are24OkDFGim3afJU7tkj IGQa1if+re+vI5BhzYwRhj0oKXzBi39M5oePd3L1dXfx83rg2FPyZBdIejsz6fR74T3JVkbd 6k2l5/3Zk2uiNMy+eBfDRgYE1E6CP28kV0nCeGKZgSVso0kGUUHud7voKqGVpMvbd0mE4pp4 PE5YJaFPjrll9miaDAvdU8LGIq5n6+aXPLKoQ/QNl6mg6ifgI6FfKILOkTizLW8E5PBSNnCm NapQ55yjm125AQ0EUmmGawEIAKJUU9+Q19oW1RK5jTf3m56j+szIc8Y9DaLC8REUKl4UZJBK BqCl6c0cukVApOD92XoU6hJPm2rLEyp/IcYcPPNTnVu8D8h9oag2L8EiFN7+2hk0xG+lwjc8 uOIZycme7AIJsBU4AZ1v63lxm2k104hwpiatgbe71GIGl7p1MX6ousP/wGzXCOF25Dx9w02C eRe7zEMfhnFjSUhzdCC9han2+KaVB7qIqNR3b8NfbwRNlwPmHqlhXffUow9OsQjSnTK8WKNR lx7xzVccXIvWP2wECFrmqmzMmXpSrmIuiWEpFwZ9x2a0Pva8dCNRiCVTK51IlRXKjaAxiN1u IUrMm6UAEQEAAYkBPAQYAQoAJgIbDBYhBPkXPLLDqup6XIofCTXXcbtuRpfPBQJb5hL4BQkN PvONAAoJEDXXcbtuRpfPCjcH/ivBsOpdpebpgLizSNU5/X4yWN5Aixsc9VBnQhGKAKnMINJQ VMpA55sD2JSPwloXYM/B3qyPJRS/9cwIuX5LDNKKOZU3Qp+TzleynM15/xea14orWYRGRict YHBM3Cnqp7OD8K6Q1uhs0fTxyJP7PZ/G0+7Corlf1DlHhDt6C2HldRPFvAvAgl6sR9Wzgcb7 rzub2HVtbJgl6YHbgyAG7x9NpXFqzx1JLAMdpt2DIYwoi+oMdRQlBIwNuKjQjCGzuXHandd3 kGvBAsyJpQ+coEep9UzwANaV28cXrFr2R4FSOcR50rBA2Nh/vqUYfpsvBvJlwuKAoV1djVHa ihNeL5E= Organization: FreeBSD Subject: zfs deadlock on r360452 relating to busy vm page Message-ID: <2bdc8563-283b-32cc-8a1a-85ff52aca99e@FreeBSD.org> Date: Tue, 12 May 2020 15:13:55 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="aIvlquH7auoI17NOayHMofyL3SITm4BOn" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 May 2020 22:14:13 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --aIvlquH7auoI17NOayHMofyL3SITm4BOn Content-Type: multipart/mixed; boundary="XElGlYQBYiLZJf7vq6TsZf2eItNVgQjfg"; protected-headers="v1" From: Bryan Drewery To: freebsd-current@FreeBSD.org Message-ID: <2bdc8563-283b-32cc-8a1a-85ff52aca99e@FreeBSD.org> Subject: zfs deadlock on r360452 relating to busy vm page --XElGlYQBYiLZJf7vq6TsZf2eItNVgQjfg Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable > panic: deadlres_td_sleep_q: possible deadlock detected for 0xfffffe25ee= fa2e00 (find), blocked for 1802392 ticks 2 stuck processes from procstat -kk before panic > 72559 101698 tail - mi_switch+0x155 sl= eepq_switch+0x11a _cv_wait+0x15a rangelock_enter+0x306 zfs_freebsd_getpag= es+0x14f VOP_GETPAGES_APV+0x59 vnode_pager_getpages+0x37 vm_pager_get_pag= es+0x4f vm_fault+0x780 vm_fault_trap+0x6e trap_pfault+0x1ee > 72985 107378 find - mi_switch+0x155 sl= eepq_switch+0x11a sleeplk+0x106 lockmgr_slock_hard+0x1f5 VOP_LOCK1_APV+0x= 40 _vn_lock+0x54 lookup+0xdd namei+0x524 vn_open_cred+0x32b kern_openat+0= x1fa filemon_wrapper_openat+0x15 amd64_syscall+0x73d The only find running was thread 107378 I couldn't record much from ddb but got locked vnodes. >=20 > db> show lockedvnods > Locked vnodes > vnode 0xfffff804de66e500: type VDIR > usecount 3, writecount 0, refcount 2 mountedhere 0 > flags () > v_object 0xfffff809459cb420 ref 0 pages 0 cleanbuf 0 dirtybuf 0 > lock type zfs: SHARED (count 1) > #0 0xffffffff80b94a0f at lockmgr_slock+0xdf > #1 0xffffffff810e2a40 at VOP_LOCK1_APV+0x40 > #2 0xffffffff80cb14f4 at _vn_lock+0x54 > #3 0xffffffff80c9b3ec at vget_finish+0x6c > #4 0xffffffff80c8051c at cache_lookup+0x57c > #5 0xffffffff80c84dad at vfs_cache_lookup+0x7d > #6 0xffffffff810df996 at VOP_LOOKUP_APV+0x56 > #7 0xffffffff80c8ee61 at lookup+0x601 > #8 0xffffffff80c8e374 at namei+0x524 > #9 0xffffffff80caa83f at kern_statat+0x7f > #10 0xffffffff80caafff at sys_fstatat+0x2f > #11 0xffffffff81065c40 at amd64_syscall+0x140 > #12 0xffffffff8103b2a0 at fast_syscall_common+0x101 > vnode 0xfffff808a08f0a00: type VDIR > usecount 6, writecount 0, refcount 2 mountedhere 0 > flags () > v_object 0xfffff801eb930000 ref 0 pages 0 cleanbuf 0 dirtybuf 0 > lock type zfs: EXCL by thread 0xfffffe24aadb6100 (pid 72267, gmake,= tid 104356) > with shared waiters pending > #0 0xffffffff80b94a0f at lockmgr_slock+0xdf > #1 0xffffffff810e2a40 at VOP_LOCK1_APV+0x40 > #2 0xffffffff80cb14f4 at _vn_lock+0x54 > #3 0xffffffff80c8e93d at lookup+0xdd > #4 0xffffffff80c8e374 at namei+0x524 > #5 0xffffffff80ca9e69 at kern_funlinkat+0xa9 > #6 0xffffffff80ca9db8 at sys_unlink+0x28 > #7 0xffffffff82780586 at filemon_wrapper_unlink+0x16 > #8 0xffffffff8106623d at amd64_syscall+0x73d > #9 0xffffffff8103b2a0 at fast_syscall_common+0x101 >=20 > vnode 0xfffff80571f29500: type VREG > usecount 6, writecount 1, refcount 2 > flags () > v_object 0xfffff806cb637c60 ref 2 pages 1 cleanbuf 0 dirtybuf 0 > lock type zfs: SHARED (count 2) > with exclusive waiters pending > #0 0xffffffff80b94a0f at lockmgr_slock+0xdf > #1 0xffffffff810e2a40 at VOP_LOCK1_APV+0x40 > #2 0xffffffff80cb14f4 at _vn_lock+0x54 > #3 0xffffffff8243af40 at zfs_lookup+0x610 > #4 0xffffffff8243b61e at zfs_freebsd_cachedlookup+0x8e > #5 0xffffffff810dfb46 at VOP_CACHEDLOOKUP_APV+0x56 > #6 0xffffffff80c84dd8 at vfs_cache_lookup+0xa8 > #7 0xffffffff810df996 at VOP_LOOKUP_APV+0x56 > #8 0xffffffff80c8ee61 at lookup+0x601 > #9 0xffffffff80c8e374 at namei+0x524 > #10 0xffffffff80caa83f at kern_statat+0x7f > #11 0xffffffff80caafff at sys_fstatat+0x2f > #12 0xffffffff8106623d at amd64_syscall+0x73d > #13 0xffffffff8103b2a0 at fast_syscall_common+0x101 It's nice how recent threads are at the top in gdb... > (kgdb) info threads > Id Target Id Frame > 1 Thread 107952 (PID=3D79390: zfs) sche= d_switch (td=3D0xfffffe26ebb36000, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 2 Thread 102764 (PID=3D73218: zfs) sche= d_switch (td=3D0xfffffe2490a12300, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 3 Thread 107378 (PID=3D72985: find) sche= d_switch (td=3D0xfffffe25eefa2e00, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 4 Thread 103940 (PID=3D72980: rm) sche= d_switch (td=3D0xfffffe2451932500, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 5 Thread 101698 (PID=3D72559: tail) sche= d_switch (td=3D0xfffffe255eac0000, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 6 Thread 103660 (PID=3D72280: timestamp) sche= d_switch (td=3D0xfffffe25f948aa00, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 7 Thread 101249 (PID=3D72280: timestamp/prefix_stdout) sche= d_switch (td=3D0xfffffe264412a100, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 8 Thread 101255 (PID=3D72280: timestamp/prefix_stderr) sche= d_switch (td=3D0xfffffe25c8e9bc00, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 9 Thread 104356 (PID=3D72267: gmake) sche= d_switch (td=3D0xfffffe24aadb6100, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 > 10 Thread 108476 (PID=3D66957: vim) sche= d_switch (td=3D0xfffffe26c8601500, flags=3D) at /usr/src/s= ys/kern/sched_ule.c:2147 The 2 threads holding shared lock on vnode 0xfffff80571f29500: The tail thread (101698) is waiting for a zfs rangelock getting pages for vnode 0xfffff80571f29500 > (kgdb) thread 5 > [Switching to thread 5 (Thread 101698)] > #0 sched_switch (td=3D0xfffffe255eac0000, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffffe255eac0000, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern/= kern_synch.c:542 > #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff810fb57dd48, pr= i=3D0) at /usr/src/sys/kern/subr_sleepqueue.c:625 > #3 0xffffffff80b57f0a in _cv_wait (cvp=3D0xfffff810fb57dd48, lock=3D0x= fffff80049a99040) at /usr/src/sys/kern/kern_condvar.c:146 > #4 0xffffffff82434ab6 in rangelock_enter_reader (rl=3D0xfffff80049a990= 18, new=3D0xfffff8022cadb100) at /usr/src/sys/cddl/contrib/opensolaris/ut= s/common/fs/zfs/zfs_rlock.c:429 > #5 rangelock_enter (rl=3D0xfffff80049a99018, off=3D, le= n=3D, type=3D) at /usr/src/sys/cddl/contrib= /opensolaris/uts/common/fs/zfs/zfs_rlock.c:477 > #6 0xffffffff82443d3f in zfs_getpages (vp=3D, ma=3D0xff= fffe259f204b18, count=3D, rbehind=3D0xfffffe259f204ac4, ra= head=3D0xfffffe259f204ad0) at /usr/src/sys/cddl/contrib/opensolaris/uts/c= ommon/fs/zfs/zfs_vnops.c:4500 > #7 zfs_freebsd_getpages (ap=3D) at /usr/src/sys/cddl/co= ntrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4567 > #8 0xffffffff810e3ab9 in VOP_GETPAGES_APV (vop=3D0xffffffff8250a1e0 , a=3D0xfffffe259f2049f0) at vnode_if.c:2644 > #9 0xffffffff80f349e7 in VOP_GETPAGES (vp=3D, m=3D, count=3D, rbehind=3D, rahead=3D) at ./vnode_if.h:1171 > #10 vnode_pager_getpages (object=3D, m=3D= , count=3D, rbehind=3D, rahead=3D)= at /usr/src/sys/vm/vnode_pager.c:743 > #11 0xffffffff80f2a93f in vm_pager_get_pages (object=3D0xfffff806cb637c= 60, m=3D0xfffffe259f204b18, count=3D1, rbehind=3D, rahead=3D= ) at /usr/src/sys/vm/vm_pager.c:305 > #12 0xffffffff80f054b0 in vm_fault_getpages (fs=3D, nera= =3D0, behindp=3D, aheadp=3D) at /usr/src/sy= s/vm/vm_fault.c:1163 > #13 vm_fault (map=3D, vaddr=3D, fault_typ= e=3D, fault_flags=3D, m_hold=3D) at /usr/src/sys/vm/vm_fault.c:1394 > #14 0xffffffff80f04bde in vm_fault_trap (map=3D0xfffffe25653949e8, vadd= r=3D, fault_type=3D, fault_flags=3D0, signo= =3D0xfffffe259f204d04, ucode=3D0xfffffe259f204d00) at /usr/src/sys/vm/vm_= fault.c:589 > #15 0xffffffff8106544e in trap_pfault (frame=3D0xfffffe259f204d40, user= mode=3D, signo=3D, ucode=3D) a= t /usr/src/sys/amd64/amd64/trap.c:821 > #16 0xffffffff81064a9c in trap (frame=3D0xfffffe259f204d40) at /usr/src= /sys/amd64/amd64/trap.c:340 > #17 > #18 0x00000000002034fc in ?? () > (kgdb) frame 11 > #11 0xffffffff80f2a93f in vm_pager_get_pages (object=3D0xfffff806cb637c= 60, m=3D0xfffffe259f204b18, count=3D1, rbehind=3D, rahead=3D= ) at /usr/src/sys/vm/vm_pager.c:305 > 305 r =3D (*pagertab[object->type]->pgo_getpages)(object, m= , count, rbehind, > (kgdb) p *object > $10 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff8114fa30 "vm = object", lo_flags =3D 627245056, lo_data =3D 0, lo_witness =3D 0x0}, rw_l= ock =3D 1}, object_list =3D {tqe_next =3D 0xfffff806cb637d68, tqe_prev =3D= 0xfffff806cb637b78}, shadow_head =3D {lh_first =3D 0x0}, shadow_list =3D= {le_next =3D 0xffffffffffffffff, > le_prev =3D 0xffffffffffffffff}, memq =3D {tqh_first =3D 0xfffffe00= 1cbca850, tqh_last =3D 0xfffffe001cbca860}, rtree =3D {rt_root =3D 184467= 41875168421969}, size =3D 1099, domain =3D {dr_policy =3D 0x0, dr_iter =3D= 0}, generation =3D 1, cleangeneration =3D 1, ref_count =3D 2, shadow_cou= nt =3D 0, memattr =3D 6 '\006', type =3D 2 '\002', > flags =3D 4096, pg_color =3D 0, paging_in_progress =3D {__count =3D 2= }, busy =3D {__count =3D 0}, resident_page_count =3D 1, backing_object =3D= 0x0, backing_object_offset =3D 0, pager_object_list =3D {tqe_next =3D 0x= 0, tqe_prev =3D 0x0}, rvq =3D {lh_first =3D 0x0}, handle =3D 0xfffff80571= f29500, un_pager =3D {vnp =3D {vnp_size =3D 4499568, > writemappings =3D 0}, devp =3D {devp_pglist =3D {tqh_first =3D 0x= 44a870, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}, sgp =3D {sgp_pglist= =3D {tqh_first =3D 0x44a870, tqh_last =3D 0x0}}, swp =3D {swp_tmpfs =3D = 0x44a870, swp_blks =3D {pt_root =3D 0}, writemappings =3D 0}}, cred =3D 0= x0, charge =3D 0, umtx_data =3D 0x0} > (kgdb) p object->handle > $11 =3D (void *) 0xfffff80571f29500 > (kgdb) p *(struct vnode *) 0xfffff80571f29500 > $18 =3D {v_type =3D VREG, v_irflag =3D 0, v_op =3D 0xffffffff8250a1e0 <= zfs_vnodeops>, v_data =3D 0xfffff80049a99000, v_mount =3D 0xfffffe247e5f3= 700, v_nmntvnodes =3D {tqe_next =3D 0xfffff8086eb38a00, tqe_prev =3D 0xff= fff80461c2d7a0}, {v_mountedhere =3D 0x0, v_unpcb =3D 0x0, v_rdev =3D 0x0,= v_fifoinfo =3D 0x0}, v_hashlist =3D {le_next =3D 0x0, > le_prev =3D 0x0}, v_cache_src =3D {lh_first =3D 0x0}, v_cache_dst =3D= {tqh_first =3D 0x0, tqh_last =3D 0xfffff80571f29550}, v_cache_dd =3D 0x0= , v_lock =3D {lock_object =3D {lo_name =3D 0xffffffff82486a37 "zfs", lo_f= lags =3D 117112832, lo_data =3D 0, lo_witness =3D 0x0}, lk_lock =3D 37, l= k_exslpfail =3D 0, lk_timo =3D 51, lk_pri =3D 96, > lk_stack =3D {depth =3D 14, pcs =3D {18446744071574211087, 18446744= 071579773504, 18446744071575377140, 18446744071600058176, 184467440716000= 59934, 18446744071579761478, 18446744071575195096, 18446744071579761046, = 18446744071575236193, 18446744071575233396, 18446744071575349311, 1844674= 4071575351295, > 18446744071579263549, 18446744071579087520, 0, 0, 0, 0}}}, v_in= terlock =3D {lock_object =3D {lo_name =3D 0xffffffff8123c142 "vnode inter= lock", lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0xfffff8123fd= 73600}, mtx_lock =3D 0}, v_vnlock =3D 0xfffff80571f29568, v_vnodelist =3D= {tqe_next =3D 0xfffff8064bd0dc80, > tqe_prev =3D 0xfffff80e250788d8}, v_lazylist =3D {tqe_next =3D 0x0,= tqe_prev =3D 0x0}, v_bufobj =3D {bo_lock =3D {lock_object =3D {lo_name =3D= 0xffffffff811fb7ab "bufobj interlock", lo_flags =3D 86179840, lo_data =3D= 0, lo_witness =3D 0xfffff8123fd7dd80}, rw_lock =3D 1}, bo_ops =3D 0xffff= ffff8191ead0 , > bo_object =3D 0xfffff806cb637c60, bo_synclist =3D {le_next =3D 0x0,= le_prev =3D 0x0}, bo_private =3D 0xfffff80571f29500, bo_clean =3D {bv_hd= =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff80571f296c0}, bv_root =3D {p= t_root =3D 0}, bv_cnt =3D 0}, bo_dirty =3D {bv_hd =3D {tqh_first =3D 0x0,= tqh_last =3D 0xfffff80571f296e0}, bv_root =3D {pt_root =3D 0}, > bv_cnt =3D 0}, bo_numoutput =3D 0, bo_flag =3D 0, bo_domain =3D 5= , bo_bsize =3D 131072}, v_pollinfo =3D 0x0, v_label =3D 0x0, v_lockf =3D = 0x0, v_rl =3D {rl_waiters =3D {tqh_first =3D 0xfffff80f2cc12708, tqh_last= =3D 0xfffff80f2cc12708}, rl_currdep =3D 0x0}, v_cstart =3D 0, v_lasta =3D= 0, v_lastw =3D 0, v_clen =3D 0, v_holdcnt =3D 2, > v_usecount =3D 6, v_iflag =3D 0, v_vflag =3D 0, v_mflag =3D 0, v_dbat= chcpu =3D -1, v_writecount =3D 1, v_hash =3D 45676874} Is that thread busying the vm object? thread 101255 (timestamp/prefix_stderr) which is also acting on vnode 0xfffff80571f29500 that the tail thread 101698 was. > (kgdb) thread > [Current thread is 8 (Thread 101255)] > (kgdb) backtrace > #0 sched_switch (td=3D0xfffffe25c8e9bc00, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern/= kern_synch.c:542 > #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffffe001cbca850, pr= i=3D84) at /usr/src/sys/kern/subr_sleepqueue.c:625 > #3 0xffffffff80f1de50 in _vm_page_busy_sleep (obj=3D, m= =3D0xfffffe001cbca850, pindex=3D, wmesg=3D,= allocflags=3D21504, locked=3Dfalse) at /usr/src/sys/vm/vm_page.c:1094 > #4 0xffffffff80f241f7 in vm_page_grab_sleep (object=3D0xfffff806cb637c= 60, m=3D, pindex=3D, wmesg=3D, allocflags=3D21504, locked=3D) at /usr/src/sys/vm/vm_page.c:4326 > #5 vm_page_acquire_unlocked (object=3D0xfffff806cb637c60, pindex=3D109= 8, prev=3D, mp=3D0xfffffe2717fc6730, allocflags=3D21504) a= t /usr/src/sys/vm/vm_page.c:4469 > #6 0xffffffff80f24c61 in vm_page_grab_valid_unlocked (mp=3D0xfffffe271= 7fc6730, object=3D0xfffff806cb637c60, pindex=3D1098, allocflags=3D21504) = at /usr/src/sys/vm/vm_page.c:4645 > #7 0xffffffff82440246 in page_busy (vp=3D0xfffff80571f29500, start=3D4= 497408, off=3D, nbytes=3D) at /usr/src/sys/= cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:414 > #8 update_pages (vp=3D0xfffff80571f29500, start=3D4497408, len=3D32, o= s=3D0xfffff8096a277400, oid=3D2209520, segflg=3D, tx=3D) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/= zfs_vnops.c:482 > #9 zfs_write (vp=3D, uio=3D, ioflag=3D0,= cr=3D, ct=3D) at /usr/src/sys/cddl/contrib= /opensolaris/uts/common/fs/zfs/zfs_vnops.c:1071 > #10 zfs_freebsd_write (ap=3D) at /usr/src/sys/cddl/contr= ib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4838 > #11 0xffffffff810e0eaf in VOP_WRITE_APV (vop=3D0xffffffff8250a1e0 , a=3D0xfffffe2717fc68c8) at vnode_if.c:925 > #12 0xffffffff80cb574c in VOP_WRITE (vp=3D0xfffff80571f29500, uio=3D0xf= ffffe2717fc6bb0, ioflag=3D8323073, cred=3D) at ./vnode_if.= h:413 > #13 vn_write (fp=3D0xfffff8048195e8c0, uio=3D, active_cr= ed=3D, flags=3D, td=3D) at /= usr/src/sys/kern/vfs_vnops.c:894 > #14 0xffffffff80cb50c3 in vn_io_fault_doio (args=3D0xfffffe2717fc6af0, = uio=3D0xfffffe2717fc6bb0, td=3D0xfffffe25c8e9bc00) at /usr/src/sys/kern/v= fs_vnops.c:959 > #15 0xffffffff80cb1c8c in vn_io_fault1 (vp=3D, uio=3D0xf= ffffe2717fc6bb0, args=3D0xfffffe2717fc6af0, td=3D0xfffffe25c8e9bc00) at /= usr/src/sys/kern/vfs_vnops.c:1077 > #16 0xffffffff80cafa32 in vn_io_fault (fp=3D0xfffff8048195e8c0, uio=3D0= xfffffe2717fc6bb0, active_cred=3D0xfffff80f2cc12708, flags=3D0, td=3D) at /usr/src/sys/kern/vfs_vnops.c:1181 > #17 0xffffffff80c34331 in fo_write (fp=3D0xfffff8048195e8c0, uio=3D0xff= fffe2717fc6bb0, active_cred=3D, flags=3D, td=3D= 0xfffffe25c8e9bc00) at /usr/src/sys/sys/file.h:326 > #18 dofilewrite (td=3D0xfffffe25c8e9bc00, fd=3D2, fp=3D0xfffff8048195e8= c0, auio=3D0xfffffe2717fc6bb0, offset=3D, flags=3D) at /usr/src/sys/kern/sys_generic.c:564 > #19 0xffffffff80c33eb0 in kern_writev (td=3D0xfffffe25c8e9bc00, fd=3D2,= auio=3D) at /usr/src/sys/kern/sys_generic.c:491 > #20 sys_write (td=3D0xfffffe25c8e9bc00, uap=3D) at /usr/= src/sys/kern/sys_generic.c:406 > #21 0xffffffff8106623d in syscallenter (td=3D) at /usr/s= rc/sys/amd64/amd64/../../kern/subr_syscall.c:150 > #22 amd64_syscall (td=3D0xfffffe25c8e9bc00, traced=3D0) at /usr/src/sys= /amd64/amd64/trap.c:1161 > #23 > #24 0x000000080043d53a in ?? () Maybe r358443 is related? > (kgdb) frame 4 > #4 0xffffffff80f241f7 in vm_page_grab_sleep (object=3D0xfffff806cb637c= 60, m=3D, pindex=3D, wmesg=3D, allocflags=3D21504, locked=3D) at /usr/src/sys/vm/vm_page.c:4326 > 4326 if (_vm_page_busy_sleep(object, m, m->pindex, wmesg, al= locflags, > (kgdb) p *object > $8 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff8114fa30 "vm o= bject", lo_flags =3D 627245056, lo_data =3D 0, lo_witness =3D 0x0}, rw_lo= ck =3D 1}, object_list =3D {tqe_next =3D 0xfffff806cb637d68, tqe_prev =3D= 0xfffff806cb637b78}, shadow_head =3D {lh_first =3D 0x0}, shadow_list =3D= {le_next =3D 0xffffffffffffffff, > le_prev =3D 0xffffffffffffffff}, memq =3D {tqh_first =3D 0xfffffe00= 1cbca850, tqh_last =3D 0xfffffe001cbca860}, rtree =3D {rt_root =3D 184467= 41875168421969}, size =3D 1099, domain =3D {dr_policy =3D 0x0, dr_iter =3D= 0}, generation =3D 1, cleangeneration =3D 1, ref_count =3D 2, shadow_cou= nt =3D 0, memattr =3D 6 '\006', type =3D 2 '\002', > flags =3D 4096, pg_color =3D 0, paging_in_progress =3D {__count =3D 2= }, busy =3D {__count =3D 0}, resident_page_count =3D 1, backing_object =3D= 0x0, backing_object_offset =3D 0, pager_object_list =3D {tqe_next =3D 0x= 0, tqe_prev =3D 0x0}, rvq =3D {lh_first =3D 0x0}, handle =3D 0xfffff80571= f29500, un_pager =3D {vnp =3D {vnp_size =3D 4499568, > writemappings =3D 0}, devp =3D {devp_pglist =3D {tqh_first =3D 0x= 44a870, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}, sgp =3D {sgp_pglist= =3D {tqh_first =3D 0x44a870, tqh_last =3D 0x0}}, swp =3D {swp_tmpfs =3D = 0x44a870, swp_blks =3D {pt_root =3D 0}, writemappings =3D 0}}, cred =3D 0= x0, charge =3D 0, umtx_data =3D 0x0} > (kgdb) frame 5 > #5 vm_page_acquire_unlocked (object=3D0xfffff806cb637c60, pindex=3D109= 8, prev=3D, mp=3D0xfffffe2717fc6730, allocflags=3D21504) a= t /usr/src/sys/vm/vm_page.c:4469 > 4469 if (!vm_page_grab_sleep(object, m, pindex, "pgn= slp", > (kgdb) p *m > $9 =3D {plinks =3D {q =3D {tqe_next =3D 0xffffffffffffffff, tqe_prev =3D= 0xffffffffffffffff}, s =3D {ss =3D {sle_next =3D 0xffffffffffffffff}}, m= emguard =3D {p =3D 18446744073709551615, v =3D 18446744073709551615}, uma= =3D {slab =3D 0xffffffffffffffff, zone =3D 0xffffffffffffffff}}, listq =3D= {tqe_next =3D 0x0, tqe_prev =3D 0xfffff806cb637ca8}, > object =3D 0xfffff806cb637c60, pindex =3D 1098, phys_addr =3D 1898840= 8832, md =3D {pv_list =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffffe001cbc= a888}, pv_gen =3D 44682, pat_mode =3D 6}, ref_count =3D 2147483648, busy_= lock =3D 1588330502, a =3D {{flags =3D 0, queue =3D 255 '\377', act_count= =3D 0 '\000'}, _bits =3D 16711680}, order =3D 13 '\r', > pool =3D 0 '\000', flags =3D 1 '\001', oflags =3D 0 '\000', psind =3D= 0 '\000', segind =3D 6 '\006', valid =3D 0 '\000', dirty =3D 0 '\000'} Pretty sure this thread is holding the rangelock from zfs_write() that tail is waiting on. So what is this thread (101255) waiting on exactly for? I'm not sure the way to track down what is using vm object 0xfffff806cb637c60. If the tail thread busied the page then they are waiting on each other I guess. If that's true then r358443 removing the write lock on the object in update_pages() could be a problem. Not sure the rest is interesting. I think they are just waiting on the locked vnode but I give it here in case I missed something. thread 101249 (timestamp/prefix_stdout) is also acting on vnode 0xfffff80571f29500 > (kgdb) thread 7 > [Switching to thread 7 (Thread 101249)] > #0 sched_switch (td=3D0xfffffe264412a100, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffffe264412a100, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern/= kern_synch.c:542 > #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff8048195e8e2, pr= i=3D119) at /usr/src/sys/kern/subr_sleepqueue.c:625 > #3 0xffffffff80bcdb6d in _sleep (ident=3D0xfffff8048195e8e2, lock=3D, priority=3D119, wmesg=3D0xffffffff8123c694 "vofflock", sbt= =3D, pr=3D0, flags=3D256) at /usr/src/sys/kern/kern_synch.= c:221 > #4 0xffffffff80cb203a in foffset_lock (fp=3D0xfffff8048195e8c0, flags=3D= ) at /usr/src/sys/kern/vfs_vnops.c:700 > #5 0xffffffff80caf909 in foffset_lock_uio (fp=3D, uio=3D= , flags=3D) at /usr/src/sys/kern/vfs_vnops.= c:748 > #6 vn_io_fault (fp=3D0xfffff8048195e8c0, uio=3D0xfffffe2719d9cbb0, act= ive_cred=3D0xfffff80786ecad00, flags=3D0, td=3D0xfffffe264412a100) at /us= r/src/sys/kern/vfs_vnops.c:1163 > #7 0xffffffff80c34331 in fo_write (fp=3D0xfffff8048195e8c0, uio=3D0xff= fffe2719d9cbb0, active_cred=3D, flags=3D, td=3D= 0xfffffe264412a100) at /usr/src/sys/sys/file.h:326 > #8 dofilewrite (td=3D0xfffffe264412a100, fd=3D1, fp=3D0xfffff8048195e8= c0, auio=3D0xfffffe2719d9cbb0, offset=3D, flags=3D) at /usr/src/sys/kern/sys_generic.c:564 > #9 0xffffffff80c33eb0 in kern_writev (td=3D0xfffffe264412a100, fd=3D1,= auio=3D) at /usr/src/sys/kern/sys_generic.c:491 > #10 sys_write (td=3D0xfffffe264412a100, uap=3D) at /usr/= src/sys/kern/sys_generic.c:406 > #11 0xffffffff8106623d in syscallenter (td=3D) at /usr/s= rc/sys/amd64/amd64/../../kern/subr_syscall.c:150 > #12 amd64_syscall (td=3D0xfffffe264412a100, traced=3D0) at /usr/src/sys= /amd64/amd64/trap.c:1161 > #13 > #14 0x000000080043d53a in ?? () > Backtrace stopped: Cannot access memory at address 0x7fffdfffddd8 > (kgdb) frame 6 > #6 vn_io_fault (fp=3D0xfffff8048195e8c0, uio=3D0xfffffe2719d9cbb0, act= ive_cred=3D0xfffff80786ecad00, flags=3D0, td=3D0xfffffe264412a100) at /us= r/src/sys/kern/vfs_vnops.c:1163 > 1163 foffset_lock_uio(fp, uio, flags); > (kgdb) p *fp > $22 =3D {f_data =3D 0xfffff80571f29500, f_ops =3D 0xffffffff81923a10 , f_cred =3D 0xfffff80786ecad00, f_vnode =3D 0xfffff80571f29500, f_t= ype =3D 1, f_vnread_flags =3D 3, f_flag =3D 2, f_count =3D 4, {f_seqcount= =3D 127, f_pipegen =3D 127}, f_nextoff =3D 4499536, f_vnun =3D {fvn_cdev= priv =3D 0x0, fvn_advice =3D 0x0}, f_offset =3D 4499536, > f_label =3D 0x0} thread 104356 (gmake) is just waiting on the lock for vnode 0xfffff80571f29500 It also is holding exclusive lock on directory vnode 0xfffff808a08f0a00 > (kgdb) thread 9 > [Switching to thread 9 (Thread 104356)] > #0 sched_switch (td=3D0xfffffe24aadb6100, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffffe24aadb6100, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern/= kern_synch.c:542 > #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff80571f29568, pr= i=3D96) at /usr/src/sys/kern/subr_sleepqueue.c:625 > #3 0xffffffff80b954f6 in sleeplk (lk=3D0xfffff80571f29568, flags=3D532= 480, ilk=3D, wmesg=3D, pri=3D, timo=3D51, queue=3D0) at /usr/src/sys/kern/kern_lock.c:295 > #4 0xffffffff80b93a1e in lockmgr_xlock_hard (lk=3D0xfffff80571f29568, = flags=3D, ilk=3D0x0, file=3D, line=3D1432, lw= a=3D0xfffff80571f29568) at /usr/src/sys/kern/kern_lock.c:841 > #5 0xffffffff810e2a40 in VOP_LOCK1_APV (vop=3D0xffffffff8250a1e0 , a=3D0xfffffe271833f4d8) at vnode_if.c:1989 > #6 0xffffffff80cb14f4 in VOP_LOCK1 (vp=3D0xfffff80571f29500, flags=3D5= 32480, file=3D0xffffffff82472ac9 "/usr/src/sys/cddl/contrib/opensolaris/u= ts/common/fs/zfs/zfs_vnops.c", line=3D1432) at ./vnode_if.h:879 > #7 _vn_lock (vp=3D0xfffff80571f29500, flags=3D532480, file=3D0xfffffff= f82472ac9 "/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vn= ops.c", line=3D1432) at /usr/src/sys/kern/vfs_vnops.c:1613 > #8 0xffffffff8243af40 in zfs_lookup_lock (dvp=3D0xfffff808a08f0a00, vp= =3D0xfffff80571f29500, name=3D0xfffffe271833f630 "copool-basic.sh.log", l= kflags=3D532480) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/z= fs/zfs_vnops.c:1432 > #9 zfs_lookup (dvp=3D0xfffff808a08f0a00, nm=3D0xfffffe271833f630 "copo= ol-basic.sh.log", vpp=3D, cnp=3D0xfffffe271833faf0, nameio= p=3D2, cr=3D, td=3D, flags=3D0, cached=3D1)= at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1= 606 > #10 0xffffffff8243b61e in zfs_freebsd_lookup (ap=3D0xfffffe271833f780, = cached=3D) a= t /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:490= 0 > #11 zfs_freebsd_cachedlookup (ap=3D0xfffffe271833f780) at /usr/src/sys/= cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4908 > #12 0xffffffff810dfb46 in VOP_CACHEDLOOKUP_APV (vop=3D0xffffffff8250a1e= 0 , a=3D0xfffffe271833f780) at vnode_if.c:180 > #13 0xffffffff80c84dd8 in VOP_CACHEDLOOKUP (dvp=3D0xfffff808a08f0a00, v= pp=3D0xfffffe271833fac0, cnp=3D0xfffffe271833faf0) at ./vnode_if.h:80 > #14 vfs_cache_lookup (ap=3D) at /usr/src/sys/kern/vfs_ca= che.c:2149 > #15 0xffffffff810df996 in VOP_LOOKUP_APV (vop=3D0xffffffff8250a1e0 , a=3D0xfffffe271833f820) at vnode_if.c:117 > #16 0xffffffff80c8ee61 in VOP_LOOKUP (dvp=3D0xfffff808a08f0a00, vpp=3D0= xfffffe271833fac0, cnp=3D0xfffffe271833faf0) at ./vnode_if.h:54 > #17 lookup (ndp=3D0xfffffe271833fa60) at /usr/src/sys/kern/vfs_lookup.c= :951 > #18 0xffffffff80c8e374 in namei (ndp=3D0xfffffe271833fa60) at /usr/src/= sys/kern/vfs_lookup.c:512 > #19 0xffffffff80ca9e69 in kern_funlinkat (td=3D0xfffffe24aadb6100, dfd=3D= -100, path=3D0x800a3982e , fd=3D, pathseg=3DUIO_USERSPACE, flag=3D, oldinum=3D0) at /usr/src/sys/kern/vfs_syscalls.c:1819 > #20 0xffffffff80ca9db8 in sys_unlink (td=3D, uap=3D) at /usr/src/sys/kern/vfs_syscalls.c:1747 > #21 0xffffffff82780586 in filemon_wrapper_unlink (td=3D, u= ap=3D0xfffffe24aadb64d8) at /usr/src/sys/dev/filemon/filemon_wrapper.c:35= 0 > #22 0xffffffff8106623d in syscallenter (td=3D) at /usr/s= rc/sys/amd64/amd64/../../kern/subr_syscall.c:150 > #23 amd64_syscall (td=3D0xfffffe24aadb6100, traced=3D0) at /usr/src/sys= /amd64/amd64/trap.c:1161 thread 108476 (vim) is waiting to lock the directory vnode 0xfffff808a08f0a00 > (kgdb) thread 10 > [Switching to thread 10 (Thread 108476)] > #0 sched_switch (td=3D0xfffffe26c8601500, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffffe26c8601500, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern/= kern_synch.c:542 > #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff808a08f0a68, pr= i=3D96) at /usr/src/sys/kern/subr_sleepqueue.c:625 > #3 0xffffffff80b954f6 in sleeplk (lk=3D0xfffff808a08f0a68, flags=3D210= 5344, ilk=3D, wmesg=3D, pri=3D, timo=3D51, queue=3D1) at /usr/src/sys/kern/kern_lock.c:295 > #4 0xffffffff80b93525 in lockmgr_slock_hard (lk=3D0xfffff808a08f0a68, = flags=3D2105344, ilk=3D, file=3D0xffffffff811fb967 "/usr/s= rc/sys/kern/vfs_subr.c", line=3D2930, lwa=3D) at /usr/src/= sys/kern/kern_lock.c:649 > #5 0xffffffff810e2a40 in VOP_LOCK1_APV (vop=3D0xffffffff8250a1e0 , a=3D0xfffffe271d46d6b8) at vnode_if.c:1989 > #6 0xffffffff80cb14f4 in VOP_LOCK1 (vp=3D0xfffff808a08f0a00, flags=3D2= 105344, file=3D0xffffffff811fb967 "/usr/src/sys/kern/vfs_subr.c", line=3D= 2930) at ./vnode_if.h:879 > #7 _vn_lock (vp=3D0xfffff808a08f0a00, flags=3D2105344, file=3D0xffffff= ff811fb967 "/usr/src/sys/kern/vfs_subr.c", line=3D2930) at /usr/src/sys/k= ern/vfs_vnops.c:1613 > #8 0xffffffff80c9b3ec in vget_finish (vp=3D0xfffff808a08f0a00, flags=3D= 2105344, vs=3DVGET_USECOUNT) at /usr/src/sys/kern/vfs_subr.c:2930 > #9 0xffffffff80c8051c in cache_lookup (dvp=3D, vpp=3D, cnp=3D, tsp=3D, ticksp=3D) at /usr/src/sys/kern/vfs_cache.c:1407 > #10 0xffffffff80c84dad in vfs_cache_lookup (ap=3D) at /u= sr/src/sys/kern/vfs_cache.c:2147 > #11 0xffffffff810df996 in VOP_LOOKUP_APV (vop=3D0xffffffff8250a1e0 , a=3D0xfffffe271d46d8a0) at vnode_if.c:117 > #12 0xffffffff80c8ee61 in VOP_LOOKUP (dvp=3D0xfffff804de66e500, vpp=3D0= xfffffe271d46da60, cnp=3D0xfffffe271d46da90) at ./vnode_if.h:54 > #13 lookup (ndp=3D0xfffffe271d46da00) at /usr/src/sys/kern/vfs_lookup.c= :951 > #14 0xffffffff80c8e374 in namei (ndp=3D0xfffffe271d46da00) at /usr/src/= sys/kern/vfs_lookup.c:512 > #15 0xffffffff80caa83f in kern_statat (td=3D0xfffffe26c8601500, flag=3D= , fd=3D, path=3D0x8049d12c0 , pathseg=3DUIO_USERSPACE, sbp=3D0xf= ffffe271d46db28, hook=3D0x0) at /usr/src/sys/kern/vfs_syscalls.c:2340 > #16 0xffffffff80caafff in sys_fstatat (td=3D, uap=3D0xffff= fe26c86018d8) at /usr/src/sys/kern/vfs_syscalls.c:2317 > #17 0xffffffff81065c40 in syscallenter (td=3D) at /usr/s= rc/sys/amd64/amd64/../../kern/subr_syscall.c:162 > #18 amd64_syscall (td=3D0xfffffe26c8601500, traced=3D0) at /usr/src/sys= /amd64/amd64/trap.c:1161 > #19 > #20 0x00000008020ba75a in ?? () Lastly the find thread (107378) is waiting to lock the same directory vnode 0xfffff808a08f0a00 > (kgdb) thread 3 > [Switching to thread 3 (Thread 107378)] > #0 sched_switch (td=3D0xfffffe25eefa2e00, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid); > (kgdb) backtrace > #0 sched_switch (td=3D0xfffffe25eefa2e00, flags=3D) at = /usr/src/sys/kern/sched_ule.c:2147 > #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern/= kern_synch.c:542 > #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff808a08f0a68, pr= i=3D96) at /usr/src/sys/kern/subr_sleepqueue.c:625 > #3 0xffffffff80b954f6 in sleeplk (lk=3D0xfffff808a08f0a68, flags=3D210= 6368, ilk=3D, wmesg=3D, pri=3D, timo=3D51, queue=3D1) at /usr/src/sys/kern/kern_lock.c:295 > #4 0xffffffff80b93525 in lockmgr_slock_hard (lk=3D0xfffff808a08f0a68, = flags=3D2106368, ilk=3D, file=3D0xffffffff811f0ff4 "/usr/s= rc/sys/kern/vfs_lookup.c", line=3D737, lwa=3D) at /usr/src= /sys/kern/kern_lock.c:649 > #5 0xffffffff810e2a40 in VOP_LOCK1_APV (vop=3D0xffffffff8250a1e0 , a=3D0xfffffe271bee5748) at vnode_if.c:1989 > #6 0xffffffff80cb14f4 in VOP_LOCK1 (vp=3D0xfffff808a08f0a00, flags=3D2= 106368, file=3D0xffffffff811f0ff4 "/usr/src/sys/kern/vfs_lookup.c", line=3D= 737) at ./vnode_if.h:879 > #7 _vn_lock (vp=3D0xfffff808a08f0a00, flags=3D2106368, file=3D0xffffff= ff811f0ff4 "/usr/src/sys/kern/vfs_lookup.c", line=3D737) at /usr/src/sys/= kern/vfs_vnops.c:1613 > #8 0xffffffff80c8e93d in lookup (ndp=3D0xfffffe271bee5a88) at /usr/src= /sys/kern/vfs_lookup.c:735 > #9 0xffffffff80c8e374 in namei (ndp=3D0xfffffe271bee5a88) at /usr/src/= sys/kern/vfs_lookup.c:512 > #10 0xffffffff80cb0bdb in vn_open_cred (ndp=3D0xfffffe271bee5a88, flagp= =3D0xfffffe271bee5bb4, cmode=3D0, vn_open_flags=3D0, cred=3D0xfffff80786e= cad00, fp=3D0xfffff802a8627690) at /usr/src/sys/kern/vfs_vnops.c:288 > #11 0xffffffff80ca8a8a in kern_openat (td=3D0xfffffe25eefa2e00, fd=3D, path=3D, pathseg=3D, flags=3D= 1048577, mode=3D) at /usr/src/sys/kern/vfs_syscalls.c:1083= > #12 0xffffffff82780415 in filemon_wrapper_openat (td=3D0xfffffe25eefa2e= 00, uap=3D0xfffffe25eefa31d8) at /usr/src/sys/dev/filemon/filemon_wrapper= =2Ec:232 > #13 0xffffffff8106623d in syscallenter (td=3D) at /usr/s= rc/sys/amd64/amd64/../../kern/subr_syscall.c:150 > #14 amd64_syscall (td=3D0xfffffe25eefa2e00, traced=3D0) at /usr/src/sys= /amd64/amd64/trap.c:1161 --=20 Regards, Bryan Drewery --XElGlYQBYiLZJf7vq6TsZf2eItNVgQjfg-- --aIvlquH7auoI17NOayHMofyL3SITm4BOn Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAEBCgB9FiEE+Rc8ssOq6npcih8JNddxu25Gl88FAl67H6lfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5 MTczQ0IyQzNBQUVBN0E1QzhBMUYwOTM1RDc3MUJCNkU0Njk3Q0YACgkQNddxu25G l88jNAgAn03Ne8XDWCpYOCRxn2Lxycotkn2u5F6TEFlVUFHPzdpSZMRiLzk7XQe0 A68xi5kEb6/lgG5gxgQ39AhwaabProrwnRvm/Jh3+a/GxCIW63a6Ir6RD30AH25W NBPPtECP3hVf+9aQDYCNdmOZ+bYH7aQVd1aM58Uk5cCYeLGH7dTmlFLIC7qVYyn4 gqQeC5cHJ8A8tXAPj3Z+J8FrmYq1ApYwXrWpfNlmYMTBqv+iJG0Z5ve+IqS2wIz2 v/34teOUSPkZyV+aRmBwdS90WZgFhc4aw6Y5BjW2+mGxOsuxsf87BLv6XzbDJRMP 8W6UNXqBBmE9ef6RxVryhFV6JBWuKA== =n3ag -----END PGP SIGNATURE----- --aIvlquH7auoI17NOayHMofyL3SITm4BOn--