Date: Tue, 12 May 2020 15:47:12 -0700 From: Bryan Drewery <bdrewery@FreeBSD.org> To: freebsd-current@FreeBSD.org Subject: Re: zfs deadlock on r360452 relating to busy vm page Message-ID: <e10d3c61-0b47-55f2-0fe1-9fafaafe7799@FreeBSD.org> In-Reply-To: <2bdc8563-283b-32cc-8a1a-85ff52aca99e@FreeBSD.org> References: <2bdc8563-283b-32cc-8a1a-85ff52aca99e@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--Y3Kw281vOe14dImLo9fNoysaIpgYWAafe
Content-Type: multipart/mixed; boundary="WGx0QEJgJvn7xIqcMq1W4goqrKA31Rpc3";
protected-headers="v1"
From: Bryan Drewery <bdrewery@FreeBSD.org>
To: freebsd-current@FreeBSD.org
Message-ID: <e10d3c61-0b47-55f2-0fe1-9fafaafe7799@FreeBSD.org>
Subject: Re: zfs deadlock on r360452 relating to busy vm page
References: <2bdc8563-283b-32cc-8a1a-85ff52aca99e@FreeBSD.org>
In-Reply-To: <2bdc8563-283b-32cc-8a1a-85ff52aca99e@FreeBSD.org>
--WGx0QEJgJvn7xIqcMq1W4goqrKA31Rpc3
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable
Trivial repro:
dd if=3D/dev/zero of=3Dblah & tail -F blah
^C
load: 0.21 cmd: tail 72381 [prev->lr_read_cv] 2.17r 0.00u 0.01s 0% 2600k=
#0 0xffffffff80bce615 at mi_switch+0x155
#1 0xffffffff80c1cfea at sleepq_switch+0x11a
#2 0xffffffff80b57f0a at _cv_wait+0x15a
#3 0xffffffff829ddab6 at rangelock_enter+0x306
#4 0xffffffff829ecd3f at zfs_freebsd_getpages+0x14f
#5 0xffffffff810e3ab9 at VOP_GETPAGES_APV+0x59
#6 0xffffffff80f349e7 at vnode_pager_getpages+0x37
#7 0xffffffff80f2a93f at vm_pager_get_pages+0x4f
#8 0xffffffff80f054b0 at vm_fault+0x780
#9 0xffffffff80f04bde at vm_fault_trap+0x6e
#10 0xffffffff8106544e at trap_pfault+0x1ee
#11 0xffffffff81064a9c at trap+0x44c
#12 0xffffffff8103a978 at calltrap+0x8
On 5/12/2020 3:13 PM, Bryan Drewery wrote:
>> panic: deadlres_td_sleep_q: possible deadlock detected for 0xfffffe25e=
efa2e00 (find), blocked for 1802392 ticks
>=20
> 2 stuck processes from procstat -kk before panic
>> 72559 101698 tail - mi_switch+0x155 s=
leepq_switch+0x11a _cv_wait+0x15a rangelock_enter+0x306 zfs_freebsd_getpa=
ges+0x14f VOP_GETPAGES_APV+0x59 vnode_pager_getpages+0x37 vm_pager_get_pa=
ges+0x4f vm_fault+0x780 vm_fault_trap+0x6e trap_pfault+0x1ee
>=20
>> 72985 107378 find - mi_switch+0x155 s=
leepq_switch+0x11a sleeplk+0x106 lockmgr_slock_hard+0x1f5 VOP_LOCK1_APV+0=
x40 _vn_lock+0x54 lookup+0xdd namei+0x524 vn_open_cred+0x32b kern_openat+=
0x1fa filemon_wrapper_openat+0x15 amd64_syscall+0x73d
>=20
>=20
> The only find running was thread 107378
>=20
> I couldn't record much from ddb but got locked vnodes.
>>
>> db> show lockedvnods
>> Locked vnodes
>> vnode 0xfffff804de66e500: type VDIR
>> usecount 3, writecount 0, refcount 2 mountedhere 0
>> flags ()
>> v_object 0xfffff809459cb420 ref 0 pages 0 cleanbuf 0 dirtybuf 0
>> lock type zfs: SHARED (count 1)
>> #0 0xffffffff80b94a0f at lockmgr_slock+0xdf
>> #1 0xffffffff810e2a40 at VOP_LOCK1_APV+0x40
>> #2 0xffffffff80cb14f4 at _vn_lock+0x54
>> #3 0xffffffff80c9b3ec at vget_finish+0x6c
>> #4 0xffffffff80c8051c at cache_lookup+0x57c
>> #5 0xffffffff80c84dad at vfs_cache_lookup+0x7d
>> #6 0xffffffff810df996 at VOP_LOOKUP_APV+0x56
>> #7 0xffffffff80c8ee61 at lookup+0x601
>> #8 0xffffffff80c8e374 at namei+0x524
>> #9 0xffffffff80caa83f at kern_statat+0x7f
>> #10 0xffffffff80caafff at sys_fstatat+0x2f
>> #11 0xffffffff81065c40 at amd64_syscall+0x140
>> #12 0xffffffff8103b2a0 at fast_syscall_common+0x101
>> vnode 0xfffff808a08f0a00: type VDIR
>> usecount 6, writecount 0, refcount 2 mountedhere 0
>> flags ()
>> v_object 0xfffff801eb930000 ref 0 pages 0 cleanbuf 0 dirtybuf 0
>> lock type zfs: EXCL by thread 0xfffffe24aadb6100 (pid 72267, gmake=
, tid 104356)
>> with shared waiters pending
>> #0 0xffffffff80b94a0f at lockmgr_slock+0xdf
>> #1 0xffffffff810e2a40 at VOP_LOCK1_APV+0x40
>> #2 0xffffffff80cb14f4 at _vn_lock+0x54
>> #3 0xffffffff80c8e93d at lookup+0xdd
>> #4 0xffffffff80c8e374 at namei+0x524
>> #5 0xffffffff80ca9e69 at kern_funlinkat+0xa9
>> #6 0xffffffff80ca9db8 at sys_unlink+0x28
>> #7 0xffffffff82780586 at filemon_wrapper_unlink+0x16
>> #8 0xffffffff8106623d at amd64_syscall+0x73d
>> #9 0xffffffff8103b2a0 at fast_syscall_common+0x101
>>
>> vnode 0xfffff80571f29500: type VREG
>> usecount 6, writecount 1, refcount 2
>> flags ()
>> v_object 0xfffff806cb637c60 ref 2 pages 1 cleanbuf 0 dirtybuf 0
>> lock type zfs: SHARED (count 2)
>> with exclusive waiters pending
>> #0 0xffffffff80b94a0f at lockmgr_slock+0xdf
>> #1 0xffffffff810e2a40 at VOP_LOCK1_APV+0x40
>> #2 0xffffffff80cb14f4 at _vn_lock+0x54
>> #3 0xffffffff8243af40 at zfs_lookup+0x610
>> #4 0xffffffff8243b61e at zfs_freebsd_cachedlookup+0x8e
>> #5 0xffffffff810dfb46 at VOP_CACHEDLOOKUP_APV+0x56
>> #6 0xffffffff80c84dd8 at vfs_cache_lookup+0xa8
>> #7 0xffffffff810df996 at VOP_LOOKUP_APV+0x56
>> #8 0xffffffff80c8ee61 at lookup+0x601
>> #9 0xffffffff80c8e374 at namei+0x524
>> #10 0xffffffff80caa83f at kern_statat+0x7f
>> #11 0xffffffff80caafff at sys_fstatat+0x2f
>> #12 0xffffffff8106623d at amd64_syscall+0x73d
>> #13 0xffffffff8103b2a0 at fast_syscall_common+0x101
>=20
> It's nice how recent threads are at the top in gdb...
>> (kgdb) info threads
>> Id Target Id Frame=
>> 1 Thread 107952 (PID=3D79390: zfs) sch=
ed_switch (td=3D0xfffffe26ebb36000, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 2 Thread 102764 (PID=3D73218: zfs) sch=
ed_switch (td=3D0xfffffe2490a12300, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 3 Thread 107378 (PID=3D72985: find) sch=
ed_switch (td=3D0xfffffe25eefa2e00, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 4 Thread 103940 (PID=3D72980: rm) sch=
ed_switch (td=3D0xfffffe2451932500, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 5 Thread 101698 (PID=3D72559: tail) sch=
ed_switch (td=3D0xfffffe255eac0000, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 6 Thread 103660 (PID=3D72280: timestamp) sch=
ed_switch (td=3D0xfffffe25f948aa00, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 7 Thread 101249 (PID=3D72280: timestamp/prefix_stdout) sch=
ed_switch (td=3D0xfffffe264412a100, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 8 Thread 101255 (PID=3D72280: timestamp/prefix_stderr) sch=
ed_switch (td=3D0xfffffe25c8e9bc00, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 9 Thread 104356 (PID=3D72267: gmake) sch=
ed_switch (td=3D0xfffffe24aadb6100, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>> 10 Thread 108476 (PID=3D66957: vim) sch=
ed_switch (td=3D0xfffffe26c8601500, flags=3D<optimized out>) at /usr/src/=
sys/kern/sched_ule.c:2147
>=20
> The 2 threads holding shared lock on vnode 0xfffff80571f29500:
>=20
> The tail thread (101698) is waiting for a zfs rangelock getting pages
> for vnode 0xfffff80571f29500
>=20
>> (kgdb) thread 5
>> [Switching to thread 5 (Thread 101698)]
>> #0 sched_switch (td=3D0xfffffe255eac0000, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid);
>> (kgdb) backtrace
>> #0 sched_switch (td=3D0xfffffe255eac0000, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern=
/kern_synch.c:542
>> #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff810fb57dd48, p=
ri=3D0) at /usr/src/sys/kern/subr_sleepqueue.c:625
>> #3 0xffffffff80b57f0a in _cv_wait (cvp=3D0xfffff810fb57dd48, lock=3D0=
xfffff80049a99040) at /usr/src/sys/kern/kern_condvar.c:146
>> #4 0xffffffff82434ab6 in rangelock_enter_reader (rl=3D0xfffff80049a99=
018, new=3D0xfffff8022cadb100) at /usr/src/sys/cddl/contrib/opensolaris/u=
ts/common/fs/zfs/zfs_rlock.c:429
>> #5 rangelock_enter (rl=3D0xfffff80049a99018, off=3D<optimized out>, l=
en=3D<optimized out>, type=3D<optimized out>) at /usr/src/sys/cddl/contri=
b/opensolaris/uts/common/fs/zfs/zfs_rlock.c:477
>> #6 0xffffffff82443d3f in zfs_getpages (vp=3D<optimized out>, ma=3D0xf=
ffffe259f204b18, count=3D<optimized out>, rbehind=3D0xfffffe259f204ac4, r=
ahead=3D0xfffffe259f204ad0) at /usr/src/sys/cddl/contrib/opensolaris/uts/=
common/fs/zfs/zfs_vnops.c:4500
>> #7 zfs_freebsd_getpages (ap=3D<optimized out>) at /usr/src/sys/cddl/c=
ontrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4567
>> #8 0xffffffff810e3ab9 in VOP_GETPAGES_APV (vop=3D0xffffffff8250a1e0 <=
zfs_vnodeops>, a=3D0xfffffe259f2049f0) at vnode_if.c:2644
>> #9 0xffffffff80f349e7 in VOP_GETPAGES (vp=3D<unavailable>, m=3D<optim=
ized out>, count=3D<unavailable>, rbehind=3D<unavailable>, rahead=3D<unav=
ailable>) at ./vnode_if.h:1171
>> #10 vnode_pager_getpages (object=3D<optimized out>, m=3D<optimized out=
>, count=3D<unavailable>, rbehind=3D<unavailable>, rahead=3D<unavailable>=
) at /usr/src/sys/vm/vnode_pager.c:743
>> #11 0xffffffff80f2a93f in vm_pager_get_pages (object=3D0xfffff806cb637=
c60, m=3D0xfffffe259f204b18, count=3D1, rbehind=3D<unavailable>, rahead=3D=
<unavailable>) at /usr/src/sys/vm/vm_pager.c:305
>> #12 0xffffffff80f054b0 in vm_fault_getpages (fs=3D<optimized out>, ner=
a=3D0, behindp=3D<optimized out>, aheadp=3D<optimized out>) at /usr/src/s=
ys/vm/vm_fault.c:1163
>> #13 vm_fault (map=3D<optimized out>, vaddr=3D<optimized out>, fault_ty=
pe=3D<optimized out>, fault_flags=3D<optimized out>, m_hold=3D<optimized =
out>) at /usr/src/sys/vm/vm_fault.c:1394
>> #14 0xffffffff80f04bde in vm_fault_trap (map=3D0xfffffe25653949e8, vad=
dr=3D<optimized out>, fault_type=3D<optimized out>, fault_flags=3D0, sign=
o=3D0xfffffe259f204d04, ucode=3D0xfffffe259f204d00) at /usr/src/sys/vm/vm=
_fault.c:589
>> #15 0xffffffff8106544e in trap_pfault (frame=3D0xfffffe259f204d40, use=
rmode=3D<optimized out>, signo=3D<optimized out>, ucode=3D<unavailable>) =
at /usr/src/sys/amd64/amd64/trap.c:821
>> #16 0xffffffff81064a9c in trap (frame=3D0xfffffe259f204d40) at /usr/sr=
c/sys/amd64/amd64/trap.c:340
>> #17 <signal handler called>
>> #18 0x00000000002034fc in ?? ()
>> (kgdb) frame 11
>> #11 0xffffffff80f2a93f in vm_pager_get_pages (object=3D0xfffff806cb637=
c60, m=3D0xfffffe259f204b18, count=3D1, rbehind=3D<unavailable>, rahead=3D=
<unavailable>) at /usr/src/sys/vm/vm_pager.c:305
>> 305 r =3D (*pagertab[object->type]->pgo_getpages)(object, =
m, count, rbehind,
>> (kgdb) p *object
>> $10 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff8114fa30 "vm=
object", lo_flags =3D 627245056, lo_data =3D 0, lo_witness =3D 0x0}, rw_=
lock =3D 1}, object_list =3D {tqe_next =3D 0xfffff806cb637d68, tqe_prev =3D=
0xfffff806cb637b78}, shadow_head =3D {lh_first =3D 0x0}, shadow_list =3D=
{le_next =3D 0xffffffffffffffff,
>> le_prev =3D 0xffffffffffffffff}, memq =3D {tqh_first =3D 0xfffffe0=
01cbca850, tqh_last =3D 0xfffffe001cbca860}, rtree =3D {rt_root =3D 18446=
741875168421969}, size =3D 1099, domain =3D {dr_policy =3D 0x0, dr_iter =3D=
0}, generation =3D 1, cleangeneration =3D 1, ref_count =3D 2, shadow_cou=
nt =3D 0, memattr =3D 6 '\006', type =3D 2 '\002',
>> flags =3D 4096, pg_color =3D 0, paging_in_progress =3D {__count =3D =
2}, busy =3D {__count =3D 0}, resident_page_count =3D 1, backing_object =3D=
0x0, backing_object_offset =3D 0, pager_object_list =3D {tqe_next =3D 0x=
0, tqe_prev =3D 0x0}, rvq =3D {lh_first =3D 0x0}, handle =3D 0xfffff80571=
f29500, un_pager =3D {vnp =3D {vnp_size =3D 4499568,
>> writemappings =3D 0}, devp =3D {devp_pglist =3D {tqh_first =3D 0=
x44a870, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}, sgp =3D {sgp_pglis=
t =3D {tqh_first =3D 0x44a870, tqh_last =3D 0x0}}, swp =3D {swp_tmpfs =3D=
0x44a870, swp_blks =3D {pt_root =3D 0}, writemappings =3D 0}}, cred =3D =
0x0, charge =3D 0, umtx_data =3D 0x0}
>> (kgdb) p object->handle
>> $11 =3D (void *) 0xfffff80571f29500
>=20
>> (kgdb) p *(struct vnode *) 0xfffff80571f29500
>> $18 =3D {v_type =3D VREG, v_irflag =3D 0, v_op =3D 0xffffffff8250a1e0 =
<zfs_vnodeops>, v_data =3D 0xfffff80049a99000, v_mount =3D 0xfffffe247e5f=
3700, v_nmntvnodes =3D {tqe_next =3D 0xfffff8086eb38a00, tqe_prev =3D 0xf=
ffff80461c2d7a0}, {v_mountedhere =3D 0x0, v_unpcb =3D 0x0, v_rdev =3D 0x0=
, v_fifoinfo =3D 0x0}, v_hashlist =3D {le_next =3D 0x0,
>> le_prev =3D 0x0}, v_cache_src =3D {lh_first =3D 0x0}, v_cache_dst =
=3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff80571f29550}, v_cache_dd =3D =
0x0, v_lock =3D {lock_object =3D {lo_name =3D 0xffffffff82486a37 "zfs", l=
o_flags =3D 117112832, lo_data =3D 0, lo_witness =3D 0x0}, lk_lock =3D 37=
, lk_exslpfail =3D 0, lk_timo =3D 51, lk_pri =3D 96,
>> lk_stack =3D {depth =3D 14, pcs =3D {18446744071574211087, 1844674=
4071579773504, 18446744071575377140, 18446744071600058176, 18446744071600=
059934, 18446744071579761478, 18446744071575195096, 18446744071579761046,=
18446744071575236193, 18446744071575233396, 18446744071575349311, 184467=
44071575351295,
>> 18446744071579263549, 18446744071579087520, 0, 0, 0, 0}}}, v_i=
nterlock =3D {lock_object =3D {lo_name =3D 0xffffffff8123c142 "vnode inte=
rlock", lo_flags =3D 16973824, lo_data =3D 0, lo_witness =3D 0xfffff8123f=
d73600}, mtx_lock =3D 0}, v_vnlock =3D 0xfffff80571f29568, v_vnodelist =3D=
{tqe_next =3D 0xfffff8064bd0dc80,
>> tqe_prev =3D 0xfffff80e250788d8}, v_lazylist =3D {tqe_next =3D 0x0=
, tqe_prev =3D 0x0}, v_bufobj =3D {bo_lock =3D {lock_object =3D {lo_name =
=3D 0xffffffff811fb7ab "bufobj interlock", lo_flags =3D 86179840, lo_data=
=3D 0, lo_witness =3D 0xfffff8123fd7dd80}, rw_lock =3D 1}, bo_ops =3D 0x=
ffffffff8191ead0 <buf_ops_bio>,
>> bo_object =3D 0xfffff806cb637c60, bo_synclist =3D {le_next =3D 0x0=
, le_prev =3D 0x0}, bo_private =3D 0xfffff80571f29500, bo_clean =3D {bv_h=
d =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffff80571f296c0}, bv_root =3D {=
pt_root =3D 0}, bv_cnt =3D 0}, bo_dirty =3D {bv_hd =3D {tqh_first =3D 0x0=
, tqh_last =3D 0xfffff80571f296e0}, bv_root =3D {pt_root =3D 0},
>> bv_cnt =3D 0}, bo_numoutput =3D 0, bo_flag =3D 0, bo_domain =3D =
5, bo_bsize =3D 131072}, v_pollinfo =3D 0x0, v_label =3D 0x0, v_lockf =3D=
0x0, v_rl =3D {rl_waiters =3D {tqh_first =3D 0xfffff80f2cc12708, tqh_las=
t =3D 0xfffff80f2cc12708}, rl_currdep =3D 0x0}, v_cstart =3D 0, v_lasta =3D=
0, v_lastw =3D 0, v_clen =3D 0, v_holdcnt =3D 2,
>> v_usecount =3D 6, v_iflag =3D 0, v_vflag =3D 0, v_mflag =3D 0, v_dba=
tchcpu =3D -1, v_writecount =3D 1, v_hash =3D 45676874}
>=20
> Is that thread busying the vm object?
>=20
>=20
> thread 101255 (timestamp/prefix_stderr) which is also acting on vnode
> 0xfffff80571f29500 that the tail thread 101698 was.
>=20
>> (kgdb) thread
>> [Current thread is 8 (Thread 101255)]
>> (kgdb) backtrace
>> #0 sched_switch (td=3D0xfffffe25c8e9bc00, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern=
/kern_synch.c:542
>> #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffffe001cbca850, p=
ri=3D84) at /usr/src/sys/kern/subr_sleepqueue.c:625
>> #3 0xffffffff80f1de50 in _vm_page_busy_sleep (obj=3D<optimized out>, =
m=3D0xfffffe001cbca850, pindex=3D<optimized out>, wmesg=3D<optimized out>=
, allocflags=3D21504, locked=3Dfalse) at /usr/src/sys/vm/vm_page.c:1094
>> #4 0xffffffff80f241f7 in vm_page_grab_sleep (object=3D0xfffff806cb637=
c60, m=3D<optimized out>, pindex=3D<optimized out>, wmesg=3D<optimized ou=
t>, allocflags=3D21504, locked=3D<error reading variable: Cannot access m=
emory at address 0x0>) at /usr/src/sys/vm/vm_page.c:4326
>> #5 vm_page_acquire_unlocked (object=3D0xfffff806cb637c60, pindex=3D10=
98, prev=3D<optimized out>, mp=3D0xfffffe2717fc6730, allocflags=3D21504) =
at /usr/src/sys/vm/vm_page.c:4469
>> #6 0xffffffff80f24c61 in vm_page_grab_valid_unlocked (mp=3D0xfffffe27=
17fc6730, object=3D0xfffff806cb637c60, pindex=3D1098, allocflags=3D21504)=
at /usr/src/sys/vm/vm_page.c:4645
>> #7 0xffffffff82440246 in page_busy (vp=3D0xfffff80571f29500, start=3D=
4497408, off=3D<optimized out>, nbytes=3D<optimized out>) at /usr/src/sys=
/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:414
>> #8 update_pages (vp=3D0xfffff80571f29500, start=3D4497408, len=3D32, =
os=3D0xfffff8096a277400, oid=3D2209520, segflg=3D<optimized out>, tx=3D<o=
ptimized out>) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs=
/zfs_vnops.c:482
>> #9 zfs_write (vp=3D<optimized out>, uio=3D<optimized out>, ioflag=3D0=
, cr=3D<optimized out>, ct=3D<optimized out>) at /usr/src/sys/cddl/contri=
b/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1071
>> #10 zfs_freebsd_write (ap=3D<optimized out>) at /usr/src/sys/cddl/cont=
rib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4838
>> #11 0xffffffff810e0eaf in VOP_WRITE_APV (vop=3D0xffffffff8250a1e0 <zfs=
_vnodeops>, a=3D0xfffffe2717fc68c8) at vnode_if.c:925
>> #12 0xffffffff80cb574c in VOP_WRITE (vp=3D0xfffff80571f29500, uio=3D0x=
fffffe2717fc6bb0, ioflag=3D8323073, cred=3D<optimized out>) at ./vnode_if=
=2Eh:413
>> #13 vn_write (fp=3D0xfffff8048195e8c0, uio=3D<optimized out>, active_c=
red=3D<optimized out>, flags=3D<optimized out>, td=3D<optimized out>) at =
/usr/src/sys/kern/vfs_vnops.c:894
>> #14 0xffffffff80cb50c3 in vn_io_fault_doio (args=3D0xfffffe2717fc6af0,=
uio=3D0xfffffe2717fc6bb0, td=3D0xfffffe25c8e9bc00) at /usr/src/sys/kern/=
vfs_vnops.c:959
>> #15 0xffffffff80cb1c8c in vn_io_fault1 (vp=3D<optimized out>, uio=3D0x=
fffffe2717fc6bb0, args=3D0xfffffe2717fc6af0, td=3D0xfffffe25c8e9bc00) at =
/usr/src/sys/kern/vfs_vnops.c:1077
>> #16 0xffffffff80cafa32 in vn_io_fault (fp=3D0xfffff8048195e8c0, uio=3D=
0xfffffe2717fc6bb0, active_cred=3D0xfffff80f2cc12708, flags=3D0, td=3D<op=
timized out>) at /usr/src/sys/kern/vfs_vnops.c:1181
>> #17 0xffffffff80c34331 in fo_write (fp=3D0xfffff8048195e8c0, uio=3D0xf=
ffffe2717fc6bb0, active_cred=3D<unavailable>, flags=3D<unavailable>, td=3D=
0xfffffe25c8e9bc00) at /usr/src/sys/sys/file.h:326
>> #18 dofilewrite (td=3D0xfffffe25c8e9bc00, fd=3D2, fp=3D0xfffff8048195e=
8c0, auio=3D0xfffffe2717fc6bb0, offset=3D<optimized out>, flags=3D<optimi=
zed out>) at /usr/src/sys/kern/sys_generic.c:564
>> #19 0xffffffff80c33eb0 in kern_writev (td=3D0xfffffe25c8e9bc00, fd=3D2=
, auio=3D<optimized out>) at /usr/src/sys/kern/sys_generic.c:491
>> #20 sys_write (td=3D0xfffffe25c8e9bc00, uap=3D<optimized out>) at /usr=
/src/sys/kern/sys_generic.c:406
>> #21 0xffffffff8106623d in syscallenter (td=3D<optimized out>) at /usr/=
src/sys/amd64/amd64/../../kern/subr_syscall.c:150
>> #22 amd64_syscall (td=3D0xfffffe25c8e9bc00, traced=3D0) at /usr/src/sy=
s/amd64/amd64/trap.c:1161
>> #23 <signal handler called>
>> #24 0x000000080043d53a in ?? ()
>=20
> Maybe r358443 is related?
>=20
>=20
>> (kgdb) frame 4
>> #4 0xffffffff80f241f7 in vm_page_grab_sleep (object=3D0xfffff806cb637=
c60, m=3D<optimized out>, pindex=3D<optimized out>, wmesg=3D<optimized ou=
t>, allocflags=3D21504, locked=3D<error reading variable: Cannot access m=
emory at address 0x0>) at /usr/src/sys/vm/vm_page.c:4326
>> 4326 if (_vm_page_busy_sleep(object, m, m->pindex, wmesg, a=
llocflags,
>> (kgdb) p *object
>> $8 =3D {lock =3D {lock_object =3D {lo_name =3D 0xffffffff8114fa30 "vm =
object", lo_flags =3D 627245056, lo_data =3D 0, lo_witness =3D 0x0}, rw_l=
ock =3D 1}, object_list =3D {tqe_next =3D 0xfffff806cb637d68, tqe_prev =3D=
0xfffff806cb637b78}, shadow_head =3D {lh_first =3D 0x0}, shadow_list =3D=
{le_next =3D 0xffffffffffffffff,
>> le_prev =3D 0xffffffffffffffff}, memq =3D {tqh_first =3D 0xfffffe0=
01cbca850, tqh_last =3D 0xfffffe001cbca860}, rtree =3D {rt_root =3D 18446=
741875168421969}, size =3D 1099, domain =3D {dr_policy =3D 0x0, dr_iter =3D=
0}, generation =3D 1, cleangeneration =3D 1, ref_count =3D 2, shadow_cou=
nt =3D 0, memattr =3D 6 '\006', type =3D 2 '\002',
>> flags =3D 4096, pg_color =3D 0, paging_in_progress =3D {__count =3D =
2}, busy =3D {__count =3D 0}, resident_page_count =3D 1, backing_object =3D=
0x0, backing_object_offset =3D 0, pager_object_list =3D {tqe_next =3D 0x=
0, tqe_prev =3D 0x0}, rvq =3D {lh_first =3D 0x0}, handle =3D 0xfffff80571=
f29500, un_pager =3D {vnp =3D {vnp_size =3D 4499568,
>> writemappings =3D 0}, devp =3D {devp_pglist =3D {tqh_first =3D 0=
x44a870, tqh_last =3D 0x0}, ops =3D 0x0, dev =3D 0x0}, sgp =3D {sgp_pglis=
t =3D {tqh_first =3D 0x44a870, tqh_last =3D 0x0}}, swp =3D {swp_tmpfs =3D=
0x44a870, swp_blks =3D {pt_root =3D 0}, writemappings =3D 0}}, cred =3D =
0x0, charge =3D 0, umtx_data =3D 0x0}
>> (kgdb) frame 5
>> #5 vm_page_acquire_unlocked (object=3D0xfffff806cb637c60, pindex=3D10=
98, prev=3D<optimized out>, mp=3D0xfffffe2717fc6730, allocflags=3D21504) =
at /usr/src/sys/vm/vm_page.c:4469
>> 4469 if (!vm_page_grab_sleep(object, m, pindex, "pg=
nslp",
>> (kgdb) p *m
>> $9 =3D {plinks =3D {q =3D {tqe_next =3D 0xffffffffffffffff, tqe_prev =3D=
0xffffffffffffffff}, s =3D {ss =3D {sle_next =3D 0xffffffffffffffff}}, m=
emguard =3D {p =3D 18446744073709551615, v =3D 18446744073709551615}, uma=
=3D {slab =3D 0xffffffffffffffff, zone =3D 0xffffffffffffffff}}, listq =3D=
{tqe_next =3D 0x0, tqe_prev =3D 0xfffff806cb637ca8},
>> object =3D 0xfffff806cb637c60, pindex =3D 1098, phys_addr =3D 189884=
08832, md =3D {pv_list =3D {tqh_first =3D 0x0, tqh_last =3D 0xfffffe001cb=
ca888}, pv_gen =3D 44682, pat_mode =3D 6}, ref_count =3D 2147483648, busy=
_lock =3D 1588330502, a =3D {{flags =3D 0, queue =3D 255 '\377', act_coun=
t =3D 0 '\000'}, _bits =3D 16711680}, order =3D 13 '\r',
>> pool =3D 0 '\000', flags =3D 1 '\001', oflags =3D 0 '\000', psind =3D=
0 '\000', segind =3D 6 '\006', valid =3D 0 '\000', dirty =3D 0 '\000'}
>=20
> Pretty sure this thread is holding the rangelock from zfs_write() that
> tail is waiting on. So what is this thread (101255) waiting on exactly
> for? I'm not sure the way to track down what is using vm object
> 0xfffff806cb637c60. If the tail thread busied the page then they are
> waiting on each other I guess. If that's true then r358443 removing the=
> write lock on the object in update_pages() could be a problem.
>=20
>=20
> Not sure the rest is interesting. I think they are just waiting on the
> locked vnode but I give it here in case I missed something.
>=20
> thread 101249 (timestamp/prefix_stdout) is also acting on vnode
> 0xfffff80571f29500
>=20
>> (kgdb) thread 7
>> [Switching to thread 7 (Thread 101249)]
>> #0 sched_switch (td=3D0xfffffe264412a100, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid);
>> (kgdb) backtrace
>> #0 sched_switch (td=3D0xfffffe264412a100, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern=
/kern_synch.c:542
>> #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff8048195e8e2, p=
ri=3D119) at /usr/src/sys/kern/subr_sleepqueue.c:625
>> #3 0xffffffff80bcdb6d in _sleep (ident=3D0xfffff8048195e8e2, lock=3D<=
optimized out>, priority=3D119, wmesg=3D0xffffffff8123c694 "vofflock", sb=
t=3D<optimized out>, pr=3D0, flags=3D256) at /usr/src/sys/kern/kern_synch=
=2Ec:221
>> #4 0xffffffff80cb203a in foffset_lock (fp=3D0xfffff8048195e8c0, flags=
=3D<optimized out>) at /usr/src/sys/kern/vfs_vnops.c:700
>> #5 0xffffffff80caf909 in foffset_lock_uio (fp=3D<optimized out>, uio=3D=
<optimized out>, flags=3D<optimized out>) at /usr/src/sys/kern/vfs_vnops.=
c:748
>> #6 vn_io_fault (fp=3D0xfffff8048195e8c0, uio=3D0xfffffe2719d9cbb0, ac=
tive_cred=3D0xfffff80786ecad00, flags=3D0, td=3D0xfffffe264412a100) at /u=
sr/src/sys/kern/vfs_vnops.c:1163
>> #7 0xffffffff80c34331 in fo_write (fp=3D0xfffff8048195e8c0, uio=3D0xf=
ffffe2719d9cbb0, active_cred=3D<unavailable>, flags=3D<unavailable>, td=3D=
0xfffffe264412a100) at /usr/src/sys/sys/file.h:326
>> #8 dofilewrite (td=3D0xfffffe264412a100, fd=3D1, fp=3D0xfffff8048195e=
8c0, auio=3D0xfffffe2719d9cbb0, offset=3D<optimized out>, flags=3D<optimi=
zed out>) at /usr/src/sys/kern/sys_generic.c:564
>> #9 0xffffffff80c33eb0 in kern_writev (td=3D0xfffffe264412a100, fd=3D1=
, auio=3D<optimized out>) at /usr/src/sys/kern/sys_generic.c:491
>> #10 sys_write (td=3D0xfffffe264412a100, uap=3D<optimized out>) at /usr=
/src/sys/kern/sys_generic.c:406
>> #11 0xffffffff8106623d in syscallenter (td=3D<optimized out>) at /usr/=
src/sys/amd64/amd64/../../kern/subr_syscall.c:150
>> #12 amd64_syscall (td=3D0xfffffe264412a100, traced=3D0) at /usr/src/sy=
s/amd64/amd64/trap.c:1161
>> #13 <signal handler called>
>> #14 0x000000080043d53a in ?? ()
>> Backtrace stopped: Cannot access memory at address 0x7fffdfffddd8
>> (kgdb) frame 6
>> #6 vn_io_fault (fp=3D0xfffff8048195e8c0, uio=3D0xfffffe2719d9cbb0, ac=
tive_cred=3D0xfffff80786ecad00, flags=3D0, td=3D0xfffffe264412a100) at /u=
sr/src/sys/kern/vfs_vnops.c:1163
>> 1163 foffset_lock_uio(fp, uio, flags);
>> (kgdb) p *fp
>> $22 =3D {f_data =3D 0xfffff80571f29500, f_ops =3D 0xffffffff81923a10 <=
vnops>, f_cred =3D 0xfffff80786ecad00, f_vnode =3D 0xfffff80571f29500, f_=
type =3D 1, f_vnread_flags =3D 3, f_flag =3D 2, f_count =3D 4, {f_seqcoun=
t =3D 127, f_pipegen =3D 127}, f_nextoff =3D 4499536, f_vnun =3D {fvn_cde=
vpriv =3D 0x0, fvn_advice =3D 0x0}, f_offset =3D 4499536,
>> f_label =3D 0x0}
>=20
> thread 104356 (gmake) is just waiting on the lock for vnode
> 0xfffff80571f29500
> It also is holding exclusive lock on directory vnode 0xfffff808a08f0a00=
>=20
>> (kgdb) thread 9
>> [Switching to thread 9 (Thread 104356)]
>> #0 sched_switch (td=3D0xfffffe24aadb6100, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid);
>> (kgdb) backtrace
>> #0 sched_switch (td=3D0xfffffe24aadb6100, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern=
/kern_synch.c:542
>> #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff80571f29568, p=
ri=3D96) at /usr/src/sys/kern/subr_sleepqueue.c:625
>> #3 0xffffffff80b954f6 in sleeplk (lk=3D0xfffff80571f29568, flags=3D53=
2480, ilk=3D<optimized out>, wmesg=3D<optimized out>, pri=3D<optimized ou=
t>, timo=3D51, queue=3D0) at /usr/src/sys/kern/kern_lock.c:295
>> #4 0xffffffff80b93a1e in lockmgr_xlock_hard (lk=3D0xfffff80571f29568,=
flags=3D<unavailable>, ilk=3D0x0, file=3D<optimized out>, line=3D1432, l=
wa=3D0xfffff80571f29568) at /usr/src/sys/kern/kern_lock.c:841
>> #5 0xffffffff810e2a40 in VOP_LOCK1_APV (vop=3D0xffffffff8250a1e0 <zfs=
_vnodeops>, a=3D0xfffffe271833f4d8) at vnode_if.c:1989
>> #6 0xffffffff80cb14f4 in VOP_LOCK1 (vp=3D0xfffff80571f29500, flags=3D=
532480, file=3D0xffffffff82472ac9 "/usr/src/sys/cddl/contrib/opensolaris/=
uts/common/fs/zfs/zfs_vnops.c", line=3D1432) at ./vnode_if.h:879
>> #7 _vn_lock (vp=3D0xfffff80571f29500, flags=3D532480, file=3D0xffffff=
ff82472ac9 "/usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_v=
nops.c", line=3D1432) at /usr/src/sys/kern/vfs_vnops.c:1613
>> #8 0xffffffff8243af40 in zfs_lookup_lock (dvp=3D0xfffff808a08f0a00, v=
p=3D0xfffff80571f29500, name=3D0xfffffe271833f630 "copool-basic.sh.log", =
lkflags=3D532480) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/=
zfs/zfs_vnops.c:1432
>> #9 zfs_lookup (dvp=3D0xfffff808a08f0a00, nm=3D0xfffffe271833f630 "cop=
ool-basic.sh.log", vpp=3D<optimized out>, cnp=3D0xfffffe271833faf0, namei=
op=3D2, cr=3D<optimized out>, td=3D<optimized out>, flags=3D0, cached=3D1=
) at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:=
1606
>> #10 0xffffffff8243b61e in zfs_freebsd_lookup (ap=3D0xfffffe271833f780,=
cached=3D<error reading variable: Cannot access memory at address 0x1>) =
at /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:49=
00
>> #11 zfs_freebsd_cachedlookup (ap=3D0xfffffe271833f780) at /usr/src/sys=
/cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:4908
>> #12 0xffffffff810dfb46 in VOP_CACHEDLOOKUP_APV (vop=3D0xffffffff8250a1=
e0 <zfs_vnodeops>, a=3D0xfffffe271833f780) at vnode_if.c:180
>> #13 0xffffffff80c84dd8 in VOP_CACHEDLOOKUP (dvp=3D0xfffff808a08f0a00, =
vpp=3D0xfffffe271833fac0, cnp=3D0xfffffe271833faf0) at ./vnode_if.h:80
>> #14 vfs_cache_lookup (ap=3D<optimized out>) at /usr/src/sys/kern/vfs_c=
ache.c:2149
>> #15 0xffffffff810df996 in VOP_LOOKUP_APV (vop=3D0xffffffff8250a1e0 <zf=
s_vnodeops>, a=3D0xfffffe271833f820) at vnode_if.c:117
>> #16 0xffffffff80c8ee61 in VOP_LOOKUP (dvp=3D0xfffff808a08f0a00, vpp=3D=
0xfffffe271833fac0, cnp=3D0xfffffe271833faf0) at ./vnode_if.h:54
>> #17 lookup (ndp=3D0xfffffe271833fa60) at /usr/src/sys/kern/vfs_lookup.=
c:951
>> #18 0xffffffff80c8e374 in namei (ndp=3D0xfffffe271833fa60) at /usr/src=
/sys/kern/vfs_lookup.c:512
>> #19 0xffffffff80ca9e69 in kern_funlinkat (td=3D0xfffffe24aadb6100, dfd=
=3D-100, path=3D0x800a3982e <error: Cannot access memory at address 0x800=
a3982e>, fd=3D<optimized out>, pathseg=3DUIO_USERSPACE, flag=3D<optimized=
out>, oldinum=3D0) at /usr/src/sys/kern/vfs_syscalls.c:1819
>> #20 0xffffffff80ca9db8 in sys_unlink (td=3D<unavailable>, uap=3D<optim=
ized out>) at /usr/src/sys/kern/vfs_syscalls.c:1747
>> #21 0xffffffff82780586 in filemon_wrapper_unlink (td=3D<unavailable>, =
uap=3D0xfffffe24aadb64d8) at /usr/src/sys/dev/filemon/filemon_wrapper.c:3=
50
>> #22 0xffffffff8106623d in syscallenter (td=3D<optimized out>) at /usr/=
src/sys/amd64/amd64/../../kern/subr_syscall.c:150
>> #23 amd64_syscall (td=3D0xfffffe24aadb6100, traced=3D0) at /usr/src/sy=
s/amd64/amd64/trap.c:1161
>=20
> thread 108476 (vim) is waiting to lock the directory vnode
> 0xfffff808a08f0a00
>=20
>> (kgdb) thread 10
>> [Switching to thread 10 (Thread 108476)]
>> #0 sched_switch (td=3D0xfffffe26c8601500, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid);
>> (kgdb) backtrace
>> #0 sched_switch (td=3D0xfffffe26c8601500, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern=
/kern_synch.c:542
>> #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff808a08f0a68, p=
ri=3D96) at /usr/src/sys/kern/subr_sleepqueue.c:625
>> #3 0xffffffff80b954f6 in sleeplk (lk=3D0xfffff808a08f0a68, flags=3D21=
05344, ilk=3D<optimized out>, wmesg=3D<optimized out>, pri=3D<optimized o=
ut>, timo=3D51, queue=3D1) at /usr/src/sys/kern/kern_lock.c:295
>> #4 0xffffffff80b93525 in lockmgr_slock_hard (lk=3D0xfffff808a08f0a68,=
flags=3D2105344, ilk=3D<optimized out>, file=3D0xffffffff811fb967 "/usr/=
src/sys/kern/vfs_subr.c", line=3D2930, lwa=3D<optimized out>) at /usr/src=
/sys/kern/kern_lock.c:649
>> #5 0xffffffff810e2a40 in VOP_LOCK1_APV (vop=3D0xffffffff8250a1e0 <zfs=
_vnodeops>, a=3D0xfffffe271d46d6b8) at vnode_if.c:1989
>> #6 0xffffffff80cb14f4 in VOP_LOCK1 (vp=3D0xfffff808a08f0a00, flags=3D=
2105344, file=3D0xffffffff811fb967 "/usr/src/sys/kern/vfs_subr.c", line=3D=
2930) at ./vnode_if.h:879
>> #7 _vn_lock (vp=3D0xfffff808a08f0a00, flags=3D2105344, file=3D0xfffff=
fff811fb967 "/usr/src/sys/kern/vfs_subr.c", line=3D2930) at /usr/src/sys/=
kern/vfs_vnops.c:1613
>> #8 0xffffffff80c9b3ec in vget_finish (vp=3D0xfffff808a08f0a00, flags=3D=
2105344, vs=3DVGET_USECOUNT) at /usr/src/sys/kern/vfs_subr.c:2930
>> #9 0xffffffff80c8051c in cache_lookup (dvp=3D<optimized out>, vpp=3D<=
optimized out>, cnp=3D<optimized out>, tsp=3D<optimized out>, ticksp=3D<o=
ptimized out>) at /usr/src/sys/kern/vfs_cache.c:1407
>> #10 0xffffffff80c84dad in vfs_cache_lookup (ap=3D<optimized out>) at /=
usr/src/sys/kern/vfs_cache.c:2147
>> #11 0xffffffff810df996 in VOP_LOOKUP_APV (vop=3D0xffffffff8250a1e0 <zf=
s_vnodeops>, a=3D0xfffffe271d46d8a0) at vnode_if.c:117
>> #12 0xffffffff80c8ee61 in VOP_LOOKUP (dvp=3D0xfffff804de66e500, vpp=3D=
0xfffffe271d46da60, cnp=3D0xfffffe271d46da90) at ./vnode_if.h:54
>> #13 lookup (ndp=3D0xfffffe271d46da00) at /usr/src/sys/kern/vfs_lookup.=
c:951
>> #14 0xffffffff80c8e374 in namei (ndp=3D0xfffffe271d46da00) at /usr/src=
/sys/kern/vfs_lookup.c:512
>> #15 0xffffffff80caa83f in kern_statat (td=3D0xfffffe26c8601500, flag=3D=
<optimized out>, fd=3D<optimized out>, path=3D0x8049d12c0 <error: Cannot =
access memory at address 0x8049d12c0>, pathseg=3DUIO_USERSPACE, sbp=3D0xf=
ffffe271d46db28, hook=3D0x0) at /usr/src/sys/kern/vfs_syscalls.c:2340
>> #16 0xffffffff80caafff in sys_fstatat (td=3D<unavailable>, uap=3D0xfff=
ffe26c86018d8) at /usr/src/sys/kern/vfs_syscalls.c:2317
>> #17 0xffffffff81065c40 in syscallenter (td=3D<optimized out>) at /usr/=
src/sys/amd64/amd64/../../kern/subr_syscall.c:162
>> #18 amd64_syscall (td=3D0xfffffe26c8601500, traced=3D0) at /usr/src/sy=
s/amd64/amd64/trap.c:1161
>> #19 <signal handler called>
>> #20 0x00000008020ba75a in ?? ()
>=20
> Lastly the find thread (107378) is waiting to lock the same directory
> vnode 0xfffff808a08f0a00
>=20
>> (kgdb) thread 3
>> [Switching to thread 3 (Thread 107378)]
>> #0 sched_switch (td=3D0xfffffe25eefa2e00, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> 2147 cpuid =3D td->td_oncpu =3D PCPU_GET(cpuid);
>> (kgdb) backtrace
>> #0 sched_switch (td=3D0xfffffe25eefa2e00, flags=3D<optimized out>) at=
/usr/src/sys/kern/sched_ule.c:2147
>> #1 0xffffffff80bce615 in mi_switch (flags=3D260) at /usr/src/sys/kern=
/kern_synch.c:542
>> #2 0xffffffff80c1cfea in sleepq_switch (wchan=3D0xfffff808a08f0a68, p=
ri=3D96) at /usr/src/sys/kern/subr_sleepqueue.c:625
>> #3 0xffffffff80b954f6 in sleeplk (lk=3D0xfffff808a08f0a68, flags=3D21=
06368, ilk=3D<optimized out>, wmesg=3D<optimized out>, pri=3D<optimized o=
ut>, timo=3D51, queue=3D1) at /usr/src/sys/kern/kern_lock.c:295
>> #4 0xffffffff80b93525 in lockmgr_slock_hard (lk=3D0xfffff808a08f0a68,=
flags=3D2106368, ilk=3D<optimized out>, file=3D0xffffffff811f0ff4 "/usr/=
src/sys/kern/vfs_lookup.c", line=3D737, lwa=3D<optimized out>) at /usr/sr=
c/sys/kern/kern_lock.c:649
>> #5 0xffffffff810e2a40 in VOP_LOCK1_APV (vop=3D0xffffffff8250a1e0 <zfs=
_vnodeops>, a=3D0xfffffe271bee5748) at vnode_if.c:1989
>> #6 0xffffffff80cb14f4 in VOP_LOCK1 (vp=3D0xfffff808a08f0a00, flags=3D=
2106368, file=3D0xffffffff811f0ff4 "/usr/src/sys/kern/vfs_lookup.c", line=
=3D737) at ./vnode_if.h:879
>> #7 _vn_lock (vp=3D0xfffff808a08f0a00, flags=3D2106368, file=3D0xfffff=
fff811f0ff4 "/usr/src/sys/kern/vfs_lookup.c", line=3D737) at /usr/src/sys=
/kern/vfs_vnops.c:1613
>> #8 0xffffffff80c8e93d in lookup (ndp=3D0xfffffe271bee5a88) at /usr/sr=
c/sys/kern/vfs_lookup.c:735
>> #9 0xffffffff80c8e374 in namei (ndp=3D0xfffffe271bee5a88) at /usr/src=
/sys/kern/vfs_lookup.c:512
>> #10 0xffffffff80cb0bdb in vn_open_cred (ndp=3D0xfffffe271bee5a88, flag=
p=3D0xfffffe271bee5bb4, cmode=3D0, vn_open_flags=3D0, cred=3D0xfffff80786=
ecad00, fp=3D0xfffff802a8627690) at /usr/src/sys/kern/vfs_vnops.c:288
>> #11 0xffffffff80ca8a8a in kern_openat (td=3D0xfffffe25eefa2e00, fd=3D<=
optimized out>, path=3D<optimized out>, pathseg=3D<optimized out>, flags=3D=
1048577, mode=3D<optimized out>) at /usr/src/sys/kern/vfs_syscalls.c:1083=
>> #12 0xffffffff82780415 in filemon_wrapper_openat (td=3D0xfffffe25eefa2=
e00, uap=3D0xfffffe25eefa31d8) at /usr/src/sys/dev/filemon/filemon_wrappe=
r.c:232
>> #13 0xffffffff8106623d in syscallenter (td=3D<optimized out>) at /usr/=
src/sys/amd64/amd64/../../kern/subr_syscall.c:150
>> #14 amd64_syscall (td=3D0xfffffe25eefa2e00, traced=3D0) at /usr/src/sy=
s/amd64/amd64/trap.c:1161
>=20
>=20
--=20
Regards,
Bryan Drewery
--WGx0QEJgJvn7xIqcMq1W4goqrKA31Rpc3--
--Y3Kw281vOe14dImLo9fNoysaIpgYWAafe
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
iQGTBAEBCgB9FiEE+Rc8ssOq6npcih8JNddxu25Gl88FAl67J3FfFIAAAAAALgAo
aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEY5
MTczQ0IyQzNBQUVBN0E1QzhBMUYwOTM1RDc3MUJCNkU0Njk3Q0YACgkQNddxu25G
l8/ELQf/fKEZ4D6Vt5ewznAe4HuX2JYe76OKkQ2W86GJ0Poc3ypv5fmZCf8r29sZ
HRuEIHd2nbd9wV7EfRvtRXeREUmJ2peL2ISnZ/aJEVVynfZLkCPSEyRkCTjDnUw5
NVImJnTEetdqbDfofanWbVGKQo+w5QB2mvo0a13U5mRvC2fw9Mw/nOqHawGZg6oI
XuhCvXCRqkdv8X52oRply8on+mbfn5LlFIQCb/OOro7KXg/PbkQC616RCbaPDh63
BqEsQKbhC3zRG4g0hnY5w336iua8cBJxeZpv6ou57jlhY7mIYM7Z6WJYeZpeWvo6
RJNtUhZ4owYUhZgrCNuGWEz3jMKo2g==
=IAV4
-----END PGP SIGNATURE-----
--Y3Kw281vOe14dImLo9fNoysaIpgYWAafe--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e10d3c61-0b47-55f2-0fe1-9fafaafe7799>