Date: Sat, 22 Mar 2008 12:29:33 +0200 From: Alex Popa <razor@dataxnet.ro> To: Max Laier <max@love2party.net> Cc: Robert Watson <rwatson@freebsd.org>, freebsd-stable@freebsd.org Subject: Re: Lock Order Reversal on 7.0-STABLE with pf and ipfw / dummynet (traces) Message-ID: <20080322102933.GA76747@dataxnet.ro> In-Reply-To: <200803152217.02568.max@love2party.net> References: <20080314192359.GA4677@dataxnet.ro> <20080315203121.I42065@fledge.watson.org> <200803152217.02568.max@love2party.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Sorry for the big delay, but here are the traces you requested.
The kernel/world combination is from the same sources as in my previous
posts, I've only changed the kernel config and recompiled it (wiped
/usr/obj, make buildkernel && make installkernel in /usr/src).
Note: I haven't had access to a serial console, and experimentation
late last night showed me that kdb commands and their output don't get
saved in dmesg... I was about to get a camera there in order to take
photos of the backtraces and such, but I remembered vidcontrol and found
the "-H -P" options to have some nice potential. Long story short, I
added something like "options SC_HISTORY_SIZE=4000" to the kernel and
here are the results.
The dmesg file is an edited output of dmesg -a > something.txt, to only
contain the actual LOR related parts. The first-lor and second-lor
files contain edited screen captures (as made by vidcontrol), each of
them with lots of debugging info for one of the two LORs. Sorry for
some wrapped lines, but I hope that's better than nothing.
Hope this helps
Alex
--
"Computer science is no more about computers
than astronomy is about telescopes" -- E. W. Dijkstra
--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="dmesg-lor.txt"
lock order reversal:
1st 0xffffffffa2a56680 pf task mtx (pf task mtx) @ /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:6729
2nd 0xffffff00013236f0 radix node head (radix node head) @ /usr/src/sys/net/route.c:147
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x539
_mtx_lock_flags() at _mtx_lock_flags+0x1f
rtalloc1() at rtalloc1+0x57
rtalloc_ign() at rtalloc_ign+0x8f
pf_route() at pf_route+0x4d3
pf_test() at pf_test+0x64c
pf_check_in() at pf_check_in+0x2b
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 ---
KDB: enter: witness_checkorder
exclusive sx so_rcv_sx r = 0 (0xffffff000140f3b8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148
shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/src/sys/net/pfil.c:73
exclusive sleep mutex pf task mtx r = 0 (0xffffffffa2a56680) locked @ /usr/src/sys/modules/pf/../../contrib/pf/net/pf.c:6729
shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/src/sys/net/pfil.c:73
Locked vnodes
0xffffff000144aba0: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xffffff0001408000 (pid 36498)
ino 2237440, on dev ufs/usr64
0xffffff00018905d0: tag ufs, type VNON
usecount 1, writecount 0, refcount 1 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xffffff0001408000 (pid 36498)
ino 2286392, on dev ufs/usr64
...
lock order reversal:
1st 0xffffffff8096eb88 PFil hook read/write mutex (PFil hook read/write mutex) @ /usr/src/sys/net/pfil.c:73
2nd 0xffffffff8096f8a8 udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:385
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x539
_mtx_lock_flags() at _mtx_lock_flags+0x1f
udp_input() at udp_input+0x1f7
ip_input() at ip_input+0xa7
dummynet_send() at dummynet_send+0xde
dummynet_io() at dummynet_io+0x587
ipfw_check_in() at ipfw_check_in+0x241
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 ---
KDB: enter: witness_checkorder
exclusive sx so_rcv_sx r = 0 (0xffffff000140f3b8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148
shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/src/sys/net/pfil.c:73
Locked vnodes
0xffffff001106a9b0: tag ufs, type VREG
usecount 4, writecount 0, refcount 6 mountedhere 0
flags (VV_TEXT)
v_object 0xffffff0011318680 ref 2 pages 35
lock type ufs: SHARED (count 1)
ino 2239451, on dev ufs/usr64
0xffffff001106a3e0: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xffffff0001820000 (pid 42488)
ino 2262052, on dev ufs/usr64
0xffffff0001eabba0: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xffffff001106f680 (pid 42307) with 1 pending
ino 2262734, on dev ufs/usr64
Starting background file system checks in 60 seconds.
...
calcru: runtime went backwards from 3023 usec to 1678 usec for pid 36225 (dhcpd)
--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="first-lor.txt"
lock order reversal:
1st 0xffffffffa2a56680 pf task mtx (pf task mtx) @ /usr/src/sys/modules/pf/../.
./contrib/pf/net/pf.c:6729
2nd 0xffffff00013236f0 radix node head (radix node head) @ /usr/src/sys/net/rou
te.c:147
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x539
_mtx_lock_flags() at _mtx_lock_flags+0x1f
rtalloc1() at rtalloc1+0x57
rtalloc_ign() at rtalloc_ign+0x8f
pf_route() at pf_route+0x4d3
pf_test() at pf_test+0x64c
pf_check_in() at pf_check_in+0x2b
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 ---
KDB: enter: witness_checkorder
[thread pid 23 tid 100022 ]
Stopped at kdb_enter+0x31: popq %rbp
db> show pcpu
cpuid = 0
curthread = 0xffffff00011bf340: pid 23 "em0 taskq"
curpcb = 0xffffffffa0555d40
fpcurthread = none
idlethread = 0xffffff000107f340: pid 12 "idle: cpu0"
spin locks held:
db> show allpcpu
Current CPU: 0
cpuid = 0
curthread = 0xffffff00011bf340: pid 23 "em0 taskq"
curpcb = 0xffffffffa0555d40
fpcurthread = none
idlethread = 0xffffff000107f340: pid 12 "idle: cpu0"
spin locks held:
cpuid = 1
curthread = 0xffffff00011bf000: pid 24 "em1 taskq"
curpcb = 0xffffffffa057ad40
fpcurthread = none
idlethread = 0xffffff000107f680: pid 11 "idle: cpu1"
spin locks held:
db> trace
Tracing pid 23 tid 100022 td 0xffffff00011bf340
kdb_enter() at kdb_enter+0x31
witness_checkorder() at witness_checkorder+0x48e
_mtx_lock_flags() at _mtx_lock_flags+0x1f
rtalloc1() at rtalloc1+0x57
rtalloc_ign() at rtalloc_ign+0x8f
pf_route() at pf_route+0x4d3
pf_test() at pf_test+0x64c
pf_check_in() at pf_check_in+0x2b
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 ---
db> alltrace
Tracing command dhcpd pid 36121 tid 100080 td 0xffffff000146f680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
bwait() at bwait+0x55
bufwait() at bufwait+0x56
bread() at bread+0x1e
ffs_read() at ffs_read+0x2f7
vn_read() at vn_read+0x1d3
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800772acc, rsp = 0x7fffffffeba8, r
bp = 0xcc5c ---
Tracing command sh pid 35283 tid 100064 td 0xffffff0001420680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_wait() at kern_wait+0x3c0
wait4() at wait4+0x33
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800923abc, rsp = 0x7fffffffd2c8,
rbp = 0x33 ---
Tracing command ntpd pid 34406 tid 100076 td 0xffffff00014b4680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x2ba
kern_kevent() at kern_kevent+0x36a
kevent() at kevent+0x85
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800bd531c, rsp = 0x7fffffffcc0
8, rbp = 0xf ---
Tracing command named pid 29869 tid 100102 td 0xffffff0001426340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800c03a4c, rsp = 0x7fffff5fbcc8
, rbp = 0x26 ---
Tracing command named pid 29869 tid 100101 td 0xffffff0001426680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x2ba
do_cv_wait() at do_cv_wait+0x312
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff7fc
dc8, rbp = 0x7fffff7fce30 ---
Tracing command named pid 29869 tid 100100 td 0xffffff00014269c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
do_cv_wait() at do_cv_wait+0x4c2
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff9fd
eb8, rbp = 0 ---
Tracing command named pid 29869 tid 100099 td 0xffffff0001420000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
do_cv_wait() at do_cv_wait+0x4c2
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffffbfe
eb8, rbp = 0 ---
Tracing command named pid 29869 tid 100075 td 0xffffff00014b49c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_sigtimedwait() at kern_sigtimedwait+0x4ea
sigwait() at sigwait+0x63
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800b7dc6c, rsp = 0x7fffffffec
48, rbp = 0x800e01120 ---
Tracing command syslogd pid 26784 tid 100091 td 0xffffff00017bd000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80081aa4c, rsp = 0x7fffffffdda8
, rbp = 0x800a1c0f0 ---
Tracing command accounting pid 24352 tid 100087 td 0xffffff00017be000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
acct_thread() at acct_thread+0x1c3
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa2a66d30, rbp = 0 ---
Tracing command devd pid 22944 tid 100060 td 0xffffff000140a000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x43796c, rsp = 0x7fffffffe928, r
bp = 0x7fffffffed50 ---
Tracing command pflogd pid 14667 tid 100063 td 0xffffff00014209c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x2ba
bpfread() at bpfread+0xde
devfs_read_f() at devfs_read_f+0x67
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffed08, r
bp = 0x1 ---
Tracing command pflogd pid 14530 tid 100066 td 0xffffff0001404340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
soreceive_generic() at soreceive_generic+0xd1a
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffe938, r
bp = 0x4 ---
Tracing command pfpurge pid 14486 tid 100062 td 0xffffff0001404680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
pf_purge_thread() at pf_purge_thread+0x60
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa29c2d30, rbp = 0 ---
Tracing command sh pid 51 tid 100050 td 0xffffff0001353680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_wait() at kern_wait+0x3c0
wait4() at wait4+0x33
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800923abc, rsp = 0x7fffffffe6d8,
rbp = 0x33 ---
Tracing command schedcpu pid 50 tid 100049 td 0xffffff00013539c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
schedcpu_thread() at schedcpu_thread+0x209
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa067ed30, rbp = 0 ---
Tracing command softdepflush pid 49 tid 100048 td 0xffffff0001355000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
softdep_flush() at softdep_flush+0x2da
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0679d30, rbp = 0 ---
Tracing command syncer pid 48 tid 100047 td 0xffffff0001355340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
sched_sync() at sched_sync+0x5da
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0674d30, rbp = 0 ---
Tracing command vnlru pid 47 tid 100046 td 0xffffff0001355680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
vnlru_proc() at vnlru_proc+0x692
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa066fd30, rbp = 0 ---
Tracing command bufdaemon pid 46 tid 100045 td 0xffffff00013559c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
buf_daemon() at buf_daemon+0x225
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa066ad30, rbp = 0 ---
Tracing command pagezero pid 45 tid 100044 td 0xffffff00012479c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
vm_pagezero() at vm_pagezero+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0665d30, rbp = 0 ---
Tracing command vmdaemon pid 44 tid 100043 td 0xffffff0001298000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0660d30, rbp = 0 ---
Tracing command pagedaemon pid 43 tid 100042 td 0xffffff0001298340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
vm_pageout() at vm_pageout+0xa62
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa065bd30, rbp = 0 ---
Tracing command dummynet pid 42 tid 100041 td 0xffffff0001298680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
msleep_spin() at msleep_spin+0x18f
taskqueue_thread_loop() at taskqueue_thread_loop+0x45
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0656d30, rbp = 0 ---
Tracing command irq1: atkbd0 pid 41 tid 100040 td 0xffffff00012989c0
fork_trampoline() at fork_trampoline
Tracing command irq7: ppbus0 ppc0 pid 40 tid 100039 td 0xffffff0001299000
fork_trampoline() at fork_trampoline
Tracing command swi0: sio pid 39 tid 100038 td 0xffffff0001299340
fork_trampoline() at fork_trampoline
Tracing command fdc0 pid 38 tid 100037 td 0xffffff0001299680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
fdc_thread() at fdc_thread+0x7b5
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0622d30, rbp = 0 ---
Tracing command irq15: ata1 pid 37 tid 100036 td 0xffffff00012999c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0613d30, rbp = 0 ---
Tracing command irq14: ata0 pid 36 tid 100035 td 0xffffff000129b000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05fbd30, rbp = 0 ---
Tracing command usb4 pid 35 tid 100034 td 0xffffff00011c0680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05e3d30, rbp = 0 ---
Tracing command usb3 pid 34 tid 100033 td 0xffffff00011c09c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05ccd30, rbp = 0 ---
Tracing command irq16: uhci3 pid 33 tid 100032 td 0xffffff0001246000
fork_trampoline() at fork_trampoline
Tracing command usb2 pid 32 tid 100031 td 0xffffff0001246340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05c1d30, rbp = 0 ---
Tracing command usb1 pid 31 tid 100030 td 0xffffff0001246680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05bbd30, rbp = 0 ---
Tracing command irq19: uhci1 pid 30 tid 100029 td 0xffffff00012469c0
fork_trampoline() at fork_trampoline
Tracing command usbtask-dr pid 29 tid 100028 td 0xffffff0001247000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
usb_task_thread() at usb_task_thread+0x99
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05b0d30, rbp = 0 ---
Tracing command usbtask-hc pid 28 tid 100027 td 0xffffff0001247340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
usb_task_thread() at usb_task_thread+0x99
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05abd30, rbp = 0 ---
Tracing command usb0 pid 27 tid 100026 td 0xffffff0001247680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05a6d30, rbp = 0 ---
Tracing command irq23: uhci0 ehci0 pid 26 tid 100025 td 0xffffff00011a6680
fork_trampoline() at fork_trampoline
Tracing command irq18: bge0 uhci2 pid 25 tid 100024 td 0xffffff00011a69c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa059bd30, rbp = 0 ---
Tracing command em1 taskq pid 24 tid 100023 td 0xffffff00011bf000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x32
trap() at trap+0x2fd
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff8040984d, rsp = 0xffffffff9fe19ff0, rbp = 0xfffff
fffa057a870 ---
_mtx_lock_sleep() at _mtx_lock_sleep+0xe0
_mtx_lock_flags() at _mtx_lock_flags+0x7a
pf_test() at pf_test+0x72
pf_check_in() at pf_check_in+0x2b
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 ---
Tracing command em0 taskq pid 23 tid 100022 td 0xffffff00011bf340
kdb_enter() at kdb_enter+0x31
witness_checkorder() at witness_checkorder+0x48e
_mtx_lock_flags() at _mtx_lock_flags+0x1f
rtalloc1() at rtalloc1+0x57
rtalloc_ign() at rtalloc_ign+0x8f
pf_route() at pf_route+0x4d3
pf_test() at pf_test+0x64c
pf_check_in() at pf_check_in+0x2b
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 ---
Tracing command irq9: acpi0 pid 22 tid 100021 td 0xffffff00011bf680
fork_trampoline() at fork_trampoline
Tracing command kqueue taskq pid 21 tid 100020 td 0xffffff00011bf9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa052bd30, rbp = 0 ---
Tracing command swi6: task queue pid 20 tid 100019 td 0xffffff00011c0000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0526d30, rbp = 0 ---
Tracing command swi6: Giant taskq pid 19 tid 100018 td 0xffffff00011c0340
fork_trampoline() at fork_trampoline
Tracing command thread taskq pid 9 tid 100017 td 0xffffff000108c9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa051cd30, rbp = 0 ---
Tracing command swi5: + pid 18 tid 100016 td 0xffffff00011a5000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0517d30, rbp = 0 ---
Tracing command swi2: cambio pid 17 tid 100015 td 0xffffff00011a5340
fork_trampoline() at fork_trampoline
Tracing command xpt_thrd pid 8 tid 100014 td 0xffffff00011a5680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
xpt_scanner_thread() at xpt_scanner_thread+0x38
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa050dd30, rbp = 0 ---
Tracing command acpi_task_2 pid 7 tid 100013 td 0xffffff00011a59c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0508d30, rbp = 0 ---
Tracing command acpi_task_1 pid 6 tid 100012 td 0xffffff00011a6000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0503d30, rbp = 0 ---
Tracing command acpi_task_0 pid 5 tid 100011 td 0xffffff00011a6340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04fed30, rbp = 0 ---
Tracing command yarrow pid 16 tid 100010 td 0xffffff0001080340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
random_kthread() at random_kthread+0x1a4
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04f5d30, rbp = 0 ---
Tracing command g_down pid 4 tid 100009 td 0xffffff0001080680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
g_io_schedule_down() at g_io_schedule_down+0x201
g_down_procbody() at g_down_procbody+0x55
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04f0d30, rbp = 0 ---
Tracing command g_up pid 3 tid 100008 td 0xffffff00010809c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
g_io_schedule_up() at g_io_schedule_up+0x119
g_up_procbody() at g_up_procbody+0x55
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04ebd30, rbp = 0 ---
Tracing command g_event pid 2 tid 100007 td 0xffffff000108c000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
g_event_procbody() at g_event_procbody+0xa9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04e6d30, rbp = 0 ---
Tracing command swi3: vm pid 15 tid 100006 td 0xffffff000108c340
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 14 tid 100005 td 0xffffff000108c680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04dcd30, rbp = 0 ---
Tracing command swi1: net pid 13 tid 100004 td 0xffffff000107f000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04d7d30, rbp = 0 ---
Tracing command idle: cpu0 pid 12 tid 100003 td 0xffffff000107f340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
critical_exit() at critical_exit+0x83
intr_execute_handlers() at intr_execute_handlers+0xdd
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff805ce6a2, rsp = 0xffffffffa04d1bc0, rbp = 0xfffff
fffa04d1bd0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x19a
sched_idletd() at sched_idletd+0x64
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04d1d30, rbp = 0 ---
Tracing command idle: cpu1 pid 11 tid 100002 td 0xffffff000107f680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
critical_exit() at critical_exit+0x83
intr_execute_handlers() at intr_execute_handlers+0xdd
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff805ce6a2, rsp = 0xffffffffa04ccbc0, rbp = 0xfffff
fffa04ccbd0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x19a
sched_idletd() at sched_idletd+0x64
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04ccd30, rbp = 0 ---
Tracing command init pid 1 tid 100001 td 0xffffff000107f9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_wait() at kern_wait+0x3c0
wait4() at wait4+0x33
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40b61c, rsp = 0x7fffffffe848, rbp
= 0x33 ---
Tracing command audit pid 10 tid 100000 td 0xffffff0001080000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_cv_wait() at _cv_wait+0x115
audit_worker() at audit_worker+0x47f
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04c2d30, rbp = 0 ---
Tracing command swapper pid 0 tid 0 td 0xffffffff808def70
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
scheduler() at scheduler+0x39e
mi_startup() at mi_startup+0x59
btext() at btext+0x2c
db> show alllocks
Process 14530 (pflogd) thread 0xffffff0001404340 (100066)
exclusive sx so_rcv_sx r = 0 (0xffffff00014713b8) locked @ /usr/src/sys/kern/uip
c_sockbuf.c:148
Process 24 (em1 taskq) thread 0xffffff00011bf000 (100023)
shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/sr
c/sys/net/pfil.c:73
Process 23 (em0 taskq) thread 0xffffff00011bf340 (100022)
exclusive sleep mutex pf task mtx r = 0 (0xffffffffa2a56680) locked @ /usr/src/s
ys/modules/pf/../../contrib/pf/net/pf.c:6729
shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/sr
c/sys/net/pfil.c:73
db> show witness
Sleep locks:
0 so_rcv_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148
7 so_rcv -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2022
8 sellck -- last acquired @ /usr/src/sys/kern/sys_generic.c:759
8 radix node head -- last acquired @ /usr/src/sys/net/route.c:147
9 rtentry -- last acquired @ /usr/src/sys/netinet/ip_output.c:594
10 ifaddr -- last acquired @ /usr/src/sys/net/route.c:287
10 rts_inq -- last acquired @ /usr/src/sys/net/netisr.c:140
11 UMA zone -- last acquired @ /usr/src/sys/vm/uma_core.c:2335
11 UMA zone -- (already displayed)
9 UMA boot pages -- last acquired @ /usr/src/sys/vm/uma_core.c:916
11 vm page queue free mutex -- last acquired @ /usr/src/sys/vm/vm_page.c:1030
9 ifnet -- last acquired @ /usr/src/sys/net/if.c:1047
11 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:212
12 eventhandler list -- last acquired @ /usr/src/sys/kern/kern_exec.c:904
10 if_addr_mtx -- last acquired @ /usr/src/sys/net/if.c:828
10 pf task mtx -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/ne
t/pf.c:6729
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
11 arc4_mtx -- last acquired @ /usr/src/sys/libkern/arc4random.c:137
11 bpf interface lock -- last acquired @ /usr/src/sys/net/bpf.c:1422
12 bpf cdev lock -- last acquired @ /usr/src/sys/net/bpf.c:1426
11 eventhandler -- (already displayed)
12 eventhandler list -- (already displayed)
8 process lock -- last acquired @ /usr/src/sys/amd64/amd64/trap.c:622
9 session -- last acquired @ /usr/src/sys/kern/kern_fork.c:582
10 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:1213
11 uidinfo struct -- last acquired @ order list:0
11 sleep mtxpool -- last acquired @ /usr/src/sys/kern/vfs_vnops.c:508
12 kqueue -- last acquired @ /usr/src/sys/kern/kern_event.c:1182
13 struct mount mtx -- last acquired @ /usr/src/sys/kern/vfs_mount.c:440
14 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:2621
15 cdev -- last acquired @ /usr/src/sys/kern/kern_conf.c:69
15 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:3040
15 Syncer mtx -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1915
10 tty -- last acquired @ /usr/src/sys/kern/kern_event.c:1664
9 sigacts -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:392
9 ktrace -- last acquired @ /usr/src/sys/kern/kern_fork.c:600
11 sleep mtxpool -- (already displayed)
11 sleep mtxpool -- (already displayed)
11 UMA zone -- (already displayed)
12 kqueue -- (already displayed)
8 process lock -- (already displayed)
2 unp_mtx -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:803
3 accept -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2837
6 so_snd -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:234
7 so_rcv -- (already displayed)
11 sleep mtxpool -- (already displayed)
7 so_rcv -- (already displayed)
7 so_rcv -- (already displayed)
6 so_snd -- (already displayed)
3 filedesc structure -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1987
14 vnode interlock -- (already displayed)
8 process lock -- (already displayed)
11 sleep mtxpool -- (already displayed)
5 Name Cache -- last acquired @ /usr/src/sys/kern/vfs_cache.c:327
14 vnode interlock -- (already displayed)
11 UMA zone -- (already displayed)
9 UMA boot pages -- (already displayed)
4 fdesc -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1472
15 cdev -- (already displayed)
4 Giant -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241
5 pipe mutex -- last acquired @ /usr/src/sys/kern/sys_pipe.c:1331
6 sigio lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:782
7 process group -- last acquired @ /usr/src/sys/kern/kern_fork.c:572
8 process lock -- (already displayed)
8 sellck -- (already displayed)
11 UMA zone -- (already displayed)
6 system map -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1995
8 vm page queue mutex -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3437
14 vnode interlock -- (already displayed)
9 pmap -- last acquired @ /usr/src/sys/amd64/amd64/pmap.c:3076
11 vm page queue free mutex -- (already displayed)
11 vm page queue free mutex -- (already displayed)
7 kmem object -- last acquired @ /usr/src/sys/vm/vm_object.c:441
11 vm page queue free mutex -- (already displayed)
8 vm page queue mutex -- (already displayed)
7 kernel object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3641
11 vm page queue free mutex -- (already displayed)
8 vm page queue mutex -- (already displayed)
11 vm page queue free mutex -- (already displayed)
9 pmap -- (already displayed)
7 KMAP ENTRY -- last acquired @ /usr/src/sys/vm/uma_core.c:2905
11 UMA zone -- (already displayed)
7 standard object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3426
11 vm page queue free mutex -- (already displayed)
14 vnode interlock -- (already displayed)
8 vm page queue mutex -- (already displayed)
8 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:225
11 UMA zone -- (already displayed)
5 UMA lock -- last acquired @ /usr/src/sys/vm/uma_core.c:1314
11 UMA zone -- (already displayed)
7 KMAP ENTRY -- (already displayed)
9 UMA boot pages -- (already displayed)
11 vm page queue free mutex -- (already displayed)
11 eventhandler -- (already displayed)
12 eventhandler list -- (already displayed)
5 kobj -- last acquired @ /usr/src/sys/kern/subr_kobj.c:307
6 kernel environment -- last acquired @ /usr/src/sys/kern/kern_environment.c:
301
7 kernel object -- (already displayed)
8 vm object_list -- (already displayed)
7 KMAP ENTRY -- (already displayed)
10 uidinfo hash -- (already displayed)
8 process lock -- (already displayed)
11 sleep mtxpool -- (already displayed)
5 evclass_mtx -- last acquired @ /usr/src/sys/security/audit/audit_bsm_klib.c
:112
5 TID lock -- last acquired @ /usr/src/sys/kern/subr_unit.c:623
7 standard object -- (already displayed)
5 intr event -- last acquired @ /usr/src/sys/kern/kern_intr.c:454
15 cdev -- (already displayed)
5 GEOM orphanage -- last acquired @ /usr/src/sys/geom/geom_event.c:201
5 ttylist -- last acquired @ /usr/src/sys/kern/tty.c:3126
10 tty -- (already displayed)
5 intr config -- last acquired @ /usr/src/sys/kern/subr_autoconf.c:73
5 taskqueue list -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:125
5 XPT lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2668
6 XPT topology lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2673
6 kernel environment -- (already displayed)
7 taskqueue -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:73
5 rman head -- last acquired @ /usr/src/sys/kern/subr_rman.c:152
5 rman -- last acquired @ /usr/src/sys/kern/subr_rman.c:539
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
5 devd -- last acquired @ /usr/src/sys/kern/subr_bus.c:489
8 sellck -- (already displayed)
8 sellck -- (already displayed)
5 ACPI semaphore -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSynch.c:30
3
5 acpi subsystem HW lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSy
nch.c:377
5 acpi subsystem GPE lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdS
ynch.c:377
7 taskqueue -- (already displayed)
5 msi -- last acquired @ /usr/src/sys/amd64/amd64/msi.c:389
9 ifnet -- (already displayed)
5 bpf global lock -- last acquired @ /usr/src/sys/net/bpf.c:1671
11 bpf interface lock -- (already displayed)
5 bounce pages lock -- last acquired @ /usr/src/sys/amd64/amd64/busdma_machde
p.c:1073
9 pmap -- (already displayed)
5 unit# allocation -- last acquired @ /usr/src/sys/kern/subr_unit.c:623
15 vnode_free_list -- (already displayed)
5 pfs_node -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_internal.h:10
3
5 pfs_fileno -- last acquired @ /usr/src/sys/kern/subr_unit.c:623
5 random reseed -- last acquired @ /usr/src/sys/dev/random/yarrow.c:278
11 arc4_mtx -- (already displayed)
5 nfsd_mtx -- last acquired @ /usr/src/sys/nfsserver/nfs_srvsock.c:796
6 so_snd -- (already displayed)
5 if_clone lock -- last acquired @ /usr/src/sys/net/if_clone.c:164
5 if_cloners lock -- last acquired @ /usr/src/sys/net/if_clone.c:252
5 domain list -- last acquired @ /usr/src/sys/kern/uipc_domain.c:228
6 pfil_head_list lock -- last acquired @ /usr/src/sys/net/pfil.c:159
5 PFil hook read/write mutex -- last acquired @ /usr/src/sys/net/pfil.c:73
6 pfil_head_list lock -- (already displayed)
10 pf task mtx -- (already displayed)
6 IPFW static rules -- last acquired @ /usr/src/sys/netinet/ip_fw2.c:2647
8 radix node head -- (already displayed)
7 if send queue -- last acquired @ /usr/src/sys/dev/em/if_em.c:949
6 network driver -- last acquired @ /usr/src/sys/dev/bge/if_bge.c:3014
10 if_addr_mtx -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
7 taskqueue -- (already displayed)
7 if send queue -- (already displayed)
5 isn_mtx -- last acquired @ /usr/src/sys/netinet/tcp_subr.c:1426
8 radix node head -- (already displayed)
6 IPFW static rules -- (already displayed)
5 lo_mtx -- last acquired @ /usr/src/sys/net/if_loop.c:161
5 ATA queue lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:177
6 ATA state lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:194
5 devstat -- last acquired @ /usr/src/sys/kern/subr_devstat.c:83
6 XPT topology lock -- (already displayed)
5 NFS iod lock -- last acquired @ /usr/src/sys/nfsclient/nfs_nfsiod.c:196
6 mountlist -- last acquired @ /usr/src/sys/kern/vfs_syscalls.c:526
13 struct mount mtx -- (already displayed)
13 struct mount mtx -- (already displayed)
5 mntid -- last acquired @ /usr/src/sys/kern/vfs_subr.c:460
6 mountlist -- (already displayed)
14 vnode interlock -- (already displayed)
5 buf queue lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1738
14 vnode interlock -- (already displayed)
5 bio queue -- last acquired @ /usr/src/sys/geom/geom_io.c:68
5 bdone lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3800
5 needsbuffer lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:321
5 FFS Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_alloc.c:1475
11 arc4_mtx -- (already displayed)
5 Name Cache -- (already displayed)
5 knlist lock for lockless objects -- last acquired @ /usr/src/sys/kern/kern_
event.c:1664
5 vfs hash -- last acquired @ /usr/src/sys/kern/vfs_hash.c:71
14 vnode interlock -- (already displayed)
5 devfs interlock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:194
14 vnode interlock -- (already displayed)
5 dirhash list -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:349
6 dirhash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:457
6 dirhash -- (already displayed)
5 pbuf mutex -- last acquired @ /usr/src/sys/vm/vm_pager.c:413
8 vm page queue mutex -- (already displayed)
7 process group -- (already displayed)
10 tty -- (already displayed)
9 session -- (already displayed)
5 Softdep Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:5022
6 system map -- (already displayed)
11 UMA zone -- (already displayed)
14 vnode interlock -- (already displayed)
6 buffer daemon lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:429
10 pf task mtx -- (already displayed)
6 buffer daemon lock -- (already displayed)
6 sigio lock -- (already displayed)
6 so_snd -- (already displayed)
5 pipe mutex -- (already displayed)
0 so_snd_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148
6 so_snd -- (already displayed)
1 unp_global_rwlock -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:770
2 unp_mtx -- (already displayed)
3 filedesc structure -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
8 radix node head -- (already displayed)
9 rtentry -- (already displayed)
10 rts_inq -- (already displayed)
0 pf_statetbl_lock -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/n
et/pf.c:979
10 pf task mtx -- (already displayed)
6 pfil_head_list lock -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
0 devfsmount -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:201
5 devfs interlock -- (already displayed)
15 vnode_free_list -- (already displayed)
14 vnode interlock -- (already displayed)
13 struct mount mtx -- (already displayed)
15 cdev -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
1 DEVFS ruleset lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_rule.c:177
0 g_disk_done -- last acquired @ /usr/src/sys/geom/geom_disk.c:198
5 bio queue -- (already displayed)
11 UMA zone -- (already displayed)
0 dummynet -- last acquired @ /usr/src/sys/netinet/ip_dummynet.c:739
0 ipqlock -- last acquired @ /usr/src/sys/netinet/ip_input.c:1086
0 sem -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1288
0 fdc lock -- last acquired @ /usr/src/sys/dev/fdc/fdc.c:803
0 if_afdata -- last acquired @ /usr/src/sys/net/if.c:578
0 GEOM topology -- last acquired @ /usr/src/sys/geom/geom_event.c:233
5 GEOM orphanage -- (already displayed)
5 devstat -- (already displayed)
5 unit# allocation -- (already displayed)
15 cdev -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
9 UMA boot pages -- (already displayed)
8 vm object_list -- (already displayed)
14 vnode interlock -- (already displayed)
7 standard object -- (already displayed)
6 system map -- (already displayed)
0 intr sources -- last acquired @ /usr/src/sys/amd64/amd64/intr_machdep.c:593
5 msi -- (already displayed)
0 audit_mtx -- last acquired @ /usr/src/sys/security/audit/audit_worker.c:448
0 uma object -- last acquired @ /usr/src/sys/vm/vm_meter.c:115
0 p_peers -- last acquired @ /usr/src/sys/kern/kern_exit.c:284
0 runningbufspace lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:340
0 umtxql -- last acquired @ /usr/src/sys/kern/kern_umtx.c:326
0 accept_filter_mtx -- last acquired @ /usr/src/sys/kern/uipc_accf.c:116
0 clone events drain lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c
:626
11 eventhandler -- (already displayed)
12 eventhandler list -- (already displayed)
15 cdev -- (already displayed)
0 ACPI PCI bus methods -- last acquired @ /usr/src/sys/dev/acpica/acpi_pcib.c:22
1
0 rtsock route_cb lock -- last acquired @ /usr/src/sys/net/rtsock.c:236
0 rawcb -- last acquired @ /usr/src/sys/net/raw_cb.c:104
7 so_rcv -- (already displayed)
11 UMA zone -- (already displayed)
0 ACPI PCI link -- last acquired @ /usr/src/sys/dev/acpica/acpi_pci_link.c:432
5 ACPI semaphore -- (already displayed)
0 ACPI root bus -- last acquired @ /usr/src/sys/dev/acpica/acpi.c:1022
5 rman -- (already displayed)
5 ACPI semaphore -- (already displayed)
6 system map -- (already displayed)
0 acct_sx -- last acquired @ /usr/src/sys/kern/kern_acct.c:351
11 eventhandler -- (already displayed)
1 proctree -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:325
2 allproc -- last acquired @ /usr/src/sys/kern/kern_fork.c:297
3 allprison -- last acquired @ /usr/src/sys/kern/kern_jail.c:945
11 sleep mtxpool -- (already displayed)
8 process lock -- (already displayed)
4 fdesc -- (already displayed)
3 filedesc structure -- (already displayed)
14 vnode interlock -- (already displayed)
3 user map -- last acquired @ /usr/src/sys/vm/vm_map.c:3111
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
8 vm object_list -- (already displayed)
7 standard object -- (already displayed)
8 vm page queue mutex -- (already displayed)
9 UMA boot pages -- (already displayed)
9 pmap -- (already displayed)
14 vnode interlock -- (already displayed)
5 buf queue lock -- (already displayed)
5 needsbuffer lock -- (already displayed)
6 system map -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
13 struct mount mtx -- (already displayed)
6 buffer daemon lock -- (already displayed)
5 Softdep Lock -- (already displayed)
11 arc4_mtx -- (already displayed)
7 process group -- (already displayed)
4 Giant -- (already displayed)
8 process lock -- (already displayed)
9 session -- (already displayed)
6 sigio lock -- (already displayed)
3 filedesc structure -- (already displayed)
8 process lock -- (already displayed)
5 FFS Lock -- (already displayed)
13 struct mount mtx -- (already displayed)
14 vnode interlock -- (already displayed)
7 standard object -- (already displayed)
5 buf queue lock -- (already displayed)
6 system map -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
4 Giant -- (already displayed)
5 needsbuffer lock -- (already displayed)
6 buffer daemon lock -- (already displayed)
5 Softdep Lock -- (already displayed)
0 vm daemon -- last acquired @ /usr/src/sys/vm/vm_pageout.c:1531
0 protect sysfilt_ops -- last acquired @ /usr/src/sys/kern/kern_event.c:758
0 so_glabel -- last acquired @ /usr/src/sys/kern/uipc_socket.c:280
0 sysctl lock -- last acquired @ /usr/src/sys/kern/kern_sysctl.c:1396
11 arc4_mtx -- (already displayed)
2 allproc -- (already displayed)
8 process lock -- (already displayed)
3 user map -- (already displayed)
15 cdev -- (already displayed)
1 filelist lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1366
3 filedesc structure -- (already displayed)
5 GEOM orphanage -- (already displayed)
4 Giant -- (already displayed)
9 ktrace -- (already displayed)
1 kernel linker -- last acquired @ /usr/src/sys/kern/kern_linker.c:1096
11 UMA zone -- (already displayed)
2 module subsystem sx lock -- last acquired @ /usr/src/sys/kern/kern_module.c:
407
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
3 filedesc structure -- (already displayed)
14 vnode interlock -- (already displayed)
5 Name Cache -- (already displayed)
5 buf queue lock -- (already displayed)
5 needsbuffer lock -- (already displayed)
5 vfs hash -- (already displayed)
15 vnode_free_list -- (already displayed)
13 struct mount mtx -- (already displayed)
5 dirhash list -- (already displayed)
11 vm page queue free mutex -- (already displayed)
6 system map -- (already displayed)
7 kernel object -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
6 dirhash -- (already displayed)
15 cdev -- (already displayed)
8 vm object_list -- (already displayed)
7 standard object -- (already displayed)
5 pbuf mutex -- (already displayed)
6 buffer daemon lock -- (already displayed)
5 kobj -- (already displayed)
1 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:741
11 UMA zone -- (already displayed)
6 system map -- (already displayed)
5 devstat -- (already displayed)
5 ttylist -- (already displayed)
8 vm object_list -- (already displayed)
5 UMA lock -- (already displayed)
15 Syncer mtx -- (already displayed)
1 unp_global_rwlock -- (already displayed)
1 tcp -- last acquired @ /usr/src/sys/netinet/tcp_usrreq.c:254
2 tcpinp -- last acquired @ /usr/src/sys/netinet/tcp_usrreq.c:255
6 so_snd -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
7 so_rcv -- (already displayed)
9 ifnet -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
1 tcp_sc_head -- last acquired @ /usr/src/sys/netinet/tcp_syncache.c:1696
1 udp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:839
2 udpinp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:843
3 in_multi_mtx -- last acquired @ /usr/src/sys/netinet/in_mcast.c:320
4 igmp_mtx -- last acquired @ /usr/src/sys/netinet/igmp.c:446
10 if_addr_mtx -- (already displayed)
10 if_addr_mtx -- (already displayed)
4 EM Core Lock -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241
5 EM TX Lock -- last acquired @ /usr/src/sys/dev/em/if_em.c:1563
7 if send queue -- (already displayed)
10 if_addr_mtx -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
9 UMA boot pages -- (already displayed)
7 taskqueue -- (already displayed)
6 network driver -- (already displayed)
6 so_snd -- (already displayed)
9 ifnet -- (already displayed)
11 arc4_mtx -- (already displayed)
8 radix node head -- (already displayed)
9 rtentry -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
6 IPFW static rules -- (already displayed)
7 if send queue -- (already displayed)
5 EM TX Lock -- (already displayed)
7 so_rcv -- (already displayed)
3 accept -- (already displayed)
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
9 UMA boot pages -- (already displayed)
8 radix node head -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
9 rtentry -- (already displayed)
6 IPFW static rules -- (already displayed)
7 if send queue -- (already displayed)
5 EM TX Lock -- (already displayed)
1 rip -- last acquired @ /usr/src/sys/netinet/raw_ip.c:613
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
2 rawinp -- last acquired @ /usr/src/sys/netinet/in_pcb.c:221
5 ACPI semaphore -- (already displayed)
3 allprison -- (already displayed)
1 cpufreq lock -- last acquired @ /usr/src/sys/kern/kern_cpu.c:413
11 UMA zone -- (already displayed)
11 vm page queue free mutex -- (already displayed)
9 UMA boot pages -- (already displayed)
11 eventhandler -- (already displayed)
12 eventhandler list -- (already displayed)
9 ifnet -- (already displayed)
1 ACPI CPU -- last acquired @ /usr/src/sys/dev/acpica/acpi_cpu.c:1167
0 ng_node -- last acquired @ order list:0
1 ng_worklist -- last acquired @ order list:0
0 802.11 com lock -- last acquired @ order list:0
0 ddp_list_mtx -- last acquired @ order list:0
1 ddp_mtx -- last acquired @ order list:0
0 slip_mtx -- last acquired @ order list:0
1 slip sc_mtx -- last acquired @ order list:0
0 unp -- last acquired @ order list:0
6 so_snd -- (already displayed)
Spin locks:
Locks which were never acquired:
swap_pager swhash
SCSI CD Changer List
MD config lock
IPFW dynamic rules
arp_inq
tcp_hc_entry
ip_inq
pfs_vncache
ppp_softc_list_mtx
tunmtx
gif_mtx
msq
semid
nfs4dev state
nfs4dev waitq
nfs4dev newq
ntfs nthash
NFS xid lock
NFS reqq lock
ACPI global lock
CAM SIMQ lock
hptlock
LED sx
LED mtx
audit_pipe_mtx
nfslock
audit_trigger_mtx
ktrace_sx
bpin lock
UUID generator mutex lock
ACPI PCI power methods
ACPI lid
ACPI HPET support
ACPI embedded controller
ACPI thermal zone
ACPI Smart Battery
ACPI cmbat
ACPI power resources
ACPI generic battery
ACPI AC adapter
MSDOSFS fileno
pmc shared lock
kqueue order
firmware table
securelevel mutex lock
encapmtx
fifo mutex
phys_pager list
dev_pager list
swapdev
swap_pager list
vm map sleep mutex
db> show lockedvnods
Locked vnodes
0xffffff0001906000: tag ufs, type VREG
usecount 1, writecount 0, refcount 6 mountedhere 0
flags ()
v_object 0xffffff00018714e0 ref 0 pages 13
lock type ufs: SHARED (count 1)
ino 2240130, on dev ufs/usr64
db> show uma
Zone Size Used Free Requests
pfosfp 40 407 97 407
pfospfen 112 696 30 696
pfiaddrpl 120 0 0 0
pfstatescrub 40 0 0 0
pffrcent 24 0 0 0
pffrcache 80 0 0 0
pffrag 80 0 0 0
pffrent 32 0 0 0
pfrkentry2 216 0 0 0
pfrkentry 216 470 52 478
pfrktable 1296 12 27 28
pfpooladdrpl 88 12 72 12
pfaltqpl 240 0 0 0
pfstatepl 392 12 18 12
pfrulepl 912 69 15 69
pfsrctrpl 152 0 0 0
FFS2 dinode 256 426 39 435
FFS1 dinode 128 0 0 0
FFS inode 192 426 34 435
Mountpoints 808 6 9 6
SWAPMETA 288 0 0 0
IPFW dynamic rule 120 0 0 0
rtentry 240 24 40 26
ripcb 288 1 25 1
sackhole 32 0 0 0
tcpreass 40 0 0 0
hostcache 136 0 0 0
syncache 120 0 0 0
tcptw 88 0 0 0
tcpcb 728 9 16 11
inpcb 288 9 30 11
udpcb 288 20 32 717
ipq 56 0 0 0
unpcb 248 10 35 16
socket 696 40 15 747
KNOTE 120 2 60 2
itimer 360 0 0 0
ksiginfo 112 58 998 58
pipe 744 1 39 495
NFSNODE 664 0 0 0
NFSMOUNT 568 0 0 0
DIRHASH 1024 37 11 37
NAMEI 1024 0 36 6609
L VFS Cache 327 0 0 0
S VFS Cache 104 427 41 903
VNODEPOLL 128 0 0 0
VNODE 496 452 44 462
ata_composite 352 0 0 0
ata_request 312 1 83 1218
g_bio 216 5 355 6008
ACL UMA zone 388 0 0 0
mbuf_ext_refcnt 4 0 0 0
mbuf_jumbo_16k 16384 0 0 0
mbuf_jumbo_9k 9216 0 0 0
mbuf_jumbo_pagesize 4096 0 0 0
mbuf_cluster 2048 1024 8 1024
mbuf 256 1 385 396
mbuf_packet 256 781 243 956
audit_record 984 0 0 0
VMSPACE 416 10 44 841
SLEEPQUEUE 64 113 111 113
UPCALL 88 0 0 0
THREAD 824 103 9 103
PROC 1128 61 38 892
umtx pi 96 0 0 0
TURNSTILE 152 113 31 113
Files 120 73 82 4722
4096 4096 345 76 4049
2048 2048 65 11 302
1024 1024 60 152 2484
512 512 537 72 2490
256 256 601 74 4734
128 128 3535 90 10183
64 64 1791 113 27916
32 32 2096 328 3186
16 16 1666 350 31457
mt_zone 1024 217 7 217
DP fakepg 120 0 0 0
MAP ENTRY 112 200 196 27758
KMAP ENTRY 112 35 97 1448
MAP 248 7 23 7
VM OBJECT 208 585 99 12763
128 Bucket 1048 24 0 24
64 Bucket 536 34 1 34
32 Bucket 280 51 5 51
16 Bucket 152 73 2 73
UMA Hash 256 7 8 7
UMA RCntSlabs 128 516 6 516
UMA Slabs 128 625 13 1160
UMA Zones 216 89 13 89
UMA Kegs 216 89 13 89
db> show malloc
Type InUse MemUse Requests
msdosfs_mount 0 0K 0
msdosfs_fat 0 0K 0
msdosfs_fileno 0 0K 0
msdosfs_node 0 0K 0
DEVFS 27 1K 28
DEVFS_RULE 36 17K 36
DEVFS2 108 2K 108
DEVFS3 241 61K 242
DEVFS1 108 54K 108
USB 91 20K 106
USBdev 21 9K 21
atkbddev 2 1K 2
USBHC 0 0K 0
UART 0 0K 0
nexusdev 4 1K 4
msi 2 1K 2
twe_commands 0 0K 0
twa_commands 0 0K 0
firewire 0 0K 0
legacydrv 0 0K 0
io_apic 1 2K 1
entropy 1024 64K 1024
ppbusdev 3 1K 3
fw_xfer 0 0K 0
memdesc 1 4K 1
madt_table 0 0K 0
scsi_cd 0 0K 0
CAM ccb queue 0 0K 0
vm_pgdata 1 128K 1
acpipwr 0 0K 0
aaccam 0 0K 0
acpi_perf 0 0K 0
UMAHash 0 0K 0
ufs_mount 12 30K 12
ufs_quota 0 0K 0
ufs_dirhash 30 6K 30
pagedep 9 65K 16
inodedep 37 521K 49
newblk 1 1K 27
bmsafemap 6 1K 11
allocdirect 7 2K 23
indirdep 1 1K 5
allocindir 1 1K 3
freefrag 1 1K 2
freeblks 1 1K 4
freefile 3 1K 8
diradd 18 2K 28
mkdir 8 1K 12
dirrem 14 1K 25
newdirblk 0 0K 0
savedino 0 0K 12
audit_trigger 0 0K 0
audit_pipe 0 0K 0
audit_pipeent 0 0K 0
audit_pipe_preselect 0 0K 0
audit_evclass 150 5K 187
audit_bsm 0 0K 0
audit_cred 0 0K 0
audit_data 0 0K 0
audit_path 0 0K 0
audit_text 0 0K 0
rpcclnt 0 0K 0
agp 0 0K 0
nfss_srvsock 0 0K 0
nfss_srvdesc 0 0K 0
nfss_daemon 1 16K 1
nfsclient_lock 0 0K 0
nfsclient_nlminfo 0 0K 0
nfsclient_req 0 0K 0
nfsclient_bigfh 0 0K 0
nfsclient_diroff 0 0K 0
nfsclient_hash 0 0K 0
nfsclient_directio 0 0K 0
nfsclient_srvsock 0 0K 0
idmap 0 0K 0
nfs4_dev 0 0K 0
syncache 1 100K 1
tcplog 0 0K 0
aacbuf 0 0K 0
hostcache 1 36K 1
IpFw/IpAcct 1 1K 1
ipfw_tbl 0 0K 0
encap_export_host 0 0K 0
dummynet 0 0K 0
in_multi 4 1K 4
ip_moptions 0 0K 0
in_msource 0 0K 0
igmp 0 0K 0
routetbl 206 83K 465
pci_link 16 2K 16
tun 0 0K 0
SCSI SES 0 0K 0
sl 0 0K 0
ppp 0 0K 0
lo 1 1K 1
gif 0 0K 0
fw_com 0 0K 0
faith 0 0K 0
arpcom 3 1K 3
clone 6 24K 6
ifnet 7 13K 7
ifaddr 52 17K 52
ether_multi 17 1K 20
BPF 10 66K 11
subr_export_host 0 0K 0
SCSI sa 0 0K 0
mount 84 4K 138
vnodemarker 0 0K 20
vnodes 1 1K 1
vfs_hash 1 512K 1
export_host 0 0K 0
cl_savebuf 0 0K 0
vfscache 1 1024K 1
biobuf 38 76K 42
soname 5 1K 228
pcb 13 9K 15
CAM XPT 10 2K 25
mbuf_tag 12 1K 247
accf 0 0K 0
ptys 0 0K 0
ttys 403 70K 403
shm 1 16K 1
sem 4 8K 4
msg 4 30K 4
ioctlops 0 0K 1994
select 0 0K 0
iov 0 0K 1177
acpicmbat 0 0K 0
acpidev 63 4K 63
ciss_data 0 0K 0
CAM periph 2 1K 9
Unitno 7 1K 9
taskqueue 15 2K 15
stack 0 0K 0
CAM SIM 1 1K 1
scsi_da 0 0K 0
sbuf 0 0K 448
scsi_ch 0 0K 0
rman 161 20K 611
acpisem 12 2K 12
mfibuf 0 0K 0
kobj 227 908K 311
md_disk 0 0K 0
eventhandler 68 6K 68
devstat 16 33K 16
bus 961 75K 4819
bus-sc 66 83K 2400
md_sectors 0 0K 0
SWAP 0 0K 0
p1003.1b 1 1K 1
umtx 112 14K 112
sysctl 0 0K 143
sysctloid 2817 138K 2817
sysctltmp 0 0K 152
LED 0 0K 0
plimit 3 1K 254
uidinfo 4 3K 4
cred 11 3K 1266
pgrp 7 1K 7
session 7 1K 7
proc 2 16K 2
subproc 162 298K 993
kbdmux 6 9K 6
mtx_pool 1 8K 1
ipsbuf 0 0K 0
module 353 45K 353
iirbuf 0 0K 0
cache 0 0K 0
devbuf 314 1777K 321
temp 34 8K 3694
ip6opt 0 0K 0
ip6ndp 0 0K 0
CAM queue 3 1K 3
free 0 0K 0
lockf 2 1K 8
linker 96 132K 125
CAM dev queue 1 1K 1
KTRACE 100 13K 100
ast_driver 0 0K 0
afd_driver 0 0K 0
prison 0 0K 0
ithread 76 13K 76
acd_driver 0 0K 0
zombie 0 0K 0
proc-args 9 1K 327
kqueue 2 3K 2
kenv 71 11K 73
filedesc 64 33K 895
filedesc_to_leader 0 0K 0
sigio 1 1K 1
ar_driver 0 0K 10
ata_pci 0 0K 0
cdev 22 6K 22
sbp 0 0K 0
if_fwip 0 0K 0
ata_dma 2 1K 2
if_fwe 0 0K 0
ad_driver 2 1K 2
ata_generic 2 2K 2
acpitask 0 0K 1
isofs_mount 0 0K 0
isofs_node 0 0K 0
isadev 5 1K 5
acpica 1865 180K 58648
GEOM 144 25K 1163
fwmem 0 0K 0
pfs_vncache 0 0K 0
pfs_nodes 20 5K 20
ntfs_mount 0 0K 0
ntfs_ntnode 0 0K 0
ntfs_fnode 0 0K 0
ntfs_dir 0 0K 0
ntfs_vattr 0 0K 0
ntfsd_resdata 0 0K 0
ntfs_vrun 0 0K 0
ntfs_decomp 0 0K 0
ntfs_nthash 1 512K 1
db> show malloc
Type InUse MemUse Requests
msdosfs_mount 0 0K 0
msdosfs_fat 0 0K 0
msdosfs_fileno 0 0K 0
msdosfs_node 0 0K 0
DEVFS 27 1K 28
DEVFS_RULE 36 17K 36
DEVFS2 108 2K 108
DEVFS3 241 61K 242
DEVFS1 108 54K 108
USB 91 20K 106
USBdev 21 9K 21
atkbddev 2 1K 2
USBHC 0 0K 0
UART 0 0K 0
nexusdev 4 1K 4
msi 2 1K 2
twe_commands 0 0K 0
twa_commands 0 0K 0
firewire 0 0K 0
legacydrv 0 0K 0
io_apic 1 2K 1
entropy 1024 64K 1024
ppbusdev 3 1K 3
fw_xfer 0 0K 0
memdesc 1 4K 1
madt_table 0 0K 0
scsi_cd 0 0K 0
CAM ccb queue 0 0K 0
vm_pgdata 1 128K 1
acpipwr 0 0K 0
aaccam 0 0K 0
acpi_perf 0 0K 0
UMAHash 0 0K 0
ufs_mount 12 30K 12
ufs_quota 0 0K 0
ufs_dirhash 30 6K 30
pagedep 9 65K 16
inodedep 37 521K 49
newblk 1 1K 27
bmsafemap 6 1K 11
allocdirect 7 2K 23
indirdep 1 1K 5
allocindir 1 1K 3
freefrag 1 1K 2
freeblks 1 1K 4
freefile 3 1K 8
diradd 18 2K 28
mkdir 8 1K 12
dirrem 14 1K 25
newdirblk 0 0K 0
savedino 0 0K 12
audit_trigger 0 0K 0
audit_pipe 0 0K 0
audit_pipeent 0 0K 0
audit_pipe_preselect 0 0K 0
audit_evclass 150 5K 187
audit_bsm 0 0K 0
audit_cred 0 0K 0
audit_data 0 0K 0
audit_path 0 0K 0
audit_text 0 0K 0
rpcclnt 0 0K 0
agp 0 0K 0
nfss_srvsock 0 0K 0
nfss_srvdesc 0 0K 0
nfss_daemon 1 16K 1
nfsclient_lock 0 0K 0
nfsclient_nlminfo 0 0K 0
nfsclient_req 0 0K 0
nfsclient_bigfh 0 0K 0
nfsclient_diroff 0 0K 0
nfsclient_hash 0 0K 0
nfsclient_directio 0 0K 0
nfsclient_srvsock 0 0K 0
idmap 0 0K 0
nfs4_dev 0 0K 0
syncache 1 100K 1
tcplog 0 0K 0
aacbuf 0 0K 0
hostcache 1 36K 1
IpFw/IpAcct 1 1K 1
ipfw_tbl 0 0K 0
encap_export_host 0 0K 0
dummynet 0 0K 0
in_multi 4 1K 4
ip_moptions 0 0K 0
in_msource 0 0K 0
igmp 0 0K 0
routetbl 206 83K 465
pci_link 16 2K 16
tun 0 0K 0
SCSI SES 0 0K 0
sl 0 0K 0
ppp 0 0K 0
lo 1 1K 1
gif 0 0K 0
fw_com 0 0K 0
faith 0 0K 0
arpcom 3 1K 3
clone 6 24K 6
ifnet 7 13K 7
ifaddr 52 17K 52
ether_multi 17 1K 20
BPF 10 66K 11
subr_export_host 0 0K 0
SCSI sa 0 0K 0
mount 84 4K 138
vnodemarker 0 0K 20
vnodes 1 1K 1
vfs_hash 1 512K 1
export_host 0 0K 0
cl_savebuf 0 0K 0
vfscache 1 1024K 1
biobuf 38 76K 42
soname 5 1K 228
pcb 13 9K 15
CAM XPT 10 2K 25
mbuf_tag 12 1K 247
accf 0 0K 0
ptys 0 0K 0
ttys 403 70K 403
shm 1 16K 1
sem 4 8K 4
msg 4 30K 4
ioctlops 0 0K 1994
select 0 0K 0
iov 0 0K 1177
acpicmbat 0 0K 0
acpidev 63 4K 63
ciss_data 0 0K 0
CAM periph 2 1K 9
Unitno 7 1K 9
taskqueue 15 2K 15
stack 0 0K 0
CAM SIM 1 1K 1
scsi_da 0 0K 0
sbuf 0 0K 448
scsi_ch 0 0K 0
rman 161 20K 611
acpisem 12 2K 12
mfibuf 0 0K 0
kobj 227 908K 311
md_disk 0 0K 0
eventhandler 68 6K 68
devstat 16 33K 16
bus 961 75K 4819
bus-sc 66 83K 2400
md_sectors 0 0K 0
SWAP 0 0K 0
p1003.1b 1 1K 1
umtx 112 14K 112
sysctl 0 0K 143
sysctloid 2817 138K 2817
sysctltmp 0 0K 152
LED 0 0K 0
plimit 3 1K 254
uidinfo 4 3K 4
cred 11 3K 1266
pgrp 7 1K 7
session 7 1K 7
proc 2 16K 2
subproc 162 298K 993
kbdmux 6 9K 6
mtx_pool 1 8K 1
ipsbuf 0 0K 0
module 353 45K 353
iirbuf 0 0K 0
cache 0 0K 0
devbuf 314 1777K 321
temp 34 8K 3694
ip6opt 0 0K 0
ip6ndp 0 0K 0
CAM queue 3 1K 3
free 0 0K 0
lockf 2 1K 8
linker 96 132K 125
CAM dev queue 1 1K 1
KTRACE 100 13K 100
ast_driver 0 0K 0
afd_driver 0 0K 0
prison 0 0K 0
ithread 76 13K 76
acd_driver 0 0K 0
zombie 0 0K 0
proc-args 9 1K 327
kqueue 2 3K 2
kenv 71 11K 73
filedesc 64 33K 895
filedesc_to_leader 0 0K 0
sigio 1 1K 1
ar_driver 0 0K 10
ata_pci 0 0K 0
cdev 22 6K 22
sbp 0 0K 0
if_fwip 0 0K 0
ata_dma 2 1K 2
if_fwe 0 0K 0
ad_driver 2 1K 2
ata_generic 2 2K 2
acpitask 0 0K 1
isofs_mount 0 0K 0
isofs_node 0 0K 0
isadev 5 1K 5
acpica 1865 180K 58648
GEOM 144 25K 1163
fwmem 0 0K 0
pfs_vncache 0 0K 0
pfs_nodes 20 5K 20
ntfs_mount 0 0K 0
ntfs_ntnode 0 0K 0
ntfs_fnode 0 0K 0
ntfs_dir 0 0K 0
ntfs_vattr 0 0K 0
ntfsd_resdata 0 0K 0
ntfs_vrun 0 0K 0
ntfs_decomp 0 0K 0
ntfs_nthash 1 512K 1
db> cont
[...]
lock order reversal:
1st 0xffffffff8096eb88 PFil hook read/write mutex (PFil hook read/write mutex)
@ /usr/src/sys/net/pfil.c:73
2nd 0xffffffff8096f8a8 udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:385
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x539
_mtx_lock_flags() at _mtx_lock_flags+0x1f
udp_input() at udp_input+0x1f7
ip_input() at ip_input+0xa7
dummynet_send() at dummynet_send+0xde
dummynet_io() at dummynet_io+0x587
ipfw_check_in() at ipfw_check_in+0x241
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 ---
KDB: enter: witness_checkorder
[thread pid 24 tid 100023 ]
Stopped at kdb_enter+0x31: popq %rbp
db> cont
--uAKRQypu60I7Lcqm
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="second-lor.txt"
lock order reversal:
1st 0xffffffff8096eb88 PFil hook read/write mutex (PFil hook read/write mutex)
@ /usr/src/sys/net/pfil.c:73
2nd 0xffffffff8096f8a8 udp (udp) @ /usr/src/sys/netinet/udp_usrreq.c:385
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
witness_checkorder() at witness_checkorder+0x539
_mtx_lock_flags() at _mtx_lock_flags+0x1f
udp_input() at udp_input+0x1f7
ip_input() at ip_input+0xa7
dummynet_send() at dummynet_send+0xde
dummynet_io() at dummynet_io+0x587
ipfw_check_in() at ipfw_check_in+0x241
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 ---
KDB: enter: witness_checkorder
[thread pid 24 tid 100023 ]
Stopped at kdb_enter+0x31: popq %rbp
db> show pcpu
cpuid = 1
curthread = 0xffffff00011bf000: pid 24 "em1 taskq"
curpcb = 0xffffffffa057ad40
fpcurthread = none
idlethread = 0xffffff000107f680: pid 11 "idle: cpu1"
spin locks held:
db> show allpcpu
Current CPU: 1
cpuid = 0
curthread = 0xffffff0001eb3000: pid 44042 "sh"
curpcb = 0xffffffffa2ad9d40
fpcurthread = none
idlethread = 0xffffff000107f340: pid 12 "idle: cpu0"
spin locks held:
cpuid = 1
curthread = 0xffffff00011bf000: pid 24 "em1 taskq"
curpcb = 0xffffffffa057ad40
fpcurthread = none
idlethread = 0xffffff000107f680: pid 11 "idle: cpu1"
spin locks held:
db> trace
Tracing pid 24 tid 100023 td 0xffffff00011bf000
kdb_enter() at kdb_enter+0x31
witness_checkorder() at witness_checkorder+0x48e
_mtx_lock_flags() at _mtx_lock_flags+0x1f
udp_input() at udp_input+0x1f7
ip_input() at ip_input+0xa7
dummynet_send() at dummynet_send+0xde
dummynet_io() at dummynet_io+0x587
ipfw_check_in() at ipfw_check_in+0x241
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 ---
db> alltrace
Tracing command sh pid 44042 tid 100110 td 0xffffff0001eb3000
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x32
trap() at trap+0x2fd
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0x80052135a, rsp = 0x7fffffffcb10, rbp = 0x51df48 ---
Tracing command sh pid 44022 tid 100124 td 0xffffff001106b9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
pipe_read() at pipe_read+0x264
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800993acc, rsp = 0x7fffffffcd98, r
bp = 0x800b3ee0e ---
Tracing command sbcl pid 42648 tid 100119 td 0xffffff001106f000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
bwait() at bwait+0x55
bufwait() at bufwait+0x56
bread() at bread+0x1e
ffs_read() at ffs_read+0x2f7
vn_read() at vn_read+0x1d3
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800951acc, rsp = 0x7fffffffecd8, r
bp = 0x7fffffffed40 ---
Tracing command python pid 42488 tid 100090 td 0xffffff0001820000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
acquire() at acquire+0x5d
_lockmgr() at _lockmgr+0x24f
ffs_lock() at ffs_lock+0x71
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46
_vn_lock() at _vn_lock+0x83
vget() at vget+0xbd
cache_lookup() at cache_lookup+0x1d0
vfs_cache_lookup() at vfs_cache_lookup+0xb0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x41
lookup() at lookup+0x43d
namei() at namei+0x304
kern_stat() at kern_stat+0x56
stat() at stat+0x22
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (188, FreeBSD ELF64, stat), rip = 0x800b1014c, rsp = 0x7fffffffb138,
rbp = 0x800e15cb0 ---
Tracing command python pid 42307 tid 100117 td 0xffffff001106f680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
bwait() at bwait+0x55
bufwait() at bufwait+0x56
bread() at bread+0x1e
ffs_blkatoff() at ffs_blkatoff+0x56
ufs_lookup() at ufs_lookup+0x553
vfs_cache_lookup() at vfs_cache_lookup+0xe5
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x41
lookup() at lookup+0x43d
namei() at namei+0x304
kern_stat() at kern_stat+0x56
stat() at stat+0x22
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (188, FreeBSD ELF64, stat), rip = 0x800b1014c, rsp = 0x7fffffffbff8,
rbp = 0 ---
Tracing command detachtty pid 42114 tid 100114 td 0xffffff0011128340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
poll() at poll+0x583
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8007c79dc, rsp = 0x7fffffffeb38,
rbp = 0x7fffffffed10 ---
Tracing command cron pid 41177 tid 100086 td 0xffffff0001821000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x2ba
kern_nanosleep() at kern_nanosleep+0x10a
nanosleep() at nanosleep+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80090730c, rsp = 0x7fffffff
ec28, rbp = 0x5 ---
Tracing command httpd pid 41081 tid 100058 td 0xffffff0001408340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe7b8,
rbp = 0x7fffffffea48 ---
Tracing command httpd pid 41067 tid 100055 td 0xffffff000129b680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8,
rbp = 0x7fffffffe808 ---
Tracing command httpd pid 40958 tid 100104 td 0xffffff0001452000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8,
rbp = 0x7fffffffe808 ---
Tracing command httpd pid 40871 tid 100103 td 0xffffff0001452340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9f8
, rbp = 0x7fffffffea68 ---
Tracing command httpd pid 40776 tid 100065 td 0xffffff00013fd340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9f8
, rbp = 0x7fffffffea68 ---
Tracing command httpd pid 40720 tid 100063 td 0xffffff00013fd9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8,
rbp = 0x7fffffffe808 ---
Tracing command httpd pid 40471 tid 100095 td 0xffffff00017c69c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe6f8,
rbp = 0x7fffffffe808 ---
Tracing command httpd pid 40372 tid 100054 td 0xffffff000129b9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe7b8,
rbp = 0x7fffffffea48 ---
Tracing command httpd pid 40301 tid 100052 td 0xffffff00014089c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9f8
, rbp = 0x7fffffffea68 ---
Tracing command sshd pid 40142 tid 100064 td 0xffffff00013fd680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8016d7a4c, rsp = 0x7fffffffe618
, rbp = 0x3 ---
Tracing command httpd pid 39930 tid 100061 td 0xffffff00014539c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fb19dc, rsp = 0x7fffffffe798,
rbp = 0x7fffffffea28 ---
Tracing command httpd pid 39568 tid 100066 td 0xffffff00013fd000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
kern_select() at kern_select+0x833
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800ffea4c, rsp = 0x7fffffffeab8
, rbp = 0 ---
Tracing command httpd pid 39491 tid 100068 td 0xffffff0001404680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8
, rbp = 0x7fffffffea48 ---
Tracing command httpd pid 39418 tid 100073 td 0xffffff000146b000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8
, rbp = 0x7fffffffea48 ---
Tracing command httpd pid 39362 tid 100059 td 0xffffff0001454000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8
, rbp = 0x7fffffffea48 ---
Tracing command httpd pid 39263 tid 100093 td 0xffffff00017c7340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8
, rbp = 0x7fffffffea48 ---
Tracing command httpd pid 39182 tid 100051 td 0xffffff0001353340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_accept() at kern_accept+0x1cf
accept() at accept+0x61
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (30, FreeBSD ELF64, accept), rip = 0x800fa454c, rsp = 0x7fffffffe9d8
, rbp = 0x7fffffffea48 ---
Tracing command httpd pid 38789 tid 100062 td 0xffffff0001453680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
kern_select() at kern_select+0x833
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800ffea4c, rsp = 0x7fffffffeab8
, rbp = 0 ---
Tracing command bgpd pid 38165 tid 100076 td 0xffffff000147b340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e39dc, rsp = 0x7fffffffec18,
rbp = 0x47e4a337 ---
Tracing command bgpd pid 38092 tid 100067 td 0xffffff00014049c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
poll() at poll+0x583
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e39dc, rsp = 0x7fffffffec38,
rbp = 0 ---
Tracing command bgpd pid 38018 tid 100069 td 0xffffff0001404340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e39dc, rsp = 0x7fffffffeca8,
rbp = 0x9515 ---
Tracing command postgres pid 37711 tid 100053 td 0xffffff0001353000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
poll() at poll+0x1d2
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8014469dc, rsp = 0x7fffffffcff8,
rbp = 0x40 ---
Tracing command postgres pid 37633 tid 100078 td 0xffffff000147a9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
kern_select() at kern_select+0x833
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd3e8
, rbp = 0 ---
Tracing command postgres pid 37595 tid 100088 td 0xffffff0001820680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
kern_select() at kern_select+0x833
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd348
, rbp = 0 ---
Tracing command postgres pid 37564 tid 100075 td 0xffffff000147b680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
kern_select() at kern_select+0x833
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd318
, rbp = 0 ---
Tracing command postgres pid 37543 tid 100070 td 0xffffff0001404000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_cv_timedwait_sig() at _cv_timedwait_sig+0x12b
kern_select() at kern_select+0x833
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x801493a4c, rsp = 0x7fffffffd9a8
, rbp = 0x6 ---
Tracing command unlinkd pid 37266 tid 100083 td 0xffffff0001454680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
pipe_read() at pipe_read+0x264
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800704acc, rsp = 0x7fffffffe808, r
bp = 0x7fffffffecd8 ---
Tracing command squid pid 36740 tid 100060 td 0xffffff0001408000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ast() at ast+0x159
Xfast_syscall() at Xfast_syscall+0xe0
--- syscall (72, FreeBSD ELF64, ovadvise), rip = 0x8009d7aac, rsp = 0x7fffffffe9
28, rbp = 0xbd0 ---
Tracing command squid pid 36671 tid 100094 td 0xffffff00017c7000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_wait() at kern_wait+0x3c0
wait4() at wait4+0x33
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800967abc, rsp = 0x7fffffffeb18,
rbp = 0x7fffffffecc8 ---
Tracing command dhcpd pid 36225 tid 100081 td 0xffffff000147a000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800772a4c, rsp = 0x7fffffffe9d8
, rbp = 0 ---
Tracing command ntpd pid 34475 tid 100096 td 0xffffff00017c6680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x2ba
kern_kevent() at kern_kevent+0x36a
kevent() at kevent+0x85
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800bd531c, rsp = 0x7fffffffcc0
8, rbp = 0xf ---
Tracing command named pid 29710 tid 100102 td 0xffffff0001452680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800c03a4c, rsp = 0x7fffff5fbcc8
, rbp = 0x26 ---
Tracing command named pid 29710 tid 100101 td 0xffffff00014529c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x2ba
do_cv_wait() at do_cv_wait+0x312
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff7fc
dc8, rbp = 0x7fffff7fce30 ---
Tracing command named pid 29710 tid 100100 td 0xffffff0001453000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
do_cv_wait() at do_cv_wait+0x4c2
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffff9fd
eb8, rbp = 0 ---
Tracing command named pid 29710 tid 100099 td 0xffffff0001453340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
do_cv_wait() at do_cv_wait+0x4c2
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x51
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800b7d94c, rsp = 0x7fffffbfe
eb8, rbp = 0 ---
Tracing command named pid 29710 tid 100087 td 0xffffff00018209c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_sigtimedwait() at kern_sigtimedwait+0x4ea
sigwait() at sigwait+0x63
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800b7dc6c, rsp = 0x7fffffffec
48, rbp = 0x800e01120 ---
Tracing command syslogd pid 26625 tid 100074 td 0xffffff000147b9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80081aa4c, rsp = 0x7fffffffdda8
, rbp = 0x800a1c0f0 ---
Tracing command accounting pid 24352 tid 100085 td 0xffffff0001821340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
acct_thread() at acct_thread+0x1c3
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa2a5cd30, rbp = 0 ---
Tracing command devd pid 22944 tid 100082 td 0xffffff00014549c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x11d
kern_select() at kern_select+0x85f
select() at select+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x43796c, rsp = 0x7fffffffe928, r
bp = 0x7fffffffed50 ---
Tracing command pflogd pid 14667 tid 100080 td 0xffffff000147a340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x12
_sleep() at _sleep+0x2ba
bpfread() at bpfread+0xde
devfs_read_f() at devfs_read_f+0x67
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffed08, r
bp = 0x1 ---
Tracing command pflogd pid 14530 tid 100056 td 0xffffff000129b340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
soreceive_generic() at soreceive_generic+0xd1a
dofileread() at dofileread+0x8d
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800940acc, rsp = 0x7fffffffe938, r
bp = 0x4 ---
Tracing command pfpurge pid 14486 tid 100071 td 0xffffff000146b680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
pf_purge_thread() at pf_purge_thread+0x60
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa29efd30, rbp = 0 ---
Tracing command sh pid 51 tid 100050 td 0xffffff0001353680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_wait() at kern_wait+0x3c0
wait4() at wait4+0x33
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800923abc, rsp = 0x7fffffffe6d8,
rbp = 0x33 ---
Tracing command schedcpu pid 50 tid 100049 td 0xffffff00013539c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
schedcpu_thread() at schedcpu_thread+0x209
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa2981d30, rbp = 0 ---
Tracing command softdepflush pid 49 tid 100048 td 0xffffff0001355000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
softdep_flush() at softdep_flush+0x2da
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0679d30, rbp = 0 ---
Tracing command syncer pid 48 tid 100047 td 0xffffff0001355340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
sched_sync() at sched_sync+0x5da
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0674d30, rbp = 0 ---
Tracing command vnlru pid 47 tid 100046 td 0xffffff0001355680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
vnlru_proc() at vnlru_proc+0x692
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa066fd30, rbp = 0 ---
Tracing command bufdaemon pid 46 tid 100045 td 0xffffff00013559c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
buf_daemon() at buf_daemon+0x225
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa066ad30, rbp = 0 ---
Tracing command pagezero pid 45 tid 100044 td 0xffffff00012479c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
vm_pagezero() at vm_pagezero+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0665d30, rbp = 0 ---
Tracing command vmdaemon pid 44 tid 100043 td 0xffffff0001298000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
vm_daemon() at vm_daemon+0x4d
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0660d30, rbp = 0 ---
Tracing command pagedaemon pid 43 tid 100042 td 0xffffff0001298340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
vm_pageout() at vm_pageout+0xa62
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa065bd30, rbp = 0 ---
Tracing command dummynet pid 42 tid 100041 td 0xffffff0001298680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
msleep_spin() at msleep_spin+0x18f
taskqueue_thread_loop() at taskqueue_thread_loop+0x45
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0656d30, rbp = 0 ---
Tracing command irq1: atkbd0 pid 41 tid 100040 td 0xffffff00012989c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0631d30, rbp = 0 ---
Tracing command irq7: ppbus0 ppc0 pid 40 tid 100039 td 0xffffff0001299000
fork_trampoline() at fork_trampoline
Tracing command swi0: sio pid 39 tid 100038 td 0xffffff0001299340
fork_trampoline() at fork_trampoline
Tracing command fdc0 pid 38 tid 100037 td 0xffffff0001299680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
fdc_thread() at fdc_thread+0x7b5
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0622d30, rbp = 0 ---
Tracing command irq15: ata1 pid 37 tid 100036 td 0xffffff00012999c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0613d30, rbp = 0 ---
Tracing command irq14: ata0 pid 36 tid 100035 td 0xffffff000129b000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05fbd30, rbp = 0 ---
Tracing command usb4 pid 35 tid 100034 td 0xffffff00011c0680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05e3d30, rbp = 0 ---
Tracing command usb3 pid 34 tid 100033 td 0xffffff00011c09c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05ccd30, rbp = 0 ---
Tracing command irq16: uhci3 pid 33 tid 100032 td 0xffffff0001246000
fork_trampoline() at fork_trampoline
Tracing command usb2 pid 32 tid 100031 td 0xffffff0001246340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05c1d30, rbp = 0 ---
Tracing command usb1 pid 31 tid 100030 td 0xffffff0001246680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05bbd30, rbp = 0 ---
Tracing command irq19: uhci1 pid 30 tid 100029 td 0xffffff00012469c0
fork_trampoline() at fork_trampoline
Tracing command usbtask-dr pid 29 tid 100028 td 0xffffff0001247000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
usb_task_thread() at usb_task_thread+0x99
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05b0d30, rbp = 0 ---
Tracing command usbtask-hc pid 28 tid 100027 td 0xffffff0001247340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
usb_task_thread() at usb_task_thread+0x99
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05abd30, rbp = 0 ---
Tracing command usb0 pid 27 tid 100026 td 0xffffff0001247680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
usb_event_thread() at usb_event_thread+0xb0
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa05a6d30, rbp = 0 ---
Tracing command irq23: uhci0 ehci0 pid 26 tid 100025 td 0xffffff00011a6680
fork_trampoline() at fork_trampoline
Tracing command irq18: bge0 uhci2 pid 25 tid 100024 td 0xffffff00011a69c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa059bd30, rbp = 0 ---
Tracing command em1 taskq pid 24 tid 100023 td 0xffffff00011bf000
kdb_enter() at kdb_enter+0x31
witness_checkorder() at witness_checkorder+0x48e
_mtx_lock_flags() at _mtx_lock_flags+0x1f
udp_input() at udp_input+0x1f7
ip_input() at ip_input+0xa7
dummynet_send() at dummynet_send+0xde
dummynet_io() at dummynet_io+0x587
ipfw_check_in() at ipfw_check_in+0x241
pfil_run_hooks() at pfil_run_hooks+0xac
ip_input() at ip_input+0x292
ether_demux() at ether_demux+0x1ac
ether_input() at ether_input+0x1bf
em_handle_rxtx() at em_handle_rxtx+0x1d2
taskqueue_run() at taskqueue_run+0x95
taskqueue_thread_loop() at taskqueue_thread_loop+0x53
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa057ad30, rbp = 0 ---
Tracing command em0 taskq pid 23 tid 100022 td 0xffffff00011bf340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
msleep_spin() at msleep_spin+0x18f
taskqueue_thread_loop() at taskqueue_thread_loop+0x45
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0555d30, rbp = 0 ---
Tracing command irq9: acpi0 pid 22 tid 100021 td 0xffffff00011bf680
fork_trampoline() at fork_trampoline
Tracing command kqueue taskq pid 21 tid 100020 td 0xffffff00011bf9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa052bd30, rbp = 0 ---
Tracing command swi6: task queue pid 20 tid 100019 td 0xffffff00011c0000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0526d30, rbp = 0 ---
Tracing command swi6: Giant taskq pid 19 tid 100018 td 0xffffff00011c0340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0521d30, rbp = 0 ---
Tracing command thread taskq pid 9 tid 100017 td 0xffffff000108c9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa051cd30, rbp = 0 ---
Tracing command swi5: + pid 18 tid 100016 td 0xffffff00011a5000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0517d30, rbp = 0 ---
Tracing command swi2: cambio pid 17 tid 100015 td 0xffffff00011a5340
fork_trampoline() at fork_trampoline
Tracing command xpt_thrd pid 8 tid 100014 td 0xffffff00011a5680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
xpt_scanner_thread() at xpt_scanner_thread+0x38
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa050dd30, rbp = 0 ---
Tracing command acpi_task_2 pid 7 tid 100013 td 0xffffff00011a59c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0508d30, rbp = 0 ---
Tracing command acpi_task_1 pid 6 tid 100012 td 0xffffff00011a6000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa0503d30, rbp = 0 ---
Tracing command acpi_task_0 pid 5 tid 100011 td 0xffffff00011a6340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_sleep() at _sleep+0x1ec
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04fed30, rbp = 0 ---
Tracing command yarrow pid 16 tid 100010 td 0xffffff0001080340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
random_kthread() at random_kthread+0x1a4
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04f5d30, rbp = 0 ---
Tracing command g_down pid 4 tid 100009 td 0xffffff0001080680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
g_io_schedule_down() at g_io_schedule_down+0x201
g_down_procbody() at g_down_procbody+0x55
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04f0d30, rbp = 0 ---
Tracing command g_up pid 3 tid 100008 td 0xffffff00010809c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
g_io_schedule_up() at g_io_schedule_up+0x119
g_up_procbody() at g_up_procbody+0x55
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04ebd30, rbp = 0 ---
Tracing command g_event pid 2 tid 100007 td 0xffffff000108c000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_timedwait() at sleepq_timedwait+0x31
_sleep() at _sleep+0x1d4
g_event_procbody() at g_event_procbody+0xa9
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04e6d30, rbp = 0 ---
Tracing command swi3: vm pid 15 tid 100006 td 0xffffff000108c340
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 14 tid 100005 td 0xffffff000108c680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04dcd30, rbp = 0 ---
Tracing command swi1: net pid 13 tid 100004 td 0xffffff000107f000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
ithread_loop() at ithread_loop+0x2d1
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04d7d30, rbp = 0 ---
Tracing command idle: cpu0 pid 12 tid 100003 td 0xffffff000107f340
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sched_idletd() at sched_idletd+0x3c
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04d1d30, rbp = 0 ---
Tracing command idle: cpu1 pid 11 tid 100002 td 0xffffff000107f680
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sched_idletd() at sched_idletd+0x3c
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04ccd30, rbp = 0 ---
Tracing command init pid 1 tid 100001 td 0xffffff000107f9c0
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_catch_signals() at sleepq_catch_signals+0x1c5
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x2aa
kern_wait() at kern_wait+0x3c0
wait4() at wait4+0x33
syscall() at syscall+0x1b5
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40b61c, rsp = 0x7fffffffe848, rbp
= 0x33 ---
Tracing command audit pid 10 tid 100000 td 0xffffff0001080000
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
sleepq_wait() at sleepq_wait+0x31
_cv_wait() at _cv_wait+0x115
audit_worker() at audit_worker+0x47f
fork_exit() at fork_exit+0x112
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffa04c2d30, rbp = 0 ---
Tracing command swapper pid 0 tid 0 td 0xffffffff808def70
sched_switch() at sched_switch+0x131
mi_switch() at mi_switch+0x186
scheduler() at scheduler+0x39e
mi_startup() at mi_startup+0x59
btext() at btext+0x2c
db> show alllocks
Process 14530 (pflogd) thread 0xffffff000129b340 (100056)
exclusive sx so_rcv_sx r = 0 (0xffffff000140f3b8) locked @ /usr/src/sys/kern/uip
c_sockbuf.c:148
Process 24 (em1 taskq) thread 0xffffff00011bf000 (100023)
shared rw PFil hook read/write mutex r = 0 (0xffffffff8096eb88) locked @ /usr/sr
c/sys/net/pfil.c:73
db> show witness
Sleep locks:
0 so_rcv_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148
7 so_rcv -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2475
13 sellck -- last acquired @ /usr/src/sys/kern/sys_generic.c:1127
8 radix node head -- last acquired @ /usr/src/sys/net/route.c:147
11 rtentry -- last acquired @ /usr/src/sys/netinet/ip_output.c:594
12 ifaddr -- last acquired @ /usr/src/sys/net/route.c:287
12 rts_inq -- last acquired @ /usr/src/sys/net/netisr.c:140
12 UMA zone -- last acquired @ /usr/src/sys/vm/uma_core.c:1874
12 vm page queue free mutex -- last acquired @ /usr/src/sys/vm/vm_page.c:1030
12 UMA zone -- (already displayed)
11 UMA boot pages -- last acquired @ /usr/src/sys/vm/uma_core.c:916
12 vm page queue free mutex -- (already displayed)
9 ifnet -- last acquired @ /usr/src/sys/net/if.c:1473
11 eventhandler -- last acquired @ /usr/src/sys/kern/subr_eventhandler.c:212
12 eventhandler list -- last acquired @ /usr/src/sys/kern/kern_exit.c:233
10 if_addr_mtx -- last acquired @ /usr/src/sys/net/if.c:828
10 pf task mtx -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/ne
t/pf.c:6729
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 UMA boot pages -- (already displayed)
11 arc4_mtx -- last acquired @ /usr/src/sys/libkern/arc4random.c:137
11 bpf interface lock -- last acquired @ /usr/src/sys/net/bpf.c:1359
12 bpf cdev lock -- last acquired @ /usr/src/sys/net/bpf.c:1363
13 sellck -- (already displayed)
11 eventhandler -- (already displayed)
11 rtentry -- (already displayed)
12 eventhandler list -- (already displayed)
8 process lock -- last acquired @ /usr/src/sys/amd64/amd64/trap.c:622
9 session -- last acquired @ /usr/src/sys/kern/kern_fork.c:582
10 uidinfo hash -- last acquired @ /usr/src/sys/kern/kern_resource.c:1213
11 uidinfo struct -- last acquired @ order list:0
11 sleep mtxpool -- last acquired @ /usr/src/sys/kern/kern_descrip.c:2139
12 kqueue -- last acquired @ /usr/src/sys/kern/kern_event.c:1021
13 struct mount mtx -- last acquired @ /usr/src/sys/kern/vfs_mount.c:440
14 vnode interlock -- last acquired @ /usr/src/sys/kern/vfs_subr.c:2186
15 cdev -- last acquired @ /usr/src/sys/kern/kern_conf.c:69
15 vnode_free_list -- last acquired @ /usr/src/sys/kern/vfs_subr.c:3012
15 Syncer mtx -- last acquired @ /usr/src/sys/kern/vfs_subr.c:1915
10 tty -- last acquired @ /usr/src/sys/kern/kern_event.c:1664
9 sigacts -- last acquired @ /usr/src/sys/kern/subr_sleepqueue.c:392
9 ktrace -- last acquired @ /usr/src/sys/kern/kern_fork.c:600
11 sleep mtxpool -- (already displayed)
11 sleep mtxpool -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
12 kqueue -- (already displayed)
11 UMA boot pages -- (already displayed)
8 process lock -- (already displayed)
2 unp_mtx -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:557
3 accept -- last acquired @ /usr/src/sys/kern/uipc_socket.c:685
6 so_snd -- last acquired @ /usr/src/sys/kern/uipc_socket.c:2474
7 so_rcv -- (already displayed)
11 sleep mtxpool -- (already displayed)
8 radix node head -- (already displayed)
11 rtentry -- (already displayed)
7 tcp_hc_entry -- last acquired @ /usr/src/sys/netinet/tcp_hostcache.c:291
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
13 sellck -- (already displayed)
12 kqueue -- (already displayed)
12 UMA zone -- (already displayed)
9 vm page queue mutex -- last acquired @ /usr/src/sys/vm/vm_fault.c:887
14 vnode interlock -- (already displayed)
10 pmap -- last acquired @ /usr/src/sys/amd64/amd64/pmap.c:3076
12 vm page queue free mutex -- (already displayed)
12 vm page queue free mutex -- (already displayed)
7 so_rcv -- (already displayed)
7 so_rcv -- (already displayed)
6 so_snd -- (already displayed)
3 filedesc structure -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1025
14 vnode interlock -- (already displayed)
8 process lock -- (already displayed)
11 sleep mtxpool -- (already displayed)
5 Name Cache -- last acquired @ /usr/src/sys/kern/vfs_cache.c:327
14 vnode interlock -- (already displayed)
12 UMA zone -- (already displayed)
11 UMA boot pages -- (already displayed)
4 fdesc -- last acquired @ /usr/src/sys/kern/kern_descrip.c:1472
15 cdev -- (already displayed)
4 Giant -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241
5 pipe mutex -- last acquired @ /usr/src/sys/kern/sys_pipe.c:572
6 sigio lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:782
7 process group -- last acquired @ /usr/src/sys/kern/kern_fork.c:572
8 process lock -- (already displayed)
9 session -- (already displayed)
13 sellck -- (already displayed)
12 UMA zone -- (already displayed)
7 system map -- last acquired @ /usr/src/sys/vm/vm_map.c:3111
9 vm page queue mutex -- (already displayed)
8 kmem object -- last acquired @ /usr/src/sys/vm/vm_kern.c:410
12 vm page queue free mutex -- (already displayed)
9 vm page queue mutex -- (already displayed)
8 kernel object -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3641
12 vm page queue free mutex -- (already displayed)
9 vm page queue mutex -- (already displayed)
12 vm page queue free mutex -- (already displayed)
10 pmap -- (already displayed)
8 KMAP ENTRY -- last acquired @ /usr/src/sys/vm/uma_core.c:414
12 UMA zone -- (already displayed)
8 standard object -- last acquired @ /usr/src/sys/vm/vm_object.c:441
12 vm page queue free mutex -- (already displayed)
14 vnode interlock -- (already displayed)
9 vm page queue mutex -- (already displayed)
9 vm object_list -- last acquired @ /usr/src/sys/vm/vm_object.c:225
12 UMA zone -- (already displayed)
9 swap_pager swhash -- last acquired @ /usr/src/sys/vm/swap_pager.c:1888
5 UMA lock -- last acquired @ /usr/src/sys/vm/uma_core.c:1492
12 UMA zone -- (already displayed)
8 KMAP ENTRY -- (already displayed)
11 UMA boot pages -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 eventhandler -- (already displayed)
12 eventhandler list -- (already displayed)
5 kobj -- last acquired @ /usr/src/sys/kern/subr_kobj.c:307
6 kernel environment -- last acquired @ /usr/src/sys/kern/kern_environment.c:
301
8 kernel object -- (already displayed)
9 vm object_list -- (already displayed)
8 KMAP ENTRY -- (already displayed)
10 uidinfo hash -- (already displayed)
8 process lock -- (already displayed)
11 sleep mtxpool -- (already displayed)
5 evclass_mtx -- last acquired @ /usr/src/sys/security/audit/audit_bsm_klib.c
:112
5 TID lock -- last acquired @ /usr/src/sys/kern/subr_unit.c:623
8 standard object -- (already displayed)
5 intr event -- last acquired @ /usr/src/sys/kern/kern_intr.c:454
15 cdev -- (already displayed)
5 GEOM orphanage -- last acquired @ /usr/src/sys/geom/geom_event.c:201
5 ttylist -- last acquired @ /usr/src/sys/kern/tty.c:2901
10 tty -- (already displayed)
5 intr config -- last acquired @ /usr/src/sys/kern/subr_autoconf.c:73
5 taskqueue list -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:125
5 XPT lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2668
6 XPT topology lock -- last acquired @ /usr/src/sys/cam/cam_xpt.c:2673
6 kernel environment -- (already displayed)
7 taskqueue -- last acquired @ /usr/src/sys/kern/subr_taskqueue.c:73
5 rman head -- last acquired @ /usr/src/sys/kern/subr_rman.c:152
5 rman -- last acquired @ /usr/src/sys/kern/subr_rman.c:539
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
5 devd -- last acquired @ /usr/src/sys/kern/subr_bus.c:489
13 sellck -- (already displayed)
13 sellck -- (already displayed)
5 ACPI semaphore -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSynch.c:30
3
5 acpi subsystem HW lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdSy
nch.c:377
5 acpi subsystem GPE lock -- last acquired @ /usr/src/sys/dev/acpica/Osd/OsdS
ynch.c:377
7 taskqueue -- (already displayed)
5 msi -- last acquired @ /usr/src/sys/amd64/amd64/msi.c:389
9 ifnet -- (already displayed)
5 bpf global lock -- last acquired @ /usr/src/sys/net/bpf.c:396
11 bpf interface lock -- (already displayed)
5 bounce pages lock -- last acquired @ /usr/src/sys/amd64/amd64/busdma_machde
p.c:1073
10 pmap -- (already displayed)
5 unit# allocation -- last acquired @ /usr/src/sys/kern/subr_unit.c:623
15 vnode_free_list -- (already displayed)
5 pfs_node -- last acquired @ /usr/src/sys/fs/pseudofs/pseudofs_internal.h:10
3
5 pfs_fileno -- last acquired @ /usr/src/sys/kern/subr_unit.c:623
6 random reseed -- last acquired @ /usr/src/sys/dev/random/yarrow.c:191
11 arc4_mtx -- (already displayed)
5 nfsd_mtx -- last acquired @ /usr/src/sys/nfsserver/nfs_srvsock.c:796
6 so_snd -- (already displayed)
5 if_clone lock -- last acquired @ /usr/src/sys/net/if_clone.c:164
5 if_cloners lock -- last acquired @ /usr/src/sys/net/if_clone.c:252
5 domain list -- last acquired @ /usr/src/sys/kern/uipc_domain.c:228
6 pfil_head_list lock -- last acquired @ /usr/src/sys/net/pfil.c:159
5 PFil hook read/write mutex -- last acquired @ /usr/src/sys/net/pfil.c:73
6 pfil_head_list lock -- (already displayed)
10 pf task mtx -- (already displayed)
6 IPFW static rules -- last acquired @ /usr/src/sys/netinet/ip_fw2.c:2647
12 UMA zone -- (already displayed)
8 radix node head -- (already displayed)
7 if send queue -- last acquired @ /usr/src/sys/dev/em/if_em.c:949
6 network driver -- last acquired @ /usr/src/sys/dev/bge/if_bge.c:3014
10 if_addr_mtx -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
7 taskqueue -- (already displayed)
7 if send queue -- (already displayed)
6 ip_inq -- last acquired @ /usr/src/sys/net/netisr.c:140
11 rtentry -- (already displayed)
6 EM TX Lock -- last acquired @ /usr/src/sys/dev/em/if_em.c:980
7 if send queue -- (already displayed)
11 bpf interface lock -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
6 dummynet -- last acquired @ /usr/src/sys/netinet/ip_dummynet.c:739
12 UMA zone -- (already displayed)
7 system map -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 eventhandler -- (already displayed)
5 isn_mtx -- last acquired @ /usr/src/sys/netinet/tcp_subr.c:1426
6 random reseed -- (already displayed)
11 arc4_mtx -- (already displayed)
8 radix node head -- (already displayed)
6 IPFW static rules -- (already displayed)
5 lo_mtx -- last acquired @ /usr/src/sys/net/if_loop.c:161
5 ATA queue lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:177
6 ATA state lock -- last acquired @ /usr/src/sys/dev/ata/ata-queue.c:194
5 devstat -- last acquired @ /usr/src/sys/kern/subr_devstat.c:83
6 XPT topology lock -- (already displayed)
5 NFS iod lock -- last acquired @ /usr/src/sys/nfsclient/nfs_nfsiod.c:196
6 mountlist -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:760
13 struct mount mtx -- (already displayed)
13 struct mount mtx -- (already displayed)
5 mntid -- last acquired @ /usr/src/sys/kern/vfs_subr.c:460
6 mountlist -- (already displayed)
14 vnode interlock -- (already displayed)
5 buf queue lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:1468
14 vnode interlock -- (already displayed)
5 bio queue -- last acquired @ /usr/src/sys/geom/geom_io.c:68
5 bdone lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:3800
5 needsbuffer lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:321
5 FFS Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1149
11 arc4_mtx -- (already displayed)
5 Name Cache -- (already displayed)
5 knlist lock for lockless objects -- last acquired @ /usr/src/sys/kern/kern_
event.c:1664
5 vfs hash -- last acquired @ /usr/src/sys/kern/vfs_hash.c:71
14 vnode interlock -- (already displayed)
5 devfs interlock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:194
14 vnode interlock -- (already displayed)
5 dirhash list -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:349
6 dirhash -- last acquired @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:368
6 dirhash -- (already displayed)
5 pbuf mutex -- last acquired @ /usr/src/sys/vm/vm_pager.c:413
9 vm page queue mutex -- (already displayed)
7 process group -- (already displayed)
10 tty -- (already displayed)
9 session -- (already displayed)
5 Softdep Lock -- last acquired @ /usr/src/sys/ufs/ffs/ffs_softdep.c:4892
7 system map -- (already displayed)
12 UMA zone -- (already displayed)
14 vnode interlock -- (already displayed)
6 buffer daemon lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:429
10 pf task mtx -- (already displayed)
6 buffer daemon lock -- (already displayed)
6 sigio lock -- (already displayed)
6 so_snd -- (already displayed)
5 pipe mutex -- (already displayed)
12 bpf cdev lock -- (already displayed)
12 kqueue -- (already displayed)
7 system map -- (already displayed)
12 UMA zone -- (already displayed)
7 so_rcv -- (already displayed)
4 protect sysfilt_ops -- last acquired @ /usr/src/sys/kern/kern_event.c:774
12 UMA zone -- (already displayed)
3 user map -- last acquired @ /usr/src/sys/vm/vm_map.c:3111
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
9 vm object_list -- (already displayed)
8 standard object -- (already displayed)
9 vm page queue mutex -- (already displayed)
10 pmap -- (already displayed)
14 vnode interlock -- (already displayed)
5 buf queue lock -- (already displayed)
5 needsbuffer lock -- (already displayed)
7 system map -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
13 struct mount mtx -- (already displayed)
6 buffer daemon lock -- (already displayed)
5 Softdep Lock -- (already displayed)
9 swap_pager swhash -- (already displayed)
2 tcpinp -- last acquired @ /usr/src/sys/netinet/tcp_input.c:479
6 so_snd -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
7 so_rcv -- (already displayed)
9 ifnet -- (already displayed)
3 tcp_sc_head -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 arc4_mtx -- (already displayed)
8 radix node head -- (already displayed)
11 rtentry -- (already displayed)
7 tcp_hc_entry -- (already displayed)
3 accept -- (already displayed)
3 so_glabel -- last acquired @ /usr/src/sys/kern/uipc_socket.c:299
8 radix node head -- (already displayed)
11 rtentry -- (already displayed)
7 tcp_hc_entry -- (already displayed)
11 arc4_mtx -- (already displayed)
5 isn_mtx -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
6 IPFW static rules -- (already displayed)
7 if send queue -- (already displayed)
6 EM TX Lock -- (already displayed)
6 ip_inq -- (already displayed)
0 so_snd_sx -- last acquired @ /usr/src/sys/kern/uipc_sockbuf.c:148
6 so_snd -- (already displayed)
1 unp_global_rwlock -- last acquired @ /usr/src/sys/kern/uipc_usrreq.c:556
2 unp_mtx -- (already displayed)
3 accept -- (already displayed)
3 so_glabel -- (already displayed)
6 so_snd -- (already displayed)
3 filedesc structure -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
8 radix node head -- (already displayed)
11 rtentry -- (already displayed)
12 rts_inq -- (already displayed)
2 tcpinp -- (already displayed)
8 standard object -- (already displayed)
14 vnode interlock -- (already displayed)
5 buf queue lock -- (already displayed)
7 system map -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
5 needsbuffer lock -- (already displayed)
0 pf_statetbl_lock -- last acquired @ /usr/src/sys/modules/pf/../../contrib/pf/n
et/pf.c:979
10 pf task mtx -- (already displayed)
6 pfil_head_list lock -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
0 devfsmount -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops.c:691
5 devfs interlock -- (already displayed)
15 vnode_free_list -- (already displayed)
14 vnode interlock -- (already displayed)
13 struct mount mtx -- (already displayed)
15 cdev -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 UMA boot pages -- (already displayed)
1 DEVFS ruleset lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_rule.c:177
8 process lock -- (already displayed)
3 user map -- (already displayed)
0 g_disk_done -- last acquired @ /usr/src/sys/geom/geom_disk.c:198
5 bio queue -- (already displayed)
12 UMA zone -- (already displayed)
0 ipqlock -- last acquired @ /usr/src/sys/netinet/ip_input.c:1086
0 semid -- last acquired @ /usr/src/sys/kern/sysv_sem.c:658
1 sem -- last acquired @ /usr/src/sys/kern/sysv_sem.c:1288
0 fdc lock -- last acquired @ /usr/src/sys/dev/fdc/fdc.c:803
0 if_afdata -- last acquired @ /usr/src/sys/net/if.c:578
0 GEOM topology -- last acquired @ /usr/src/sys/geom/geom_event.c:233
5 GEOM orphanage -- (already displayed)
5 devstat -- (already displayed)
5 unit# allocation -- (already displayed)
15 cdev -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
11 UMA boot pages -- (already displayed)
9 vm object_list -- (already displayed)
14 vnode interlock -- (already displayed)
8 standard object -- (already displayed)
7 system map -- (already displayed)
0 intr sources -- last acquired @ /usr/src/sys/amd64/amd64/intr_machdep.c:593
5 msi -- (already displayed)
0 audit_mtx -- last acquired @ /usr/src/sys/security/audit/audit_worker.c:448
0 uma object -- last acquired @ /usr/src/sys/vm/vm_meter.c:115
0 p_peers -- last acquired @ /usr/src/sys/kern/kern_exit.c:284
0 runningbufspace lock -- last acquired @ /usr/src/sys/kern/vfs_bio.c:340
0 umtxql -- last acquired @ /usr/src/sys/kern/kern_umtx.c:326
0 accept_filter_mtx -- last acquired @ /usr/src/sys/kern/uipc_accf.c:116
0 ACPI PCI bus methods -- last acquired @ /usr/src/sys/dev/acpica/acpi_pcib.c:22
1
0 rtsock route_cb lock -- last acquired @ /usr/src/sys/net/rtsock.c:191
0 rawcb -- last acquired @ /usr/src/sys/net/raw_usrreq.c:81
7 so_rcv -- (already displayed)
12 UMA zone -- (already displayed)
0 ACPI PCI link -- last acquired @ /usr/src/sys/dev/acpica/acpi_pci_link.c:432
5 ACPI semaphore -- (already displayed)
0 ACPI root bus -- last acquired @ /usr/src/sys/dev/acpica/acpi.c:1022
5 rman -- (already displayed)
5 ACPI semaphore -- (already displayed)
7 system map -- (already displayed)
0 acct_sx -- last acquired @ /usr/src/sys/kern/kern_acct.c:351
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 eventhandler -- (already displayed)
5 TID lock -- (already displayed)
9 vm object_list -- (already displayed)
7 system map -- (already displayed)
8 standard object -- (already displayed)
1 proctree -- last acquired @ /usr/src/sys/kern/kern_fork.c:571
2 allproc -- last acquired @ /usr/src/sys/kern/kern_fork.c:297
3 allprison -- last acquired @ /usr/src/sys/kern/kern_jail.c:945
11 sleep mtxpool -- (already displayed)
8 process lock -- (already displayed)
4 fdesc -- (already displayed)
3 filedesc structure -- (already displayed)
14 vnode interlock -- (already displayed)
3 user map -- (already displayed)
11 arc4_mtx -- (already displayed)
7 process group -- (already displayed)
4 Giant -- (already displayed)
8 process lock -- (already displayed)
9 session -- (already displayed)
6 sigio lock -- (already displayed)
2 clone events drain lock -- last acquired @ /usr/src/sys/fs/devfs/devfs_vnops
.c:626
11 eventhandler -- (already displayed)
12 eventhandler list -- (already displayed)
15 cdev -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
3 filedesc structure -- (already displayed)
8 process lock -- (already displayed)
5 FFS Lock -- (already displayed)
13 struct mount mtx -- (already displayed)
14 vnode interlock -- (already displayed)
5 buf queue lock -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
4 Giant -- (already displayed)
5 needsbuffer lock -- (already displayed)
6 buffer daemon lock -- (already displayed)
5 Softdep Lock -- (already displayed)
0 vm daemon -- last acquired @ /usr/src/sys/vm/vm_pageout.c:1531
0 sysctl lock -- last acquired @ /usr/src/sys/kern/kern_sysctl.c:1396
11 arc4_mtx -- (already displayed)
2 allproc -- (already displayed)
8 process lock -- (already displayed)
3 user map -- (already displayed)
15 cdev -- (already displayed)
1 filelist lock -- last acquired @ /usr/src/sys/kern/kern_descrip.c:2179
3 filedesc structure -- (already displayed)
5 GEOM orphanage -- (already displayed)
4 Giant -- (already displayed)
9 ktrace -- (already displayed)
1 kernel linker -- last acquired @ /usr/src/sys/kern/kern_linker.c:1096
12 UMA zone -- (already displayed)
2 module subsystem sx lock -- last acquired @ /usr/src/sys/kern/kern_module.c:
407
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
3 filedesc structure -- (already displayed)
14 vnode interlock -- (already displayed)
5 Name Cache -- (already displayed)
5 buf queue lock -- (already displayed)
5 needsbuffer lock -- (already displayed)
5 vfs hash -- (already displayed)
15 vnode_free_list -- (already displayed)
13 struct mount mtx -- (already displayed)
5 dirhash list -- (already displayed)
12 vm page queue free mutex -- (already displayed)
7 system map -- (already displayed)
8 kernel object -- (already displayed)
5 bio queue -- (already displayed)
5 bdone lock -- (already displayed)
6 dirhash -- (already displayed)
15 cdev -- (already displayed)
9 vm object_list -- (already displayed)
8 standard object -- (already displayed)
5 pbuf mutex -- (already displayed)
6 buffer daemon lock -- (already displayed)
5 kobj -- (already displayed)
1 malloc -- last acquired @ /usr/src/sys/kern/kern_malloc.c:741
12 UMA zone -- (already displayed)
7 system map -- (already displayed)
5 devstat -- (already displayed)
5 ttylist -- (already displayed)
9 vm object_list -- (already displayed)
5 UMA lock -- (already displayed)
15 Syncer mtx -- (already displayed)
1 unp_global_rwlock -- (already displayed)
1 tcp -- last acquired @ /usr/src/sys/netinet/tcp_timer.c:128
2 tcpinp -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
8 radix node head -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
11 rtentry -- (already displayed)
6 IPFW static rules -- (already displayed)
7 if send queue -- (already displayed)
6 EM TX Lock -- (already displayed)
3 accept -- (already displayed)
11 sleep mtxpool -- (already displayed)
3 so_glabel -- (already displayed)
6 ip_inq -- (already displayed)
3 tcp_sc_head -- (already displayed)
1 udp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:839
2 udpinp -- last acquired @ /usr/src/sys/netinet/udp_usrreq.c:843
3 in_multi_mtx -- last acquired @ /usr/src/sys/netinet/in_mcast.c:320
4 igmp_mtx -- last acquired @ /usr/src/sys/netinet/igmp.c:446
10 if_addr_mtx -- (already displayed)
10 if_addr_mtx -- (already displayed)
4 EM Core Lock -- last acquired @ /usr/src/sys/kern/kern_timeout.c:241
6 EM TX Lock -- (already displayed)
10 if_addr_mtx -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 UMA boot pages -- (already displayed)
7 taskqueue -- (already displayed)
6 network driver -- (already displayed)
6 so_snd -- (already displayed)
9 ifnet -- (already displayed)
11 arc4_mtx -- (already displayed)
8 radix node head -- (already displayed)
11 rtentry -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
6 IPFW static rules -- (already displayed)
7 if send queue -- (already displayed)
6 EM TX Lock -- (already displayed)
7 so_rcv -- (already displayed)
3 accept -- (already displayed)
6 ip_inq -- (already displayed)
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 UMA boot pages -- (already displayed)
8 radix node head -- (already displayed)
5 PFil hook read/write mutex -- (already displayed)
11 rtentry -- (already displayed)
6 IPFW static rules -- (already displayed)
7 if send queue -- (already displayed)
6 EM TX Lock -- (already displayed)
1 rip -- last acquired @ /usr/src/sys/netinet/raw_ip.c:638
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
2 rawinp -- last acquired @ /usr/src/sys/netinet/raw_ip.c:639
7 so_rcv -- (already displayed)
5 ACPI semaphore -- (already displayed)
3 allprison -- (already displayed)
1 cpufreq lock -- last acquired @ /usr/src/sys/kern/kern_cpu.c:413
12 UMA zone -- (already displayed)
12 vm page queue free mutex -- (already displayed)
11 UMA boot pages -- (already displayed)
11 eventhandler -- (already displayed)
12 eventhandler list -- (already displayed)
9 ifnet -- (already displayed)
1 ACPI CPU -- last acquired @ /usr/src/sys/dev/acpica/acpi_cpu.c:1167
8 radix node head -- (already displayed)
14 vnode interlock -- (already displayed)
3 filedesc structure -- (already displayed)
0 ng_node -- last acquired @ order list:0
1 ng_worklist -- last acquired @ order list:0
0 802.11 com lock -- last acquired @ order list:0
0 ddp_list_mtx -- last acquired @ order list:0
1 ddp_mtx -- last acquired @ order list:0
0 slip_mtx -- last acquired @ order list:0
1 slip sc_mtx -- last acquired @ order list:0
0 unp -- last acquired @ order list:0
6 so_snd -- (already displayed)
Spin locks:
Locks which were never acquired:
SCSI CD Changer List
MD config lock
IPFW dynamic rules
arp_inq
pfs_vncache
ppp_softc_list_mtx
tunmtx
gif_mtx
msq
nfs4dev state
nfs4dev waitq
nfs4dev newq
ntfs nthash
NFS xid lock
NFS reqq lock
ACPI global lock
CAM SIMQ lock
hptlock
LED sx
LED mtx
audit_pipe_mtx
nfslock
audit_trigger_mtx
ktrace_sx
bpin lock
UUID generator mutex lock
ACPI PCI power methods
ACPI lid
ACPI HPET support
ACPI embedded controller
ACPI thermal zone
ACPI Smart Battery
ACPI cmbat
ACPI power resources
ACPI generic battery
ACPI AC adapter
MSDOSFS fileno
pmc shared lock
kqueue order
firmware table
securelevel mutex lock
encapmtx
fifo mutex
phys_pager list
dev_pager list
swapdev
swap_pager list
vm map sleep mutex
db> show lockedvnods
Locked vnodes
0xffffff001106a9b0: tag ufs, type VREG
usecount 4, writecount 0, refcount 6 mountedhere 0
flags (VV_TEXT)
v_object 0xffffff0011318680 ref 2 pages 35
lock type ufs: SHARED (count 1)
ino 2239451, on dev ufs/usr64
0xffffff001106a3e0: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xffffff0001820000 (pid 42488)
ino 2262052, on dev ufs/usr64
0xffffff0001eabba0: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
lock type ufs: EXCL (count 1) by thread 0xffffff001106f680 (pid 42307) with
1 pending
ino 2262734, on dev ufs/usr64
db> show uma
Zone Size Used Free Requests
pfosfp 40 407 97 407
pfospfen 112 696 30 696
pfiaddrpl 120 0 0 0
pfstatescrub 40 0 0 0
pffrcent 24 0 0 0
pffrcache 80 0 0 0
pffrag 80 0 0 0
pffrent 32 0 0 0
pfrkentry2 216 0 0 0
pfrkentry 216 2322 72 2330
pfrktable 1296 12 27 1880
pfpooladdrpl 88 12 72 12
pfaltqpl 240 0 0 0
pfstatepl 392 460 10 466
pfrulepl 912 69 15 69
pfsrctrpl 152 0 0 0
FFS2 dinode 256 1209 66 1232
FFS1 dinode 128 0 0 0
FFS inode 192 1209 51 1232
Mountpoints 808 6 9 6
SWAPMETA 288 0 0 0
IPFW dynamic rule 120 0 0 0
rtentry 240 73 55 139
ripcb 288 1 38 2
sackhole 32 0 0 0
tcpreass 40 0 0 0
hostcache 136 2 54 2
syncache 120 0 93 86
tcptw 88 19 65 19
tcpcb 728 38 27 115
inpcb 288 57 21 115
udpcb 288 23 29 743
ipq 56 0 0 0
unpcb 248 23 22 36
socket 696 99 26 899
KNOTE 120 26 67 21403
itimer 360 0 0 0
ksiginfo 112 84 972 84
pipe 744 6 34 551
NFSNODE 664 0 0 0
NFSMOUNT 568 0 0 0
DIRHASH 1024 56 12 56
NAMEI 1024 2 34 18355
L VFS Cache 327 0 0 0
S VFS Cache 104 1400 112 2069
VNODEPOLL 128 0 0 0
VNODE 496 1240 16 1264
ata_composite 352 0 0 0
ata_request 312 2 94 4738
g_bio 216 10 332 18896
ACL UMA zone 388 0 0 0
mbuf_ext_refcnt 4 0 336 60
mbuf_jumbo_16k 16384 0 0 0
mbuf_jumbo_9k 9216 0 0 0
mbuf_jumbo_pagesize 4096 0 14 36
mbuf_cluster 2048 1152 8 1152
mbuf 256 1 392 9470
mbuf_packet 256 793 359 12023
audit_record 984 0 0 0
VMSPACE 416 45 18 1009
SLEEPQUEUE 64 145 135 145
UPCALL 88 0 0 0
THREAD 824 129 15 129
PROC 1128 96 36 1060
umtx pi 96 0 0 0
TURNSTILE 152 145 23 145
Files 120 239 102 8378
4096 4096 382 37 4229
2048 2048 65 85 2246
1024 1024 62 150 3212
512 512 625 26 6508
256 256 790 80 6895
128 128 9011 153 16531
64 64 1988 196 51814
32 32 2139 285 3341
16 16 1685 331 33999
mt_zone 1024 217 7 217
DP fakepg 120 0 0 0
MAP ENTRY 112 5149 1748 42364
KMAP ENTRY 112 36 96 1495
MAP 248 7 23 7
VM OBJECT 208 2182 122 17689
128 Bucket 1048 43 2 43
64 Bucket 536 42 0 42
32 Bucket 280 59 11 59
16 Bucket 152 80 20 80
UMA Hash 256 6 9 7
UMA RCntSlabs 128 594 15 594
UMA Slabs 128 674 22 1224
UMA Zones 216 89 13 89
UMA Kegs 216 89 13 89
db> show malloc
Type InUse MemUse Requests
msdosfs_mount 0 0K 0
msdosfs_fat 0 0K 0
msdosfs_fileno 0 0K 0
msdosfs_node 0 0K 0
DEVFS 27 1K 28
DEVFS_RULE 36 17K 36
DEVFS2 108 2K 108
DEVFS3 244 61K 245
DEVFS1 111 56K 111
USB 91 20K 106
USBdev 21 9K 21
atkbddev 2 1K 2
USBHC 0 0K 0
UART 0 0K 0
nexusdev 4 1K 4
msi 2 1K 2
twe_commands 0 0K 0
twa_commands 0 0K 0
firewire 0 0K 0
legacydrv 0 0K 0
io_apic 1 2K 1
entropy 1024 64K 1024
ppbusdev 3 1K 3
fw_xfer 0 0K 0
memdesc 1 4K 1
madt_table 0 0K 0
scsi_cd 0 0K 0
CAM ccb queue 0 0K 0
vm_pgdata 1 128K 1
acpipwr 0 0K 0
aaccam 0 0K 0
acpi_perf 0 0K 0
UMAHash 1 1K 1
ufs_mount 12 30K 12
ufs_quota 0 0K 0
ufs_dirhash 51 10K 51
pagedep 9 65K 24
inodedep 63 528K 89
newblk 1 1K 5144
bmsafemap 11 2K 18
allocdirect 41 11K 222
indirdep 6 1K 10
allocindir 4634 580K 4921
freefrag 1 1K 373
freeblks 12 3K 17
freefile 12 1K 22
diradd 32 2K 59
mkdir 4 1K 12
dirrem 12 1K 42
newdirblk 0 0K 0
savedino 0 0K 11
audit_trigger 0 0K 0
audit_pipe 0 0K 0
audit_pipeent 0 0K 0
audit_pipe_preselect 0 0K 0
audit_evclass 150 5K 187
audit_bsm 0 0K 0
audit_cred 0 0K 0
audit_data 0 0K 0
audit_path 0 0K 0
audit_text 0 0K 0
rpcclnt 0 0K 0
agp 0 0K 0
nfss_srvsock 0 0K 0
nfss_srvdesc 0 0K 0
nfss_daemon 1 16K 1
nfsclient_lock 0 0K 0
nfsclient_nlminfo 0 0K 0
nfsclient_req 0 0K 0
nfsclient_bigfh 0 0K 0
nfsclient_diroff 0 0K 0
nfsclient_hash 0 0K 0
nfsclient_directio 0 0K 0
nfsclient_srvsock 0 0K 0
idmap 0 0K 0
nfs4_dev 0 0K 0
syncache 1 100K 1
tcplog 0 0K 0
aacbuf 0 0K 0
hostcache 1 36K 1
IpFw/IpAcct 697 78K 697
ipfw_tbl 31 8K 31
encap_export_host 0 0K 0
dummynet 54 108K 54
in_multi 4 1K 4
ip_moptions 0 0K 0
in_msource 0 0K 0
igmp 0 0K 0
routetbl 341 97K 4442
pci_link 16 2K 16
tun 0 0K 0
SCSI SES 0 0K 0
sl 0 0K 0
ppp 0 0K 0
lo 1 1K 1
gif 0 0K 0
fw_com 0 0K 0
faith 0 0K 0
arpcom 3 1K 3
clone 6 24K 6
ifnet 7 13K 7
ifaddr 52 17K 52
ether_multi 17 1K 20
BPF 14 74K 15
subr_export_host 0 0K 0
SCSI sa 0 0K 0
mount 84 4K 138
vnodemarker 0 0K 29
vnodes 1 1K 1
vfs_hash 1 512K 1
export_host 0 0K 0
cl_savebuf 0 0K 72
vfscache 1 1024K 1
biobuf 17 34K 87
soname 8 1K 1326
pcb 24 9K 39
CAM XPT 10 2K 25
mbuf_tag 30 3K 18347
accf 0 0K 0
ptys 1 1K 1
ttys 513 85K 731
shm 6 26K 8
sem 4 8K 4
msg 4 30K 4
ioctlops 0 0K 3983
select 0 0K 0
iov 0 0K 3038
acpicmbat 0 0K 0
acpidev 63 4K 63
ciss_data 0 0K 0
CAM periph 2 1K 9
Unitno 7 1K 9
taskqueue 15 2K 15
stack 0 0K 0
CAM SIM 1 1K 1
scsi_da 0 0K 0
sbuf 0 0K 450
scsi_ch 0 0K 0
rman 161 20K 611
acpisem 12 2K 12
mfibuf 0 0K 0
kobj 227 908K 311
md_disk 0 0K 0
eventhandler 68 6K 68
devstat 16 33K 16
bus 961 75K 4819
bus-sc 66 83K 2400
md_sectors 0 0K 0
SWAP 0 0K 0
p1003.1b 1 1K 1
umtx 144 18K 144
sysctl 0 0K 192
sysctloid 2817 138K 2817
sysctltmp 0 0K 175
LED 0 0K 0
plimit 7 2K 412
uidinfo 9 3K 18
cred 34 9K 2477
pgrp 22 3K 26
session 22 3K 24
proc 2 16K 2
subproc 230 455K 1194
kbdmux 6 9K 6
mtx_pool 1 8K 1
ipsbuf 0 0K 0
module 353 45K 353
iirbuf 0 0K 0
cache 0 0K 0
devbuf 314 1777K 321
temp 34 8K 4594
ip6opt 0 0K 0
ip6ndp 0 0K 0
CAM queue 3 1K 3
free 0 0K 0
lockf 3 1K 10
linker 96 132K 125
CAM dev queue 1 1K 1
KTRACE 100 13K 100
ast_driver 0 0K 0
afd_driver 0 0K 0
prison 0 0K 0
ithread 76 13K 76
acd_driver 0 0K 0
zombie 0 0K 0
proc-args 28 3K 401
kqueue 34 39K 54
kenv 71 11K 73
filedesc 102 64K 1074
filedesc_to_leader 0 0K 0
sigio 1 1K 1
ar_driver 0 0K 10
ata_pci 0 0K 0
cdev 24 6K 24
sbp 0 0K 0
if_fwip 0 0K 0
ata_dma 2 1K 2
if_fwe 0 0K 0
ad_driver 2 1K 2
ata_generic 2 2K 2
acpitask 0 0K 1
isofs_mount 0 0K 0
isofs_node 0 0K 0
isadev 5 1K 5
acpica 1865 180K 58648
GEOM 144 25K 1163
fwmem 0 0K 0
pfs_vncache 0 0K 0
pfs_nodes 20 5K 20
ntfs_mount 0 0K 0
ntfs_ntnode 0 0K 0
ntfs_fnode 0 0K 0
ntfs_dir 0 0K 0
ntfs_vattr 0 0K 0
ntfsd_resdata 0 0K 0
ntfs_vrun 0 0K 0
ntfs_decomp 0 0K 0
ntfs_nthash 1 512K 1
db> cont
Starting background file system checks in 60 seconds.
--uAKRQypu60I7Lcqm--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080322102933.GA76747>