Date: Tue, 5 Jun 2007 14:22:38 -0700 From: "Matt Reimer" <mattjreimer@gmail.com> To: freebsd-current@freebsd.org Subject: arcmsr crash Message-ID: <f383264b0706051422s6579746ap53a9206c36491dae@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Once a week or so we're seeing a panic with a -current kernel built
just before the gcc 4.2 import (maybe three weeks ago). The box has a
Supermicro X7DBE/X7DBE+ motherboard with two Xeon 5160s, 16G RAM, and
an Areca 1220 controller with eight 500G disks connected.
Does this indicate that the arcmsr driver is at fault:
Tracing command irq16: arcmsr0 pid 26 tid 100018 td 0xffffff040fc5b000
cpustop_handler() at cpustop_handler+0x35
ipi_nmi_handler() at ipi_nmi_handler+0x2e
trap() at trap+0x365
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff8041ab11, rsp = 0xffffffffab59eff0, rbp
= 0xffffffffac0a37d0 ---
siocnclose() at siocnclose+0x21
sio_cnputc() at sio_cnputc+0x89
cnputc() at cnputc+0x6a
putchar() at putchar+0x5f
kvprintf() at kvprintf+0xd45
printf() at printf+0xe1
panic() at panic+0x145
xpt_done() at xpt_done+0x14a
arcmsr_interrupt() at arcmsr_interrupt+0x2df
ithread_loop() at ithread_loop+0x108
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0a3d30, rbp = 0 ---
Following is a bunch of ddb output and dmesg. The initial KDB
backtrace seems to have a panic message intermingled somehow. It looks
like the arcmsr driver caused a panic, which in turn fails because of
a lock violation.
What else can I do to track down the problem?
TIA,
Matt
KDB: stack backtrace:
pandb_trace_self_wrapper() at idb_trace_self_wrapper+0x3a
c: vfs_badlock() at Bvfs_badlock+0x95
adassert_vop_locked() at assert_vop_locked+0x77
taiVOP_READ_APV() at lVOP_READ_APV+0x8a
q Nvn_read() at Evn_read+0x244
XTdofileread() at (dofileread+0x94
0xkern_readv() at fkern_readv+0x60
ffread() at fread+0x4a
ff0syscall() at 0syscall+0x20b
00Xfast_syscall() at aXfast_syscall+0xab
5--- syscall (3d, FreeBSD ELF64, read), rip = 0x800c7aa2c, rsp =
0x7fffffffc758, rbp = 0 ---
6VOP_READ: 0xffffff0336319798 is not locked but should be
KDB: enter: lock violation
2[thread pid 66384 tid 100138 ]
Stopped at kdb_enter+0x2f: nop
db> trace
Tracing pid 66384 tid 100138 td 0xffffff0345b6f2c0
kdb_enter() at kdb_enter+0x2f
assert_vop_locked() at assert_vop_locked+0x77
VOP_READ_APV() at VOP_READ_APV+0x8a
vn_read() at vn_read+0x244
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800c7aa2c, rsp =
0x7fffffffc758, rbp = 0 ---
db> ps
pid ppid pgrp uid state wmesg wchan cmd
66384 66292 687 1003 R CPU 1 perl5.8.8
66383 687 687 80 S lockf 0xffffff030c65d280 httpd
66331 687 687 80 S select 0xffffffff8069a830 httpd
66292 687 687 80 S kqread 0xffffff02afe39600 httpd
66290 687 687 80 S select 0xffffffff8069a830 httpd
66288 687 687 80 S select 0xffffffff8069a830 httpd
66279 687 687 80 S select 0xffffffff8069a830 httpd
66277 687 687 80 S select 0xffffffff8069a830 httpd
66275 687 687 80 S select 0xffffffff8069a830 httpd
66274 687 687 80 S kqread 0xffffff02c8c1b700 httpd
66267 687 687 80 S select 0xffffffff8069a830 httpd
66210 687 687 80 S lockf 0xffffff02d96ba580 httpd
66206 687 687 80 S select 0xffffffff8069a830 httpd
66203 687 687 80 S select 0xffffffff8069a830 httpd
66031 687 687 80 S select 0xffffffff8069a830 httpd
53467 53465 53465 1003 S select 0xffffffff8069a830 sftp-server
53465 53464 53465 1003 Ss pause 0xffffff024debbac8 csh
53464 53462 53462 1003 S select 0xffffffff8069a830 sshd
53462 699 53462 0 Ss sbwait 0xffffff03b3932984 sshd
3194 77246 3194 0 S+ select 0xffffffff8069a830 ssh
69351 77246 69351 0 T+ emacs
38702 77246 38702 0 T+ ping
37157 37146 37157 0 S+ ttyin 0xffffff0000a5a010 bash
37146 37142 37146 1005 S+ wait 0xffffff02db456000 su
37142 37141 37142 1005 Ss+ wait 0xffffff030b361380 bash
37141 37123 37123 1005 S select 0xffffffff8069a830 sshd
37123 699 37123 0 Ss sbwait 0xffffff034d8a0c3c sshd
77253 77246 77253 0 T+ emacs
77246 77235 77246 0 S+ wait 0xffffff034cb40000 bash
77235 77054 77235 1001 S+ wait 0xffffff004ba44380 su
77054 77001 77054 1001 S+ wait 0xffffff02cff28700 bash
77001 77000 77001 1001 Ss+ wait 0xffffff007b41aa80 sh
77000 76971 76971 1001 S select 0xffffffff8069a830 sshd
76971 699 76971 0 Ss sbwait 0xffffff02e3cc4c3c sshd
831 830 831 1005 Ss+ ttyin 0xffffff00009f7010 bash
830 827 827 1005 S select 0xffffffff8069a830 sshd
827 699 827 0 Ss sbwait 0xffffff0349f216cc sshd
765 1 765 0 Ss+ ttyin 0xffffff0000a84810 getty
764 1 764 0 Ss+ ttyin 0xffffff0000a82410 getty
763 1 763 0 Ss+ ttyin 0xffffff0000b00410 getty
762 1 762 0 Ss+ ttyin 0xffffff0000a84410 getty
761 1 761 0 Ss+ ttyin 0xffffff040fa5c410 getty
760 1 760 0 Ss+ ttyin 0xffffff0000abfc10 getty
759 1 759 0 Ss+ ttyin 0xffffff0000abf810 getty
758 1 758 0 Ss+ ttyin 0xffffff0000abe810 getty
757 1 757 0 Ss+ ttyin 0xffffff0000abe410 getty
742 1 742 0 Ss select 0xffffffff8069a830 inetd
715 1 715 0 Ss nanslp 0xffffffff80609bc8 cron
709 1 709 25 Ss pause 0xffffff008bb63ac8 sendmail
705 1 705 0 Ss select 0xffffffff8069a830 sendmail
699 1 699 0 Ss select 0xffffffff8069a830 sshd
687 1 687 0 Ss nanslp 0xffffffff80609bc8 httpd
675 644 643 88 S+ (threaded) mysqld
100283 S ucond 0xffffff003fe30800 mysqld
100282 S ucond 0xffffff0000afc580 mysqld
100281 S ucond 0xffffff03059b9900 mysqld
100264 S ucond 0xffffff02c9816680 mysqld
100103 S sigwait 0xffffffffae6b69f8 mysqld
100100 S ucond 0xffffff01151a9900 mysqld
100102 S select 0xffffffff8069a830 mysqld
100101 S select 0xffffffff8069a830 mysqld
100099 S ucond 0xffffff01151a9a00 mysqld
100098 S ucond 0xffffff0000b15c00 mysqld
100097 S ucond 0xffffff0000b15d00 mysqld
100096 S ucond 0xffffff0000d1d100 mysqld
100091 S select 0xffffffff8069a830 initial thread
644 1 643 88 S+ wait 0xffffff010b0b1000 sh
536 1 536 0 Ss select 0xffffffff8069a830 syslogd
469 1 469 0 Ss select 0xffffffff8069a830 devd
110 1 110 0 Ss pause 0xffffff006635c748 adjkerntz
38 0 0 0 SL sdflush 0xffffffff806a9308 [softdepflush]
37 0 0 0 SL syncer 0xffffffff80609840 [syncer]
36 0 0 0 SL vlruwt 0xffffff040fbfe000 [vnlru]
35 0 0 0 SL psleep 0xffffffff8069b09c [bufdaemon]
34 0 0 0 SL pgzero 0xffffffff806ba2a0 [pagezero]
33 0 0 0 SL pollid 0xffffffff8060898c [idlepoll]
32 0 0 0 SL psleep 0xffffffff806aa14c [vmdaemon]
31 0 0 0 SL psleep 0xffffffff806aa104 [pagedaemon]
30 0 0 0 WL [swi0: sio]
29 0 0 0 WL [irq1: atkbd0]
28 0 0 0 WL [irq15: ata1]
27 0 0 0 WL [irq14: ata0]
26 0 0 0 RL CPU 2 [irq16: arcmsr0]
25 0 0 0 WL [irq19: em1]
24 0 0 0 RL [irq18: em0]
23 0 0 0 WL [irq9: acpi0]
22 0 0 0 SL - 0xffffff00009d2580 [acpi_task_2]
9 0 0 0 SL - 0xffffff00009d2580 [acpi_task_1]
8 0 0 0 SL - 0xffffff00009d2580 [acpi_task_0]
7 0 0 0 SL - 0xffffff00009d9a80 [kqueue taskq]
21 0 0 0 WL [swi6: task queue]
20 0 0 0 WL [swi6: Giant taskq]
6 0 0 0 SL - 0xffffff00009d9d00 [thread taskq]
19 0 0 0 WL [swi5: +]
18 0 0 0 RL [swi2: cambio]
5 0 0 0 SL ccb_scan 0xffffffff805ef440 [xpt_thrd]
17 0 0 0 SL - 0xffffffff80609848 [yarrow]
4 0 0 0 SL - 0xffffffff80605b18 [g_down]
3 0 0 0 SL - 0xffffffff80605b10 [g_up]
2 0 0 0 SL - 0xffffffff80605b00 [g_event]
16 0 0 0 WL [swi1: net]
15 0 0 0 WL [swi3: vm]
14 0 0 0 RL [swi4: clock sio]
13 0 0 0 RL CPU 0 [idle: cpu0]
12 0 0 0 RL [idle: cpu1]
11 0 0 0 RL [idle: cpu2]
10 0 0 0 RL CPU 3 [idle: cpu3]
1 0 1 0 SLs wait 0xffffff040fc98000 [init]
0 0 0 0 WLs [swapper]
66360 66331 687 1003 Z perl5.8.8
db> show pcpu
cpuid = 1
curthread = 0xffffff0345b6f2c0: pid 66384 "perl5.8.8"
curpcb = 0xffffffffae76fd40
fpcurthread = 0xffffff0345b6f2c0: pid 66384 "perl5.8.8"
idlethread = 0xffffff040fc93b00: pid 12 "idle: cpu1"
spin locks held:
db> show allpcpu
Current CPU: 1
cpuid = 0
curthread = 0xffffff040fc56000: pid 13 "idle: cpu0"
curpcb = 0xffffffffac06fd40
fpcurthread = none
idlethread = 0xffffff040fc56000: pid 13 "idle: cpu0"
spin locks held:
cpuid = 1
curthread = 0xffffff0345b6f2c0: pid 66384 "perl5.8.8"
curpcb = 0xffffffffae76fd40
fpcurthread = 0xffffff0345b6f2c0: pid 66384 "perl5.8.8"
idlethread = 0xffffff040fc93b00: pid 12 "idle: cpu1"
spin locks held:
cpuid = 2
curthread = 0xffffff040fc5b000: pid 26 "irq16: arcmsr0"
curpcb = 0xffffffffac0a3d40
fpcurthread = none
idlethread = 0xffffff040fc93840: pid 11 "idle: cpu2"
spin locks held:
cpuid = 3
curthread = 0xffffff040fc93580: pid 10 "idle: cpu3"
curpcb = 0xffffffffac060d40
fpcurthread = none
idlethread = 0xffffff040fc93580: pid 10 "idle: cpu3"
spin locks held:
db> show locks
db> show alllocks
Process 53462 (sshd) thread 0xffffff008ba852c0 (100238)
exclusive sx so_rcv_sx r = 0 (0xffffff03b3932928) locked @
/usr/src/sys/kern/uipc_sockbuf.c:141
Process 37123 (sshd) thread 0xffffff03084d5580 (100259)
exclusive sx so_rcv_sx r = 0 (0xffffff034d8a0be0) locked @
/usr/src/sys/kern/uipc_sockbuf.c:141
Process 76971 (sshd) thread 0xffffff034db08b00 (100123)
exclusive sx so_rcv_sx r = 0 (0xffffff02e3cc4be0) locked @
/usr/src/sys/kern/uipc_sockbuf.c:141
Process 827 (sshd) thread 0xffffff034da09840 (100119)
exclusive sx so_rcv_sx r = 0 (0xffffff0349f21670) locked @
/usr/src/sys/kern/uipc_sockbuf.c:141
Process 18 (swi2: cambio) thread 0xffffff040fc5b840 (100015)
exclusive sleep mutex Giant r = 0 (0xffffffff80608820) locked @
/usr/src/sys/cam/cam_xpt.c:7107
db> show lockedvnods
Locked vnodes
db> alltrace
Tracing command perl5.8.8 pid 66384 tid 100138 td 0xffffff0345b6f2c0
kdb_enter() at kdb_enter+0x2f
assert_vop_locked() at assert_vop_locked+0x77
VOP_READ_APV() at VOP_READ_APV+0x8a
vn_read() at vn_read+0x244
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800c7aa2c, rsp =
0x7fffffffc758, rbp = 0 ---
Tracing command httpd pid 66383 tid 100199 td 0xffffff030581f000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
lf_advlock() at lf_advlock+0x29c
VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0x9b
flock() at flock+0x125
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (131, FreeBSD ELF64, flock), rip = 0x800fb6e4c, rsp =
0x7fffffffeb38, rbp = 0x804cb1028 ---
Tracing command httpd pid 66331 tid 100171 td 0xffffff02e2c00000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffff82c8, rbp = 0x801219400 ---
Tracing command httpd pid 66292 tid 100213 td 0xffffff02e2c002c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_sleep() at _sleep+0x34d
kern_kevent() at kern_kevent+0x2e9
kevent() at kevent+0x90
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x8010083cc, rsp =
0x7fffffff86e8, rbp = 0x7fffffff875c ---
Tracing command httpd pid 66290 tid 100353 td 0xffffff02dd7fa840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe818, rbp = 0x801219400 ---
Tracing command httpd pid 66288 tid 100056 td 0xffffff00663f8000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe818, rbp = 0x801219400 ---
Tracing command httpd pid 66279 tid 100125 td 0xffffff034db08580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe818, rbp = 0x801219400 ---
Tracing command httpd pid 66277 tid 100321 td 0xffffff037e09e2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe818, rbp = 0x801219400 ---
Tracing command httpd pid 66275 tid 100287 td 0xffffff02ff4982c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe818, rbp = 0x801219400 ---
Tracing command httpd pid 66274 tid 100166 td 0xffffff032cd14000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_kevent() at kern_kevent+0x2e9
kevent() at kevent+0x90
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x8010083cc, rsp =
0x7fffffffeb08, rbp = 0x7fffffffeb74 ---
Tracing command httpd pid 66267 tid 100290 td 0xffffff03b6a46840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe8a8, rbp = 0x801219400 ---
Tracing command httpd pid 66210 tid 100231 td 0xffffff033ff762c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
lf_advlock() at lf_advlock+0x29c
VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0x9b
flock() at flock+0x125
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (131, FreeBSD ELF64, flock), rip = 0x800fb6e4c, rsp =
0x7fffffffeb38, rbp = 0x804cb1028 ---
Tracing command httpd pid 66206 tid 100379 td 0xffffff022511fb00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe818, rbp = 0x801219400 ---
Tracing command httpd pid 66203 tid 100205 td 0xffffff030939db00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe818, rbp = 0x801219400 ---
Tracing command httpd pid 66031 tid 100149 td 0xffffff034600d000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
poll() at poll+0x538
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800fd006c, rsp =
0x7fffffffe8a8, rbp = 0x801219400 ---
Tracing command sftp-server pid 53467 tid 100078 td 0xffffff00663f8b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800d0a9ac, rsp =
0x7fffffffab68, rbp = 0x80151e100 ---
Tracing command csh pid 53465 tid 100359 td 0xffffff030e579580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_sigsuspend() at kern_sigsuspend+0xa0
sigsuspend() at sigsuspend+0x40
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80094a94c, rsp =
0x7fffffffe5f8, rbp = 0x5b7c08 ---
Tracing command sshd pid 53464 tid 100192 td 0xffffff02ff080580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8016e79ac, rsp =
0x7fffffffe008, rbp = 0x7fffffffe078 ---
Tracing command sshd pid 53462 tid 100238 td 0xffffff008ba852c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
soreceive_generic() at soreceive_generic+0xf31
soo_read() at soo_read+0x37
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x8016e7a2c, rsp =
0x7fffffffe008, rbp = 0x4 ---
Tracing command ssh pid 3194 tid 100383 td 0xffffff022511f000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800f349ac, rsp =
0x7fffffffbef8, rbp = 0x7fffffffbff8 ---
Tracing command emacs pid 69351 tid 100122 td 0xffffff034da09000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
thread_suspend_check() at thread_suspend_check+0x12d
userret() at userret+0x12c
syscall() at syscall+0x2c6
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (0, FreeBSD ELF64, nosys), rip = 0x800baba3c, rsp =
0x7fffffffe398, rbp = 0x3 ---
Tracing command ping pid 38702 tid 100241 td 0xffffff008bb04840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
kern_select() at kern_select+0x4fc
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80093f9ac, rsp =
0x7ffffffee6d8, rbp = 0x7ffffffee880 ---
Tracing command bash pid 37157 tid 100345 td 0xffffff032f666840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
ptsread() at ptsread+0x52
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800bf4a2c, rsp =
0x7fffffffe6f8, rbp = 0 ---
Tracing command su pid 37146 tid 100380 td 0xffffff022511f840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009c8d8c, rsp =
0x7fffffffe6e8, rbp = 0x800b6f8a0 ---
Tracing command bash pid 37142 tid 100237 td 0xffffff008ba85580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b7cd8c, rsp =
0x7fffffffe828, rbp = 0x800e315b0 ---
Tracing command sshd pid 37141 tid 100401 td 0xffffff0296e13580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8016e79ac, rsp =
0x7fffffffe008, rbp = 0x7fffffffe078 ---
Tracing command sshd pid 37123 tid 100259 td 0xffffff03084d5580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
soreceive_generic() at soreceive_generic+0xf31
soo_read() at soo_read+0x37
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x8016e7a2c, rsp =
0x7fffffffe008, rbp = 0x4 ---
Tracing command emacs pid 77253 tid 100127 td 0xffffff034db08000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
thread_suspend_check() at thread_suspend_check+0x12d
userret() at userret+0x12c
syscall() at syscall+0x2c6
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (0, FreeBSD ELF64, nosys), rip = 0x800baba3c, rsp =
0x7fffffffe3f8, rbp = 0x3 ---
Tracing command bash pid 77246 tid 100128 td 0xffffff034d8c7b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b7cd8c, rsp =
0x7fffffffe788, rbp = 0x800e32560 ---
Tracing command su pid 77235 tid 100049 td 0xffffff004ba462c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009c8d8c, rsp =
0x7fffffffe6d8, rbp = 0x800b6f8a0 ---
Tracing command bash pid 77054 tid 100275 td 0xffffff02b13e6580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b7cd8c, rsp =
0x7fffffffe7c8, rbp = 0x800e32110 ---
Tracing command sh pid 77001 tid 100071 td 0xffffff007b99c2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800931d8c, rsp =
0x7fffffffe948, rbp = 0x12cc9 ---
Tracing command sshd pid 77000 tid 100193 td 0xffffff02ff0802c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8016e79ac, rsp =
0x7fffffffe008, rbp = 0x7fffffffe078 ---
Tracing command sshd pid 76971 tid 100123 td 0xffffff034db08b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
soreceive_generic() at soreceive_generic+0xf31
soo_read() at soo_read+0x37
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x8016e7a2c, rsp =
0x7fffffffe008, rbp = 0x4 ---
Tracing command bash pid 831 tid 100131 td 0xffffff034d8c72c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
ptsread() at ptsread+0x52
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800bf4a2c, rsp =
0x7fffffffe778, rbp = 0 ---
Tracing command sshd pid 830 tid 100129 td 0xffffff034d8c7840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8016e79ac, rsp =
0x7fffffffe008, rbp = 0x7fffffffe078 ---
Tracing command sshd pid 827 tid 100119 td 0xffffff034da09840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
soreceive_generic() at soreceive_generic+0xf31
soo_read() at soo_read+0x37
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x8016e7a2c, rsp =
0x7fffffffe008, rbp = 0x4 ---
Tracing command getty pid 765 tid 100105 td 0xffffff027a490580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 764 tid 100055 td 0xffffff004ba65000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 763 tid 100116 td 0xffffff010b1482c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 762 tid 100114 td 0xffffff010b148840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 761 tid 100112 td 0xffffff027a5ef000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 760 tid 100111 td 0xffffff027a5ef2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 759 tid 100110 td 0xffffff027a5ef580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 758 tid 100108 td 0xffffff027a5efb00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command getty pid 757 tid 100106 td 0xffffff027a4902c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
ttysleep() at ttysleep+0x1b
ttread() at ttread+0x312
giant_read() at giant_read+0x51
devfs_read_f() at devfs_read_f+0x6c
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800829a2c, rsp =
0x7fffffffeda8, rbp = 0x5088c0 ---
Tracing command inetd pid 742 tid 100067 td 0xffffff004ba46b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800a3e9ac, rsp =
0x7fffffffdae8, rbp = 0x800c39080 ---
Tracing command cron pid 715 tid 100115 td 0xffffff010b148580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_sleep() at _sleep+0x34d
kern_nanosleep() at kern_nanosleep+0x130
nanosleep() at nanosleep+0x59
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80081121c, rsp =
0x7fffffffec28, rbp = 0x46657ea8 ---
Tracing command sendmail pid 709 tid 100083 td 0xffffff008bb26840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_sigsuspend() at kern_sigsuspend+0xa0
sigsuspend() at sigsuspend+0x40
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x800d0294c, rsp =
0x7fffffffcef8, rbp = 0 ---
Tracing command sendmail pid 705 tid 100061 td 0xffffff00663b7000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
kern_select() at kern_select+0x4fc
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x800d8f9ac, rsp =
0x7fffffffc298, rbp = 0x3 ---
Tracing command sshd pid 699 tid 100086 td 0xffffff010b027b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8016e79ac, rsp =
0x7fffffffe0e8, rbp = 0x9 ---
Tracing command httpd pid 687 tid 100036 td 0xffffff040fc74840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_sleep() at _sleep+0x34d
kern_nanosleep() at kern_nanosleep+0x130
nanosleep() at nanosleep+0x59
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80100721c, rsp =
0x7fffffffeae8, rbp = 0x801219400 ---
Tracing command mysqld pid 675 tid 100283 td 0xffffff027f70bb00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7ffffeaf2d88, rbp = 0 ---
Tracing command mysqld pid 675 tid 100282 td 0xffffff040fbfd2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7ffffeb33d88, rbp = 0 ---
Tracing command mysqld pid 675 tid 100281 td 0xffffff033ff76000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7ffffeb74d88, rbp = 0 ---
Tracing command mysqld pid 675 tid 100264 td 0xffffff02ffe94580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7ffffebb5d88, rbp = 0 ---
Tracing command mysqld pid 675 tid 100103 td 0xffffff027a490b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_sigtimedwait() at kern_sigtimedwait+0x1fc
sigwait() at sigwait+0x74
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (429, FreeBSD ELF64, sigwait), rip = 0x8012a312c, rsp =
0x7ffffebf6f28, rbp = 0x801501e10 ---
Tracing command mysqld pid 675 tid 100100 td 0xffffff027ab31000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7ffffedf7e98, rbp = 0 ---
Tracing command mysqld pid 675 tid 100102 td 0xffffff040fbfd580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
kern_select() at kern_select+0x4fc
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013309ac, rsp =
0x7ffffeff8f08, rbp = 0 ---
Tracing command mysqld pid 675 tid 100101 td 0xffffff040fbfd840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x2b
_cv_timedwait_sig() at _cv_timedwait_sig+0x166
kern_select() at kern_select+0x4fc
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013309ac, rsp =
0x7fffff1f9f38, rbp = 0 ---
Tracing command mysqld pid 675 tid 100099 td 0xffffff027ab312c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7fffff5fbbf8, rbp = 0 ---
Tracing command mysqld pid 675 tid 100098 td 0xffffff040f9d9000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7fffff7fcbf8, rbp = 0 ---
Tracing command mysqld pid 675 tid 100097 td 0xffffff040f9d92c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7fffff9fdbf8, rbp = 0 ---
Tracing command mysqld pid 675 tid 100096 td 0xffffff040f9d9580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
do_cv_wait() at do_cv_wait+0x5c7
__umtx_op_cv_wait() at __umtx_op_cv_wait+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8012a2e0c, rsp =
0x7fffffbfebf8, rbp = 0 ---
Tracing command mysqld pid 675 tid 100091 td 0xffffff010afe6b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x8013309ac, rsp =
0x7fffffffe518, rbp = 0xf ---
Tracing command sh pid 644 tid 100092 td 0xffffff010afe6840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800931d8c, rsp =
0x7fffffffe228, rbp = 0x283 ---
Tracing command syslogd pid 536 tid 100090 td 0xffffff010b027000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x80082c9ac, rsp =
0x7fffffffd988, rbp = 0x800a1a0f0 ---
Tracing command devd pid 469 tid 100060 td 0xffffff00663b72c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_cv_wait_sig() at _cv_wait_sig+0x158
kern_select() at kern_select+0x80f
select() at select+0x3e
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (93, FreeBSD ELF64, select), rip = 0x437ebc, rsp =
0x7fffffffe938, rbp = 0x7fffffffe960 ---
Tracing command adjkerntz pid 110 tid 100066 td 0xffffff00663d6000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_sigsuspend() at kern_sigsuspend+0xa0
sigsuspend() at sigsuspend+0x40
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (341, FreeBSD ELF64, sigsuspend), rip = 0x80068a94c, rsp =
0x7fffffffed18, rbp = 0x6270 ---
Tracing command softdepflush pid 38 tid 100042 td 0xffffff040f97b840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
softdep_flush() at softdep_flush+0x330
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac1fad30, rbp = 0 ---
Tracing command syncer pid 37 tid 100043 td 0xffffff040f97b580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
sched_sync() at sched_sync+0x557
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac1f5d30, rbp = 0 ---
Tracing command vnlru pid 36 tid 100044 td 0xffffff040f97b2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
vnlru_proc() at vnlru_proc+0x6de
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac1f0d30, rbp = 0 ---
Tracing command bufdaemon pid 35 tid 100026 td 0xffffff040fc7db00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
buf_daemon() at buf_daemon+0x136
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0d0d30, rbp = 0 ---
Tracing command pagezero pid 34 tid 100027 td 0xffffff040fc7d840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
vm_pagezero() at vm_pagezero+0x74
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0cbd30, rbp = 0 ---
Tracing command idlepoll pid 33 tid 100028 td 0xffffff040fc7d580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
poll_idle() at poll_idle+0x93
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0c6d30, rbp = 0 ---
Tracing command vmdaemon pid 32 tid 100029 td 0xffffff040fc7d2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
vm_daemon() at vm_daemon+0x48
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0c1d30, rbp = 0 ---
Tracing command pagedaemon pid 31 tid 100030 td 0xffffff040fa75b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
vm_pageout() at vm_pageout+0x28a
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac19ad30, rbp = 0 ---
Tracing command swi0: sio pid 30 tid 100031 td 0xffffff040fa75840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac195d30, rbp = 0 ---
Tracing command irq1: atkbd0 pid 29 tid 100032 td 0xffffff040fa75580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac190d30, rbp = 0 ---
Tracing command irq15: ata1 pid 28 tid 100033 td 0xffffff040fa752c0
fork_trampoline() at fork_trampoline
Tracing command irq14: ata0 pid 27 tid 100034 td 0xffffff040fa75000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac186d30, rbp = 0 ---
Tracing command irq16: arcmsr0 pid 26 tid 100018 td 0xffffff040fc5b000
cpustop_handler() at cpustop_handler+0x35
ipi_nmi_handler() at ipi_nmi_handler+0x2e
trap() at trap+0x365
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff8041ab11, rsp = 0xffffffffab59eff0, rbp
= 0xffffffffac0a37d0 ---
siocnclose() at siocnclose+0x21
sio_cnputc() at sio_cnputc+0x89
cnputc() at cnputc+0x6a
putchar() at putchar+0x5f
kvprintf() at kvprintf+0xd45
printf() at printf+0xe1
panic() at panic+0x145
xpt_done() at xpt_done+0x14a
arcmsr_interrupt() at arcmsr_interrupt+0x2df
ithread_loop() at ithread_loop+0x108
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0a3d30, rbp = 0 ---
Tracing command irq19: em1 pid 25 tid 100019 td 0xffffff040fc7ab00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac09cd30, rbp = 0 ---
Tracing command irq18: em0 pid 24 tid 100020 td 0xffffff040fc7a840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac097d30, rbp = 0 ---
Tracing command irq9: acpi0 pid 23 tid 100021 td 0xffffff040fc71b00
fork_trampoline() at fork_trampoline
Tracing command acpi_task_2 pid 22 tid 100022 td 0xffffff040fc71840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0e4d30, rbp = 0 ---
Tracing command acpi_task_1 pid 9 tid 100023 td 0xffffff040fc71580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0dfd30, rbp = 0 ---
Tracing command acpi_task_0 pid 8 tid 100024 td 0xffffff040fc712c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0dad30, rbp = 0 ---
Tracing command kqueue taskq pid 7 tid 100025 td 0xffffff040fc71000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0d5d30, rbp = 0 ---
Tracing command swi6: task queue pid 21 tid 100011 td 0xffffff040fc56580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac079d30, rbp = 0 ---
Tracing command swi6: Giant taskq pid 20 tid 100012 td 0xffffff040fc562c0
fork_trampoline() at fork_trampoline
Tracing command thread taskq pid 6 tid 100013 td 0xffffff040fc7d000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
taskqueue_thread_loop() at taskqueue_thread_loop+0x71
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0bcd30, rbp = 0 ---
Tracing command swi5: + pid 19 tid 100014 td 0xffffff040fc5bb00
fork_trampoline() at fork_trampoline
Tracing command swi2: cambio pid 18 tid 100015 td 0xffffff040fc5b840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sched_add() at sched_add+0x6d7
intr_event_schedule_thread() at intr_event_schedule_thread+0xbc
intr_execute_handlers() at intr_execute_handlers+0x182
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff80156c77, rsp = 0xffffffffac0b2ad0, rbp
= 0xffffffffac0b2bd0 ---
camisr_runqueue() at camisr_runqueue+0x257
camisr() at camisr+0xaf
ithread_loop() at ithread_loop+0x108
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0b2d30, rbp = 0 ---
Tracing command xpt_thrd pid 5 tid 100016 td 0xffffff040fc5b580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait() at sleepq_wait+0x40
_sleep() at _sleep+0x21a
xpt_scanner_thread() at xpt_scanner_thread+0x38
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0add30, rbp = 0 ---
Tracing command yarrow pid 17 tid 100017 td 0xffffff040fc5b2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
random_kthread() at random_kthread+0x100
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac0a8d30, rbp = 0 ---
Tracing command g_down pid 4 tid 100005 td 0xffffff040fc93000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
g_io_schedule_down() at g_io_schedule_down+0x213
g_down_procbody() at g_down_procbody+0x7a
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac056d30, rbp = 0 ---
Tracing command g_up pid 3 tid 100006 td 0xffffff040fc7a580
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
g_io_schedule_up() at g_io_schedule_up+0x147
g_up_procbody() at g_up_procbody+0x7a
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac092d30, rbp = 0 ---
Tracing command g_event pid 2 tid 100007 td 0xffffff040fc7a2c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_timedwait() at sleepq_timedwait+0x40
_sleep() at _sleep+0x3b3
g_event_procbody() at g_event_procbody+0xaf
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac08dd30, rbp = 0 ---
Tracing command swi1: net pid 16 tid 100008 td 0xffffff040fc7a000
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac088d30, rbp = 0 ---
Tracing command swi3: vm pid 15 tid 100009 td 0xffffff040fc56b00
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 14 tid 100010 td 0xffffff040fc56840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ithread_loop() at ithread_loop+0x340
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac07ed30, rbp = 0 ---
Tracing command idle: cpu0 pid 13 tid 100000 td 0xffffff040fc56000
cpustop_handler() at cpustop_handler+0x35
ipi_nmi_handler() at ipi_nmi_handler+0x2e
trap() at trap+0x365
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff80284240, rsp = 0xffffffff806c56d0, rbp
= 0xffffffffac06fc20 ---
sched_runnable() at sched_runnable+0x20
cpu_idle() at cpu_idle+0x26
sched_idletd() at sched_idletd+0x2f
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac06fd30, rbp = 0 ---
Tracing command idle: cpu1 pid 12 tid 100001 td 0xffffff040fc93b00
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
ipi_bitmap_handler() at ipi_bitmap_handler+0x91
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x76
--- interrupt, rip = 0xffffffff803da5d6, rsp = 0xffffffffac06abe0, rbp
= 0xffffffffac06abf0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a0
sched_idletd() at sched_idletd+0x34
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac06ad30, rbp = 0 ---
Tracing command idle: cpu2 pid 11 tid 100002 td 0xffffff040fc93840
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sched_add() at sched_add+0x6d7
intr_event_schedule_thread() at intr_event_schedule_thread+0xbc
intr_execute_handlers() at intr_execute_handlers+0x182
Xapic_isr1() at Xapic_isr1+0x7f
--- interrupt, rip = 0xffffffff803da5d6, rsp = 0xffffffffac065be0, rbp
= 0xffffffffac065bf0 ---
acpi_cpu_c1() at acpi_cpu_c1+0x6
acpi_cpu_idle() at acpi_cpu_idle+0x1a0
sched_idletd() at sched_idletd+0x34
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac065d30, rbp = 0 ---
Tracing command idle: cpu3 pid 10 tid 100003 td 0xffffff040fc93580
cpustop_handler() at cpustop_handler+0x35
ipi_nmi_handler() at ipi_nmi_handler+0x2e
trap() at trap+0x365
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff803e878c, rsp = 0xffffffffab5a3ff0, rbp
= 0xffffffffac060c40 ---
cpu_idle() at cpu_idle+0x2c
fork_exit() at fork_exit+0xaa
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffffffac060d30, rbp = 0 ---
Tracing command init pid 1 tid 100004 td 0xffffff040fc932c0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
sleepq_switch() at sleepq_switch+0xcb
sleepq_wait_sig() at sleepq_wait_sig+0x28
_sleep() at _sleep+0x2ee
kern_wait() at kern_wait+0x39e
wait4() at wait4+0x38
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c4dc, rsp =
0x7fffffffe8a8, rbp = 0x60 ---
Tracing command swapper pid 0 tid 0 td 0xffffffff80605fa0
sched_switch() at sched_switch+0xe1
mi_switch() at mi_switch+0x1c7
scheduler() at scheduler+0x2eb
mi_startup() at mi_startup+0xd3
btext() at btext+0x2c
db> call boot(0)
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x3a
mi_switch() at mi_switch+0x2bf
boot() at boot+0x3e
db_fncall() at db_fncall+0xb4
db_command_loop() at db_command_loop+0x319
db_trap() at db_trap+0x63
kdb_trap() at kdb_trap+0xa8
trap() at trap+0x20a
calltrap() at calltrap+0x8
--- trap 0x3, rip = 0xffffffff8028ffcf, rsp = 0xffffffffae76f910, rbp
= 0xffffffffae76f920 ---
kdb_enter() at kdb_enter+0x2f
assert_vop_locked() at assert_vop_locked+0x77
VOP_READ_APV() at VOP_READ_APV+0x8a
vn_read() at vn_read+0x244
dofileread() at dofileread+0x94
kern_readv() at kern_readv+0x60
read() at read+0x4a
syscall() at syscall+0x20b
Xfast_syscall() at Xfast_syscall+0xab
--- syscall (3, FreeBSD ELF64, read), rip = 0x800c7aa2c, rsp =
0x7fffffffc758, rbp = 0 ---
db> call cpu_reset(0)
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 1
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT #6: Tue May 22 14:07:46 PDT 2007
mreimer@gandalf.vpop.net:/usr/obj/usr/src/sys/GANDALF
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz (3000.02-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,<b15>,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 2
usable memory = 17170817024 (16375 MB)
avail memory = 16629559296 (15859 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 6
cpu3 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
acpi0: <PTLTD RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
cpu2: <ACPI CPU> on acpi0
acpi_throttle2: <ACPI CPU Throttling> on cpu2
acpi_throttle2: failed to attach P_CNT
device_attach: acpi_throttle2 attach returned 6
cpu3: <ACPI CPU> on acpi0
acpi_throttle3: <ACPI CPU Throttling> on cpu3
acpi_throttle3: failed to attach P_CNT
device_attach: acpi_throttle3 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.2 on pci3
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci6: <ACPI PCI bus> on pcib6
em0: <Intel(R) PRO/1000 Network Connection Version - 6.5.0> port
0x2000-0x201f mem 0xd8400000-0xd841ffff irq 18 at device 0.0 on pci6
em0: Ethernet address: 00:30:48:33:96:a4
em0: [ITHREAD]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.5.0> port
0x2020-0x203f mem 0xd8420000-0xd843ffff irq 19 at device 0.1 on pci6
em1: Ethernet address: 00:30:48:33:96:a5
em1: [ITHREAD]
pcib7: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <PCI-PCI bridge> at device 0.0 on pci9
pci10: <PCI bus> on pcib10
arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xd8700000-0xd8700fff,0xd8000000-0xd83fffff irq 16 at device 14.0 on pci10
ARECA RAID ADAPTER0: Driver Version 1.20.00.14 2007-2-05
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.42 2007-1-15
arcmsr0: [ITHREAD]
pcib11: <PCI-PCI bridge> at device 0.2 on pci9
pci11: <PCI bus> on pcib11
pci0: <base peripheral> at device 8.0 (no driver attached)
pcib12: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci12: <ACPI PCI bus> on pcib12
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib13: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci13: <ACPI PCI bus> on pcib13
vgapci0: <VGA-compatible display> port 0x3000-0x30ff mem
0xd0000000-0xd7ffffff,0xd8600000-0xd860ffff irq 18 at device 1.0 on
pci13
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on
pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
Waiting 5 seconds for SCSI devices to settle
acd0: DMA limited to UDMA33, controller found non-ATA66 cable
acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ16> at ata0-slave UDMA33
acd0: FAILURE - READ_BIG ILLEGAL REQUEST asc=0x64 ascq=0x00
GEOM_LABEL: Label for provider acd0 is iso9660/FreeBSD_Install.
acd0: FAILURE - READ_BIG ILLEGAL REQUEST asc=0x64 ascq=0x00
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <Areca gandalf-FreeBSD6 R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da0: 19072MB (39060480 512 byte sectors: 255H 63S/T 2431C)
da1 at arcmsr0 bus 0 target 0 lun 1
da1: <Areca gandalf-FreeBSD7 R001> Fixed Direct Access SCSI-5 device
da1: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da1: 19072MB (39060480 512 byte sectors: 255H 63S/T 2431C)
da2 at arcmsr0 bus 0 target 0 lun 2
da2: <Areca gandalf-data R001> Fixed Direct Access SCSI-5 device
da2: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da2: 2761840MB (5656249344 512 byte sectors: 255H 63S/T 352085C)
da3 at arcmsr0 bus 0 target 0 lun 3
da3: <Areca gandalf-swap R001> Fixed Direct Access SCSI-5 device
da3: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da3: 61032MB (124993536 512 byte sectors: 255H 63S/T 7780C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/da1s1a
WARNING: reducing size to maximum of 67108864 blocks per swap unit
ipfw2 (+ipv6) initialized, divert loadable, rule-based forwarding
disabled, default to deny, logging disabled
em0: link state changed to UP
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?f383264b0706051422s6579746ap53a9206c36491dae>