Date: 19 Dec 2005 14:32:31 -0600 From: rand@meridian-enviro.com (Douglas K. Rand) To: freebsd-stable <freebsd-stable@freebsd.org> Subject: Re: indefinite wait buffer: Does this indicate hardware issue? Message-ID: <87d5jsj09s.fsf@delta.meridian-enviro.com> In-Reply-To: <a78074950512160849w70335610xd2cb0c3b61ed92a2@mail.gmail.com> References: <a78074950512160849w70335610xd2cb0c3b61ed92a2@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
** On Sat, 17 Dec 2005 00:49:48 +0800, Xin LI <delphij@gmail.com> said:
Xin> I have a box indicating the following sometimes:
Xin> "swap_pager: indefinite wait buffer: bufobj: 0, blkno: 262169, size: 4096"
We are having a very similar problem that we've been trying to
diagnose off and on for a while. About 20% of the time the system will
emit very similar messages. In each case processes trying to write to
disks get stuck in the "ufs" state.
This is a dual CPU AMD Athlon(tm) MP 1600+ system on a Tyan 2460 mobo
with 512 MB of RAM and an ICP Vortex GDT8546RZ SATA RAID controller.
Here is the result of a C-t on an interactive shell via a serial
console:
load: 0.00 cmd: csh 547 [ufs] 0.02u 0.01s 0% 2516k
With the kernel debugger a few processes will be stuck in the same state:
db> ps
pid proc uid ppid pgrp flag stat wmesg wchan cmd
647 c1a08000 0 645 647 0000110 [SLPQ ufs 0xc1856c08][SLP] cron
646 c1883418 2 644 646 0000110 [SLPQ biord 0xcbcff650][SLP] cron
645 c1a0a830 0 493 493 0000000 [SLPQ ppwait 0xc1a0a830][SLP] cron
644 c1887c48 0 493 493 0000000 [SLPQ ppwait 0xc1887c48][SLP] cron
643 c1a0a418 0 632 642 0004002 [SLPQ wdrain 0xc06a26c4][SLP] bsdtar
632 c1a08a3c 0 631 632 0004002 [SLPQ pause 0xc1a08a70][SLP] tcsh
631 c1834624 1001 616 631 0004102 [SLPQ wait 0xc1834624][SLP] su
616 c188720c 1001 615 616 0004002 [SLPQ pause 0xc1887240][SLP] tcsh
615 c1a0a624 1001 613 613 0000100 [SLPQ select 0xc06a2144][SLP] sshd
613 c1883624 0 471 613 0004100 [SLPQ sbwait 0xc1b3dbc8][SLP] sshd
547 c1671a3c 0 537 547 0004002 [SLPQ ufs 0xc1856c08][SLP] csh
537 c1883a3c 0 1 537 0004102 [SLPQ wait 0xc1883a3c][SLP] login
536 c1887624 0 1 536 0004002 [SLPQ ttyin 0xc16e8810][SLP] getty
535 c1671624 0 1 535 0004002 [SLPQ ttyin 0xc16e8410][SLP] getty
534 c1a0aa3c 0 1 534 0004002 [SLPQ ttyin 0xc16dac10][SLP] getty
533 c1671c48 0 1 533 0004002 [SLPQ ttyin 0xc16e7010][SLP] getty
532 c1a0ac48 0 1 532 0004002 [SLPQ ttyin 0xc16e7410][SLP] getty
531 c1887a3c 0 1 531 0004002 [SLPQ ttyin 0xc16e7810][SLP] getty
530 c1887000 0 1 530 0004002 [SLPQ ttyin 0xc16e7c10][SLP] getty
529 c183420c 0 1 529 0004002 [SLPQ ttyin 0xc16e8010][SLP] getty
513 c1a0a20c 0 1 513 0000000 [SLPQ select 0xc06a2144][SLP] inetd
493 c1a08830 0 1 493 0000000 [SLPQ ufs 0xc1856c08][SLP] cron
481 c1883c48 25 1 481 0000100 [SLPQ pause 0xc1883c7c][SLP] sendmail
477 c167120c 0 1 477 0000100 [SLPQ pause 0xc1671240][SLP] sendmail
471 c1834000 0 1 471 0000100 [SLPQ select 0xc06a2144][SLP] sshd
349 c1887418 0 1 349 0000000 [SLPQ ufs 0xc1856c08][SLP] ypbind
336 c1834418 0 1 336 0000000 [SLPQ select 0xc06a2144][SLP] rpcbind
317 c1887830 0 1 317 0000000 [SLPQ select 0xc06a2144][SLP] syslogd
283 c1671830 0 1 283 0000000 [SLPQ select 0xc06a2144][SLP] devd
228 c1883000 65 1 228 0000100 [SLPQ select 0xc06a2144][SLP] dhclient
208 c188320c 0 1 53 0000002 [SLPQ select 0xc06a2144][SLP] dhclient
52 c1834a3c 0 0 0 0000204 [SLPQ - 0xd5526d08][SLP] schedcpu
51 c1834c48 0 0 0 0000204 [SLPQ - 0xc06aa4cc][SLP] nfsiod 3
50 c161b624 0 0 0 0000204 [SLPQ - 0xc06aa4c8][SLP] nfsiod 2
49 c161b830 0 0 0 0000204 [SLPQ - 0xc06aa4c4][SLP] nfsiod 1
48 c161ba3c 0 0 0 0000204 [SLPQ - 0xc06aa4c0][SLP] nfsiod 0
47 c161bc48 0 0 0 0000204 [SLPQ vlruwt 0xc161bc48][SLP] vnlru
46 c1670000 0 0 0 0000204 [SLPQ getblk 0xcbc36578][SLP] syncer
45 c167020c 0 0 0 0000204 [SLPQ psleep 0xc06a268c][SLP] bufdaemon
44 c1670418 0 0 0 000020c [SLPQ pgzero 0xc06b09c4][SLP] pagezero
9 c1670624 0 0 0 0000204 [SLPQ psleep 0xc06b0514][SLP] vmdaemon
8 c1670830 0 0 0 0000204 [SLPQ psleep 0xc06b04d0][SLP] pagedaemon
43 c1670a3c 0 0 0 0000204 [IWAIT] swi0: sio
7 c1670c48 0 0 0 0000204 [SLPQ - 0xc16dd43c][SLP] fdc0
6 c1671000 0 0 0 0000204 [SLPQ - 0xc166f280][SLP] kqueue taskq
42 c160fc48 0 0 0 0000204 [IWAIT] swi5:+
5 c161a000 0 0 0 0000204 [SLPQ - 0xc15c1280][SLP] thread taskq
41 c161a20c 0 0 0 0000204 [IWAIT] swi6:+
40 c161a418 0 0 0 0000204 [IWAIT] swi6: task queue
39 c161a624 0 0 0 0000204 [IWAIT] swi2: cambio
38 c161a830 0 0 0 0000204 [SLPQ - 0xc0698140][SLP] yarrow
4 c161aa3c 0 0 0 0000204 [SLPQ - 0xc0698b08][SLP] g_down
3 c161ac48 0 0 0 0000204 [SLPQ - 0xc0698b04][SLP] g_up
2 c161b000 0 0 0 0000204 [SLPQ - 0xc0698afc][SLP] g_event
37 c161b20c 0 0 0 0000204 [IWAIT] swi3: vm
36 c161b418 0 0 0 000020c [IWAIT] swi4: clock sio
35 c1600624 0 0 0 0000204 [IWAIT] swi1: net
34 c1600830 0 0 0 0000204 [IWAIT] irq23:
33 c1600a3c 0 0 0 0000204 [IWAIT] irq22:
32 c1600c48 0 0 0 0000204 [IWAIT] irq21:
31 c160f000 0 0 0 0000204 [IWAIT] irq20:
30 c160f20c 0 0 0 0000204 [IWAIT] irq19:
29 c160f418 0 0 0 0000204 [IWAIT] irq18:
28 c160f624 0 0 0 0000204 [IWAIT] irq17: fxp0
27 c160f830 0 0 0 0000204 [IWAIT] irq16: iir0
26 c160fa3c 0 0 0 0000204 [IWAIT] irq15: ata1
25 c15cc20c 0 0 0 0000204 [IWAIT] irq14: ata0
24 c15cc418 0 0 0 0000204 [IWAIT] irq13:
23 c15cc624 0 0 0 0000204 [IWAIT] irq12:
22 c15cc830 0 0 0 0000204 [IWAIT] irq11:
21 c15cca3c 0 0 0 0000204 [IWAIT] irq10:
20 c15ccc48 0 0 0 0000204 [IWAIT] irq9:
19 c1600000 0 0 0 0000204 [IWAIT] irq8:
18 c160020c 0 0 0 0000204 [IWAIT] irq7:
17 c1600418 0 0 0 0000204 [IWAIT] irq6: fdc0
16 c15c7000 0 0 0 0000204 [IWAIT] irq5:
15 c15c720c 0 0 0 0000204 [IWAIT] irq4: sio0
14 c15c7418 0 0 0 0000204 [IWAIT] irq3: sio1
13 c15c7624 0 0 0 0000204 [IWAIT] irq0:
12 c15c7830 0 0 0 0000204 [IWAIT] irq1: atkbd0
11 c15c7a3c 0 0 0 000020c [CPU 0] idle: cpu0
10 c15c7c48 0 0 0 000020c [CPU 1] idle: cpu1
1 c15cc000 0 0 1 0004200 [SLPQ wait 0xc15cc000][SLP] init
0 c0698c00 0 0 0 0000200 [IWAIT] swapper
On another thread someone suggested show lockedvnods and alltrace, but
I'm unsure how to interpert the output:
db> show lockedvnods
Locked vnodes
0xc1856bb0: tag ufs, type VDIR
usecount 56, writecount 0, refcount 59 mountedhere 0
flags (VV_ROOT)
v_object 0xc189e8c4 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc1884c00 (pid 646) with 4 pending
ino 2, on dev da0s1a
0xc19ed220: tag ufs, type VREG
usecount 1, writecount 1, refcount 4 mountedhere 0
flags ()
v_object 0xc18a9ce4 ref 0 pages 5
lock type ufs: EXCL (count 1) by thread 0xc1673d80 (pid 317)
ino 211971, on dev da0s1e
0xc1943770: tag syncer, type VNON
usecount 1, writecount 0, refcount 2 mountedhere 0
flags ()
lock type syncer: EXCL (count 1) by thread 0xc161c480 (pid 46)
0xc1b2e660: tag ufs, type VDIR
usecount 5, writecount 0, refcount 7 mountedhere 0
flags ()
v_object 0xc1b3c108 ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc161c480 (pid 46)
ino 7418881, on dev da0s1h
0xc1d3add0: tag ufs, type VREG
usecount 1, writecount 1, refcount 2257 mountedhere 0
flags ()
v_object 0xc1d4e210 ref 0 pages 9020
lock type ufs: EXCL (count 1) by thread 0xc1885d80 (pid 643)
ino 7420444, on dev da0s1h
db> alltrace
Tracing command cron pid 647 tid 100077 td 0xc1673900
sched_switch(c1673900,0,1) at sched_switch+0x14b
mi_switch(1,0,c1673900,0,0) at mi_switch+0x1ba
sleepq_switch(c1856c08) at sleepq_switch+0x133
sleepq_wait(c1856c08,0,0,c1673900,c1856c08) at sleepq_wait+0xb
msleep(c1856c08,c069a58c,50,c063ff41,0) at msleep+0x345
acquire(d554da48,40,60000,c1673900,0) at acquire+0x7b
lockmgr(c1856c08,3002,c1856c2c,c1673900,d554da6c) at lockmgr+0x3fe
vop_stdlock(d554dab4,c0676a40,d554dab4,d554da7c,c05bca98) at vop_stdlock+0x1e
VOP_LOCK_APV(c0676f80,d554dab4,d554da90,c062243b,d554dab4) at VOP_LOCK_APV+0x43
ffs_lock(d554dab4,1002,c1856bb0,d554dad0,c051e198) at ffs_lock+0x10
VOP_LOCK_APV(c0676a40,d554dab4) at VOP_LOCK_APV+0x43
vn_lock(c1856bb0,1002,c1673900) at vn_lock+0xf4
lookup(d554dba0,11,0,d554dbc8,41) at lookup+0xc3
namei(d554dba0,bfbfe3c0,0,0,c1856bb0) at namei+0x376
kern_lstat(c1673900,bfbfe3c0,0,d554dc74) at kern_lstat+0x47
lstat(c1673900,d554dd04,2,2,216) at lstat+0x1b
syscall(3b,3b,3b,0,0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (190, FreeBSD ELF32, lstat), eip = 0x280ca36b, esp = 0xbfbfe2ec, ebp = 0xbfbfe388 ---
Tracing command cron pid 646 tid 100065 td 0xc1884c00
sched_switch(c1884c00,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884c00,0,c06407f5) at mi_switch+0x1ba
sleepq_switch(cbcff650) at sleepq_switch+0x133
sleepq_wait(cbcff650,0,0,c1884c00,4c) at sleepq_wait+0xb
msleep(cbcff650,c06a2760,4c,c06407f5,0) at msleep+0x345
bwait(cbcff650,4c,c06407f5) at bwait+0x5f
bufwait(cbcff650,1,0,0,0) at bufwait+0x1a
breadn(c1856bb0,0,0,800,0) at breadn+0x1e6
bread(c1856bb0,0,0,800,0) at bread+0x20
ffs_blkatoff(c1856bb0,0,0,0,d69b79e8) at ffs_blkatoff+0x9e
ufs_lookup(d69b7a84) at ufs_lookup+0x2cd
VOP_CACHEDLOOKUP_APV(c0676a40,d69b7a84) at VOP_CACHEDLOOKUP_APV+0x38
vfs_cache_lookup(d69b7b1c,c1856bb0,c1b0b700,d69b7b38,c050b579) at vfs_cache_lookup+0xb2
VOP_LOOKUP_APV(c0676a40,d69b7b1c) at VOP_LOOKUP_APV+0x43
lookup(d69b7ba0,0,0,d69b7bc8,b4d3cb1e) at lookup+0x421
namei(d69b7ba0,bfbfe3c0,0,0,c1856bb0) at namei+0x376
kern_lstat(c1884c00,bfbfe3c0,0,d69b7c74) at kern_lstat+0x47
lstat(c1884c00,d69b7d04,2,1,216) at lstat+0x1b
syscall(3b,3b,3b,2,0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (190, FreeBSD ELF32, lstat), eip = 0x280ca36b, esp = 0xbfbfe2ec, ebp = 0xbfbfe388 ---
Tracing command cron pid 645 tid 100088 td 0xc1885a80
sched_switch(c1885a80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1885a80,0,c1b0b300) at mi_switch+0x1ba
sleepq_switch(c1a0a830) at sleepq_switch+0x133
sleepq_wait(c1a0a830,0,0,c1885a80,c1a08068) at sleepq_wait+0xb
msleep(c1a0a830,c1a08068,5c,c063aab6,0) at msleep+0x345
fork1(c1885a80,80000034,0,d69d2cd4,c1885a80) at fork1+0x123b
vfork(c1885a80,d69d2d04,0,1,246) at vfork+0x1b
syscall(3b,3b,3b,1,8052040) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (66, FreeBSD ELF32, vfork), eip = 0x280ad184, esp = 0xbfbfe850, ebp = 0xbfbfedf8 ---
Tracing command cron pid 644 tid 100076 td 0xc1673a80
sched_switch(c1673a80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1673a80,0,c1b0b700) at mi_switch+0x1ba
sleepq_switch(c1887c48) at sleepq_switch+0x133
sleepq_wait(c1887c48,0,0,c1673a80,c1883480) at sleepq_wait+0xb
msleep(c1887c48,c1883480,5c,c063aab6,0) at msleep+0x345
fork1(c1673a80,80000034,0,d5550cd4,d5550ce0) at fork1+0x123b
vfork(c1673a80,d5550d04,0,0,246) at vfork+0x1b
syscall(3b,3b,3b,0,8052040) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (66, FreeBSD ELF32, vfork), eip = 0x280ad184, esp = 0xbfbfe850, ebp = 0xbfbfedf8 ---
Tracing command bsdtar pid 643 tid 100086 td 0xc1885d80
sched_switch(c1885d80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1885d80,0,4) at mi_switch+0x1ba
sleepq_switch(c06a26c4) at sleepq_switch+0x133
sleepq_wait(c06a26c4,0,0,c1885d80,64000004) at sleepq_wait+0xb
msleep(c06a26c4,c06a26e0,44,c06404c9,0) at msleep+0x345
waitrunningbufspace(c0f57398,cbcf6818,d69d89a0,c05010c3,cbc09a28) at waitrunningbufspace+0x66
bufwrite(cbc09a28) at bufwrite+0x121
bawrite(cbc09a28,c16d7cc0,c1d3ae4c,c1d3ae4c,c1d3ae4c) at bawrite+0x13
cluster_wbuild(c1d3add0,4000,8cc,0,8) at cluster_wbuild+0x714
cluster_write(c1d3add0,cbcf6818,2330000,0,7f) at cluster_write+0x4db
ffs_write(d69d8bf0,0,0,d69d8ba4,4) at ffs_write+0x508
VOP_WRITE_APV(c0676a40,d69d8bf0) at VOP_WRITE_APV+0xce
vn_write(c186f948,d69d8cbc,c1b05a80,0,c1885d80) at vn_write+0x1ea
dofilewrite(c1885d80,3,c186f948,d69d8cbc,ffffffff) at dofilewrite+0x4f
kern_writev(c1885d80,3,d69d8cbc,8067b48,4b8) at kern_writev+0x3b
write(c1885d80,d69d8d04,3,5dd,292) at write+0x45
syscall(3b,3b,3b,0,232fcb8) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (4, FreeBSD ELF32, write), eip = 0x280fa06b, esp = 0xbfbfe4cc, ebp = 0xbfbfe508 ---
Tracing command tcsh pid 632 tid 100082 td 0xc1a09480
sched_switch(c1a09480,0,1) at sched_switch+0x14b
mi_switch(1,0,c1a09480,0,0) at mi_switch+0x1ba
sleepq_switch(c1a08a70,c1a09480,d69f1c88,c04c0d16,c1a08a70) at sleepq_switch+0x133
sleepq_wait_sig(c1a08a70,0,100,c1a09480,c1a08a70) at sleepq_wait_sig+0xc
msleep(c1a08a70,c1a08aa4,168,c063cae1,0) at msleep+0x336
kern_sigsuspend(c1a09480,2,0,0,0) at kern_sigsuspend+0xbf
sigsuspend(c1a09480,d69f1d04,1,2,202) at sigsuspend+0x33
syscall(3b,3b,3b,2901,bfbf0240) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2814e94b, esp = 0xbfbf022c, ebp = 0xbfbf0258 ---
Tracing command su pid 631 tid 100059 td 0xc1672780
sched_switch(c1672780,0,1) at sched_switch+0x14b
mi_switch(1,0,c1672780,0,0) at mi_switch+0x1ba
sleepq_switch(c1834624,c1672780,d552cc38,c04c0d16,c1834624) at sleepq_switch+0x133
sleepq_wait_sig(c1834624,0,100,c1672780,0) at sleepq_wait_sig+0xc
msleep(c1834624,c183468c,15c,c063c9be,0) at msleep+0x336
kern_wait(c1672780,278,d552cc84,2,0) at kern_wait+0x8ec
wait4(c1672780,d552cd04,4,2,206) at wait4+0x2a
syscall(3b,3b,3b,28155f40,bfbfe36a) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x280cd00b, esp = 0xbfbfe22c, ebp = 0xbfbfe248 ---
Tracing command tcsh pid 616 tid 100071 td 0xc1884300
sched_switch(c1884300,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884300,0,0) at mi_switch+0x1ba
sleepq_switch(c1887240,c1884300,d69a5c88,c04c0d16,c1887240) at sleepq_switch+0x133
sleepq_wait_sig(c1887240,0,100,c1884300,c1887240) at sleepq_wait_sig+0xc
msleep(c1887240,c1887274,168,c063cae1,0) at msleep+0x336
kern_sigsuspend(c1884300,2,0,0,0) at kern_sigsuspend+0xbf
sigsuspend(c1884300,d69a5d04,1,6,202) at sigsuspend+0x33
syscall(3b,3b,3b,101,bfbf28d0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2814e94b, esp = 0xbfbf28bc, ebp = 0xbfbf28e8 ---
Tracing command sshd pid 615 tid 100087 td 0xc1885c00
sched_switch(c1885c00,0,1) at sched_switch+0x14b
mi_switch(1,0,0,0,c06a2144) at mi_switch+0x1ba
sleepq_switch(c06a2144,0,d69d5b54,c0491dae,c06a2144) at sleepq_switch+0x133
sleepq_wait_sig(c06a2144,c06a2144,c06a2144,c06a2120,c063ea71) at sleepq_wait_sig+0xc
cv_wait_sig(c06a2144,c06a2120,4,d69d5ba4,d69d5b80) at cv_wait_sig+0x18a
kern_select(c1885c00,b,8079bf0,8079c00,0) at kern_select+0x5c4
select(c1885c00,d69d5d04,5,26,202) at select+0x44
syscall(3b,3b,3b,0,bfbfe564) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x282c390b, esp = 0xbfbfe4ec, ebp = 0xbfbfe538 ---
Tracing command sshd pid 613 tid 100066 td 0xc1884a80
sched_switch(c1884a80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884a80,0,0) at mi_switch+0x1ba
sleepq_switch(c1b3dbc8,c1884a80,d69b4bc0,c04c0d16,c1b3dbc8) at sleepq_switch+0x133
sleepq_wait_sig(c1b3dbc8,0,100,c1884a80,c1b3db20) at sleepq_wait_sig+0xc
msleep(c1b3dbc8,c1b3db94,158,c063fb82,0) at msleep+0x336
sbwait(c1b3db70,1,4,0,0) at sbwait+0x35
soreceive(c1b3db20,0,d69b4cbc,0,0) at soreceive+0x283
soo_read(c187a288,d69b4cbc,c1b06100,0,c1884a80) at soo_read+0x5d
dofileread(c1884a80,5,c187a288,d69b4cbc,ffffffff) at dofileread+0x52
kern_readv(c1884a80,5,d69b4cbc,bfbfe518,4) at kern_readv+0x36
read(c1884a80,d69b4d04,3,1,286) at read+0x45
syscall(3b,3b,3b,4,0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x282c408b, esp = 0xbfbfe4bc, ebp = 0xbfbfe4e8 ---
Tracing command csh pid 547 tid 100054 td 0xc1673000
sched_switch(c1673000,0,1) at sched_switch+0x14b
mi_switch(1,0,c1673000,0,0) at mi_switch+0x1ba
sleepq_switch(c1856c08) at sleepq_switch+0x133
sleepq_wait(c1856c08,0,0,c1673000,c1856c08) at sleepq_wait+0xb
msleep(c1856c08,c069a58c,50,c063ff41,0) at msleep+0x345
acquire(d553ba48,40,60000,c1673000,0) at acquire+0x7b
lockmgr(c1856c08,3002,c1856c2c,c1673000,d553ba6c) at lockmgr+0x3fe
vop_stdlock(d553bab4,c0676a40,d553bab4,d553ba7c,c05bca98) at vop_stdlock+0x1e
VOP_LOCK_APV(c0676f80,d553bab4,d553ba90,c062243b,d553bab4) at VOP_LOCK_APV+0x43
ffs_lock(d553bab4,1002,c1856bb0,d553bad0,c051e198) at ffs_lock+0x10
VOP_LOCK_APV(c0676a40,d553bab4) at VOP_LOCK_APV+0x43
vn_lock(c1856bb0,1002,c1673000) at vn_lock+0xf4
lookup(d553bc00,c1673154,0,d553bc28,d553bb74) at lookup+0xc3
namei(d553bc00,0,1,d553bbcc,c0610875) at namei+0x376
kern_stat(c1673000,8132408,0,d553bc74) at kern_stat+0x35
stat(c1673000,d553bd04,2,1,282) at stat+0x1b
syscall(3b,3b,3b,bfbf6c40,8132408) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x2814f3ab, esp = 0xbfbf66ec, ebp = 0xbfbf6bf8 ---
Tracing command login pid 537 tid 100068 td 0xc1884780
sched_switch(c1884780,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884780,0,0) at mi_switch+0x1ba
sleepq_switch(c1883a3c,c1884780,d69aec38,c04c0d16,c1883a3c) at sleepq_switch+0x133
sleepq_wait_sig(c1883a3c,0,100,c1884780,0) at sleepq_wait_sig+0xc
msleep(c1883a3c,c1883aa4,15c,c063c9be,0) at msleep+0x336
kern_wait(c1884780,223,d69aec84,0,0) at kern_wait+0x8ec
wait4(c1884780,d69aed04,4,1,206) at wait4+0x2a
syscall(3b,3b,3b,0,bfbfefa7) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x280ce00b, esp = 0xbfbfedac, ebp = 0xbfbfedc8 ---
Tracing command getty pid 536 tid 100073 td 0xc1884000
sched_switch(c1884000,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884000,1,0) at mi_switch+0x1ba
sleepq_switch(c16e8810,c1884000,d699f9d4,c04c0d16,c16e8810) at sleepq_switch+0x133
sleepq_wait_sig(c16e8810,0,100,c1884000,c16e8800) at sleepq_wait_sig+0xc
msleep(c16e8810,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16e8800,c16e8810,159,c063f4de,0) at ttysleep+0x21
ttread(c16e8800,d699fcbc,0) at ttread+0x513
ttyread(c16eae00,d699fcbc,0,d699fc18,c04927a4) at ttyread+0x2f
scread(c16eae00,d699fcbc,0,0,d699fc44) at scread+0x22
giant_read(c16eae00,d699fcbc,0,1,c16eae00) at giant_read+0x4c
devfs_read_f(c187ac60,d699fcbc,c15c1080,0,c1884000) at devfs_read_f+0x62
dofileread(c1884000,0,c187ac60,d699fcbc,ffffffff) at dofileread+0x52
kern_readv(c1884000,0,d699fcbc,bfbfee3f,1) at kern_readv+0x36
read(c1884000,d699fd04,3,0,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command getty pid 535 tid 100052 td 0xc1673300
sched_switch(c1673300,0,1) at sched_switch+0x14b
mi_switch(1,0,c1673300,1,0) at mi_switch+0x1ba
sleepq_switch(c16e8410,c1673300,d55419d4,c04c0d16,c16e8410) at sleepq_switch+0x133
sleepq_wait_sig(c16e8410,0,100,c1673300,c16e8400) at sleepq_wait_sig+0xc
msleep(c16e8410,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16e8400,c16e8410,159,c063f4de,0) at ttysleep+0x21
ttread(c16e8400,d5541cbc,0) at ttread+0x513
ttyread(c1674700,d5541cbc,0,d5541c18,c04927a4) at ttyread+0x2f
scread(c1674700,d5541cbc,0,0,d5541c44) at scread+0x22
giant_read(c1674700,d5541cbc,0,1,c1674700) at giant_read+0x4c
devfs_read_f(c187a3a8,d5541cbc,c15c1080,0,c1673300) at devfs_read_f+0x62
dofileread(c1673300,0,c187a3a8,d5541cbc,ffffffff) at dofileread+0x52
kern_readv(c1673300,0,d5541cbc,bfbfee3f,1) at kern_readv+0x36
read(c1673300,d5541d04,3,0,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command getty pid 534 tid 100089 td 0xc1885900
sched_switch(c1885900,0,1) at sched_switch+0x14b
mi_switch(1,0,c1885900,1,0) at mi_switch+0x1ba
sleepq_switch(c16dac10,c1885900,d69cf9d4,c04c0d16,c16dac10) at sleepq_switch+0x133
sleepq_wait_sig(c16dac10,0,100,c1885900,c16dac00) at sleepq_wait_sig+0xc
msleep(c16dac10,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16dac00,c16dac10,159,c063f4de,0) at ttysleep+0x21
ttread(c16dac00,d69cfcbc,0) at ttread+0x513
ttyread(c1674e00,d69cfcbc,0,d69cfc18,c04927a4) at ttyread+0x2f
scread(c1674e00,d69cfcbc,0,0,d69cfc44) at scread+0x22
giant_read(c1674e00,d69cfcbc,0,1,c1674e00) at giant_read+0x4c
devfs_read_f(c186f750,d69cfcbc,c15c1080,0,c1885900) at devfs_read_f+0x62
dofileread(c1885900,0,c186f750,d69cfcbc,ffffffff) at dofileread+0x52
kern_readv(c1885900,0,d69cfcbc,bfbfee3f,1) at kern_readv+0x36
read(c1885900,d69cfd04,3,1,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command getty pid 533 tid 100055 td 0xc1672d80
sched_switch(c1672d80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1672d80,1,0) at mi_switch+0x1ba
sleepq_switch(c16e7010,c1672d80,d55389d4,c04c0d16,c16e7010) at sleepq_switch+0x133
sleepq_wait_sig(c16e7010,0,100,c1672d80,c16e7000) at sleepq_wait_sig+0xc
msleep(c16e7010,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16e7000,c16e7010,159,c063f4de,0) at ttysleep+0x21
ttread(c16e7000,d5538cbc,0) at ttread+0x513
ttyread(c1674d00,d5538cbc,0,d5538c18,c04927a4) at ttyread+0x2f
scread(c1674d00,d5538cbc,0,0,d5538c44) at scread+0x22
giant_read(c1674d00,d5538cbc,0,1,c1674d00) at giant_read+0x4c
devfs_read_f(c186f558,d5538cbc,c15c1080,0,c1672d80) at devfs_read_f+0x62
dofileread(c1672d80,0,c186f558,d5538cbc,ffffffff) at dofileread+0x52
kern_readv(c1672d80,0,d5538cbc,bfbfee3f,1) at kern_readv+0x36
read(c1672d80,d5538d04,3,2,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command getty pid 532 tid 100090 td 0xc1885780
sched_switch(c1885780,0,1) at sched_switch+0x14b
mi_switch(1,0,c1885780,1,0) at mi_switch+0x1ba
sleepq_switch(c16e7410,c1885780,d69cc9d4,c04c0d16,c16e7410) at sleepq_switch+0x133
sleepq_wait_sig(c16e7410,0,100,c1885780,c16e7400) at sleepq_wait_sig+0xc
msleep(c16e7410,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16e7400,c16e7410,159,c063f4de,0) at ttysleep+0x21
ttread(c16e7400,d69cccbc,0) at ttread+0x513
ttyread(c1674c00,d69cccbc,0,d69ccc18,c04927a4) at ttyread+0x2f
scread(c1674c00,d69cccbc,0,0,d69ccc44) at scread+0x22
giant_read(c1674c00,d69cccbc,0,1,c1674c00) at giant_read+0x4c
devfs_read_f(c1870870,d69cccbc,c15c1080,0,c1885780) at devfs_read_f+0x62
dofileread(c1885780,0,c1870870,d69cccbc,ffffffff) at dofileread+0x52
kern_readv(c1885780,0,d69cccbc,bfbfee3f,1) at kern_readv+0x36
read(c1885780,d69ccd04,3,0,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command getty pid 531 tid 100075 td 0xc1673c00
sched_switch(c1673c00,0,1) at sched_switch+0x14b
mi_switch(1,0,c1673c00,1,0) at mi_switch+0x1ba
sleepq_switch(c16e7810,c1673c00,d55539d4,c04c0d16,c16e7810) at sleepq_switch+0x133
sleepq_wait_sig(c16e7810,0,100,c1673c00,c16e7800) at sleepq_wait_sig+0xc
msleep(c16e7810,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16e7800,c16e7810,159,c063f4de,0) at ttysleep+0x21
ttread(c16e7800,d5553cbc,0) at ttread+0x513
ttyread(c1674b00,d5553cbc,0,d5553c18,c04927a4) at ttyread+0x2f
scread(c1674b00,d5553cbc,0,0,d5553c44) at scread+0x22
giant_read(c1674b00,d5553cbc,0,1,c1674b00) at giant_read+0x4c
devfs_read_f(c187a240,d5553cbc,c15c1080,0,c1673c00) at devfs_read_f+0x62
dofileread(c1673c00,0,c187a240,d5553cbc,ffffffff) at dofileread+0x52
kern_readv(c1673c00,0,d5553cbc,bfbfee3f,1) at kern_readv+0x36
read(c1673c00,d5553d04,3,0,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command getty pid 530 tid 100070 td 0xc1884480
sched_switch(c1884480,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884480,1,0) at mi_switch+0x1ba
sleepq_switch(c16e7c10,c1884480,d69a89d4,c04c0d16,c16e7c10) at sleepq_switch+0x133
sleepq_wait_sig(c16e7c10,0,100,c1884480,c16e7c00) at sleepq_wait_sig+0xc
msleep(c16e7c10,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16e7c00,c16e7c10,159,c063f4de,0) at ttysleep+0x21
ttread(c16e7c00,d69a8cbc,0) at ttread+0x513
ttyread(c1674900,d69a8cbc,0,d69a8c18,c04927a4) at ttyread+0x2f
scread(c1674900,d69a8cbc,0,0,d69a8c44) at scread+0x22
giant_read(c1674900,d69a8cbc,0,1,c1674900) at giant_read+0x4c
devfs_read_f(c187a120,d69a8cbc,c15c1080,0,c1884480) at devfs_read_f+0x62
dofileread(c1884480,0,c187a120,d69a8cbc,ffffffff) at dofileread+0x52
kern_readv(c1884480,0,d69a8cbc,bfbfee3f,1) at kern_readv+0x36
read(c1884480,d69a8d04,3,0,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command getty pid 529 tid 100057 td 0xc1672a80
sched_switch(c1672a80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1672a80,1,0) at mi_switch+0x1ba
sleepq_switch(c16e8010,c1672a80,d55329d4,c04c0d16,c16e8010) at sleepq_switch+0x133
sleepq_wait_sig(c16e8010,0,100,c1672a80,c16e8000) at sleepq_wait_sig+0xc
msleep(c16e8010,0,159,c063f4de,0) at msleep+0x336
ttysleep(c16e8000,c16e8010,159,c063f4de,0) at ttysleep+0x21
ttread(c16e8000,d5532cbc,0) at ttread+0x513
ttyread(c1674800,d5532cbc,0,d5532c18,c04927a4) at ttyread+0x2f
scread(c1674800,d5532cbc,0,0,d5532c44) at scread+0x22
giant_read(c1674800,d5532cbc,0,1,c1674800) at giant_read+0x4c
devfs_read_f(c187a090,d5532cbc,c15c1080,0,c1672a80) at devfs_read_f+0x62
dofileread(c1672a80,0,c187a090,d5532cbc,ffffffff) at dofileread+0x52
kern_readv(c1672a80,0,d5532cbc,bfbfee3f,1) at kern_readv+0x36
read(c1672a80,d5532d04,3,0,292) at read+0x45
syscall(3b,3b,3b,0,804f8a0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (3, FreeBSD ELF32, read), eip = 0x280c808b, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command inetd pid 513 tid 100085 td 0xc1a09000
sched_switch(c1a09000,0,1) at sched_switch+0x14b
mi_switch(1,0,0,0,c06a2144) at mi_switch+0x1ba
sleepq_switch(c06a2144,0,d69e8b54,c0491dae,c06a2144) at sleepq_switch+0x133
sleepq_wait_sig(c06a2144,c06a2144,c06a2144,c06a2120,c063ea71) at sleepq_wait_sig+0xc
cv_wait_sig(c06a2144,c06a2120,4,d69e8ba4,0) at cv_wait_sig+0x18a
kern_select(c1a09000,5,bfbfe8c0,0,0) at kern_select+0x5c4
select(c1a09000,d69e8d04,5,0,206) at select+0x44
syscall(3b,3b,3b,0,bfbfee50) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x280da90b, esp = 0xbfbfe82c, ebp = 0xbfbfee88 ---
Tracing command cron pid 493 tid 100081 td 0xc1a09600
sched_switch(c1a09600,0,1) at sched_switch+0x14b
mi_switch(1,0,c1a09600,0,0) at mi_switch+0x1ba
sleepq_switch(c1856c08) at sleepq_switch+0x133
sleepq_wait(c1856c08,0,0,c1a09600,c1856c08) at sleepq_wait+0xb
msleep(c1856c08,c069a58c,50,c063ff41,0) at msleep+0x345
acquire(d69f4a48,40,60000,c1a09600,0) at acquire+0x7b
lockmgr(c1856c08,3002,c1856c2c,c1a09600,d69f4a6c) at lockmgr+0x3fe
vop_stdlock(d69f4ab4,c0676a40,d69f4ab4,d69f4a7c,c05bca98) at vop_stdlock+0x1e
VOP_LOCK_APV(c0676f80,d69f4ab4,d69f4a90,c062243b,d69f4ab4) at VOP_LOCK_APV+0x43
ffs_lock(d69f4ab4,1002,c1856bb0,d69f4ad0,c051e198) at ffs_lock+0x10
VOP_LOCK_APV(c0676a40,d69f4ab4) at VOP_LOCK_APV+0x43
vn_lock(c1856bb0,1002,c1a09600) at vn_lock+0xf4
lookup(d69f4c00,c060c336,0,d69f4c28,11) at lookup+0xc3
namei(d69f4c00,48,1cc01,241c0,0) at namei+0x376
kern_stat(c1a09600,804e66d,0,d69f4c74) at kern_stat+0x35
stat(c1a09600,d69f4d04,2,0,292) at stat+0x1b
syscall(3b,3b,3b,bfbfee80,bfbfeee4) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (188, FreeBSD ELF32, stat), eip = 0x280ca3ab, esp = 0xbfbfeb5c, ebp = 0xbfbfee58 ---
Tracing command sendmail pid 481 tid 100069 td 0xc1884600
sched_switch(c1884600,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884600,0,0) at mi_switch+0x1ba
sleepq_switch(c1883c7c,c1884600,d69abc88,c04c0d16,c1883c7c) at sleepq_switch+0x133
sleepq_wait_sig(c1883c7c,0,100,c1884600,c1883c7c) at sleepq_wait_sig+0xc
msleep(c1883c7c,c1883cb0,168,c063cae1,0) at msleep+0x336
kern_sigsuspend(c1884600,0,0,0,0) at kern_sigsuspend+0xbf
sigsuspend(c1884600,d69abd04,1,0,202) at sigsuspend+0x33
syscall(3b,3b,3b,0,bfbfd020) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827a94b, esp = 0xbfbfd00c, ebp = 0xbfbfd038 ---
Tracing command sendmail pid 477 tid 100050 td 0xc1673600
sched_switch(c1673600,0,1) at sched_switch+0x14b
mi_switch(1,0,c1673600,0,0) at mi_switch+0x1ba
sleepq_switch(c1671240,c1673600,d5547c88,c04c0d16,c1671240) at sleepq_switch+0x133
sleepq_wait_sig(c1671240,0,100,c1673600,c1671240) at sleepq_wait_sig+0xc
msleep(c1671240,c1671274,168,c063cae1,0) at msleep+0x336
kern_sigsuspend(c1673600,0,0,0,0) at kern_sigsuspend+0xbf
sigsuspend(c1673600,d5547d04,1,0,202) at sigsuspend+0x33
syscall(3b,3b,3b,0,bfbfd030) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x2827a94b, esp = 0xbfbfd01c, ebp = 0xbfbfd048 ---
Tracing command sshd pid 471 tid 100056 td 0xc1672c00
sched_switch(c1672c00,0,1) at sched_switch+0x14b
mi_switch(1,0,0,0,c06a2144) at mi_switch+0x1ba
sleepq_switch(c06a2144,0,d5535b54,c0491dae,c06a2144) at sleepq_switch+0x133
sleepq_wait_sig(c06a2144,c06a2144,c06a2144,c06a2120,c063ea71) at sleepq_wait_sig+0xc
cv_wait_sig(c06a2144,c06a2120,4,d5535ba4,4) at cv_wait_sig+0x18a
kern_select(c1672c00,7,80798e0,0,0) at kern_select+0x5c4
select(c1672c00,d5535d04,5,0,202) at select+0x44
syscall(3b,3b,3b,8073d60,80798e0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x282c390b, esp = 0xbfbfe5ac, ebp = 0xbfbfee98 ---
Tracing command ypbind pid 349 tid 100072 td 0xc1884180
sched_switch(c1884180,0,1) at sched_switch+0x14b
mi_switch(1,0,c1884180,0,0) at mi_switch+0x1ba
sleepq_switch(c1856c08) at sleepq_switch+0x133
sleepq_wait(c1856c08,0,0,c1884180,c1856c08) at sleepq_wait+0xb
msleep(c1856c08,c069a58c,50,c063ff41,0) at msleep+0x345
acquire(d69a28f8,40,60000,c1884180,0) at acquire+0x7b
lockmgr(c1856c08,3002,c1856c2c,c1884180,d69a291c) at lockmgr+0x3fe
vop_stdlock(d69a2964,c0676a40,d69a2964,d69a292c,c05bca98) at vop_stdlock+0x1e
VOP_LOCK_APV(c0676f80,d69a2964,d69a2940,c062243b,d69a2964) at VOP_LOCK_APV+0x43
ffs_lock(d69a2964,1002,c1856bb0,d69a2980,c051e198) at ffs_lock+0x10
VOP_LOCK_APV(c0676a40,d69a2964) at VOP_LOCK_APV+0x43
vn_lock(c1856bb0,1002,c1884180) at vn_lock+0xf4
lookup(d69a2bcc,4,0,d69a2bf4,6) at lookup+0xc3
namei(d69a2bcc,d69a2a68,c04a6a10,c1884180,d69a2aa4) at namei+0x376
vn_open_cred(d69a2bcc,d69a2ccc,1a4,c15c1080,7) at vn_open_cred+0x2ad
vn_open(d69a2bcc,d69a2ccc,1a4,7,c04de086) at vn_open+0x1e
kern_open(c1884180,28135537,0,1,1b6) at kern_open+0xb6
open(c1884180,d69a2d04,3,0,296) at open+0x1a
syscall(3b,3b,3b,4,28150160) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (5, FreeBSD ELF32, open), eip = 0x280bb04b, esp = 0xbfbfe8bc, ebp = 0xbfbfe8e8 ---
Tracing command rpcbind pid 336 tid 100058 td 0xc1672900
sched_switch(c1672900,0,1) at sched_switch+0x14b
mi_switch(1,0,0,0,c06a2144) at mi_switch+0x1ba
sleepq_switch(c06a2144,0,0,d552fb94,c0492160) at sleepq_switch+0x133
sleepq_timedwait_sig(c06a2144,0,c06a2144,c06a2144,7531) at sleepq_timedwait_sig+0xd
cv_timedwait_sig(c06a2144,c06a2120,7531,38,7) at cv_timedwait_sig+0x19c
poll(c1672900,d552fd04,3,0,296) at poll+0x4ec
syscall(3b,3b,3b,8051390,7) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (209, FreeBSD ELF32, poll), eip = 0x280d31cb, esp = 0xbfbfcd9c, ebp = 0xbfbfee48 ---
Tracing command syslogd pid 317 tid 100074 td 0xc1673d80
sched_switch(c1673d80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1673d80,0,200012) at mi_switch+0x1ba
sleepq_switch(c06a26c4) at sleepq_switch+0x133
sleepq_wait(c06a26c4,0,0,c1673d80,a00200a4) at sleepq_wait+0xb
msleep(c06a26c4,c06a26e0,44,c06404c9,0) at msleep+0x345
waitrunningbufspace(cbcfd500,cbcfd500,d5556c40,c05010c3,cbcfd500) at waitrunningbufspace+0x66
bufwrite(cbcfd500) at bufwrite+0x121
bawrite(cbcfd500) at bawrite+0x13
ffs_syncvnode(c19ed220,1,4,d5556c94,c0621fcc) at ffs_syncvnode+0x28d
ffs_fsync(d5556cc0) at ffs_fsync+0x12
VOP_FSYNC_APV(c0676a40,d5556cc0) at VOP_FSYNC_APV+0x38
fsync(c1673d80,d5556d04,1,2,296) at fsync+0x170
syscall(3b,3b,3b,10,0) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (95, FreeBSD ELF32, fsync), eip = 0x280ca8eb, esp = 0xbfbfe86c, ebp = 0xbfbfe888 ---
Tracing command devd pid 283 tid 100053 td 0xc1673180
sched_switch(c1673180,0,1) at sched_switch+0x14b
mi_switch(1,0,0,0,c06a2144) at mi_switch+0x1ba
sleepq_switch(c06a2144,0,d553eb54,c0491dae,c06a2144) at sleepq_switch+0x133
sleepq_wait_sig(c06a2144,c06a2144,c06a2144,c06a2120,c063ea71) at sleepq_wait_sig+0xc
cv_wait_sig(c06a2144,c06a2120,4,d553eba4,d553eb78) at cv_wait_sig+0x18a
kern_select(c1673180,5,bfbfe9d0,0,0) at kern_select+0x5c4
select(c1673180,d553ed04,5,0,292) at select+0x44
syscall(3b,3b,3b,1,bfbfea50) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (93, FreeBSD ELF32, select), eip = 0x806060f, esp = 0xbfbfe98c, ebp = 0xbfbfee78 ---
Tracing command dhclient pid 228 tid 100063 td 0xc1885000
sched_switch(c1885000,0,1) at sched_switch+0x14b
mi_switch(1,0,0,0,c06a2144) at mi_switch+0x1ba
sleepq_switch(c06a2144,0,0,d69bdb94,c0492160) at sleepq_switch+0x133
sleepq_timedwait_sig(c06a2144,0,c06a2144,c06a2144,84c831) at sleepq_timedwait_sig+0xd
cv_timedwait_sig(c06a2144,c06a2120,84c831,10,2) at cv_timedwait_sig+0x19c
poll(c1885000,d69bdd04,3,0,292) at poll+0x4ec
syscall(3b,3b,3b,80701d0,2) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (209, FreeBSD ELF32, poll), eip = 0x280c51cb, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command dhclient pid 208 tid 100064 td 0xc1884d80
sched_switch(c1884d80,0,1) at sched_switch+0x14b
mi_switch(1,0,0,0,c06a2144) at mi_switch+0x1ba
sleepq_switch(c06a2144,0,d69bab98,c0491dae,c06a2144) at sleepq_switch+0x133
sleepq_wait_sig(c06a2144,c06a2144,c06a2144,c06a2120,c063ea71) at sleepq_wait_sig+0xc
cv_wait_sig(c06a2144,c06a2120,8,1,0) at cv_wait_sig+0x18a
poll(c1884d80,d69bad04,3,0,292) at poll+0x502
syscall(3b,3b,3b,4,bfbfee30) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (209, FreeBSD ELF32, poll), eip = 0x280c51cb, esp = 0xbfbfee1c, ebp = 0xbfbfee48 ---
Tracing command schedcpu pid 52 tid 100061 td 0xc1672480
sched_switch(c1672480,0,1) at sched_switch+0x14b
mi_switch(1,0,c1672480,0,0) at mi_switch+0x1ba
sleepq_switch(d5526d08,c1672480,d5526cec,c04c0cfe,d5526d08) at sleepq_switch+0x133
sleepq_timedwait(d5526d08,0,0,c1672480,d5526d08) at sleepq_timedwait+0xc
msleep(d5526d08,0,60,c0638487,3e8) at msleep+0x31e
schedcpu_thread(0,d5526d38) at schedcpu_thread+0x32
fork_exit(c04cbb44,0,d5526d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5526d6c, ebp = 0 ---
Tracing command nfsiod 3 pid 51 tid 100062 td 0xc1672300
sched_switch(c1672300,0,1) at sched_switch+0x14b
mi_switch(1,0,c1672300,1,0) at mi_switch+0x1ba
sleepq_switch(c06aa4cc,c1672300,d5523ce0,c04c0d16,c06aa4cc) at sleepq_switch+0x133
sleepq_wait_sig(c06aa4cc,0,100,c1672300,c1834c48) at sleepq_wait_sig+0xc
msleep(c06aa4cc,0,15c,c0638487,0) at msleep+0x336
nfssvc_iod(c06a9e6c,d5523d38) at nfssvc_iod+0xba
fork_exit(c0581c58,c06a9e6c,d5523d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5523d6c, ebp = 0 ---
Tracing command nfsiod 2 pid 50 tid 100038 td 0xc161ca80
sched_switch(c161ca80,0,1) at sched_switch+0x14b
mi_switch(1,0,c161ca80,1,0) at mi_switch+0x1ba
sleepq_switch(c06aa4c8,c161ca80,d5514ce0,c04c0d16,c06aa4c8) at sleepq_switch+0x133
sleepq_wait_sig(c06aa4c8,0,100,c161ca80,c161b624) at sleepq_wait_sig+0xc
msleep(c06aa4c8,0,15c,c0638487,0) at msleep+0x336
nfssvc_iod(c06a9e68,d5514d38) at nfssvc_iod+0xba
fork_exit(c0581c58,c06a9e68,d5514d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5514d6c, ebp = 0 ---
Tracing command nfsiod 1 pid 49 tid 100039 td 0xc161c900
sched_switch(c161c900,0,1) at sched_switch+0x14b
mi_switch(1,0,c161c900,1,0) at mi_switch+0x1ba
sleepq_switch(c06aa4c4,c161c900,d5511ce0,c04c0d16,c06aa4c4) at sleepq_switch+0x133
sleepq_wait_sig(c06aa4c4,0,100,c161c900,c161b830) at sleepq_wait_sig+0xc
msleep(c06aa4c4,0,15c,c0638487,0) at msleep+0x336
nfssvc_iod(c06a9e64,d5511d38) at nfssvc_iod+0xba
fork_exit(c0581c58,c06a9e64,d5511d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5511d6c, ebp = 0 ---
Tracing command nfsiod 0 pid 48 tid 100040 td 0xc161c780
sched_switch(c161c780,0,1) at sched_switch+0x14b
mi_switch(1,0,c161c780,1,0) at mi_switch+0x1ba
sleepq_switch(c06aa4c0,c161c780,d550ece0,c04c0d16,c06aa4c0) at sleepq_switch+0x133
sleepq_wait_sig(c06aa4c0,0,100,c161c780,0) at sleepq_wait_sig+0xc
msleep(c06aa4c0,0,15c,c0638487,0) at msleep+0x336
nfssvc_iod(c06a9e60,d550ed38) at nfssvc_iod+0xba
fork_exit(c0581c58,c06a9e60,d550ed38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd550ed6c, ebp = 0 ---
Tracing command vnlru pid 47 tid 100041 td 0xc161c600
sched_switch(c161c600,0,1) at sched_switch+0x14b
mi_switch(1,0,c161c600,1,0) at mi_switch+0x1ba
sleepq_switch(c161bc48,c161c600,d550bcd8,c04c0cfe,c161bc48) at sleepq_switch+0x133
sleepq_timedwait(c161bc48,0,0,c161c600,4f00b) at sleepq_timedwait+0xc
msleep(c161bc48,c06a2aa0,250,c0641464,3e8,c06a2b20) at msleep+0x31e
vnlru_proc(0,d550bd38) at vnlru_proc+0x117
fork_exit(c050f9c4,0,d550bd38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd550bd6c, ebp = 0 ---
Tracing command syncer pid 46 tid 100042 td 0xc161c480
sched_switch(c161c480,0,1) at sched_switch+0x14b
mi_switch(1,0,c161c480,1,0) at mi_switch+0x1ba
sleepq_switch(cbc36578) at sleepq_switch+0x133
sleepq_wait(cbc36578,0,0,c161c480,cbc36578) at sleepq_wait+0xb
msleep(cbc36578,c069a034,50,c0640783,0) at msleep+0x345
acquire(d5508ae0,120,60000,c161c480,0) at acquire+0x7b
lockmgr(cbc36578,202122,c193d8fc,c161c480) at lockmgr+0x3fe
getblk(c193d880,710f240,0,4000,0) at getblk+0x1a5
breadn(c193d880,710f240,0,4000,0) at breadn+0x31
bread(c193d880,710f240,0,4000,0) at bread+0x20
ffs_update(c1b2e660,0,2032,1,0) at ffs_update+0x21c
ffs_syncvnode(c1b2e660,3) at ffs_syncvnode+0x450
ffs_sync(c17b3800,3,c161c480,c17b3800,2) at ffs_sync+0x1f9
sync_fsync(d5508cc0) at sync_fsync+0x177
VOP_FSYNC_APV(c066dac0,d5508cc0) at VOP_FSYNC_APV+0x38
sync_vnode(c1943830,c161c480) at sync_vnode+0x153
sched_sync(0,d5508d38) at sched_sync+0x259
fork_exit(c05115d0,0,d5508d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5508d6c, ebp = 0 ---
Tracing command bufdaemon pid 45 tid 100043 td 0xc161c300
sched_switch(c161c300,0,1) at sched_switch+0x14b
mi_switch(1,0,c161c300,1,0) at mi_switch+0x1ba
sleepq_switch(c06a268c,c161c300,d5505cf0,c04c0cfe,c06a268c) at sleepq_switch+0x133
sleepq_timedwait(c06a268c,0,0,c161c300,4) at sleepq_timedwait+0xc
msleep(c06a268c,c06a26a0,44,c064069e,3e8) at msleep+0x31e
buf_daemon(0,d5505d38) at buf_daemon+0x173
fork_exit(c0502cc8,0,d5505d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5505d6c, ebp = 0 ---
Tracing command pagezero pid 44 tid 100044 td 0xc161c180
sched_switch(c161c180,0,1) at sched_switch+0x14b
mi_switch(1,0,c161c180,0,0) at mi_switch+0x1ba
sleepq_switch(c06b09c4,c161c180,d5502cf0,c04c0cfe,c06b09c4) at sleepq_switch+0x133
sleepq_timedwait(c06b09c4,0,0,c161c180,c161c180) at sleepq_timedwait+0xc
msleep(c06b09c4,c06b04a0,2ff,c064c4da,493e0) at msleep+0x31e
vm_pagezero(0,d5502d38) at vm_pagezero+0x97
fork_exit(c05e3678,0,d5502d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5502d6c, ebp = 0 ---
Tracing command vmdaemon pid 9 tid 100045 td 0xc161c000
sched_switch(c161c000,0,1) at sched_switch+0x14b
mi_switch(1,0,c161c000,1,0) at mi_switch+0x1ba
sleepq_switch(c06b0514) at sleepq_switch+0x133
sleepq_wait(c06b0514,0,0,c161c000,c1670624) at sleepq_wait+0xb
msleep(c06b0514,0,68,c064069e,0) at msleep+0x345
vm_daemon(0,d54ffd38) at vm_daemon+0x50
fork_exit(c05e249c,0,d54ffd38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54ffd6c, ebp = 0 ---
Tracing command pagedaemon pid 8 tid 100046 td 0xc1610d80
sched_switch(c1610d80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1610d80,0,0) at mi_switch+0x1ba
sleepq_switch(c06b04d0,c1610d80,d54fccec,c04c0cfe,c06b04d0) at sleepq_switch+0x133
sleepq_timedwait(c06b04d0,0,0,c1610d80,4) at sleepq_timedwait+0xc
msleep(c06b04d0,c06b04a0,44,c064069e,1388) at msleep+0x31e
vm_pageout(0,d54fcd38) at vm_pageout+0x280
fork_exit(c05e210c,0,d54fcd38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54fcd6c, ebp = 0 ---
Tracing command swi0: sio pid 43 tid 100047 td 0xc1610c00
sched_switch(c1610c00,0,1) at sched_switch+0x14b
mi_switch(1,0) at mi_switch+0x1ba
ithread_loop(c16ed880,d54f9d38) at ithread_loop+0x2a2
fork_exit(c04a6ae4,c16ed880,d54f9d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54f9d6c, ebp = 0 ---
Tracing command fdc0 pid 7 tid 100048 td 0xc1610a80
sched_switch(c1610a80,0,1) at sched_switch+0x14b
mi_switch(1,0,c1610a80,0,0) at mi_switch+0x1ba
sleepq_switch(c16dd43c,c1610a80,d54f6ca8,c04c0cfe,c16dd43c) at sleepq_switch+0x133
sleepq_timedwait(c16dd43c,0,0,c1610a80,c16dd43c) at sleepq_timedwait+0xc
msleep(c16dd43c,c16dd4f0,4c,c0638487,3e8) at msleep+0x31e
fdc_worker(c16dd400,c1670c48,c05ed680,0,d54f6d24) at fdc_worker+0x25a
fdc_thread(c16dd400,d54f6d38) at fdc_thread+0x8e
fork_exit(c05ed680,c16dd400,d54f6d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54f6d6c, ebp = 0 ---
Tracing command kqueue taskq pid 6 tid 100049 td 0xc1610900
sched_switch(c1610900,0,1) at sched_switch+0x14b
mi_switch(1,0,c1610900,0,0) at mi_switch+0x1ba
sleepq_switch(c166f280) at sleepq_switch+0x133
sleepq_wait(c166f280,0,0,c1610900,c166f280) at sleepq_wait+0xb
msleep(c166f280,c166f29c,5c,c0638487,0,c166f280) at msleep+0x345
taskqueue_thread_loop(c06993c0,d54f3d38) at taskqueue_thread_loop+0x52
fork_exit(c04da954,c06993c0,d54f3d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54f3d6c, ebp = 0 ---
Tracing command swi5:+ pid 42 tid 100027 td 0xc1610180
fork_trampoline() at fork_trampoline
Tracing command thread taskq pid 5 tid 100028 td 0xc1610000
sched_switch(c1610000,0,1) at sched_switch+0x14b
mi_switch(1,0,c1610000,0,0) at mi_switch+0x1ba
sleepq_switch(c15c1280) at sleepq_switch+0x133
sleepq_wait(c15c1280,0,0,c1610000,c15c1280) at sleepq_wait+0xb
msleep(c15c1280,c15c129c,5c,c0638487,0,c15c1280) at msleep+0x345
taskqueue_thread_loop(c06a0c08,d54e1d38) at taskqueue_thread_loop+0x52
fork_exit(c04da954,c06a0c08,d54e1d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54e1d6c, ebp = 0 ---
Tracing command swi6:+ pid 41 tid 100029 td 0xc15cdd80
fork_trampoline() at fork_trampoline
Tracing command swi6: task queue pid 40 tid 100030 td 0xc15cdc00
sched_switch(c15cdc00,0,1) at sched_switch+0x14b
mi_switch(1,0) at mi_switch+0x1ba
ithread_loop(c15c1400,d42f0d38) at ithread_loop+0x2a2
fork_exit(c04a6ae4,c15c1400,d42f0d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42f0d6c, ebp = 0 ---
Tracing command swi2: cambio pid 39 tid 100031 td 0xc15cda80
sched_switch(c15cda80,0,1) at sched_switch+0x14b
mi_switch(1,0) at mi_switch+0x1ba
ithread_loop(c15c1500,d42edd38) at ithread_loop+0x2a2
fork_exit(c04a6ae4,c15c1500,d42edd38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42edd6c, ebp = 0 ---
Tracing command yarrow pid 38 tid 100032 td 0xc15cd900
sched_switch(c15cd900,0,1) at sched_switch+0x14b
mi_switch(1,0,c15cd900,0,c0698144) at mi_switch+0x1ba
sleepq_switch(c0698140,c15cd900,d42eacdc,c04c0cfe,c0698140) at sleepq_switch+0x133
sleepq_timedwait(c0698140,0,0,c15cd900,44) at sleepq_timedwait+0xc
msleep(c0698140,0,44,c0638487,64) at msleep+0x31e
random_kthread(0,d42ead38) at random_kthread+0x215
fork_exit(c0476078,0,d42ead38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42ead6c, ebp = 0 ---
Tracing command g_down pid 4 tid 100033 td 0xc15cd780
sched_switch(c15cd780,0,1) at sched_switch+0x14b
mi_switch(1,0,c15cd780,0,0) at mi_switch+0x1ba
sleepq_switch(c0698b08,c15cd780,d42e7cd8,c04c0cfe,c0698b08) at sleepq_switch+0x133
sleepq_timedwait(c0698b08,0,0,c15cd780,0) at sleepq_timedwait+0xc
msleep(c0698b08,c06989e8,24c,c0638487,64) at msleep+0x31e
g_io_schedule_down(c15cd780) at g_io_schedule_down+0x56
g_down_procbody(0,d42e7d38) at g_down_procbody+0x92
fork_exit(c0482cf8,0,d42e7d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42e7d6c, ebp = 0 ---
Tracing command g_up pid 3 tid 100034 td 0xc15cd600
sched_switch(c15cd600,0,1) at sched_switch+0x14b
mi_switch(1,0,c15cd600,0,0) at mi_switch+0x1ba
sleepq_switch(c0698b04,c15cd600,d42e4cdc,c04c0cfe,c0698b04) at sleepq_switch+0x133
sleepq_timedwait(c0698b04,0,0,c15cd600,0) at sleepq_timedwait+0xc
msleep(c0698b04,c0698a28,24c,c0638487,64) at msleep+0x31e
g_io_schedule_up(c15cd600) at g_io_schedule_up+0xcc
g_up_procbody(0,d42e4d38) at g_up_procbody+0x92
fork_exit(c0482c60,0,d42e4d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42e4d6c, ebp = 0 ---
Tracing command g_event pid 2 tid 100035 td 0xc15cd480
sched_switch(c15cd480,0,1) at sched_switch+0x14b
mi_switch(1,0,c15cd480,0,0) at mi_switch+0x1ba
sleepq_switch(c0698afc,c15cd480,d42e1ce4,c04c0cfe,c0698afc) at sleepq_switch+0x133
sleepq_timedwait(c0698afc,0,0,c15cd480,66666667) at sleepq_timedwait+0xc
msleep(c0698afc,0,4c,c0638487,64) at msleep+0x31e
g_event_procbody(0,d42e1d38) at g_event_procbody+0xca
fork_exit(c0482d90,0,d42e1d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42e1d6c, ebp = 0 ---
Tracing command swi3: vm pid 37 tid 100036 td 0xc161cd80
fork_trampoline() at fork_trampoline
Tracing command swi4: clock sio pid 36 tid 100037 td 0xc161cc00
sched_switch(c161cc00,0,1) at sched_switch+0x14b
mi_switch(1,0) at mi_switch+0x1ba
ithread_loop(c15b4280,d5517d38) at ithread_loop+0x2a2
fork_exit(c04a6ae4,c15b4280,d5517d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd5517d6c, ebp = 0 ---
Tracing command swi1: net pid 35 tid 100017 td 0xc15c9900
sched_switch(c15c9900,0,1) at sched_switch+0x14b
mi_switch(1,0) at mi_switch+0x1ba
ithread_loop(c15b4300,d42ccd38) at ithread_loop+0x2a2
fork_exit(c04a6ae4,c15b4300,d42ccd38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42ccd6c, ebp = 0 ---
Tracing command irq23: pid 34 tid 100018 td 0xc15c9780
fork_trampoline() at fork_trampoline
Tracing command irq22: pid 33 tid 100019 td 0xc15c9600
fork_trampoline() at fork_trampoline
Tracing command irq21: pid 32 tid 100020 td 0xc15c9480
fork_trampoline() at fork_trampoline
Tracing command irq20: pid 31 tid 100021 td 0xc15c9300
fork_trampoline() at fork_trampoline
Tracing command irq19: pid 30 tid 100022 td 0xc15c9180
fork_trampoline() at fork_trampoline
Tracing command irq18: pid 29 tid 100023 td 0xc1610780
fork_trampoline() at fork_trampoline
Tracing command irq17: fxp0 pid 28 tid 100024 td 0xc1610600
sched_switch(c063e049,d54edce4,c04c0f0a,13f574b6,302c63e) at sched_switch+0x14b
kdb_backtrace(13f574b6,302c63e,ffc03014,c1610600,bbd) at kdb_backtrace+0x21
mi_switch(1,0) at mi_switch+0xf6
ithread_loop(c15b4680,d54edd38) at ithread_loop+0x2a2
fork_exit(c04a6ae4,c15b4680,d54edd38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54edd6c, ebp = 0 ---
Tracing command irq16: iir0 pid 27 tid 100025 td 0xc1610480
sched_switch(c1610480,0,1) at sched_switch+0x14b
mi_switch(1,0) at mi_switch+0x1ba
ithread_loop(c15b4700,d54ead38) at ithread_loop+0x2a2
fork_exit(c04a6ae4,c15b4700,d54ead38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd54ead6c, ebp = 0 ---
Tracing command irq15: ata1 pid 26 tid 100026 td 0xc1610300
fork_trampoline() at fork_trampoline
Tracing command irq14: ata0 pid 25 tid 100008 td 0xc15c8300
fork_trampoline() at fork_trampoline
Tracing command irq13: pid 24 tid 100009 td 0xc15c8180
fork_trampoline() at fork_trampoline
Tracing command irq12: pid 23 tid 100010 td 0xc15c8000
fork_trampoline() at fork_trampoline
Tracing command irq11: pid 22 tid 100011 td 0xc15cd300
fork_trampoline() at fork_trampoline
Tracing command irq10: pid 21 tid 100012 td 0xc15cd180
fork_trampoline() at fork_trampoline
Tracing command irq9: pid 20 tid 100013 td 0xc15cd000
fork_trampoline() at fork_trampoline
Tracing command irq8: pid 19 tid 100014 td 0xc15c9d80
fork_trampoline() at fork_trampoline
Tracing command irq7: pid 18 tid 100015 td 0xc15c9c00
fork_trampoline() at fork_trampoline
Tracing command irq6: fdc0 pid 17 tid 100016 td 0xc15c9a80
fork_trampoline() at fork_trampoline
Tracing command irq5: pid 16 tid 100000 td 0xc15c9000
fork_trampoline() at fork_trampoline
Tracing command irq4: sio0 pid 15 tid 100001 td 0xc15c8d80
fork_trampoline() at fork_trampoline
Tracing command irq3: sio1 pid 14 tid 100002 td 0xc15c8c00
fork_trampoline() at fork_trampoline
Tracing command irq0: pid 13 tid 100003 td 0xc15c8a80
fork_trampoline() at fork_trampoline
Tracing command irq1: atkbd0 pid 12 tid 100004 td 0xc15c8900
fork_trampoline() at fork_trampoline
Tracing command idle: cpu0 pid 11 tid 100005 td 0xc15c8780
kdb_enter(c064d658) at kdb_enter+0x2b
siointr1(c16e9400) at siointr1+0xd5
siointr(c16e9400) at siointr+0x5e
intr_execute_handlers(c15fa090,d42abcb4,4,d42abcf8,c0606603) at intr_execute_handlers+0x85
lapic_handle_intr(34) at lapic_handle_intr+0x2e
Xapic_isr1() at Xapic_isr1+0x33
--- interrupt, eip = 0xc060d7bd, esp = 0xd42abcf8, ebp = 0xd42abcf8 ---
cpu_idle_default(d42abd10,c04a5fed,c15c7a3c,c04a5f58,d42abd24) at cpu_idle_default+0x5
cpu_idle(c15c7a3c,c04a5f58,d42abd24,c04a5d01,0) at cpu_idle+0x28
idle_proc(0,d42abd38) at idle_proc+0x95
fork_exit(c04a5f58,0,d42abd38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42abd6c, ebp = 0 ---
Tracing command idle: cpu1 pid 10 tid 100006 td 0xc15c8600
sched_switch(c15c8600,c1610600,6) at sched_switch+0x14b
mi_switch(6,c1610600,c1610754,c1610600,d42a8c3c) at mi_switch+0x1ba
maybe_preempt(c1610600) at maybe_preempt+0xc4
sched_add(c1610600,4,c15c8600,c1610600,c15b4680) at sched_add+0xf0
setrunqueue(c1610600,4) at setrunqueue+0x63
ithread_schedule(c15b4680,11,c15c8600,c15c8600,2) at ithread_schedule+0xcf
intr_execute_handlers(c15fa1c8,d42a8cb4,11,d42a8cf8,c0606653) at intr_execute_handlers+0xc5
lapic_handle_intr(41) at lapic_handle_intr+0x2e
Xapic_isr2() at Xapic_isr2+0x33
--- interrupt, eip = 0xc060d7bd, esp = 0xd42a8cf8, ebp = 0xd42a8cf8 ---
cpu_idle_default(d42a8d10,c04a5fed,c15c7c48,c04a5f58,d42a8d24) at cpu_idle_default+0x5
cpu_idle(c15c7c48,c04a5f58,d42a8d24,c04a5d01,0) at cpu_idle+0x28
idle_proc(0,d42a8d38) at idle_proc+0x95
fork_exit(c04a5f58,0,d42a8d38) at fork_exit+0x71
fork_trampoline() at fork_trampoline+0x8
--- trap 0x1, eip = 0, esp = 0xd42a8d6c, ebp = 0 ---
Tracing command init pid 1 tid 100007 td 0xc15c8480
sched_switch(c15c8480,0,1) at sched_switch+0x14b
mi_switch(1,0,c15c8480,0,0) at mi_switch+0x1ba
sleepq_switch(c15cc000,c15c8480,d42a5c38,c04c0d16,c15cc000) at sleepq_switch+0x133
sleepq_wait_sig(c15cc000,0,100,c15c8480,0) at sleepq_wait_sig+0xc
msleep(c15cc000,c15cc068,15c,c063c9be,0) at msleep+0x336
kern_wait(c15c8480,ffffffff,d42a5c84,0,0) at kern_wait+0x8ec
wait4(c15c8480,d42a5d04,4,1,282) at wait4+0x2a
syscall(3b,3b,3b,bfbfef24,bfbfef18) at syscall+0x29b
Xint0x80_syscall() at Xint0x80_syscall+0x1f
--- syscall (7, FreeBSD ELF32, wait4), eip = 0x8051f43, esp = 0xbfbfed9c, ebp = 0xbfbfedb8 ---
Tracing command swapper pid 0 tid 0 td 0xc0698e20
sched_switch(c0698e20,0,1) at sched_switch+0x14b
mi_switch(1,0) at mi_switch+0x1ba
scheduler(0,81ec00,81e000,0,c042f5d5) at scheduler+0x262
mi_startup() at mi_startup+0x96
begin() at begin+0x2c
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?87d5jsj09s.fsf>