From owner-freebsd-fs@FreeBSD.ORG Sat Oct 24 17:46:59 2009 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1535E106566C; Sat, 24 Oct 2009 17:46:59 +0000 (UTC) (envelope-from stef-list@memberwebs.com) Received: from memberwebs.com (memberwebs.com [94.75.203.95]) by mx1.freebsd.org (Postfix) with ESMTP id EF46D8FC19; Sat, 24 Oct 2009 17:46:57 +0000 (UTC) Received: from [172.27.5.159] (unknown [172.27.5.159]) by memberwebs.com (Postfix) with ESMTP id B41F283E4C6; Sat, 24 Oct 2009 17:46:52 +0000 (UTC) Message-ID: <4AE33D89.6050807@memberwebs.com> Date: Sat, 24 Oct 2009 12:46:49 -0500 From: Stef Walter User-Agent: Thunderbird 2.0.0.23 (X11/20090817) MIME-Version: 1.0 To: freebsd-fs@freebsd.org References: <4AD77230.3030803@memberwebs.com> In-Reply-To: <4AD77230.3030803@memberwebs.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: pjd@FreeBSD.org Subject: Re: Deadlock after canceled zfs recv X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: stef@memberwebs.com List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 24 Oct 2009 17:46:59 -0000 Stef Walter wrote: > I'm running the latest RELENG_8, and been doing some pre-production > stress testing. > > I can deadlock (reproduceable) after a canceled ssh + zfs recv. Here's > how I reproduce the problem: Pawel, I hope it's okay if I CC you on this problem. I'd like to contribute and help get this problem sorted out before the 8.0 release. I've done a deadlock post mortem on this problem. It's below. There are several zfs processes waiting on locks. Is there anything I can do to further research the problem. Thanks in advance. Cheers, Stef ---------------------------------------------------------------------- DEADLOCK POSTMORTEM db> show pcpu cpuid = 0 dynamic pcpu = 0x68e280 curthread = 0xc596fb40: pid 11 "idle: cpu0" curpcb = 0xc54cdd90 fpcurthread = none idlethread = 0xc596fb40: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x68e280 curthread = 0xc596fb40: pid 11 "idle: cpu0" curpcb = 0xc54cdd90 fpcurthread = none idlethread = 0xc596fb40: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db> show locks db> show alllocks db> show lockedvnods Locked vnodes db> alltrace Tracing command zfs pid 1559 tid 100126 td 0xc673ab40 sched_switch(c673ab40,0,104,191,ecf50dd2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c673ab40,0,c0c7a69a,24b,c673ab40,...) at sleepq_switch+0x15f sleepq_wait(c97710a8,0,e8188ab4,1,0,...) at sleepq_wait+0x63 _cv_wait(c97710a8,c9771078,208,206,c97710a8,...) at _cv_wait+0x240 dsl_dataset_hold_ref(c6b0f9c0,e8188b58,c6268de1,241,e8188b28,...) at dsl_dataset_hold_ref+0xa0 dsl_dataset_hold(d1be8000,c6b0f9c0,e8188b58,c086d87d,1,...) at dsl_dataset_hold+0x10c dmu_objset_open(d1be8000,5,9,e8188b84,e8188b94,...) at dmu_objset_open+0xc3 zfs_ioc_objset_stats(d1be8000,24,d1be800e,0,d1be8938,...) at zfs_ioc_objset_stats+0x38 zfs_ioc_dataset_list_next(d1be8000,0,0,0,0,...) at zfs_ioc_dataset_list_next+0x11f zfsdev_ioctl(c6136100,cc285a13,d1be8000,3,c673ab40,...) at zfsdev_ioctl+0x7d devfs_ioctl_f(c5ed3a10,cc285a13,d1be8000,c60ff680,c673ab40,...) at devfs_ioctl_f+0xf8 kern_ioctl(c673ab40,3,cc285a13,d1be8000,173ab40,...) at kern_ioctl+0x1fd ioctl(c673ab40,e8188cf8,c,e8188d38,c0d5c928,...) at ioctl+0x134 syscall(e8188d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfc9cc, ebp = 0xbfbfd638 --- Tracing command zfs pid 1558 tid 100124 td 0xc6122480 sched_switch(c6122480,0,104,191,fc4c78d2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6122480,0,c0c7a69a,26e,2,...) at sleepq_switch+0x15f sleepq_timedwait(c0dcac64,0,c62689d8,2,0,...) at sleepq_timedwait+0x6b _sleep(c0dcac64,0,0,c62689d8,1,...) at _sleep+0x339 pause(c62689d8,1,c9771200,e818250c,c61a1ec2,...) at pause+0x47 dnode_special_close(c9446000,c626870f,c61a0c60,c9771200,c9771200,...) at dnode_special_close+0x2c dmu_objset_evict(c9771000,c9771200,1cb,0,c6265be3,...) at dmu_objset_evict+0x82 dsl_dataset_destroy(c9771000,c6268520,16,e8182984,e8182944,...) at dsl_dataset_destroy+0x24c dmu_recv_abort_cleanup(e8182984,c626d723,1cb,c6265810,e818292c,...) at dmu_recv_abort_cleanup+0x2f dmu_recv_stream(e8182984,c5ed4dc8,e818299c,0,0,...) at dmu_recv_stream+0x9de zfs_ioc_recv(cfa8f000,0,0,0,0,...) at zfs_ioc_recv+0x2d4 zfsdev_ioctl(c6136100,cc285a1c,cfa8f000,3,c6122480,...) at zfsdev_ioctl+0x7d devfs_ioctl_f(c5ed4700,cc285a1c,cfa8f000,c8488d00,c6122480,...) at devfs_ioctl_f+0xf8 kern_ioctl(c6122480,3,cc285a1c,cfa8f000,1000000,...) at kern_ioctl+0x1fd ioctl(c6122480,e8182cf8,c,c0c8e3e7,c0d5c928,...) at ioctl+0x134 syscall(e8182d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfbcfc, ebp = 0xbfbfbd28 --- Tracing command zfskern pid 1521 tid 100117 td 0xc61226c0 sched_switch(c61226c0,0,104,191,a1331866,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c61226c0,0,c0c7a69a,26e,c61226c0,...) at sleepq_switch+0x15f sleepq_timedwait(c65a65dc,0,e8164c28,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c65a65dc,c65a6584,7530,c65a65dc,c65a6580,...) at _cv_timedwait+0x250 txg_thread_wait(7530,0,3b9aca00,0,3e8,...) at txg_thread_wait+0x44 txg_sync_thread(c65a6400,e8164d38,c0c717a4,343,c611d550,...) at txg_sync_thread+0x122 fork_exit(c61d37e0,c65a6400,e8164d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8164d70, ebp = 0 --- Tracing command zfskern pid 1521 tid 100116 td 0xc6122900 sched_switch(c6122900,0,104,191,a133372e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6122900,0,c0c7a69a,24b,c6122900,...) at sleepq_switch+0x15f sleepq_wait(c65a65ec,0,e8160c50,1,0,...) at sleepq_wait+0x63 _cv_wait(c65a65ec,c65a6584,c65a65f4,c65a65ec,c65a6580,...) at _cv_wait+0x240 txg_thread_wait(0,0,c626d723,18d,10,...) at txg_thread_wait+0x87 txg_quiesce_thread(c65a6400,e8160d38,c0c717a4,343,c611d550,...) at txg_quiesce_thread+0xb5 fork_exit(c61d3070,c65a6400,e8160d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8160d70, ebp = 0 --- Tracing command zfskern pid 1521 tid 100114 td 0xc6122d80 sched_switch(c6122d80,0,104,191,a12ac4fe,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c6122d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541908,4c,c62713d8,0,0,...) at sleepq_wait+0x63 _sleep(c6541908,c654191c,24c,c62713d8,0,...) at _sleep+0x36b vdev_geom_worker(c6541900,e8154d38,c0c717a4,343,c611d550,...) at vdev_geom_worker+0x124 fork_exit(c621a170,c6541900,e8154d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8154d70, ebp = 0 --- Tracing command zfskern pid 1521 tid 100087 td 0xc6120b40 sched_switch(c6120b40,0,104,191,7f209aca,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120b40,0,c0c7a69a,26e,c6120b40,...) at sleepq_switch+0x15f sleepq_timedwait(c627e534,0,e8109c24,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c627e534,c627e520,3e8,1131,0,...) at _cv_timedwait+0x250 l2arc_feed_thread(0,e8109d38,c0c717a4,343,c611d550,...) at l2arc_feed_thread+0x131 fork_exit(c618f090,0,e8109d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8109d70, ebp = 0 --- Tracing command zfskern pid 1521 tid 100082 td 0xc61216c0 sched_switch(c61216c0,0,104,191,7f20c4da,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c61216c0,0,c0c7a69a,26e,c61216c0,...) at sleepq_switch+0x15f sleepq_timedwait(c627614c,0,c57f6c80,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c627614c,c6276138,3e8,0,3e8,...) at _cv_timedwait+0x250 arc_reclaim_thread(0,c57f6d38,c0c717a4,343,c611d550,...) at arc_reclaim_thread+0x39f fork_exit(c6191f00,0,c57f6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc57f6d70, ebp = 0 --- Tracing command bash pid 1519 tid 100080 td 0xc6121b40 sched_switch(c6121b40,0,104,191,ebff50ce,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200 sleepq_switch(c6121b40,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c611daa0,5c,c0c7cf0b,100,0,...) at sleepq_wait_sig+0x17 _sleep(c611daa0,c611db28,15c,c0c7cf0b,0,...) at _sleep+0x354 kern_wait(c6121b40,ffffffff,c57f0c74,6,0,...) at kern_wait+0xb76 wait4(c6121b40,c57f0cf8,10,c0c7cdfa,c0d5c404,...) at wait4+0x3b syscall(c57f0d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x80e1f4f, esp = 0xbfbfe9dc, ebp = 0xbfbfe9f8 --- Tracing command sshd pid 1516 tid 100075 td 0xc5ede480 sched_switch(c5ede480,0,104,191,ebe40d02,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede480,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c087111a,c5e7cd50,0,c0c74a49,c5ede480,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5e7cd64,0,c57cda7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e7cd64,c5e7cd50,c0c7c95a,5d1,c5ed30e0,...) at _cv_wait_sig+0x240 seltdwait(c5ed30e0,58,c5f40200,c5ede480,58,...) at seltdwait+0xa2 kern_select(c5ede480,9,286030c4,286030cc,0,0,20,65,280fdd50) at kern_select+0x4f4 select(c5ede480,c57cdcf8,14,c0c5e8da,c0d5cd6c,...) at select+0x66 syscall(c57cdd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfde5c, ebp = 0xbfbfdea8 --- Tracing command login pid 1514 tid 100070 td 0xc5ee0000 sched_switch(c5ee0000,0,104,191,2065f36,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0000,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c087111a,c59ede04,0,c0c74a49,c5ee0000,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ede70,0,c57bcb0c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ede70,c59ede04,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ede00,c59ede70,1,c57bcb87,c57bcb94,...) at tty_wait+0x71 ttydisc_read(c59ede00,c57bcc58,0,9e,0,...) at ttydisc_read+0xef ttydev_read(c5afd400,c57bcc58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed39a0,c57bcc58,c596b100,0,c5ee0000,...) at devfs_read_f+0x7e dofileread(c57bcc58,ffffffff,ffffffff,0,c5ed39a0,...) at dofileread+0x96 kern_readv(c5ee0000,0,c57bcc58,c57bcc78,1,...) at kern_readv+0x58 read(c5ee0000,c57bccf8,c,c0c7cefa,c0d5c394,...) at read+0x4f syscall(c57bcd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x281af253, esp = 0xbfbfe6bc, ebp = 0xbfbfe958 --- Tracing command getty pid 1513 tid 100083 td 0xc6121480 sched_switch(c6121480,0,104,191,51bbee4e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121480,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c6121480,c6121480,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5c7d470,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5c7d470,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c5c7d400,c5c7d470,c57f9c58,1,0,...) at tty_wait+0x71 ttydisc_read(c5c7d400,c57f9c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85900,c57f9c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed33f0,c57f9c58,c596b100,0,c6121480,...) at devfs_read_f+0x7e dofileread(c57f9c58,ffffffff,ffffffff,0,c5ed33f0,...) at dofileread+0x96 kern_readv(c6121480,0,c57f9c58,c57f9c78,1,...) at kern_readv+0x58 read(c6121480,c57f9cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(c57f9d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1512 tid 100077 td 0xc5ede000 sched_switch(c5ede000,0,104,191,519be396,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede000,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c5ede000,c5ede000,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59eda70,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59eda70,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59eda00,c59eda70,c57e7c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59eda00,c57e7c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85700,c57e7c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed38c0,c57e7c58,c596b100,0,c5ede000,...) at devfs_read_f+0x7e dofileread(c57e7c58,ffffffff,ffffffff,0,c5ed38c0,...) at dofileread+0x96 kern_readv(c5ede000,0,c57e7c58,c57e7c78,1,...) at kern_readv+0x58 read(c5ede000,c57e7cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(c57e7d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1511 tid 100069 td 0xc5ee0240 sched_switch(c5ee0240,0,104,191,517bb2e2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0240,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c5ee0240,c5ee0240,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed670,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed670,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed600,c59ed670,c57b8c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed600,c57b8c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85e00,c57b8c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3268,c57b8c58,c596b100,0,c5ee0240,...) at devfs_read_f+0x7e dofileread(c57b8c58,ffffffff,ffffffff,0,c5ed3268,...) at dofileread+0x96 kern_readv(c5ee0240,0,c57b8c58,c57b8c78,1,...) at kern_readv+0x58 read(c5ee0240,c57b8cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(c57b8d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1510 tid 100081 td 0xc6121900 sched_switch(c6121900,0,104,191,515ba0da,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121900,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c6121900,c6121900,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed470,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed470,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed400,c59ed470,c57f3c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed400,c57f3c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c86100,c57f3c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3ce8,c57f3c58,c596b100,0,c6121900,...) at devfs_read_f+0x7e dofileread(c57f3c58,ffffffff,ffffffff,0,c5ed3ce8,...) at dofileread+0x96 kern_readv(c6121900,0,c57f3c58,c57f3c78,1,...) at kern_readv+0x58 read(c6121900,c57f3cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(c57f3d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1509 tid 100063 td 0xc5e56240 sched_switch(c5e56240,0,104,191,513b103e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e56240,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c5e56240,c5e56240,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59eca70,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59eca70,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59eca00,c59eca70,c5799c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59eca00,c5799c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85b00,c5799c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3818,c5799c58,c596b100,0,c5e56240,...) at devfs_read_f+0x7e dofileread(c5799c58,ffffffff,ffffffff,0,c5ed3818,...) at dofileread+0x96 kern_readv(c5e56240,0,c5799c58,c5799c78,1,...) at kern_readv+0x58 read(c5e56240,c5799cf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(c5799d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1508 tid 100086 td 0xc6120d80 sched_switch(c6120d80,0,104,191,511a69a2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120d80,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c6120d80,c6120d80,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed270,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed270,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed200,c59ed270,e80fcc58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed200,e80fcc58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85600,e80fcc58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3310,e80fcc58,c596b100,0,c6120d80,...) at devfs_read_f+0x7e dofileread(e80fcc58,ffffffff,ffffffff,0,c5ed3310,...) at dofileread+0x96 kern_readv(c6120d80,0,e80fcc58,e80fcc78,1,...) at kern_readv+0x58 read(c6120d80,e80fccf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(e80fcd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1507 tid 100085 td 0xc6121000 sched_switch(c6121000,0,104,191,51fdf8aa,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121000,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c6121000,c6121000,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59edc70,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59edc70,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59edc00,c59edc70,c57ffc58,1,0,...) at tty_wait+0x71 ttydisc_read(c59edc00,c57ffc58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c86000,c57ffc58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3e70,c57ffc58,c596b100,0,c6121000,...) at devfs_read_f+0x7e dofileread(c57ffc58,ffffffff,ffffffff,0,c5ed3e70,...) at dofileread+0x96 kern_readv(c6121000,0,c57ffc58,c57ffc78,1,...) at kern_readv+0x58 read(c6121000,c57ffcf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(c57ffd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1506 tid 100084 td 0xc6121240 sched_switch(c6121240,0,104,191,50fa4d42,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121240,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,c6121240,c6121240,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed870,0,c0c7ec9a,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed870,c0dca5b0,c0c7e645,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed800,c59ed870,c57fcc58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed800,c57fcc58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85a00,c57fcc58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3a80,c57fcc58,c596b100,0,c6121240,...) at devfs_read_f+0x7e dofileread(c57fcc58,ffffffff,ffffffff,0,c5ed3a80,...) at dofileread+0x96 kern_readv(c6121240,0,c57fcc58,c57fcc78,1,...) at kern_readv+0x58 read(c6121240,c57fccf8,c,c0c8e3e7,c0d5c394,...) at read+0x4f syscall(c57fcd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command cron pid 1455 tid 100065 td 0xc5ee0b40 sched_switch(c5ee0b40,0,104,191,153fd976,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200 sleepq_switch(c5ee0b40,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(ea61,c08b8560,c5ee0b40,0,100,...) at sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c0dcae04,5c,c0c7759a,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c0dcae04,0,15c,c0c7759a,ea61,...) at _sleep+0x31e kern_nanosleep(c5ee0b40,c57a1c64,c57a1c6c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c5ee0b40,c57a1cf8,8,c0c7d048,c0d5dd80,...) at nanosleep+0x6f syscall(c57a1d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28179c2f, esp = 0xbfbfec8c, ebp = 0xbfbfecb8 --- Tracing command sendmail pid 1449 tid 100064 td 0xc5ee0d80 sched_switch(c5ee0d80,0,104,191,22a85a16,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,68,...) at mi_switch+0x200 sleepq_switch(c5ee0d80,0,c0c7a69a,18b,68,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59deaf8,68,c0c2a55d,100,0,...) at sleepq_wait_sig+0x17 _sleep(c59deaf8,c59deb28,168,c0c2a55d,0,...) at _sleep+0x354 kern_sigsuspend(c5ee0d80,0,0,0,0,...) at kern_sigsuspend+0xe4 sigsuspend(c5ee0d80,c579dcf8,4,c0c7cdfa,c0d5e88c,...) at sigsuspend+0x4d syscall(c579dd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28332a3b, esp = 0xbfbfcf9c, ebp = 0xbfbfcfc8 --- Tracing command sendmail pid 1443 tid 100067 td 0xc5ee06c0 sched_switch(c5ee06c0,0,104,191,ecb7afa2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee06c0,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c57b0a4c,c087111a,c5e5cb50,0,c5ee06c0,...) at sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c5e5cb64,0,c57b0a7c,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5e5cb64,c5e5cb50,1389,5d1,c5ed3bd0,...) at _cv_timedwait_sig+0x250 seltdwait(c57b0c28,c57b0c30,c60ff500,c5ee06c0,c188b014,...) at seltdwait+0x8a kern_select(c5ee06c0,5,bfbfc520,0,0,c57b0c70,20,5,0) at kern_select+0x4f4 select(c5ee06c0,c57b0cf8,14,c0c7d31e,c0d5cd6c,...) at select+0x66 syscall(c57b0d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283d81d3, esp = 0xbfbfc48c, ebp = 0xbfbfcfb8 --- Tracing command sshd pid 1436 tid 100071 td 0xc5eded80 sched_switch(c5eded80,0,104,191,4d9d04fe,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5eded80,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c087111a,c5f2d850,0,c0c74a49,c5eded80,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2d864,0,c57c0a7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2d864,c5f2d850,c0c7c95a,5d1,c5ed3850,...) at _cv_wait_sig+0x240 seltdwait(c5ed3850,58,c6100b00,c5eded80,0,...) at seltdwait+0xa2 kern_select(c5eded80,5,286090ac,0,0,0,20,65,a) at kern_select+0x4f4 select(c5eded80,c57c0cf8,14,c0c7d770,c0d5cd6c,...) at select+0x66 syscall(c57c0d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfdf1c, ebp = 0xbfbfee38 --- Tracing command syslogd pid 1206 tid 100074 td 0xc5ede6c0 sched_switch(c5ede6c0,0,104,191,b65fba92,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede6c0,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c087111a,c5e7bc10,0,c0c74a49,c5ede6c0,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5e7bc24,0,c57caa7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e7bc24,c5e7bc10,c0c7c95a,5d1,c5ed32d8,...) at _cv_wait_sig+0x240 seltdwait(c5ed32d8,58,c596b100,c5ede6c0,0,...) at seltdwait+0xa2 kern_select(c5ede6c0,9,282290a8,0,0,0,20,65,281a4ad8) at kern_select+0x4f4 select(c5ede6c0,c57cacf8,14,c0c7cefa,c0d5cd6c,...) at select+0x66 syscall(c57cad38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281921d3, esp = 0xbfbfde7c, ebp = 0xbfbfee28 --- Tracing command dhclient pid 1157 tid 100068 td 0xc5ee0480 sched_switch(c5ee0480,0,104,191,125256de,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0480,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c57b4aa8,c087111a,c5e7cdd0,0,c5ee0480,...) at sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c5e7cde4,0,c57b4ad8,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5e7cde4,c5e7cdd0,5265431,5d1,c57b4b64,...) at _cv_timedwait_sig+0x250 seltdwait(c57b4c5c,c57b4c64,4df,c5ee0480,c57b4b5c,...) at seltdwait+0x8a poll(c5ee0480,c57b4cf8,c,c0c15d47,c0d5da1c,...) at poll+0x300 syscall(c57b4d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command dhclient pid 1120 tid 100073 td 0xc5ede900 sched_switch(c5ede900,0,104,191,8b5d5aba,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede900,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c087111a,c5f2e5d0,0,c0c74a49,c5ede900,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2e5e4,0,c57c7ad8,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2e5e4,c5f2e5d0,c0c7c95a,5d1,c57c7b5c,...) at _cv_wait_sig+0x240 seltdwait(c5f6692c,c0c7c95a,4df,c5ede900,c57c7b5c,...) at seltdwait+0xa2 poll(c5ede900,c57c7cf8,c,c57c7cb8,c0d5da1c,...) at poll+0x300 syscall(c57c7d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command devd pid 971 tid 100072 td 0xc5edeb40 sched_switch(c5edeb40,0,104,191,7c9b5d12,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5edeb40,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c087111a,c5f2d650,0,c0c74a49,c5edeb40,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2d664,0,c57c4a7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2d664,c5f2d650,c0c7c95a,5d1,c5ed34d0,...) at _cv_wait_sig+0x240 seltdwait(c5ed34d0,58,c596b100,c5edeb40,c0c6e5c4,...) at seltdwait+0xa2 kern_select(c5edeb40,6,bfbfe9b0,0,0,0,20,65,10) at kern_select+0x4f4 select(c5edeb40,c57c4cf8,14,c0c15d47,c0d5cd6c,...) at select+0x66 syscall(c57c4d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x80880e3, esp = 0xbfbfe97c, ebp = 0xbfbfee58 --- Tracing command moused pid 954 tid 100066 td 0xc5ee0900 sched_switch(c5ee0900,0,104,191,5e2c030e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0900,0,c0c7a69a,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c087111a,c5f2ed50,0,c0c74a49,c5ee0900,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2ed64,0,c57aca7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2ed64,c5f2ed50,c0c7c95a,5d1,c5ed3000,...) at _cv_wait_sig+0x240 seltdwait(c5ed3000,58,c596b100,c5ee0900,0,...) at seltdwait+0xa2 kern_select(c5ee0900,7,bfbfea14,0,0,0,20,65,0) at kern_select+0x4f4 select(c5ee0900,c57accf8,14,c0c8e3e7,c0d5cd6c,...) at select+0x66 syscall(c57acd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281ac1d3, esp = 0xbfbfe99c, ebp = 0xbfbfeb38 --- Tracing command flowcleaner pid 21 tid 100062 td 0xc5e56480 sched_switch(c5e56480,0,104,191,8f896f2e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e56480,0,c0c7a69a,26e,c5e56480,...) at sleepq_switch+0x15f sleepq_timedwait(c0f37888,0,c5791cc4,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c0f37888,c0f37890,2710,3ec,0,...) at _cv_timedwait+0x250 flowtable_cleaner(0,c5791d38,c0c717a4,343,c5ea1000,...) at flowtable_cleaner+0x1af fork_exit(c0927ec0,0,c5791d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5791d70, ebp = 0 --- Tracing command softdepflush pid 20 tid 100061 td 0xc5e566c0 sched_switch(c5e566c0,0,104,191,a95525e6,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200 sleepq_switch(c5e566c0,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f42f80,44,c0c9cf37,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f42f80,c0f42f24,44,c0c9cf37,3e8,...) at _sleep+0x339 softdep_flush(0,c578ed38,c0c717a4,343,c5ea12a8,...) at softdep_flush+0x244 fork_exit(c0ab1150,0,c578ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc578ed70, ebp = 0 --- Tracing command vnlru pid 19 tid 100060 td 0xc5e56900 sched_switch(c5e56900,0,104,191,a954e79e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,50,...) at mi_switch+0x200 sleepq_switch(c5e56900,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c5ea1550,50,c0c847ac,0,0,...) at sleepq_timedwait+0x6b _sleep(c5ea1550,c0f37654,250,c0c847ac,3e8,...) at _sleep+0x339 vnlru_proc(0,c578bd38,c0c717a4,343,c5ea1550,...) at vnlru_proc+0xe7 fork_exit(c09138c0,0,c578bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc578bd70, ebp = 0 --- Tracing command syncer pid 18 tid 100059 td 0xc5e56b40 sched_switch(c5e56b40,0,104,191,a954bed6,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e56b40,0,c0c7a69a,26e,c5e56b40,...) at sleepq_switch+0x15f sleepq_timedwait(c0f37694,0,c5788c88,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c0f37694,c0f37680,3e8,6cc,4e20,...) at _cv_timedwait+0x250 sched_sync(0,c5788d38,c0c717a4,343,c5ea17f8,...) at sched_sync+0x502 fork_exit(c0912cf0,0,c5788d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5788d70, ebp = 0 --- Tracing command bufdaemon pid 17 tid 100058 td 0xc5e56d80 sched_switch(c5e56d80,0,104,191,a954977e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200 sleepq_switch(c5e56d80,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f373c8,44,c0c81d22,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f373c8,c0f373cc,44,c0c81d22,3e8,...) at _sleep+0x339 buf_daemon(0,c5785d38,c0c717a4,343,c5ea1aa0,...) at buf_daemon+0x138 fork_exit(c08fb060,0,c5785d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5785d70, ebp = 0 --- Tracing command pagezero pid 16 tid 100057 td 0xc5e5b000 sched_switch(c5e5b000,0,104,191,8e2607de,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e5b000,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f43c54,0,c0ca2840,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f43c54,c0f43810,0,c0ca2840,493e0,...) at _sleep+0x339 vm_pagezero(0,c5782d38,c0c717a4,343,c5ea1d48,...) at vm_pagezero+0xdc fork_exit(c0aee490,0,c5782d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5782d70, ebp = 0 --- Tracing command vmdaemon pid 15 tid 100056 td 0xc5e5b240 sched_switch(c5e5b240,0,104,191,2a76132,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,68,...) at mi_switch+0x200 sleepq_switch(c5e5b240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c0f4387c,68,c0c81d22,0,0,...) at sleepq_wait+0x63 _sleep(c0f4387c,c0f43880,68,c0c81d22,0,...) at _sleep+0x36b vm_daemon(0,c577fd38,c0c717a4,343,c596e2a8,...) at vm_daemon+0x59 fork_exit(c0ae88e0,0,c577fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc577fd70, ebp = 0 --- Tracing command pagedaemon pid 9 tid 100055 td 0xc5e5b480 sched_switch(c5e5b480,0,104,191,5b168876,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200 sleepq_switch(c5e5b480,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f43844,44,c0c81d22,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f43844,c0f43810,44,c0c81d22,1388,...) at _sleep+0x339 vm_pageout(0,c577cd38,c0c717a4,343,c596e550,...) at vm_pageout+0x2bb fork_exit(c0ae9780,0,c577cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc577cd70, ebp = 0 --- Tracing command usb pid 14 tid 100054 td 0xc5e5b6c0 sched_switch(c5e5b6c0,0,104,191,5ff31610,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e5b6c0,0,c0c7a69a,24b,c5e5b6c0,...) at sleepq_switch+0x15f sleepq_wait(c5aead0c,0,c5779cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aead0c,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5aead04,c5779d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5aead04,c5779d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5779d70, ebp = 0 --- Tracing command usb pid 14 tid 100053 td 0xc5e5b900 sched_switch(c5e5b900,0,104,191,730c5dd2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e5b900,0,c0c7a69a,24b,c5e5b900,...) at sleepq_switch+0x15f sleepq_wait(c5aeacdc,0,c5776cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aeacdc,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5aeacd4,c5776d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5aeacd4,c5776d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5776d70, ebp = 0 --- Tracing command usb pid 14 tid 100052 td 0xc5e5bb40 sched_switch(c5e5bb40,0,104,191,a1274526,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e5bb40,0,c0c7a69a,24b,c5e5bb40,...) at sleepq_switch+0x15f sleepq_wait(c5aeacac,0,c5773cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aeacac,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5aeaca4,c5773d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5aeaca4,c5773d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5773d70, ebp = 0 --- Tracing command usb pid 14 tid 100051 td 0xc5e5bd80 sched_switch(c5e5bd80,0,104,191,5a5f3bf8,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e5bd80,0,c0c7a69a,24b,c5e5bd80,...) at sleepq_switch+0x15f sleepq_wait(c5aeac7c,0,c5770cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aeac7c,c5aeadac,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5aeac74,c5770d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5aeac74,c5770d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5770d70, ebp = 0 --- Tracing command usb pid 14 tid 100050 td 0xc5b5f6c0 sched_switch(c5b5f6c0,0,104,191,5a5f14f4,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b5f6c0,0,c0c7a69a,24b,c5b5f6c0,...) at sleepq_switch+0x15f sleepq_wait(c5ad7dac,0,c576dcb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ad7dac,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ad7da4,c576dd38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ad7da4,c576dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc576dd70, ebp = 0 --- Tracing command usb pid 14 tid 100049 td 0xc5b5f900 sched_switch(c5b5f900,0,104,191,6ef6c8da,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b5f900,0,c0c7a69a,24b,c5b5f900,...) at sleepq_switch+0x15f sleepq_wait(c5ad7d7c,0,c576acb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ad7d7c,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ad7d74,c576ad38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ad7d74,c576ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc576ad70, ebp = 0 --- Tracing command usb pid 14 tid 100048 td 0xc5b5fb40 sched_switch(c5b5fb40,0,104,191,544dcdcc,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b5fb40,0,c0c7a69a,24b,c5b5fb40,...) at sleepq_switch+0x15f sleepq_wait(c5ad7d4c,0,c5767cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ad7d4c,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ad7d44,c5767d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ad7d44,c5767d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5767d70, ebp = 0 --- Tracing command usb pid 14 tid 100047 td 0xc5b5fd80 sched_switch(c5b5fd80,0,104,191,544da8c4,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b5fd80,0,c0c7a69a,24b,c5b5fd80,...) at sleepq_switch+0x15f sleepq_wait(c5ad7d1c,0,c5764cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ad7d1c,c5ad7e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ad7d14,c5764d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ad7d14,c5764d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5764d70, ebp = 0 --- Tracing command usb pid 14 tid 100046 td 0xc5e55000 sched_switch(c5e55000,0,104,191,544d83f0,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e55000,0,c0c7a69a,24b,c5e55000,...) at sleepq_switch+0x15f sleepq_wait(c5ac5dac,0,c5761cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5dac,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ac5da4,c5761d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ac5da4,c5761d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5761d70, ebp = 0 --- Tracing command usb pid 14 tid 100045 td 0xc5e55240 sched_switch(c5e55240,0,104,191,6e88eef6,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e55240,0,c0c7a69a,24b,c5e55240,...) at sleepq_switch+0x15f sleepq_wait(c5ac5d7c,0,c575ecb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5d7c,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ac5d74,c575ed38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ac5d74,c575ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc575ed70, ebp = 0 --- Tracing command usb pid 14 tid 100044 td 0xc5e55480 sched_switch(c5e55480,0,104,191,4e39af0c,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e55480,0,c0c7a69a,24b,c5e55480,...) at sleepq_switch+0x15f sleepq_wait(c5ac5d4c,0,c575bcb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5d4c,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ac5d44,c575bd38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ac5d44,c575bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc575bd70, ebp = 0 --- Tracing command usb pid 14 tid 100043 td 0xc5e556c0 sched_switch(c5e556c0,0,104,191,4e398d00,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e556c0,0,c0c7a69a,24b,c5e556c0,...) at sleepq_switch+0x15f sleepq_wait(c5ac5d1c,0,c5758cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5d1c,c5ac5e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ac5d14,c5758d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ac5d14,c5758d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5758d70, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc5e55900 sched_switch(c5e55900,0,104,191,6f1cda86,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e55900,0,c0c7a69a,24b,c5e55900,...) at sleepq_switch+0x15f sleepq_wait(c5ab0dac,0,c5755cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ab0dac,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ab0da4,c5755d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ab0da4,c5755d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5755d70, ebp = 0 --- Tracing command usb pid 14 tid 100041 td 0xc5e55b40 sched_switch(c5e55b40,0,104,191,6f1d1da2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e55b40,0,c0c7a69a,24b,c5e55b40,...) at sleepq_switch+0x15f sleepq_wait(c5ab0d7c,0,c5752cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ab0d7c,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ab0d74,c5752d38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ab0d74,c5752d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5752d70, ebp = 0 --- Tracing command usb pid 14 tid 100040 td 0xc5e55d80 sched_switch(c5e55d80,0,104,191,5e2bc3d2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e55d80,0,c0c7a69a,24b,c5e55d80,...) at sleepq_switch+0x15f sleepq_wait(c5ab0d4c,0,c574fcb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ab0d4c,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ab0d44,c574fd38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ab0d44,c574fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc574fd70, ebp = 0 --- Tracing command usb pid 14 tid 100039 td 0xc5e56000 sched_switch(c5e56000,0,104,191,3a0de216,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5e56000,0,c0c7a69a,24b,c5e56000,...) at sleepq_switch+0x15f sleepq_wait(c5ab0d1c,0,c574ccb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ab0d1c,c5ab0e4c,c0c650ce,6a,c0dceb40,...) at _cv_wait+0x240 usb_process(c5ab0d14,c574cd38,c0c717a4,343,c596e7f8,...) at usb_process+0x193 fork_exit(c07b7150,c5ab0d14,c574cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc574cd70, ebp = 0 --- Tracing command sctp_iterator pid 8 tid 100038 td 0xc5a85d80 sched_switch(c5a85d80,0,104,191,804aa3ce,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5a85d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c0f3917c,0,c0c8fb50,0,0,...) at sleepq_wait+0x63 _sleep(c0f3917c,c0f39090,0,c0c8fb50,0,...) at _sleep+0x36b sctp_iterator_thread(0,c5749d38,c0c717a4,343,c596eaa0,...) at sctp_iterator_thread+0x60 fork_exit(c099c420,0,c5749d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5749d70, ebp = 0 --- Tracing command fdc0 pid 7 tid 100034 td 0xc5b5d6c0 sched_switch(c5b5d6c0,0,104,191,b1e97d3e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c5b5d6c0,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c59ec23c,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b _sleep(c59ec23c,c59ec2f0,4c,c0c6c015,3e8,...) at _sleep+0x339 fdc_thread(c59ec200,c573cd38,c0c717a4,343,c596ed48,...) at fdc_thread+0x2be fork_exit(c0b6ba00,c59ec200,c573cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc573cd70, ebp = 0 --- Tracing command fw0_probe pid 6 tid 100031 td 0xc5b5dd80 sched_switch(c5b5dd80,0,104,191,7c72a4a,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200 sleepq_switch(c5b5dd80,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5b62000,5c,c0c6c015,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5b62000,c5b66488,15c,c0c6c015,0,...) at _sleep+0x354 fw_bus_probe_thread(c5b62000,c5558d38,c0c717a4,343,c59de000,...) at fw_bus_probe_thread+0xa08 fork_exit(c0654500,c5b62000,c5558d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5558d70, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100015 td 0xc59716c0 sched_switch(c59716c0,0,104,191,481a07e0,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c59716c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c0d97a54,4c,c0c117a4,0,0,...) at sleepq_wait+0x63 _sleep(c0d97a54,c0d97a6c,4c,c0c117a4,0,...) at _sleep+0x36b xpt_scanner_thread(0,c54f2d38,c0c717a4,343,c59de2a8,...) at xpt_scanner_thread+0x4a fork_exit(c0482f30,0,c54f2d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54f2d70, ebp = 0 --- Tracing command yarrow pid 13 tid 100011 td 0xc59b5000 sched_switch(c59b5000,0,104,191,f525b556,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b5000,0,c0c7a69a,26e,2,...) at sleepq_switch+0x15f sleepq_timedwait(c0dcac64,0,c0c6c015,2,0,...) at sleepq_timedwait+0x6b _sleep(c0dcac64,0,0,c0c6c015,64,...) at _sleep+0x339 pause(c0c6c015,64,c0c58f7a,111,0,...) at pause+0x47 random_kthread(0,c54e6d38,c0c717a4,343,c59de550,...) at random_kthread+0x1ef fork_exit(c0730080,0,c54e6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54e6d70, ebp = 0 --- Tracing command g_down pid 4 tid 100009 td 0xc59b5480 sched_switch(c59b5480,0,104,191,f124675e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c59b5480,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dc8a24,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dc8a24,c0dc8988,24c,c0c6c015,64,...) at _sleep+0x339 g_io_schedule_down(c59b5480,0,c0c6d64b,74,0,...) at g_io_schedule_down+0x6b g_down_procbody(0,c54e0d38,c0c717a4,343,c596d000,...) at g_down_procbody+0x8d fork_exit(c081e2f0,0,c54e0d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54e0d70, ebp = 0 --- Tracing command g_up pid 3 tid 100008 td 0xc596f000 sched_switch(c596f000,0,104,191,f1248b8e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c596f000,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dc8a20,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dc8a20,c0dc89a8,24c,c0c6c015,64,...) at _sleep+0x339 g_io_schedule_up(c596f000,0,c0c6d64b,5d,0,...) at g_io_schedule_up+0x133 g_up_procbody(0,c54ddd38,c0c717a4,343,c596d2a8,...) at g_up_procbody+0x8d fork_exit(c081e380,0,c54ddd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54ddd70, ebp = 0 --- Tracing command g_event pid 2 tid 100007 td 0xc596f240 sched_switch(c596f240,0,104,191,f525e4ee,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c596f240,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dc8a18,4c,c0c6c015,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dc8a18,0,4c,c0c6c015,64,...) at _sleep+0x339 g_event_procbody(0,c54dad38,c0c717a4,343,c596d550,...) at g_event_procbody+0xcb fork_exit(c081e410,0,c54dad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54dad70, ebp = 0 --- Tracing command intr pid 12 tid 100037 td 0xc5b5d000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100036 td 0xc5b5d240 sched_switch(c5b5d240,0,109,191,f631f0a6,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c5c77170,...) at mi_switch+0x200 ithread_loop(c5c802f0,c5743d38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c5c802f0,c5743d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5743d70, ebp = 0 --- Tracing command intr pid 12 tid 100035 td 0xc5b5d480 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100033 td 0xc5b5d900 sched_switch(c5b5d900,0,109,191,59d95806,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b42f0,...) at mi_switch+0x200 ithread_loop(c5c6a630,c572fd38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c5c6a630,c572fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc572fd70, ebp = 0 --- Tracing command intr pid 12 tid 100032 td 0xc5b5db40 sched_switch(c5b5db40,0,109,191,479c27be,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b4370,...) at mi_switch+0x200 ithread_loop(c5c6a4a0,c56c4d38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c5c6a4a0,c56c4d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56c4d70, ebp = 0 --- Tracing command intr pid 12 tid 100029 td 0xc5b5f240 sched_switch(c5b5f240,0,109,191,47dd9b8c,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b41f0,...) at mi_switch+0x200 ithread_loop(c5b028e0,c554fd38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c5b028e0,c554fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc554fd70, ebp = 0 --- Tracing command intr pid 12 tid 100027 td 0xc59b56c0 sched_switch(c59b56c0,0,109,191,e51b0c76,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b3ef0,...) at mi_switch+0x200 ithread_loop(c5b02980,c5546d38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c5b02980,c5546d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5546d70, ebp = 0 --- Tracing command intr pid 12 tid 100026 td 0xc59b5900 sched_switch(c59b5900,0,109,191,a1270286,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b3df0,...) at mi_switch+0x200 ithread_loop(c5aa95c0,c552ed38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c5aa95c0,c552ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc552ed70, ebp = 0 --- Tracing command intr pid 12 tid 100025 td 0xc59b5b40 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100024 td 0xc59b5d80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xc5a85000 sched_switch(c5a85000,0,109,191,6f1c989a,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b4270,...) at mi_switch+0x200 ithread_loop(c5aac820,c5521d38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c5aac820,c5521d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5521d70, ebp = 0 --- Tracing command intr pid 12 tid 100022 td 0xc5a85240 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100020 td 0xc5a856c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100016 td 0xc5971480 sched_switch(c5971480,0,109,191,a12a276a,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c5a83970,...) at mi_switch+0x200 ithread_loop(c590a980,c54f5d38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c590a980,c54f5d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54f5d70, ebp = 0 --- Tracing command intr pid 12 tid 100013 td 0xc5971b40 sched_switch(c5971b40,0,109,191,8a3a33da,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b3070,...) at mi_switch+0x200 ithread_loop(c590a9b0,c54ecd38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c590a9b0,c54ecd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54ecd70, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc5971d80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100006 td 0xc596f480 sched_switch(c596f480,0,109,191,fc4c5766,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b3c70,...) at mi_switch+0x200 ithread_loop(c596c0c0,c54d7d38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c596c0c0,c54d7d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54d7d70, ebp = 0 --- Tracing command intr pid 12 tid 100005 td 0xc596f6c0 sched_switch(c596f6c0,0,109,191,4b847aea,...) at sched_switch+0x406 mi_switch(109,0,c0c71a2b,4f6,c59b3cf0,...) at mi_switch+0x200 ithread_loop(c596c0d0,c54d4d38,c0c717a4,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c0859f70,c596c0d0,c54d4d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54d4d70, ebp = 0 --- Tracing command intr pid 12 tid 100004 td 0xc596f900 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xc596fb40 kdb_enter(c0c1f62f,c0c5fa9d,80000000,c5afd380,0,...) at kdb_enter+0x3a uart_intr(c5afd300,c596fb40,c5953cd0,c596ad80,4,...) at uart_intr+0x126 intr_event_handle(c596ad80,c54cdc34,0,1f4,c5c7d000,...) at intr_event_handle+0x5c intr_execute_handlers(c5953cd0,c54cdc34,0,c54cdc74,c0b96044,...) at intr_execute_handlers+0x49 lapic_handle_intr(3b,c54cdc34) at lapic_handle_intr+0x4c Xapic_isr1() at Xapic_isr1+0x34 --- interrupt, eip = 0xc0b8a525, esp = 0xc54cdc74, ebp = 0xc54cdc74 --- acpi_cpu_c1(c54c0008,c54c0028,0,c596fb40,c0dceb40,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(c54cdcb4,c0ba194b,1,c54cdcf8,c08a31be,...) at acpi_cpu_idle+0x11c cpu_idle_acpi(1,c54cdcf8,c08a31be,1,0,...) at cpu_idle_acpi+0x1b cpu_idle(1,0,c0c77e8a,9e9,c596fb40,...) at cpu_idle+0x1b sched_idletd(0,c54cdd38,c0c717a4,343,c596daa0,...) at sched_idletd+0x23e fork_exit(c08a2f80,0,c54cdd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54cdd70, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc596fd80 sched_switch(c596fd80,0,104,191,6b6f0d6a,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,5c,...) at mi_switch+0x200 sleepq_switch(c596fd80,0,c0c7a69a,18b,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7a69a,15b,0,100,100,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c596dd48,5c,c0c7cf0b,100,0,...) at sleepq_wait_sig+0x17 _sleep(c596dd48,c596ddd0,15c,c0c7cf0b,0,...) at _sleep+0x354 kern_wait(c596fd80,ffffffff,c54c9c74,0,0,...) at kern_wait+0xb76 wait4(c596fd80,c54c9cf8,10,c0c7cd3b,c0d5c404,...) at wait4+0x3b syscall(c54c9d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b8f, esp = 0xbfbfe90c, ebp = 0xbfbfe928 --- Tracing command audit pid 10 tid 100001 td 0xc5971000 sched_switch(c5971000,0,104,191,48159710,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5971000,0,c0c7a69a,24b,c5971000,...) at sleepq_switch+0x15f sleepq_wait(c0f428a0,0,c54c6c9c,1,0,...) at sleepq_wait+0x63 _cv_wait(c0f428a0,c0f42884,c0c9a679,194,0,...) at _cv_wait+0x240 audit_worker(0,c54c6d38,c0c717a4,343,c596e000,...) at audit_worker+0x84 fork_exit(c0a84b10,0,c54c6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54c6d70, ebp = 0 --- Tracing command kernel pid 0 tid 100130 td 0xc673a000 sched_switch(c673a000,0,104,191,fea55740,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c673a000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(d4a446c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(d4a446c0,d4a446d8,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(ce9352f0,e8194d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,ce9352f0,e8194d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8194d70, ebp = 0 --- Tracing command kernel pid 0 tid 100115 td 0xc6122b40 sched_switch(c6122b40,0,104,191,9e7765e8,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6122b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6542280,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6542280,c6542298,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5b07c70,e8157d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5b07c70,e8157d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8157d70, ebp = 0 --- Tracing command kernel pid 0 tid 100113 td 0xc6545000 sched_switch(c6545000,0,104,191,a12b1c96,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6545000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541700,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541700,c6541718,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f3b0,e8151d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f3b0,e8151d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8151d70, ebp = 0 --- Tracing command kernel pid 0 tid 100112 td 0xc6545240 sched_switch(c6545240,0,104,191,96bcc54e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6545240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c65416c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c65416c0,c65416d8,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4e0,e814ed38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4e0,e814ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe814ed70, ebp = 0 --- Tracing command kernel pid 0 tid 100111 td 0xc6545480 sched_switch(c6545480,0,104,191,96bca382,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6545480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541680,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541680,c6541698,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5e9a1d0,e814bd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5e9a1d0,e814bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe814bd70, ebp = 0 --- Tracing command kernel pid 0 tid 100110 td 0xc65456c0 sched_switch(c65456c0,0,104,191,96bc803e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c65456c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c65412c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c65412c0,c65412d8,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f470,e8148d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f470,e8148d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8148d70, ebp = 0 --- Tracing command kernel pid 0 tid 100109 td 0xc6546480 sched_switch(c6546480,0,104,191,96bc5daa,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6546480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541280,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541280,c6541298,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5e9a700,e8145d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5e9a700,e8145d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8145d70, ebp = 0 --- Tracing command kernel pid 0 tid 100108 td 0xc6120000 sched_switch(c6120000,0,104,191,96bc39da,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541240,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541240,c6541258,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f520,e8142d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f520,e8142d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8142d70, ebp = 0 --- Tracing command kernel pid 0 tid 100107 td 0xc6120240 sched_switch(c6120240,0,104,191,a0dc8922,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541800,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541800,c6541818,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e813fd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f500,e813fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe813fd70, ebp = 0 --- Tracing command kernel pid 0 tid 100106 td 0xc6545900 sched_switch(c6545900,0,104,191,9fc3328a,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6545900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e813cd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e813cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe813cd70, ebp = 0 --- Tracing command kernel pid 0 tid 100105 td 0xc6545b40 sched_switch(c6545b40,0,104,191,9fc30fca,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6545b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8139d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e8139d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8139d70, ebp = 0 --- Tracing command kernel pid 0 tid 100104 td 0xc6545d80 sched_switch(c6545d80,0,104,191,9fc38dc2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6545d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8136d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e8136d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8136d70, ebp = 0 --- Tracing command kernel pid 0 tid 100103 td 0xc6546000 sched_switch(c6546000,0,104,191,9fc3aa6e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6546000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8133d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e8133d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8133d70, ebp = 0 --- Tracing command kernel pid 0 tid 100102 td 0xc6546240 sched_switch(c6546240,0,104,191,9fc37132,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6546240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8130d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e8130d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8130d70, ebp = 0 --- Tracing command kernel pid 0 tid 100101 td 0xc6120480 sched_switch(c6120480,0,104,191,9fc2dec2,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e812dd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e812dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe812dd70, ebp = 0 --- Tracing command kernel pid 0 tid 100100 td 0xc61206c0 sched_switch(c61206c0,0,104,191,9fc3cc86,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c61206c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e811bd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e811bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe811bd70, ebp = 0 --- Tracing command kernel pid 0 tid 100099 td 0xc6120900 sched_switch(c6120900,0,104,191,9fc35486,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541840,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541840,c6541858,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8118d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4f0,e8118d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8118d70, ebp = 0 --- Tracing command kernel pid 0 tid 100098 td 0xc6547d80 sched_switch(c6547d80,0,104,191,4b17e030,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6547d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e8115d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e8115d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8115d70, ebp = 0 --- Tracing command kernel pid 0 tid 100097 td 0xc6547b40 sched_switch(c6547b40,0,104,191,4af7fcd8,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6547b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e812ad38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e812ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe812ad70, ebp = 0 --- Tracing command kernel pid 0 tid 100096 td 0xc65466c0 sched_switch(c65466c0,0,104,191,4adc90e8,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c65466c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e8127d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e8127d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8127d70, ebp = 0 --- Tracing command kernel pid 0 tid 100095 td 0xc6546900 sched_switch(c6546900,0,104,191,47f2818e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6546900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e8124d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e8124d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8124d70, ebp = 0 --- Tracing command kernel pid 0 tid 100094 td 0xc6546b40 sched_switch(c6546b40,0,104,191,47d25caa,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6546b40,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e8121d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e8121d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8121d70, ebp = 0 --- Tracing command kernel pid 0 tid 100093 td 0xc6546d80 sched_switch(c6546d80,0,104,191,4bdfd712,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6546d80,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e811ed38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e811ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe811ed70, ebp = 0 --- Tracing command kernel pid 0 tid 100092 td 0xc6547000 sched_switch(c6547000,0,104,191,4ba7157e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6547000,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e8112d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e8112d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8112d70, ebp = 0 --- Tracing command kernel pid 0 tid 100091 td 0xc6547240 sched_switch(c6547240,0,104,191,4b7c372e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6547240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6541880,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c6541880,c6541898,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e810fd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4d0,e810fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe810fd70, ebp = 0 --- Tracing command kernel pid 0 tid 100090 td 0xc6547480 sched_switch(c6547480,0,104,191,a4c861da,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6547480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c65418c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c65418c0,c65418d8,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f490,e810cd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f490,e810cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe810cd70, ebp = 0 --- Tracing command kernel pid 0 tid 100089 td 0xc65476c0 sched_switch(c65476c0,0,104,191,a128ad06,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c65476c0,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5e5d680,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c5e5d680,c5e5d698,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4c0,e815ad38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5f3f4c0,e815ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe815ad70, ebp = 0 --- Tracing command kernel pid 0 tid 100088 td 0xc6547900 sched_switch(c6547900,0,104,191,a0e7711e,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6547900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5e7c100,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c5e7c100,c5e7c118,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5c4d360,e815dd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5c4d360,e815dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe815dd70, ebp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xc5b5f000 sched_switch(c5b5f000,0,104,191,f94b86a6,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b5f000,0,c0c7a69a,24b,c5b5f000,...) at sleepq_switch+0x15f sleepq_wait(c5abeb80,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63 msleep_spin(c5abeb80,c5abeb98,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d taskqueue_thread_loop(c5b6649c,c5555d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0x94 fork_exit(c08baf50,c5b6649c,c5555d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5555d70, ebp = 0 --- Tracing command kernel pid 0 tid 100028 td 0xc5b5f480 sched_switch(c5b5f480,0,104,191,39b65fdc,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b5f480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5b05500,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c5b05500,c5b05518,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c5b5b074,c554bd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c5b5b074,c554bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc554bd70, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc5a85480 sched_switch(c5a85480,0,104,191,737dd5ea,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5a85480,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5a66e80,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c5a66e80,c5a66e98,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c0ddd5c8,c5504d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c0ddd5c8,c5504d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5504d70, ebp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xc5a85900 sched_switch(c5a85900,0,104,191,609f5440,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5a85900,0,c0c7a69a,24b,c5a85900,...) at sleepq_switch+0x15f sleepq_wait(c5a67240,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63 msleep_spin(c5a67240,c5a67258,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d taskqueue_thread_loop(c0d9a860,c54fed38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0x94 fork_exit(c08baf50,c0d9a860,c54fed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54fed70, ebp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xc5a85b40 sched_switch(c5a85b40,0,104,191,609f3204,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5a85b40,0,c0c7a69a,24b,c5a85b40,...) at sleepq_switch+0x15f sleepq_wait(c5a67240,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63 msleep_spin(c5a67240,c5a67258,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d taskqueue_thread_loop(c0d9a860,c54fbd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0x94 fork_exit(c08baf50,c0d9a860,c54fbd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54fbd70, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc5971240 sched_switch(c5971240,0,104,191,609f07ac,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5971240,0,c0c7a69a,24b,c5971240,...) at sleepq_switch+0x15f sleepq_wait(c5a67240,0,c0c76ece,c0c6c015,0,...) at sleepq_wait+0x63 msleep_spin(c5a67240,c5a67258,c0c6c015,0,c0c74a49,...) at msleep_spin+0x21d taskqueue_thread_loop(c0d9a860,c54f8d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0x94 fork_exit(c08baf50,c0d9a860,c54f8d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54f8d70, ebp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xc5971900 sched_switch(c5971900,0,104,191,609dbb5c,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5971900,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5a673c0,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c5a673c0,c5a673d8,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c0dc9384,c54efd38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c0dc9384,c54efd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54efd70, ebp = 0 --- Tracing command kernel pid 0 tid 100010 td 0xc59b5240 sched_switch(c59b5240,0,104,191,73b9f616,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b5240,0,c0c7a69a,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5955c40,0,c0c6c015,0,0,...) at sleepq_wait+0x63 _sleep(c5955c40,c5955c58,0,c0c6c015,0,...) at _sleep+0x36b taskqueue_thread_loop(c0ddc060,c54e3d38,c0c717a4,343,c0dc8ae0,...) at taskqueue_thread_loop+0xba fork_exit(c08baf50,c0ddc060,c54e3d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54e3d70, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc0dc8d90 sched_switch(c0dc8d90,0,104,191,8f890d82,...) at sched_switch+0x406 mi_switch(104,0,c0c7a69a,1d6,44,...) at mi_switch+0x200 sleepq_switch(c0dc8d90,0,c0c7a69a,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dc8ae0,44,c0c78703,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dc8ae0,0,44,c0c78703,2710,...) at _sleep+0x339 scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x23e mi_startup() at mi_startup+0x96 begin() at begin+0x2c ---------------------------------------------------------------------- TOP OF DMESG KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC1 #1: Sat Oct 24 18:12:41 UTC 2009 root@box8.ws.local:/usr/obj/usr/src/sys/WITHDDB WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2400.10-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 Features=0xbfebfbff Features2=0x4400 real memory = 2147483648 (2048 MB) avail memory = 2086248448 (1989 MB) ---------------------------------------------------------------------- KERNEL CONFIG include GENERIC ident WITHDDB options KDB options DDB options BREAK_TO_DEBUGGER makeoptions DEBUG=-g options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC