Date: Fri, 30 Oct 2009 19:29:42 -0600 From: Stef Walter <stef-list@memberwebs.com> To: Pawel Jakub Dawidek <pjd@FreeBSD.org> Cc: freebsd-fs@freebsd.org, stef@memberwebs.com Subject: Re: 8.0-RC2: ZFS deadlock with zfs receive Message-ID: <4AEB9306.5070501@memberwebs.com> In-Reply-To: <20091030232805.GA2996@garage.freebsd.pl> References: <804B79F6-27CE-40D2-8AB8-6FC378F448FA@sarenet.es> <4AEA0EAD.1050302@memberwebs.com> <20091030232805.GA2996@garage.freebsd.pl>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a multi-part message in MIME format. --------------070204000802060007010703 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Pawel Jakub Dawidek wrote: > On Thu, Oct 29, 2009 at 03:52:45PM -0600, Stef Walter wrote: >> Borja Marcos wrote: >>> I've been sending some alltraces to pjd about this easy to reproduce >>> problem. I am using zfs send/zfs receive to replicate a dataset from one >>> server to another. At 1 minute intervals, an incremental snapshot is >>> sent to update the dataset copy. If there is reading activity on the >>> dataset copy, a deadlock can happen rendering ZFS and all the FS >>> subsystem unusable. I've tried with 8.0RC2 and it still happens. > > I was able to reproduce it, but I don't have fix yet. Thanks for taking your time to look at it. Super appreciated. >> FWIW, another (or the same) zfs recv deadlock I've been trying to get to >> the bottom of: >> >> http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html > > Could you guys recompile your kernel after uncommenting line: > > #CFLAGS+=-DDEBUG=1 > > in sys/modules/zfs/Makefile? > > You should see panic on assertion instead of deadlock, I believe. Rebuilt with that change... You're right, now I get this assertion: login: panic: solaris assert: list_head(&osi->os_dnodes) == osi->os_meta_dnode (0xc9060488 == 0xca922000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c, line: 459 Full details and backtraces attached. Thanks again, Stef --------------070204000802060007010703 Content-Type: text/plain; name="zfs-recv-cancelled-deadlock-assertion.txt" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="zfs-recv-cancelled-deadlock-assertion.txt" http://lists.freebsd.org/pipermail/freebsd-fs/2009-October/006999.html ---------------------------------------------------------------------- The Assertion login: panic: solaris assert: list_head(&osi->os_dnodes) == osi->os_meta_dnode (0xc9060488 == 0xca922000), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c, line: 459 cpuid = 0 KDB: enter: panic [thread pid 1552 tid 100121 ] Stopped at kdb_enter+0x3a: movl $0,kdb_why db> bt Tracing pid 1552 tid 100121 td 0xc658f480 kdb_enter(c0c79511,c0c79511,c629f3cf,e81794d8,0,...) at kdb_enter+0x3a panic(c629f3cf,c62a3486,c9060488,0,c629f660,...) at panic+0x136 dmu_objset_evict(c7346200,c72b0200,1cb,0,c629cc75,...) at dmu_objset_evict+0x299 dsl_dataset_destroy(c7346200,c62a2ebd,16,e8179984,e8179944,...) at dsl_dataset_destroy+0x24c dmu_recv_abort_cleanup(e8179984,c62ad24a,1cb,c629c778,e817992c,...) at dmu_recv_abort_cleanup+0x2f dmu_recv_stream(e8179984,c5ed4690,e817999c,0,0,...) at dmu_recv_stream+0xa0e zfs_ioc_recv(caab8000,0,0,0,0,...) at zfs_ioc_recv+0x2d4 zfsdev_ioctl(c5f67300,cc285a1c,caab8000,3,c658f480,...) at zfsdev_ioctl+0x7d devfs_ioctl_f(c5ed4dc8,cc285a1c,caab8000,c981c280,c658f480,...) at devfs_ioctl_f+0xf8 kern_ioctl(c658f480,3,cc285a1c,caab8000,1000000,...) at kern_ioctl+0x1fd ioctl(c658f480,e8179cf8,c,c0c916cc,c0d5ff48,...) at ioctl+0x134 syscall(e8179d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfbcfc, ebp = 0xbfbfbd28 --- db> show pcpu cpuid = 0 dynamic pcpu = 0x68ac00 curthread = 0xc658f480: pid 1552 "zfs" curpcb = 0xe8179d90 fpcurthread = none idlethread = 0xc596fb40: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db> show allpcpu Current CPU: 0 cpuid = 0 dynamic pcpu = 0x68ac00 curthread = 0xc658f480: pid 1552 "zfs" curpcb = 0xe8179d90 fpcurthread = none idlethread = 0xc596fb40: pid 11 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: db> show locks exclusive sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7346294) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:745 shared sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7a29494) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:519 exclusive sx zfsvfs->z_online_recv_lock (zfsvfs->z_online_recv_lock) r = 0 (0xd2940198) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2484 db> show alllocks Process 1552 (zfs) thread 0xc658f480 (100121) exclusive sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7346294) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:745 shared sx ds->ds_rwlock (ds->ds_rwlock) r = 0 (0xc7a29494) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:519 exclusive sx zfsvfs->z_online_recv_lock (zfsvfs->z_online_recv_lock) r = 0 (0xd2940198) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ioctl.c:2484 db> show lockedvnods Locked vnodes db> alltrace Tracing command zfs pid 1552 tid 100121 td 0xc658f480 kdb_enter(c0c79511,c0c79511,c629f3cf,e81794d8,0,...) at kdb_enter+0x3a panic(c629f3cf,c62a3486,c9060488,0,c629f660,...) at panic+0x136 dmu_objset_evict(c7346200,c72b0200,1cb,0,c629cc75,...) at dmu_objset_evict+0x299 dsl_dataset_destroy(c7346200,c62a2ebd,16,e8179984,e8179944,...) at dsl_dataset_destroy+0x24c dmu_recv_abort_cleanup(e8179984,c62ad24a,1cb,c629c778,e817992c,...) at dmu_recv_abort_cleanup+0x2f dmu_recv_stream(e8179984,c5ed4690,e817999c,0,0,...) at dmu_recv_stream+0xa0e zfs_ioc_recv(caab8000,0,0,0,0,...) at zfs_ioc_recv+0x2d4 zfsdev_ioctl(c5f67300,cc285a1c,caab8000,3,c658f480,...) at zfsdev_ioctl+0x7d devfs_ioctl_f(c5ed4dc8,cc285a1c,caab8000,c981c280,c658f480,...) at devfs_ioctl_f+0xf8 kern_ioctl(c658f480,3,cc285a1c,caab8000,1000000,...) at kern_ioctl+0x1fd ioctl(c658f480,e8179cf8,c,c0c916cc,c0d5ff48,...) at ioctl+0x134 syscall(e8179d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (54, FreeBSD ELF32, ioctl), eip = 0x282211f3, esp = 0xbfbfbcfc, ebp = 0xbfbfbd28 --- Tracing command zfskern pid 1527 tid 100117 td 0xc61226c0 sched_switch(c61226c0,0,104,191,b9d041d4,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c61226c0,0,c0c7d8d5,26e,c61226c0,...) at sleepq_switch+0x15f sleepq_timedwait(c669addc,0,e8166c08,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c669addc,c669ad84,7530,1,c669addc,...) at _cv_timedwait+0x250 txg_thread_wait(7530,0,3b9aca00,0,202c7066,...) at txg_thread_wait+0xab txg_sync_thread(c669ac00,e8166d38,c0c749df,343,c611e000,...) at txg_sync_thread+0x480 fork_exit(c61fd6c0,c669ac00,e8166d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8166d70, ebp = 0 --- Tracing command zfskern pid 1527 tid 100116 td 0xc6122900 sched_switch(c6122900,0,104,191,b9d06478,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6122900,0,c0c7d8d5,24b,c6122900,...) at sleepq_switch+0x15f sleepq_wait(c669adec,0,e8162c38,1,0,...) at sleepq_wait+0x63 _cv_wait(c669adec,c669ad84,c669adf4,1,c669adec,...) at _cv_wait+0x240 txg_thread_wait(0,0,c62ad24a,18d,c08c441b,...) at txg_thread_wait+0x4f txg_quiesce_thread(c669ac00,e8162d38,c0c749df,343,c611e000,...) at txg_quiesce_thread+0xa0 fork_exit(c61fce20,c669ac00,e8162d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8162d70, ebp = 0 --- Tracing command zfskern pid 1527 tid 100114 td 0xc6122d80 sched_switch(c6122d80,0,104,191,b9c5a414,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c6122d80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c65882c8,4c,c62b6b3a,0,0,...) at sleepq_wait+0x63 _sleep(c65882c8,c65882dc,24c,c62b6b3a,0,...) at _sleep+0x36b vdev_geom_worker(c65882c0,e8156d38,c0c749df,343,c611e000,...) at vdev_geom_worker+0x124 fork_exit(c6251040,c65882c0,e8156d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8156d70, ebp = 0 --- Tracing command zfskern pid 1527 tid 100087 td 0xc6120b40 sched_switch(c6120b40,0,104,191,7710eb92,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120b40,0,c0c7d8d5,26e,c6120b40,...) at sleepq_switch+0x15f sleepq_timedwait(c62c2ff4,0,e810bc24,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c62c2ff4,c62c2fe0,3e8,1131,0,...) at _cv_timedwait+0x250 l2arc_feed_thread(0,e810bd38,c0c749df,343,c611e000,...) at l2arc_feed_thread+0x149 fork_exit(c61a4fa0,0,e810bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe810bd70, ebp = 0 --- Tracing command zfskern pid 1527 tid 100078 td 0xc6122000 sched_switch(c6122000,0,104,191,7711309a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6122000,0,c0c7d8d5,26e,c6122000,...) at sleepq_switch+0x15f sleepq_timedwait(c62babcc,0,c57eac80,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c62babcc,c62babb8,3e8,0,3e8,...) at _cv_timedwait+0x250 arc_reclaim_thread(0,c57ead38,c0c749df,343,c611e000,...) at arc_reclaim_thread+0x3b7 fork_exit(c61a8ac0,0,c57ead38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc57ead70, ebp = 0 --- Tracing command bash pid 1525 tid 100076 td 0xc5ede240 sched_switch(c5ede240,0,104,191,788fbd72,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5f2ae04,0,c0c77c84,c5ede240,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2ae70,0,c57e4b0c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2ae70,c5f2ae04,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c5f2ae00,c5f2ae70,c57e4c58,1,0,...) at tty_wait+0x71 ttydisc_read(c5f2ae00,c57e4c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c6137000,c57e4c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed4738,c57e4c58,c5f40300,0,c5ede240,...) at devfs_read_f+0x7e dofileread(c57e4c58,ffffffff,ffffffff,0,c5ed4738,...) at dofileread+0x96 kern_readv(c5ede240,0,c57e4c58,c57e4c78,1,...) at kern_readv+0x58 read(c5ede240,c57e4cf8,c,c5ede240,c0d5f9b4,...) at read+0x4f syscall(c57e4d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x8127b93, esp = 0xbfbfe03c, ebp = 0xbfbfe068 --- Tracing command sshd pid 1521 tid 100079 td 0xc6121d80 sched_switch(c6121d80,0,104,191,788df0ca,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5e58710,0,c0c77c84,c6121d80,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5e58724,0,c57eda7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5e58724,c5e58710,c0c7fb95,5d1,c5ed4770,...) at _cv_wait_sig+0x240 seltdwait(c5ed4770,58,c60ff600,c6121d80,200246,...) at seltdwait+0xa2 kern_select(c6121d80,9,286030c8,286030cc,0,0,20,65,280fdd50) at kern_select+0x4f4 select(c6121d80,c57edcf8,14,c0c61635,c0d6038c,...) at select+0x66 syscall(c57edd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfde5c, ebp = 0xbfbfdea8 --- Tracing command getty pid 1518 tid 100071 td 0xc5eded80 sched_switch(c5eded80,0,104,191,d92742d4,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5eded80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5c7f404,0,c0c77c84,c5eded80,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5c7f470,0,c57c0b0c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5c7f470,c5c7f404,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c5c7f400,c5c7f470,c57c0c58,1,0,...) at tty_wait+0x71 ttydisc_read(c5c7f400,c57c0c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5b00a00,c57c0c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3850,c57c0c58,c596b100,0,c5eded80,...) at devfs_read_f+0x7e dofileread(c57c0c58,ffffffff,ffffffff,0,c5ed3850,...) at dofileread+0x96 kern_readv(c5eded80,0,c57c0c58,c57c0c78,1,...) at kern_readv+0x58 read(c5eded80,c57c0cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c57c0d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedbc, ebp = 0xbfbfede8 --- Tracing command getty pid 1517 tid 100083 td 0xc6121480 sched_switch(c6121480,0,104,191,914789b2,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c6121480,c6121480,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59eca70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59eca70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59eca00,c59eca70,c57f9c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59eca00,c57f9c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85a00,c57f9c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3498,c57f9c58,c596b100,0,c6121480,...) at devfs_read_f+0x7e dofileread(c57f9c58,ffffffff,ffffffff,0,c5ed3498,...) at dofileread+0x96 kern_readv(c6121480,0,c57f9c58,c57f9c78,1,...) at kern_readv+0x58 read(c6121480,c57f9cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c57f9d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1516 tid 100077 td 0xc5ede000 sched_switch(c5ede000,0,104,191,9126c18e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede000,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c5ede000,c5ede000,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed270,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed270,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed200,c59ed270,c57e7c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed200,c57e7c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5b00100,c57e7c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3af0,c57e7c58,c596b100,0,c5ede000,...) at devfs_read_f+0x7e dofileread(c57e7c58,ffffffff,ffffffff,0,c5ed3af0,...) at dofileread+0x96 kern_readv(c5ede000,0,c57e7c58,c57e7c78,1,...) at kern_readv+0x58 read(c5ede000,c57e7cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c57e7d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1515 tid 100074 td 0xc5ede6c0 sched_switch(c5ede6c0,0,104,191,9105c10a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede6c0,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c5ede6c0,c5ede6c0,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59edc70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59edc70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59edc00,c59edc70,c57cac58,1,0,...) at tty_wait+0x71 ttydisc_read(c59edc00,c57cac58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85800,c57cac58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3930,c57cac58,c596b100,0,c5ede6c0,...) at devfs_read_f+0x7e dofileread(c57cac58,ffffffff,ffffffff,0,c5ed3930,...) at dofileread+0x96 kern_readv(c5ede6c0,0,c57cac58,c57cac78,1,...) at kern_readv+0x58 read(c5ede6c0,c57cacf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c57cad38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1514 tid 100081 td 0xc6121900 sched_switch(c6121900,0,104,191,90e4c0de,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c6121900,c6121900,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed670,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed670,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed600,c59ed670,c57f3c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed600,c57f3c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85700,c57f3c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3ce8,c57f3c58,c596b100,0,c6121900,...) at devfs_read_f+0x7e dofileread(c57f3c58,ffffffff,ffffffff,0,c5ed3ce8,...) at dofileread+0x96 kern_readv(c6121900,0,c57f3c58,c57f3c78,1,...) at kern_readv+0x58 read(c6121900,c57f3cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c57f3d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1513 tid 100063 td 0xc5b06240 sched_switch(c5b06240,0,104,191,90c3f93e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b06240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c5b06240,c5b06240,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ede70,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ede70,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ede00,c59ede70,c5799c58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ede00,c5799c58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85300,c5799c58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3620,c5799c58,c596b100,0,c5b06240,...) at devfs_read_f+0x7e dofileread(c5799c58,ffffffff,ffffffff,0,c5ed3620,...) at dofileread+0x96 kern_readv(c5b06240,0,c5799c58,c5799c78,1,...) at kern_readv+0x58 read(c5b06240,c5799cf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c5799d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1512 tid 100086 td 0xc6120d80 sched_switch(c6120d80,0,104,191,90a2bcba,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c6120d80,c6120d80,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed470,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed470,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed400,c59ed470,e80fcc58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed400,e80fcc58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85e00,e80fcc58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3bd0,e80fcc58,c596b100,0,c6120d80,...) at devfs_read_f+0x7e dofileread(e80fcc58,ffffffff,ffffffff,0,c5ed3bd0,...) at dofileread+0x96 kern_readv(c6120d80,0,e80fcc58,e80fcc78,1,...) at kern_readv+0x58 read(c6120d80,e80fccf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(e80fcd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1511 tid 100085 td 0xc6121000 sched_switch(c6121000,0,104,191,908225fa,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121000,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c6121000,c6121000,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c59ed870,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c59ed870,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c59ed800,c59ed870,c57ffc58,1,0,...) at tty_wait+0x71 ttydisc_read(c59ed800,c57ffc58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c85900,c57ffc58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3508,c57ffc58,c596b100,0,c6121000,...) at devfs_read_f+0x7e dofileread(c57ffc58,ffffffff,ffffffff,0,c5ed3508,...) at dofileread+0x96 kern_readv(c6121000,0,c57ffc58,c57ffc78,1,...) at kern_readv+0x58 read(c6121000,c57ffcf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c57ffd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command getty pid 1510 tid 100084 td 0xc6121240 sched_switch(c6121240,0,104,191,918e02aa,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6121240,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,c6121240,c6121240,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5c7f070,0,c0c81ed5,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5c7f070,c0dce270,c0c81880,4d5,0,...) at _cv_wait_sig+0x240 tty_wait(c5c7f000,c5c7f070,c57fcc58,1,0,...) at tty_wait+0x71 ttydisc_read(c5c7f000,c57fcc58,0,9e,0,...) at ttydisc_read+0x22c ttydev_read(c5c86000,c57fcc58,0,0,1,...) at ttydev_read+0xaa devfs_read_f(c5ed3700,c57fcc58,c596b100,0,c6121240,...) at devfs_read_f+0x7e dofileread(c57fcc58,ffffffff,ffffffff,0,c5ed3700,...) at dofileread+0x96 kern_readv(c6121240,0,c57fcc58,c57fcc78,1,...) at kern_readv+0x58 read(c6121240,c57fccf8,c,c0c916cc,c0d5f9b4,...) at read+0x4f syscall(c57fcd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x2818f253, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command cron pid 1459 tid 100067 td 0xc5ee06c0 sched_switch(c5ee06c0,0,104,191,7c0f71e0,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200 sleepq_switch(c5ee06c0,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(c351,c08bad00,c5ee06c0,0,100,...) at sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c0dceac4,5c,c0c7a7d5,100,0,...) at sleepq_timedwait_sig+0x1a _sleep(c0dceac4,0,15c,c0c7a7d5,c351,...) at _sleep+0x31e kern_nanosleep(c5ee06c0,c57b0c64,c57b0c6c,32,0,...) at kern_nanosleep+0xc1 nanosleep(c5ee06c0,c57b0cf8,8,c0c80283,c0d613a0,...) at nanosleep+0x6f syscall(c57b0d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28179c2f, esp = 0xbfbfec8c, ebp = 0xbfbfecb8 --- Tracing command sendmail pid 1453 tid 100069 td 0xc5ee0240 sched_switch(c5ee0240,0,104,191,61ca0cea,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,68,...) at mi_switch+0x200 sleepq_switch(c5ee0240,0,c0c7d8d5,18b,68,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f1a058,68,c0c2d13b,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5f1a058,c5f1a088,168,c0c2d13b,0,...) at _sleep+0x354 kern_sigsuspend(c5ee0240,0,0,0,0,...) at kern_sigsuspend+0xe4 sigsuspend(c5ee0240,c57b8cf8,4,c0c80035,c0d61eac,...) at sigsuspend+0x4d syscall(c57b8d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28332a3b, esp = 0xbfbfcf9c, ebp = 0xbfbfcfc8 --- Tracing command sendmail pid 1447 tid 100066 td 0xc5ee0900 sched_switch(c5ee0900,0,104,191,54c1b16c,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c57aca4c,c08738da,c5f2d650,0,c5ee0900,...) at sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c5f2d664,0,c57aca7c,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5f2d664,c5f2d650,1389,5d1,c5ed32a0,...) at _cv_timedwait_sig+0x250 seltdwait(c57acc28,c57acc30,c6100600,c5ee0900,c188b014,...) at seltdwait+0x8a kern_select(c5ee0900,5,bfbfc520,0,0,c57acc70,20,5,0) at kern_select+0x4f4 select(c5ee0900,c57accf8,14,c0c80559,c0d6038c,...) at select+0x66 syscall(c57acd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283d81d3, esp = 0xbfbfc48c, ebp = 0xbfbfcfb8 --- Tracing command sshd pid 1440 tid 100068 td 0xc5ee0480 sched_switch(c5ee0480,0,104,191,7cd515d2,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5f2d010,0,c0c77c84,c5ee0480,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2d024,0,c57b4a7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2d024,c5f2d010,c0c7fb95,5d1,c5ed34d0,...) at _cv_wait_sig+0x240 seltdwait(c5ed34d0,58,c6100000,c5ee0480,0,...) at seltdwait+0xa2 kern_select(c5ee0480,5,286090ac,0,0,0,20,65,a) at kern_select+0x4f4 select(c5ee0480,c57b4cf8,14,c0c809ab,c0d6038c,...) at select+0x66 syscall(c57b4d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283cc1d3, esp = 0xbfbfdf1c, ebp = 0xbfbfee38 --- Tracing command syslogd pid 1210 tid 100072 td 0xc5edeb40 sched_switch(c5edeb40,0,104,191,b0d575ee,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5edeb40,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5c4bd50,0,c0c77c84,c5edeb40,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5c4bd64,0,c57c4a7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5c4bd64,c5c4bd50,c0c7fb95,5d1,c5ed38c0,...) at _cv_wait_sig+0x240 seltdwait(c5ed38c0,58,c596b100,c5edeb40,0,...) at seltdwait+0xa2 kern_select(c5edeb40,9,282290a8,0,0,0,20,65,281a4ad8) at kern_select+0x4f4 select(c5edeb40,c57c4cf8,14,c0c80135,c0d6038c,...) at select+0x66 syscall(c57c4d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281921d3, esp = 0xbfbfde7c, ebp = 0xbfbfee28 --- Tracing command dhclient pid 1161 tid 100075 td 0xc5ede480 sched_switch(c5ede480,0,104,191,5075fdca,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede480,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c57cdaa8,c08738da,c5f2d4d0,0,c5ede480,...) at sleepq_catch_signals+0xb7 sleepq_timedwait_sig(c5f2d4e4,0,c57cdad8,101,0,...) at sleepq_timedwait_sig+0x1a _cv_timedwait_sig(c5f2d4e4,c5f2d4d0,5265819,5d1,c57cdb64,...) at _cv_timedwait_sig+0x250 seltdwait(c57cdc5c,c57cdc64,4df,c5ede480,c57cdb5c,...) at seltdwait+0x8a poll(c5ede480,c57cdcf8,c,c0c18927,c0d6103c,...) at poll+0x300 syscall(c57cdd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command dhclient pid 1120 tid 100064 td 0xc5ee0d80 sched_switch(c5ee0d80,0,104,191,c880666e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0d80,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5f2e950,0,c0c77c84,c5ee0d80,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2e964,0,c579dad8,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2e964,c5f2e950,c0c7fb95,5d1,c579db5c,...) at _cv_wait_sig+0x240 seltdwait(c5f6692c,c0c7fb95,4df,c5ee0d80,c579db5c,...) at seltdwait+0xa2 poll(c5ee0d80,c579dcf8,c,c579dcb8,c0d6103c,...) at poll+0x300 syscall(c579dd38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2813122f, esp = 0xbfbfedcc, ebp = 0xbfbfedf8 --- Tracing command devd pid 971 tid 100065 td 0xc5ee0b40 sched_switch(c5ee0b40,0,104,191,1d25f46e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ee0b40,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5f2de10,0,c0c77c84,c5ee0b40,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2de24,0,c57a1a7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2de24,c5f2de10,c0c7fb95,5d1,c5ed3380,...) at _cv_wait_sig+0x240 seltdwait(c5ed3380,58,c596b100,c5ee0b40,c0c717ff,...) at seltdwait+0xa2 kern_select(c5ee0b40,6,bfbfe9b0,0,0,0,20,c0dd2800,b) at kern_select+0x4f4 select(c5ee0b40,c57a1cf8,14,c57a1ca0,c0d6038c,...) at select+0x66 syscall(c57a1d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x80880e3, esp = 0xbfbfe97c, ebp = 0xbfbfee58 --- Tracing command moused pid 954 tid 100073 td 0xc5ede900 sched_switch(c5ede900,0,104,191,b66153a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ede900,0,c0c7d8d5,18b,0,...) at sleepq_switch+0x15f sleepq_catch_signals(c08738da,c5f2d810,0,c0c77c84,c5ede900,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5f2d824,0,c57c7a7c,101,0,...) at sleepq_wait_sig+0x17 _cv_wait_sig(c5f2d824,c5f2d810,c0c7fb95,5d1,c5ed3b28,...) at _cv_wait_sig+0x240 seltdwait(c5ed3b28,58,c596b100,c5ede900,0,...) at seltdwait+0xa2 kern_select(c5ede900,7,bfbfea14,0,0,0,20,65,0) at kern_select+0x4f4 select(c5ede900,c57c7cf8,14,c0c916cc,c0d6038c,...) at select+0x66 syscall(c57c7d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281ac1d3, esp = 0xbfbfe99c, ebp = 0xbfbfeb38 --- Tracing command flowcleaner pid 21 tid 100062 td 0xc5b06480 sched_switch(c5b06480,0,104,191,8776d332,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b06480,0,c0c7d8d5,26e,c5b06480,...) at sleepq_switch+0x15f sleepq_timedwait(c0f3b548,0,c5791cc4,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c0f3b548,c0f3b550,2710,3f0,0,...) at _cv_timedwait+0x250 flowtable_cleaner(0,c5791d38,c0c749df,343,c5e9b000,...) at flowtable_cleaner+0x1bf fork_exit(c092a660,0,c5791d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5791d70, ebp = 0 --- Tracing command softdepflush pid 20 tid 100061 td 0xc5b066c0 sched_switch(c5b066c0,0,104,191,9a91c59a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200 sleepq_switch(c5b066c0,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f46c40,44,c0ca021c,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f46c40,c0f46be4,44,c0ca021c,3e8,...) at _sleep+0x339 softdep_flush(0,c578ed38,c0c749df,343,c5e9b2a8,...) at softdep_flush+0x244 fork_exit(c0ab3a60,0,c578ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc578ed70, ebp = 0 --- Tracing command vnlru pid 19 tid 100060 td 0xc5b06900 sched_switch(c5b06900,0,104,191,9a9157a6,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,50,...) at mi_switch+0x200 sleepq_switch(c5b06900,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c5e9b550,50,c0c879e7,0,0,...) at sleepq_timedwait+0x6b _sleep(c5e9b550,c0f3b314,250,c0c879e7,3e8,...) at _sleep+0x339 vnlru_proc(0,c578bd38,c0c749df,343,c5e9b550,...) at vnlru_proc+0xe7 fork_exit(c0916060,0,c578bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc578bd70, ebp = 0 --- Tracing command syncer pid 18 tid 100059 td 0xc5b06b40 sched_switch(c5b06b40,0,104,191,77355a82,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b06b40,0,c0c7d8d5,26e,c5b06b40,...) at sleepq_switch+0x15f sleepq_timedwait(c0f3b354,0,c5788c88,1,0,...) at sleepq_timedwait+0x6b _cv_timedwait(c0f3b354,c0f3b340,3e8,6cc,4e20,...) at _cv_timedwait+0x250 sched_sync(0,c5788d38,c0c749df,343,c5e9b7f8,...) at sched_sync+0x502 fork_exit(c0915490,0,c5788d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5788d70, ebp = 0 --- Tracing command bufdaemon pid 17 tid 100058 td 0xc5b06d80 sched_switch(c5b06d80,0,104,191,9a911762,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200 sleepq_switch(c5b06d80,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f3b088,44,c0c84f5d,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f3b088,c0f3b08c,44,c0c84f5d,3e8,...) at _sleep+0x339 buf_daemon(0,c5785d38,c0c749df,343,c5e9baa0,...) at buf_daemon+0x138 fork_exit(c08fd800,0,c5785d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5785d70, ebp = 0 --- Tracing command pagezero pid 16 tid 100057 td 0xc5c82000 sched_switch(c5c82000,0,104,191,e6d3fcae,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5c82000,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f47914,0,c0ca5b25,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f47914,c0f474d0,0,c0ca5b25,493e0,...) at _sleep+0x339 vm_pagezero(0,c5782d38,c0c749df,343,c5e9bd48,...) at vm_pagezero+0xdc fork_exit(c0af0df0,0,c5782d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5782d70, ebp = 0 --- Tracing command vmdaemon pid 15 tid 100056 td 0xc5c82240 sched_switch(c5c82240,0,104,191,794357b6,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,68,...) at mi_switch+0x200 sleepq_switch(c5c82240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c0f4753c,68,c0c84f5d,0,0,...) at sleepq_wait+0x63 _sleep(c0f4753c,c0f47540,68,c0c84f5d,0,...) at _sleep+0x36b vm_daemon(0,c577fd38,c0c749df,343,c596e2a8,...) at vm_daemon+0x59 fork_exit(c0aeb240,0,c577fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc577fd70, ebp = 0 --- Tracing command pagedaemon pid 9 tid 100055 td 0xc5c82480 sched_switch(c5c82480,0,104,191,41e649ea,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200 sleepq_switch(c5c82480,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0f47504,44,c0c84f5d,0,0,...) at sleepq_timedwait+0x6b _sleep(c0f47504,c0f474d0,44,c0c84f5d,1388,...) at _sleep+0x339 vm_pageout(0,c577cd38,c0c749df,343,c596e550,...) at vm_pageout+0x2bb fork_exit(c0aec0e0,0,c577cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc577cd70, ebp = 0 --- Tracing command sctp_iterator pid 8 tid 100054 td 0xc5c826c0 sched_switch(c5c826c0,0,104,191,be4bf2fa,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5c826c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c0f3ce3c,0,c0c92e35,0,0,...) at sleepq_wait+0x63 _sleep(c0f3ce3c,c0f3cd50,0,c0c92e35,0,...) at _sleep+0x36b sctp_iterator_thread(0,c5779d38,c0c749df,343,c596e7f8,...) at sctp_iterator_thread+0x60 fork_exit(c099ed00,0,c5779d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5779d70, ebp = 0 --- Tracing command fdc0 pid 7 tid 100050 td 0xc5ad66c0 sched_switch(c5ad66c0,0,104,191,76ec9a8a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c5ad66c0,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c59ec23c,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b _sleep(c59ec23c,c59ec2f0,4c,c0c6f250,3e8,...) at _sleep+0x339 fdc_thread(c59ec200,c576cd38,c0c749df,343,c596eaa0,...) at fdc_thread+0x2be fork_exit(c0b6e360,c59ec200,c576cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc576cd70, ebp = 0 --- Tracing command fw0_probe pid 6 tid 100047 td 0xc5ad6d80 sched_switch(c5ad6d80,0,104,191,397c48de,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200 sleepq_switch(c5ad6d80,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c5b6e000,5c,c0c6f250,100,0,...) at sleepq_wait_sig+0x17 _sleep(c5b6e000,c5b72488,15c,c0c6f250,0,...) at _sleep+0x354 fw_bus_probe_thread(c5b6e000,c5588d38,c0c749df,343,c596ed48,...) at fw_bus_probe_thread+0xa08 fork_exit(c0654740,c5b6e000,c5588d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5588d70, ebp = 0 --- Tracing command usb pid 14 tid 100042 td 0xc5b05900 sched_switch(c5b05900,0,104,191,ab8bccb2,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b05900,0,c0c7d8d5,24b,c5b05900,...) at sleepq_switch+0x15f sleepq_wait(c5aedd0c,0,c555ecb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aedd0c,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aedd04,c555ed38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aedd04,c555ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc555ed70, ebp = 0 --- Tracing command usb pid 14 tid 100041 td 0xc5b05b40 sched_switch(c5b05b40,0,104,191,1e3c062e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b05b40,0,c0c7d8d5,24b,c5b05b40,...) at sleepq_switch+0x15f sleepq_wait(c5aedcdc,0,c555bcb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aedcdc,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aedcd4,c555bd38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aedcd4,c555bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc555bd70, ebp = 0 --- Tracing command usb pid 14 tid 100040 td 0xc5b05d80 sched_switch(c5b05d80,0,104,191,b9c1c834,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b05d80,0,c0c7d8d5,24b,c5b05d80,...) at sleepq_switch+0x15f sleepq_wait(c5aedcac,0,c5558cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aedcac,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aedca4,c5558d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aedca4,c5558d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5558d70, ebp = 0 --- Tracing command usb pid 14 tid 100039 td 0xc5b06000 sched_switch(c5b06000,0,104,191,8b9b48a0,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b06000,0,c0c7d8d5,24b,c5b06000,...) at sleepq_switch+0x15f sleepq_wait(c5aedc7c,0,c5555cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aedc7c,c5aeddac,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aedc74,c5555d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aedc74,c5555d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5555d70, ebp = 0 --- Tracing command usb pid 14 tid 100037 td 0xc5ad4000 sched_switch(c5ad4000,0,104,191,8b9b1c18,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad4000,0,c0c7d8d5,24b,c5ad4000,...) at sleepq_switch+0x15f sleepq_wait(c5adadac,0,c554dcb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5adadac,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5adada4,c554dd38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5adada4,c554dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc554dd70, ebp = 0 --- Tracing command usb pid 14 tid 100036 td 0xc5ad4240 sched_switch(c5ad4240,0,104,191,ddca875a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad4240,0,c0c7d8d5,24b,c5ad4240,...) at sleepq_switch+0x15f sleepq_wait(c5adad7c,0,c554acb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5adad7c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5adad74,c554ad38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5adad74,c554ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc554ad70, ebp = 0 --- Tracing command usb pid 14 tid 100035 td 0xc5ad4480 sched_switch(c5ad4480,0,104,191,8589b4b4,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad4480,0,c0c7d8d5,24b,c5ad4480,...) at sleepq_switch+0x15f sleepq_wait(c5adad4c,0,c5547cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5adad4c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5adad44,c5547d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5adad44,c5547d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5547d70, ebp = 0 --- Tracing command usb pid 14 tid 100034 td 0xc5ad46c0 sched_switch(c5ad46c0,0,104,191,85898ec8,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad46c0,0,c0c7d8d5,24b,c5ad46c0,...) at sleepq_switch+0x15f sleepq_wait(c5adad1c,0,c5544cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5adad1c,c5adae4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5adad14,c5544d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5adad14,c5544d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5544d70, ebp = 0 --- Tracing command usb pid 14 tid 100032 td 0xc5ad4b40 sched_switch(c5ad4b40,0,104,191,8589586c,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad4b40,0,c0c7d8d5,24b,c5ad4b40,...) at sleepq_switch+0x15f sleepq_wait(c5ac5dac,0,c553dcb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5dac,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5ac5da4,c553dd38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5ac5da4,c553dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc553dd70, ebp = 0 --- Tracing command usb pid 14 tid 100031 td 0xc5ad4d80 sched_switch(c5ad4d80,0,104,191,dd5c7726,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad4d80,0,c0c7d8d5,24b,c5ad4d80,...) at sleepq_switch+0x15f sleepq_wait(c5ac5d7c,0,c553acb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5d7c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5ac5d74,c553ad38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5ac5d74,c553ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc553ad70, ebp = 0 --- Tracing command usb pid 14 tid 100030 td 0xc5ad6000 sched_switch(c5ad6000,0,104,191,7f759cb0,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad6000,0,c0c7d8d5,24b,c5ad6000,...) at sleepq_switch+0x15f sleepq_wait(c5ac5d4c,0,c5537cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5d4c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5ac5d44,c5537d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5ac5d44,c5537d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5537d70, ebp = 0 --- Tracing command usb pid 14 tid 100029 td 0xc5ad6240 sched_switch(c5ad6240,0,104,191,7f757954,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5ad6240,0,c0c7d8d5,24b,c5ad6240,...) at sleepq_switch+0x15f sleepq_wait(c5ac5d1c,0,c5534cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5ac5d1c,c5ac5e4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5ac5d14,c5534d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5ac5d14,c5534d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5534d70, ebp = 0 --- Tracing command usb pid 14 tid 100027 td 0xc59b56c0 sched_switch(c59b56c0,0,104,191,de33bc1e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b56c0,0,c0c7d8d5,24b,c59b56c0,...) at sleepq_switch+0x15f sleepq_wait(c5aaedac,0,c552dcb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aaedac,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aaeda4,c552dd38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aaeda4,c552dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc552dd70, ebp = 0 --- Tracing command usb pid 14 tid 100026 td 0xc59b5900 sched_switch(c59b5900,0,104,191,de342e32,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b5900,0,c0c7d8d5,24b,c59b5900,...) at sleepq_switch+0x15f sleepq_wait(c5aaed7c,0,c552acb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aaed7c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aaed74,c552ad38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aaed74,c552ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc552ad70, ebp = 0 --- Tracing command usb pid 14 tid 100025 td 0xc59b5b40 sched_switch(c59b5b40,0,104,191,72ec5146,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b5b40,0,c0c7d8d5,24b,c59b5b40,...) at sleepq_switch+0x15f sleepq_wait(c5aaed4c,0,c5527cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aaed4c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aaed44,c5527d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aaed44,c5527d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5527d70, ebp = 0 --- Tracing command usb pid 14 tid 100024 td 0xc59b5d80 sched_switch(c59b5d80,0,104,191,72edde2a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b5d80,0,c0c7d8d5,24b,c59b5d80,...) at sleepq_switch+0x15f sleepq_wait(c5aaed1c,0,c5524cb8,1,0,...) at sleepq_wait+0x63 _cv_wait(c5aaed1c,c5aaee4c,c0c67ee9,6a,c0dd2800,...) at _cv_wait+0x240 usb_process(c5aaed14,c5524d38,c0c749df,343,c59d9000,...) at usb_process+0x193 fork_exit(c07b7c80,c5aaed14,c5524d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5524d70, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100018 td 0xc5a85b40 sched_switch(c5a85b40,0,104,191,795df7fc,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c5a85b40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c0d9b0d4,4c,c0c14384,0,0,...) at sleepq_wait+0x63 _sleep(c0d9b0d4,c0d9b0ec,4c,c0c14384,0,...) at _sleep+0x36b xpt_scanner_thread(0,c54fbd38,c0c749df,343,c59d92a8,...) at xpt_scanner_thread+0x4a fork_exit(c0483010,0,c54fbd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54fbd70, ebp = 0 --- Tracing command yarrow pid 13 tid 100011 td 0xc59b5000 sched_switch(c59b5000,0,104,191,d456c068,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b5000,0,c0c7d8d5,26e,2,...) at sleepq_switch+0x15f sleepq_timedwait(c0dce924,0,c0c6f250,2,0,...) at sleepq_timedwait+0x6b _sleep(c0dce924,0,0,c0c6f250,64,...) at _sleep+0x339 pause(c0c6f250,64,c0c5bcd5,111,0,...) at pause+0x47 random_kthread(0,c54e6d38,c0c749df,343,c59d9550,...) at random_kthread+0x1ef fork_exit(c07308f0,0,c54e6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54e6d70, ebp = 0 --- Tracing command g_down pid 4 tid 100009 td 0xc59b5480 sched_switch(c59b5480,0,104,191,d6a10fd4,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c59b5480,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dcc6e4,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dcc6e4,c0dcc648,24c,c0c6f250,64,...) at _sleep+0x339 g_io_schedule_down(c59b5480,0,c0c70886,74,0,...) at g_io_schedule_down+0x6b g_down_procbody(0,c54e0d38,c0c749df,343,c596d000,...) at g_down_procbody+0x8d fork_exit(c0820a20,0,c54e0d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54e0d70, ebp = 0 --- Tracing command g_up pid 3 tid 100008 td 0xc596f000 sched_switch(c596f000,0,104,191,d6a139cc,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c596f000,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dcc6e0,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dcc6e0,c0dcc668,24c,c0c6f250,64,...) at _sleep+0x339 g_io_schedule_up(c596f000,0,c0c70886,5d,0,...) at g_io_schedule_up+0x133 g_up_procbody(0,c54ddd38,c0c749df,343,c596d2a8,...) at g_up_procbody+0x8d fork_exit(c0820ab0,0,c54ddd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54ddd70, ebp = 0 --- Tracing command g_event pid 2 tid 100007 td 0xc596f240 sched_switch(c596f240,0,104,191,d4571590,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,4c,...) at mi_switch+0x200 sleepq_switch(c596f240,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dcc6d8,4c,c0c6f250,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dcc6d8,0,4c,c0c6f250,64,...) at _sleep+0x339 g_event_procbody(0,c54dad38,c0c749df,343,c596d550,...) at g_event_procbody+0xcb fork_exit(c0820b40,0,c54dad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54dad70, ebp = 0 --- Tracing command intr pid 12 tid 100053 td 0xc5c82900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100052 td 0xc5c82b40 sched_switch(c5c82b40,0,109,191,d922a5cc,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c5c79070,...) at mi_switch+0x200 ithread_loop(c5c842e0,c5773d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c5c842e0,c5773d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5773d70, ebp = 0 --- Tracing command intr pid 12 tid 100051 td 0xc5c82d80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100049 td 0xc5ad6900 sched_switch(c5ad6900,0,109,191,8409d254,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b42f0,...) at mi_switch+0x200 ithread_loop(c5c6c620,c575fd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c5c6c620,c575fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc575fd70, ebp = 0 --- Tracing command intr pid 12 tid 100048 td 0xc5ad6b40 sched_switch(c5ad6b40,0,109,191,766e7c92,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b4370,...) at mi_switch+0x200 ithread_loop(c5c6c490,c56f4d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c5c6c490,c56f4d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc56f4d70, ebp = 0 --- Tracing command intr pid 12 tid 100045 td 0xc5b05240 sched_switch(c5b05240,0,109,191,79566bac,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b41f0,...) at mi_switch+0x200 ithread_loop(c5b078e0,c557fd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c5b078e0,c557fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc557fd70, ebp = 0 --- Tracing command intr pid 12 tid 100043 td 0xc5b056c0 sched_switch(c5b056c0,0,109,191,78975ef2,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b3ef0,...) at mi_switch+0x200 ithread_loop(c5b07980,c5576d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c5b07980,c5576d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5576d70, ebp = 0 --- Tracing command intr pid 12 tid 100038 td 0xc5a85d80 sched_switch(c5a85d80,0,109,191,b9c1897c,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b3df0,...) at mi_switch+0x200 ithread_loop(c5aaa740,c5552d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c5aaa740,c5552d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5552d70, ebp = 0 --- Tracing command intr pid 12 tid 100033 td 0xc5ad4900 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100028 td 0xc5ad6480 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100023 td 0xc5a85000 sched_switch(c5a85000,0,109,191,de33447e,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b4270,...) at mi_switch+0x200 ithread_loop(c5aa9820,c5521d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c5aa9820,c5521d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5521d70, ebp = 0 --- Tracing command intr pid 12 tid 100022 td 0xc5a85240 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100020 td 0xc5a856c0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xc5a85900 sched_switch(c5a85900,0,109,191,b9c50404,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c5a838f0,...) at mi_switch+0x200 ithread_loop(c590aa40,c54fed38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c590aa40,c54fed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54fed70, ebp = 0 --- Tracing command intr pid 12 tid 100013 td 0xc5971b40 sched_switch(c5971b40,0,109,191,387ac886,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b3070,...) at mi_switch+0x200 ithread_loop(c590a9b0,c54ecd38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c590a9b0,c54ecd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54ecd70, ebp = 0 --- Tracing command intr pid 12 tid 100012 td 0xc5971d80 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100006 td 0xc596f480 sched_switch(c596f480,0,109,191,d7c526fc,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b3c70,...) at mi_switch+0x200 ithread_loop(c596c0c0,c54d7d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c596c0c0,c54d7d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54d7d70, ebp = 0 --- Tracing command intr pid 12 tid 100005 td 0xc596f6c0 sched_switch(c596f6c0,0,109,191,7b7ba1ea,...) at sched_switch+0x406 mi_switch(109,0,c0c74c66,4f6,c59b3cf0,...) at mi_switch+0x200 ithread_loop(c596c0d0,c54d4d38,c0c749df,343,c596d7f8,...) at ithread_loop+0x1f6 fork_exit(c085c730,c596c0d0,c54d4d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54d4d70, ebp = 0 --- Tracing command intr pid 12 tid 100004 td 0xc596f900 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xc596fb40 sched_switch(c596fb40,0,608,18c,b9c14e64,...) at sched_switch+0x406 mi_switch(608,0,c0c79c0b,cf,0,...) at mi_switch+0x200 critical_exit(c5953f7c,c596fb40,c5953f7c,c59b3d80,17,...) at critical_exit+0xa8 intr_event_handle(c59b3d80,c54cdc7c,c54cdc74,2710,c0dd2800,...) at intr_event_handle+0xba intr_execute_handlers(c5953f7c,c54cdc7c,0,c54cdcf8,c0b98a14,...) at intr_execute_handlers+0x49 lapic_handle_intr(34,c54cdc7c) at lapic_handle_intr+0x4c Xapic_isr1() at Xapic_isr1+0x34 --- interrupt, eip = 0xc08a594c, esp = 0xc54cdcbc, ebp = 0xc54cdcf8 --- sched_idletd(0,c54cdd38,c0c749df,343,c596daa0,...) at sched_idletd+0x21c fork_exit(c08a5730,0,c54cdd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54cdd70, ebp = 0 --- Tracing command init pid 1 tid 100002 td 0xc596fd80 sched_switch(c596fd80,0,104,191,7ccb36e6,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,5c,...) at mi_switch+0x200 sleepq_switch(c596fd80,0,c0c7d8d5,18b,5c,...) at sleepq_switch+0x15f sleepq_catch_signals(c0c7d8d5,15b,0,100,100,...) at sleepq_catch_signals+0xb7 sleepq_wait_sig(c596dd48,5c,c0c80146,100,0,...) at sleepq_wait_sig+0x17 _sleep(c596dd48,c596ddd0,15c,c0c80146,0,...) at _sleep+0x354 kern_wait(c596fd80,ffffffff,c54c9c74,0,0,...) at kern_wait+0xb76 wait4(c596fd80,c54c9cf8,10,c0c7ff76,c0d5fa24,...) at wait4+0x3b syscall(c54c9d38) at syscall+0x2b4 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x8054b8f, esp = 0xbfbfe90c, ebp = 0xbfbfe928 --- Tracing command audit pid 10 tid 100001 td 0xc5971000 sched_switch(c5971000,0,104,191,7959b734,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5971000,0,c0c7d8d5,24b,c5971000,...) at sleepq_switch+0x15f sleepq_wait(c0f46560,0,c54c6c9c,1,0,...) at sleepq_wait+0x63 _cv_wait(c0f46560,c0f46544,c0c9d95e,194,c0d67918,...) at _cv_wait+0x240 audit_worker(0,c54c6d38,c0c749df,343,c596e000,...) at audit_worker+0x84 fork_exit(c0a87420,0,c54c6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54c6d70, ebp = 0 --- Tracing command kernel pid 0 tid 100118 td 0xc6122240 sched_switch(c6122240,0,104,191,c19cab6c,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6122240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(d34c0cc0,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(d34c0cc0,d34c0cd8,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c72d5c80,e8170d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c72d5c80,e8170d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8170d70, ebp = 0 --- Tracing command kernel pid 0 tid 100115 td 0xc6122b40 sched_switch(c6122b40,0,104,191,7673f806,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6122b40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587940,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587940,c6587958,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5e6cc40,e8159d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5e6cc40,e8159d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8159d70, ebp = 0 --- Tracing command kernel pid 0 tid 100113 td 0xc658c000 sched_switch(c658c000,0,104,191,b9c61070,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658c000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588240,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588240,c6588258,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f390,e8153d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f390,e8153d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8153d70, ebp = 0 --- Tracing command kernel pid 0 tid 100112 td 0xc658c240 sched_switch(c658c240,0,104,191,6f062614,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658c240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588200,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588200,c6588218,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5b0c390,e8150d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5b0c390,e8150d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8150d70, ebp = 0 --- Tracing command kernel pid 0 tid 100111 td 0xc658c480 sched_switch(c658c480,0,104,191,6f0602b0,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658c480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588180,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588180,c6588198,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f450,e814dd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f450,e814dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe814dd70, ebp = 0 --- Tracing command kernel pid 0 tid 100110 td 0xc658c6c0 sched_switch(c658c6c0,0,104,191,6f05dd6c,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658c6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588140,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588140,c6588158,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4c0,e814ad38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4c0,e814ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe814ad70, ebp = 0 --- Tracing command kernel pid 0 tid 100109 td 0xc658d240 sched_switch(c658d240,0,104,191,6f0351a8,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658d240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588100,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588100,c6588118,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f460,e8147d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f460,e8147d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8147d70, ebp = 0 --- Tracing command kernel pid 0 tid 100108 td 0xc658d480 sched_switch(c658d480,0,104,191,6f032cf4,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658d480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c65880c0,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c65880c0,c65880d8,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f570,e8144d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f570,e8144d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8144d70, ebp = 0 --- Tracing command kernel pid 0 tid 100107 td 0xc6120000 sched_switch(c6120000,0,104,191,b97558e6,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588080,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588080,c6588098,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f510,e8141d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f510,e8141d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8141d70, ebp = 0 --- Tracing command kernel pid 0 tid 100106 td 0xc6120240 sched_switch(c6120240,0,104,191,b85eff06,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e813ed38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e813ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe813ed70, ebp = 0 --- Tracing command kernel pid 0 tid 100105 td 0xc658c900 sched_switch(c658c900,0,104,191,b85e93c2,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658c900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e813bd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e813bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe813bd70, ebp = 0 --- Tracing command kernel pid 0 tid 100104 td 0xc658cb40 sched_switch(c658cb40,0,104,191,b85f7bde,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658cb40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e8138d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e8138d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8138d70, ebp = 0 --- Tracing command kernel pid 0 tid 100103 td 0xc658cd80 sched_switch(c658cd80,0,104,191,b85f588e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658cd80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e8135d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e8135d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8135d70, ebp = 0 --- Tracing command kernel pid 0 tid 100102 td 0xc658d000 sched_switch(c658d000,0,104,191,b85f3b6a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658d000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e8132d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e8132d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8132d70, ebp = 0 --- Tracing command kernel pid 0 tid 100101 td 0xc6120480 sched_switch(c6120480,0,104,191,b85f1fb2,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e812fd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e812fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe812fd70, ebp = 0 --- Tracing command kernel pid 0 tid 100100 td 0xc61206c0 sched_switch(c61206c0,0,104,191,b85ec786,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c61206c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e8120d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e8120d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8120d70, ebp = 0 --- Tracing command kernel pid 0 tid 100099 td 0xc6120900 sched_switch(c6120900,0,104,191,b85f9ab6,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c6120900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6588000,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6588000,c6588018,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f500,e811dd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f500,e811dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe811dd70, ebp = 0 --- Tracing command kernel pid 0 tid 100098 td 0xc658ed80 sched_switch(c658ed80,0,104,191,89d1724a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658ed80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e811ad38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e811ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe811ad70, ebp = 0 --- Tracing command kernel pid 0 tid 100097 td 0xc658eb40 sched_switch(c658eb40,0,104,191,89a3cffe,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658eb40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8117d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e8117d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8117d70, ebp = 0 --- Tracing command kernel pid 0 tid 100096 td 0xc658d6c0 sched_switch(c658d6c0,0,104,191,89838012,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658d6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e812cd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e812cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe812cd70, ebp = 0 --- Tracing command kernel pid 0 tid 100095 td 0xc658d900 sched_switch(c658d900,0,104,191,8af49d86,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658d900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8129d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e8129d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8129d70, ebp = 0 --- Tracing command kernel pid 0 tid 100094 td 0xc658db40 sched_switch(c658db40,0,104,191,8ab29942,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658db40,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8126d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e8126d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8126d70, ebp = 0 --- Tracing command kernel pid 0 tid 100093 td 0xc658dd80 sched_switch(c658dd80,0,104,191,8a832bb8,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658dd80,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8123d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e8123d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8123d70, ebp = 0 --- Tracing command kernel pid 0 tid 100092 td 0xc658e000 sched_switch(c658e000,0,104,191,8a1a880e,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658e000,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8114d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e8114d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8114d70, ebp = 0 --- Tracing command kernel pid 0 tid 100091 td 0xc658e240 sched_switch(c658e240,0,104,191,89f70072,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658e240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e80,c6587e98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4f0,e8111d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4f0,e8111d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe8111d70, ebp = 0 --- Tracing command kernel pid 0 tid 100090 td 0xc658e480 sched_switch(c658e480,0,104,191,7c37d874,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658e480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e40,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e40,c6587e58,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f4d0,e810ed38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f4d0,e810ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe810ed70, ebp = 0 --- Tracing command kernel pid 0 tid 100089 td 0xc658e6c0 sched_switch(c658e6c0,0,104,191,b9c3495c,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658e6c0,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587e00,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587e00,c6587e18,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f490,e815cd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f490,e815cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe815cd70, ebp = 0 --- Tracing command kernel pid 0 tid 100088 td 0xc658e900 sched_switch(c658e900,0,104,191,b9838a5a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c658e900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c6587dc0,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c6587dc0,c6587dd8,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5f3f2b0,e815fd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5f3f2b0,e815fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe815fd70, ebp = 0 --- Tracing command kernel pid 0 tid 100046 td 0xc5b05000 sched_switch(c5b05000,0,104,191,d30d63e4,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b05000,0,c0c7d8d5,24b,c5b05000,...) at sleepq_switch+0x15f sleepq_wait(c5b08540,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63 msleep_spin(c5b08540,c5b08558,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d taskqueue_thread_loop(c5b7249c,c5585d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94 fork_exit(c08bd6f0,c5b7249c,c5585d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5585d70, ebp = 0 --- Tracing command kernel pid 0 tid 100044 td 0xc5b05480 sched_switch(c5b05480,0,104,191,6b34f380,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5b05480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5abec80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c5abec80,c5abec98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c5b61074,c557bd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c5b61074,c557bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc557bd70, ebp = 0 --- Tracing command kernel pid 0 tid 100021 td 0xc5a85480 sched_switch(c5a85480,0,104,191,aa033aea,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5a85480,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5a5ee80,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c5a5ee80,c5a5ee98,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c0de1288,c5504d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c0de1288,c5504d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc5504d70, ebp = 0 --- Tracing command kernel pid 0 tid 100017 td 0xc5971240 sched_switch(c5971240,0,104,191,91e1f390,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5971240,0,c0c7d8d5,24b,c5971240,...) at sleepq_switch+0x15f sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63 msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d taskqueue_thread_loop(c0d9dee0,c54f8d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94 fork_exit(c08bd6f0,c0d9dee0,c54f8d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54f8d70, ebp = 0 --- Tracing command kernel pid 0 tid 100016 td 0xc5971480 sched_switch(c5971480,0,104,191,91e1d060,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5971480,0,c0c7d8d5,24b,c5971480,...) at sleepq_switch+0x15f sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63 msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d taskqueue_thread_loop(c0d9dee0,c54f5d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94 fork_exit(c08bd6f0,c0d9dee0,c54f5d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54f5d70, ebp = 0 --- Tracing command kernel pid 0 tid 100015 td 0xc59716c0 sched_switch(c59716c0,0,104,191,91e1ab74,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59716c0,0,c0c7d8d5,24b,c59716c0,...) at sleepq_switch+0x15f sleepq_wait(c5a5f380,0,c0c7a109,c0c6f250,0,...) at sleepq_wait+0x63 msleep_spin(c5a5f380,c5a5f398,c0c6f250,0,c0c77c84,...) at msleep_spin+0x21d taskqueue_thread_loop(c0d9dee0,c54f2d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0x94 fork_exit(c08bd6f0,c0d9dee0,c54f2d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54f2d70, ebp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xc5971900 sched_switch(c5971900,0,104,191,91e061e0,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c5971900,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5a5f3c0,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c5a5f3c0,c5a5f3d8,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c0dcd044,c54efd38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c0dcd044,c54efd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54efd70, ebp = 0 --- Tracing command kernel pid 0 tid 100010 td 0xc59b5240 sched_switch(c59b5240,0,104,191,aa3f995a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,0,...) at mi_switch+0x200 sleepq_switch(c59b5240,0,c0c7d8d5,24b,0,...) at sleepq_switch+0x15f sleepq_wait(c5955c40,0,c0c6f250,0,0,...) at sleepq_wait+0x63 _sleep(c5955c40,c5955c58,0,c0c6f250,0,...) at _sleep+0x36b taskqueue_thread_loop(c0ddfd20,c54e3d38,c0c749df,343,c0dcc7a0,...) at taskqueue_thread_loop+0xba fork_exit(c08bd6f0,c0ddfd20,c54e3d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xc54e3d70, ebp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xc0dcca50 sched_switch(c0dcca50,0,104,191,8776839a,...) at sched_switch+0x406 mi_switch(104,0,c0c7d8d5,1d6,44,...) at mi_switch+0x200 sleepq_switch(c0dcca50,0,c0c7d8d5,26e,0,...) at sleepq_switch+0x15f sleepq_timedwait(c0dcc7a0,44,c0c7b93e,0,0,...) at sleepq_timedwait+0x6b _sleep(c0dcc7a0,0,44,c0c7b93e,2710,...) at _sleep+0x339 scheduler(0,141ec00,141ec00,141e000,1425000,...) at scheduler+0x23e mi_startup() at mi_startup+0x96 begin() at begin+0x2c ---------------------------------------------------------------------- Head of dmesg WARNING: console at 0xc0d5a000 has no name KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC2 #4: Sat Oct 31 02:28:58 UTC 2009 root@box8.ws.local:/usr/obj/usr/src/sys/WITHDDB WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 2.40GHz (2400.10-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf27 Stepping = 7 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4400<CNXT-ID,xTPR> real memory = 2147483648 (2048 MB) avail memory = 2086248448 (1989 MB) ---------------------------------------------------------------------- Kernel Config include GENERIC ident WITHDDB options KDB options DDB options BREAK_TO_DEBUGGER makeoptions DEBUG=-g options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC --------------070204000802060007010703--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4AEB9306.5070501>