From owner-freebsd-bugs@FreeBSD.ORG Thu Dec 27 13:20:02 2007 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E78BB16A41B for ; Thu, 27 Dec 2007 13:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D301F13C457 for ; Thu, 27 Dec 2007 13:20:02 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id lBRDK2Du006785 for ; Thu, 27 Dec 2007 13:20:02 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id lBRDK2R8006784; Thu, 27 Dec 2007 13:20:02 GMT (envelope-from gnats) Date: Thu, 27 Dec 2007 13:20:02 GMT Message-Id: <200712271320.lBRDK2R8006784@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org From: Per olof Ljungmark Cc: Subject: Re: kern/118044: panic: spin lock held too long X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Per olof Ljungmark List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Dec 2007 13:20:03 -0000 The following reply was made to PR kern/118044; it has been noted by GNATS. From: Per olof Ljungmark To: Maxime Henrion Cc: bug-followup@FreeBSD.org Subject: Re: kern/118044: panic: spin lock held too long Date: Thu, 27 Dec 2007 14:16:16 +0100 Maxime Henrion wrote: > Hi Per, > > > There is a suspicion that the problem you are experiencing might > be fixed by a patch that has just been committed to HEAD. Could > you please give it a try? It's a fairly small patch, you just need > to apply revision 1.125 of sys/net/route.c and revision 1.68 of > sys/net/route.h. > > Let me know if you need some help in doing this testing. Hi Maxime, Thank you for your kind attantion to this problem. Did as per your advise, result below. System was rebuilt from sources today + patched route.c/h. savecore: no dumps found Initial i386 initialization:. Additional ABI support:. Starting named. Setting date via ntp. lock order reversal: 1st 0xc58370d8 rtentry (rtentry) @ /usr/src/sys/net/route.c:196 2nd 0xc538d07c radix node head (radix node head) @ /usr/src/sys/net/route.c:692 KDB: stack backtrace: db_trace_self_wrapper(c0a9fc45,e5797a70,c078792e,c0aa2117,c538d07c,...) at db_trace_self_wrapper+0x26 kdb_backtrace(c0aa2117,c538d07c,c0aa2178,c0aa2178,c0aaaa4b,...) at kdb_backtrace+0x29 witness_checkorder(c538d07c,9,c0aaaa4b,2b4,c5214d58,...) at witness_checkorder+0x6de _mtx_lock_flags(c538d07c,0,c0aaaa4b,2b4,c0a9bb4b,...) at _mtx_lock_flags+0xbc rtrequest1(1,e5797b0c,e5797b3c,e5797b48,c08911ac,...) at rtrequest1+0x4f rtredirect(e5797bbc,e5797bac,0,16,e5797b9c,...) at rtredirect+0x126 icmp_input(c5738000,14,246,c0b9aca0,e5797c0c,...) at icmp_input+0x50f ip_input(c5738000,14e,800,c5250400,800,...) at ip_input+0x650 netisr_dispatch(2,c5738000,10,3,0,...) at netisr_dispatch+0x73 ether_demux(c5250400,c5738000,3,0,3,...) at ether_demux+0x1f1 ether_input(c5250400,c5738000,c0a6b43a,bc4,c5250400,...) at ether_input+0x37f bge_intr(c526d000,0,c0a999ef,471,c515e564,...) at bge_intr+0x7da ithread_loop(c5268640,e5797d38,c0a99763,2ea,c5263558,...) at ithread_loop+0x1b5 fork_exit(c07328a0,c5268640,e5797d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5797d70, ebp = 0 --- 27 Dec 14:09:38 ntpdate[648]: adjust time server 192.168.1.2 offset -0.408824 sec Clearing /tmp (X related). Starting local daemons:. Updating motd. Mounting late file systems:. Starting ntpd. postfix/postfix-script: starting the Postfix mail system Starting mysql. Performing sanity check on apache22 configuration: Syntax OK Starting apache22. Configuring syscons: keymap keyrate font8x16 font8x14 font8x8 blanktime. Starting sshd. Starting cron. Local package initialization:. Starting default moused:. Starting inetd. Starting background file system checks in 60 seconds. Thu Dec 27 14:09:43 CET 2007 FreeBSD/i386 (ruben.i.inter-sonic.com) (ttyd0) login: rtfree: 0xc5722bb8 has 1 refs Dec 27 14:10:19 ruben su: peo to root on /dev/ttyp0 rtfree: 0xc5722bb8 has 1 refs rtfree: 0xc5722bb8 has 1 refs rtfree: 0xc5722bb8 has 1 refs KDB: enter: Line break on console [thread pid 33 tid 100037 ] Stopped at kdb_enter+0x32: leave db> wh Tracing pid 33 tid 100037 td 0xc5214cc0 kdb_enter(c0a8e6e4,c536f8f8,c0c1b0e8,8,c536f88c,...) at kdb_enter+0x32 siointr1(c0c1b0e8,0,c0ac563d,56f,e57979dc,...) at siointr1+0x11c siointr(c536f800,c0bb0844,2,c5214cc0,c5111280,...) at siointr+0x32 intr_execute_handlers(c5107cd0,e57979fc,e5797a58,c09eb844,3b,...) at intr_execute_handlers+0xec lapic_handle_intr(3b,e57979fc) at lapic_handle_intr+0x3f Xapic_isr1() at Xapic_isr1+0x34 --- interrupt, eip = 0xc07429c2, esp = 0xe5797a3c, ebp = 0xe5797a58 --- _mtx_lock_sleep(c538d07c,c5214cc0,0,c0aaaa4b,93,...) at _mtx_lock_sleep+0xa2 _mtx_lock_flags(c538d07c,0,c0aaaa4b,93,c0a9bb4b,...) at _mtx_lock_flags+0xef rtalloc1(e5797bac,1,0,e5797b20,c07426fc,...) at rtalloc1+0x63 rtredirect(e5797bbc,e5797bac,0,26,e5797b9c,...) at rtredirect+0x35e icmp_input(c57d1700,14,246,c0b9aca0,e5797c0c,...) at icmp_input+0x50f ip_input(c57d1700,14e,800,c5250400,800,...) at ip_input+0x650 netisr_dispatch(2,c57d1700,10,3,0,...) at netisr_dispatch+0x73 ether_demux(c5250400,c57d1700,3,0,3,...) at ether_demux+0x1f1 ether_input(c5250400,c57d1700,c0a6b43a,bc4,c5250400,...) at ether_input+0x37f bge_intr(c526d000,0,c0a999ef,471,c515e564,...) at bge_intr+0x7da ithread_loop(c5268640,e5797d38,c0a99763,2ea,c5263558,...) at ithread_loop+0x1b5 fork_exit(c07328a0,c5268640,e5797d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5797d70, ebp = 0 --- db> ps pid ppid pgrp uid state wmesg wchan cmd 954 949 954 0 S+ ttywri 0xc5355838 dmesg 949 948 949 0 S+ pause 0xc56ef30c csh 948 944 948 1001 S+ wait 0xc5af0804 su 944 943 944 1001 Ss+ pause 0xc5540b10 csh 943 940 940 1001 R CPU 1 sshd 940 852 940 0 Ss sbwait 0xc58a8560 sshd 936 835 835 80 S accept 0xc58a866a httpd 935 835 835 80 S accept 0xc58a866a httpd 934 835 835 80 S accept 0xc58a866a httpd 933 835 835 80 S accept 0xc58a866a httpd 932 835 835 80 S accept 0xc58a866a httpd 931 1 931 0 Ss+ ttyin 0xc5357c10 getty 930 1 930 0 Ss+ ttyin 0xc538a810 getty 929 1 929 0 Ss+ ttyin 0xc538ac10 getty 928 1 928 0 Ss+ ttyin 0xc5389810 getty 927 1 927 0 Ss+ ttyin 0xc538b410 getty 926 1 926 0 Ss+ ttyin 0xc538c410 getty 925 1 925 0 Ss+ ttyin 0xc538c010 getty 924 1 924 0 Ss+ ttyin 0xc537fc10 getty 923 1 923 0 Ss+ ttyin 0xc5380410 getty 921 919 54 0 S+ nanslp 0xc0bb0824 sleep 920 1 54 0 S+ piperd 0xc586d948 logger 919 918 54 0 S+ wait 0xc559f2ac sh 918 1 54 0 S+ wait 0xc5598558 sh 902 1 902 0 Ss select 0xc0c01ddc inetd 875 1 875 0 Ss select 0xc0c01ddc moused 859 1 859 0 Ss nanslp 0xc0bb0824 cron 852 1 852 0 Ss select 0xc0c01ddc sshd 835 1 835 0 Ss select 0xc0c01ddc httpd 833 808 805 88 S+ (threaded) mysqld 100139 S ucond 0xc561f2c0 mysqld 100138 S sigwait 0xebe8cbe0 mysqld 100135 S ucond 0xc561f240 mysqld 100137 S select 0xc0c01ddc mysqld 100136 S select 0xc0c01ddc mysqld 100134 S ucond 0xc561f140 mysqld 100133 S ucond 0xc561f100 mysqld 100132 S ucond 0xc55ae740 mysqld 100131 S ucond 0xc55ae700 mysqld 100086 S select 0xc0c01ddc initial thread 808 1 805 88 S+ wait 0xc585fab0 sh 803 795 795 125 S kqread 0xc587da00 qmgr 802 795 795 125 S kqread 0xc55aab80 pickup 795 1 795 0 Ss kqread 0xc5564080 master 728 1 728 0 Ss select 0xc0c01ddc ntpd 643 1 643 53 Ss (threaded) named 100130 S select 0xc0c01ddc named 100129 S ucond 0xc555d240 named 100128 S ucond 0xc555d200 named 100127 S ucond 0xc555d1c0 named 100126 S ucond 0xc555d180 named 100125 S ucond 0xc555d140 named 100107 S sigwait 0xe9b14be0 named 578 1 578 0 Ss select 0xc0c01ddc syslogd 517 1 517 0 Ss select 0xc0c01ddc devd 321 1 321 65 Ss select 0xc0c01ddc dhclient 301 1 54 0 S+ select 0xc0c01ddc dhclient 53 0 0 0 SL sdflush 0xc0c0db44 [softdepflush] 52 0 0 0 SL syncer 0xc0bb064c [syncer] 51 0 0 0 SL vlruwt 0xc5374000 [vnlru] 50 0 0 0 SL psleep 0xc0c02244 [bufdaemon] 49 0 0 0 SL pgzero 0xc0c0e700 [pagezero] 48 0 0 0 SL psleep 0xc0c0e318 [vmdaemon] 47 0 0 0 SL psleep 0xc0c0e2e0 [pagedaemon] 46 0 0 0 SL waiting_ 0xc0c03fac [sctp_iterator] 45 0 0 0 WL [irq7: ppc0] 44 0 0 0 SL - 0xc531ce3c [fdc0] 43 0 0 0 WL [swi0: sio] 42 0 0 0 WL [irq12: psm0] 41 0 0 0 WL [irq1: atkbd0] 40 0 0 0 WL [irq15: ata1] 39 0 0 0 WL [irq14: ata0] 38 0 0 0 SL idle 0xc531e000 [aic_recovery1] 37 0 0 0 WL [irq22: ahd1] 36 0 0 0 SL idle 0xc52d5000 [aic_recovery0] 35 0 0 0 SL aifthd 0xc5263000 [aac0aif] 34 0 0 0 WL [irq18: aac0] 33 0 0 0 RL CPU 0 [irq21: bge0] 32 0 0 0 SL usbevt 0xc5210210 [usb2] 31 0 0 0 WL [irq23: ehci0] 30 0 0 0 SL usbevt 0xc5245210 [usb1] 29 0 0 0 WL [irq19: uhci1 ahd0] 28 0 0 0 SL usbtsk 0xc0bae2f4 [usbtask-dr] 27 0 0 0 SL usbtsk 0xc0bae2e0 [usbtask-hc] 26 0 0 0 SL usbevt 0xc524b210 [usb0] 25 0 0 0 WL [irq16: uhci0] 24 0 0 0 WL [irq9: acpi0] 23 0 0 0 SL - 0xc5212600 [kqueue taskq] 22 0 0 0 WL [swi6: task queue] 21 0 0 0 WL [swi6: Giant taskq] 9 0 0 0 SL - 0xc5212880 [thread taskq] 8 0 0 0 SL - 0xc5212900 [acpi_task_2] 7 0 0 0 SL - 0xc5212900 [acpi_task_1] 6 0 0 0 SL - 0xc5212900 [acpi_task_0] 20 0 0 0 WL [swi5: +] 19 0 0 0 WL [swi2: cambio] 5 0 0 0 SL ccb_scan 0xc0b7f514 [xpt_thrd] 18 0 0 0 SL - 0xc0bb0654 [yarrow] 4 0 0 0 SL - 0xc0bae6cc [g_down] 3 0 0 0 SL - 0xc0bae6c8 [g_up] 2 0 0 0 SL - 0xc0bae6c0 [g_event] 17 0 0 0 WL [swi3: vm] 16 0 0 0 RL CPU 3 [swi4: clock sio] 15 0 0 0 WL [swi1: net] 14 0 0 0 RL [idle: cpu0] 13 0 0 0 RL [idle: cpu1] 12 0 0 0 RL CPU 2 [idle: cpu2] 11 0 0 0 RL [idle: cpu3] 1 0 1 0 SLs wait 0xc5114ab0 [init] 10 0 0 0 SL audit_wo 0xc0c0d5b4 [audit] 0 0 0 0 WLs [swapper] db> show allpcpu Current CPU: 0 cpuid = 0 curthread = 0xc5214cc0: pid 33 "irq21: bge0" curpcb = 0xe5797d90 fpcurthread = none idlethread = 0xc5117000: pid 14 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: exclusive spin mutex sio r = 0 (0xc0c1b0e8) locked @ /usr/src/sys/dev/sio/sio.c:1391 cpuid = 1 curthread = 0xc5861440: pid 943 "sshd" curpcb = 0xebe1fd90 fpcurthread = none idlethread = 0xc5115cc0: pid 13 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 curthread = 0xc5115aa0: pid 12 "idle: cpu2" curpcb = 0xe3d32d90 fpcurthread = none idlethread = 0xc5115aa0: pid 12 "idle: cpu2" APIC ID = 6 currentldt = 0x50 spin locks held: cpuid = 3 curthread = 0xc5117cc0: pid 16 "swi4: clock sio" curpcb = 0xe3d4ad90 fpcurthread = none idlethread = 0xc5115880: pid 11 "idle: cpu3" APIC ID = 7 currentldt = 0x50 spin locks held: db> alltrace Tracing command dmesg pid 954 tid 100095 td 0xc56ee000 sched_switch(c56ee000,0,1,180,3869c968,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c56ee000,0,c0aa0842,19b,c5355838,...) at sleepq_switch+0xf0 sleepq_catch_signals(ebdafa70,c07423dd,c0bb8700,4,5a,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c5355838,0,c0a9d757,da,0,...) at sleepq_timedwait_sig+0x17 _sleep(c5355838,0,15a,c0aa420e,493e0,...) at _sleep+0x331 ttysleep(c5355800,c5355838,15a,c0aa420e,493e0,...) at ttysleep+0x39 ttwrite(c5355800,ebdafc60,0,ebdafbbc,c071e148,...) at ttwrite+0x5aa ptswrite(c5afb000,ebdafc60,0,174,41,...) at ptswrite+0x38 giant_write(c5afb000,ebdafc60,0,0,c0b56b60,...) at giant_write+0x48 devfs_write_f(c55a4438,ebdafc60,c5af9000,0,c56ee000,...) at devfs_write_f+0x98 dofilewrite(ebdafc60,ffffffff,ffffffff,0,c55a4438,...) at dofilewrite+0x95 kern_writev(c56ee000,1,ebdafc60,28201041,0,...) at kern_writev+0x58 write(c56ee000,ebdafcfc,c,c0a8d9fa,c0b49ae0,...) at write+0x4f syscall(ebdafd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip = 0x2815b423, esp = 0xbfbfe40c, ebp = 0xbfbfe428 --- Tracing command csh pid 949 tid 100096 td 0xc56e9cc0 sched_switch(c56e9cc0,0,1,180,371d3b46,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c56e9cc0,0,c0aa0842,19b,c56ef30c,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,68,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c56ef30c,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c56ef30c,c56ef33c,168,c0a5fd27,0,...) at _sleep+0x359 kern_sigsuspend(c56e9cc0,0,0,0,0,...) at kern_sigsuspend+0xe4 sigsuspend(c56e9cc0,ebdaccfc,4,c0aa2849,c0b4ba78,...) at sigsuspend+0x4d syscall(ebdacd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166efb, esp = 0xbfbfe73c, ebp = 0xbfbfe7a8 --- Tracing command su pid 948 tid 100143 td 0xc5861660 sched_switch(c5861660,0,1,180,d92ee264,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5861660,0,c0aa0842,19b,c5af0804,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c5af0804,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c5af0804,c5af0894,15c,c0aa296f,0,...) at _sleep+0x359 kern_wait(c5861660,3b5,ebe22c2c,2,0,...) at kern_wait+0xa66 wait4(c5861660,ebe22cfc,10,ebe22d38,c0b49b28,...) at wait4+0x3b syscall(ebe22d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x2810ad7b, esp = 0xbfbfe79c, ebp = 0xbfbfe7b8 --- Tracing command csh pid 944 tid 100059 td 0xc5265880 sched_switch(c5265880,0,1,180,77538768,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5265880,0,c0aa0842,19b,c5540b10,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,68,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c5540b10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c5540b10,c5540b40,168,c0a5fd27,0,...) at _sleep+0x359 kern_sigsuspend(c5265880,0,0,0,0,...) at kern_sigsuspend+0xe4 sigsuspend(c5265880,e580bcfc,4,c0aa2849,c0b4ba78,...) at sigsuspend+0x4d syscall(e580bd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (341, FreeBSD ELF32, sigsuspend), eip = 0x28166efb, esp = 0xbfbfe80c, ebp = 0xbfbfe878 --- Tracing command sshd pid 943 tid 100144 td 0xc5861440 cpustop_handler(2,ebe1f890,c0a04a50,38,c0aa17fa,...) at cpustop_handler+0x32 ipi_nmi_handler(38,c0aa17fa,6ce,c0c01c00,13,...) at ipi_nmi_handler+0x2f trap(ebe1f89c) at trap+0x30 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc07429c2, esp = 0xebe1f8dc, ebp = 0xebe1f8f8 --- _mtx_lock_sleep(c5722c18,c5861440,0,c0aaaa4b,c4,...) at _mtx_lock_sleep+0xa2 _mtx_lock_flags(c5722c18,0,c0aaaa4b,c4,c0786d22,...) at _mtx_lock_flags+0xef rtalloc1(ebe1fa04,1,0,100,ebe1fa00,...) at rtalloc1+0x225 rtalloc_ign(ebe1fa00,0,c0aaf933,7f,c5861440,...) at rtalloc_ign+0xe6 ip_output(c5893500,0,ebe1fa00,0,0,...) at ip_output+0x355 tcp_output(c588a570,c5bb3c00,c0ab3c29,30b,c58887a4,...) at tcp_output+0x13f0 tcp_usr_send(c5903318,0,c5bb3c00,0,0,...) at tcp_usr_send+0x25a sosend_generic(c5903318,0,ebe1fc60,0,0,...) at sosend_generic+0x50d sosend(c5903318,0,ebe1fc60,0,0,...) at sosend+0x54 soo_write(c5853240,ebe1fc60,c5976800,0,c5861440,...) at soo_write+0x4b dofilewrite(ebe1fc60,ffffffff,ffffffff,0,c5853240,...) at dofilewrite+0x95 kern_writev(c5861440,3,ebe1fc60,28533110,0,...) at kern_writev+0x58 write(c5861440,ebe1fcfc,c,c0a5633b,c0b49ae0,...) at write+0x4f syscall(ebe1fd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (4, FreeBSD ELF32, write), eip = 0x283e3423, esp = 0xbfbfe5cc, ebp = 0xbfbfe5e8 --- Tracing command sshd pid 940 tid 100145 td 0xc5861220 sched_switch(c5861220,0,1,180,94a3b5cc,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5861220,0,c0aa0842,19b,c58a8560,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,58,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c58a8560,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c58a8560,c58a8518,158,c0aa5515,0) at _sleep+0x359 sbwait(c58a84f4,4,c0aa562a,5ae,c58a8518,...) at sbwait+0x76 soreceive_generic(c58a84a4,0,ebe1cc60,0,0,...) at soreceive_generic+0x42f soreceive(c58a84a4,0,ebe1cc60,0,0,...) at soreceive+0x4d soo_read(c55e76c0,ebe1cc60,c5976300,0,c5861220,...) at soo_read+0x3b dofileread(ebe1cc60,ffffffff,ffffffff,0,c55e76c0,...) at dofileread+0x96 kern_readv(c5861220,5,ebe1cc60,bfbfe64c,4,...) at kern_readv+0x58 read(c5861220,ebe1ccfc,c,c0ab7dcb,c0b49ac8,...) at read+0x4f syscall(ebe1cd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x283e3443, esp = 0xbfbfe60c, ebp = 0xbfbfe628 --- Tracing command httpd pid 936 tid 100105 td 0xc5543880 sched_switch(c5543880,0,1,180,21cc628c,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5543880,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,58,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359 kern_accept(c5543880,3,e9b1ac70,e9b1ac6c,e9b1ac68,...) at kern_accept+0x12b accept(c5543880,e9b1acfc,c,e9b1ad38,c0b49d50,...) at accept+0x8f syscall(e9b1ad38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 --- Tracing command httpd pid 935 tid 100147 td 0xc56eecc0 sched_switch(c56eecc0,0,1,180,216c709c,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c56eecc0,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,58,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359 kern_accept(c56eecc0,3,ebdc1c70,ebdc1c6c,ebdc1c68,...) at kern_accept+0x12b accept(c56eecc0,ebdc1cfc,c,ebdc1d38,c0b49d50,...) at accept+0x8f syscall(ebdc1d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 --- Tracing command httpd pid 934 tid 100148 td 0xc5af5000 sched_switch(c5af5000,0,1,180,207a8436,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5af5000,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,58,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359 kern_accept(c5af5000,3,ebecec70,ebecec6c,ebecec68,...) at kern_accept+0x12b accept(c5af5000,ebececfc,c,ebeced38,c0b49d50,...) at accept+0x8f syscall(ebeced38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 --- Tracing command httpd pid 933 tid 100149 td 0xc5af4cc0 sched_switch(c5af4cc0,0,1,180,2035600c,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5af4cc0,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,58,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359 kern_accept(c5af4cc0,3,ebecbc70,ebecbc6c,ebecbc68,...) at kern_accept+0x12b accept(c5af4cc0,ebecbcfc,c,ebecbd38,c0b49d50,...) at accept+0x8f syscall(ebecbd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 --- Tracing command httpd pid 932 tid 100150 td 0xc5af4aa0 sched_switch(c5af4aa0,0,1,180,1f5dfefc,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5af4aa0,0,c0aa0842,19b,c58a866a,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,58,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c58a866a,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c58a866a,c0c020c0,158,c0a6ee93,0,...) at _sleep+0x359 kern_accept(c5af4aa0,3,ebec8c70,ebec8c6c,ebec8c68,...) at kern_accept+0x12b accept(c5af4aa0,ebec8cfc,c,ebec8d38,c0b49d50,...) at accept+0x8f syscall(ebec8d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (30, FreeBSD ELF32, accept), eip = 0x282d138f, esp = 0xbfbfeb7c, ebp = 0xbfbfeb98 --- Tracing command getty pid 931 tid 100064 td 0xc5543aa0 sched_switch(c5543aa0,0,1,180,aa94c4ec,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5543aa0,0,c0aa0842,19b,c5357c10,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c5357c10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c5357c10,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c5357c00,c5357c10,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c5357c00,e9b1dc60,0,e9b1dbb8,c071e1c8,...) at ttread+0x4e7 ttyread(c5372100,e9b1dc60,0,168,0,...) at ttyread+0x39 giant_read(c5372100,e9b1dc60,0,0,1,...) at giant_read+0x48 devfs_read_f(c56563f0,e9b1dc60,c50f4300,0,c5543aa0,...) at devfs_read_f+0x71 dofileread(e9b1dc60,ffffffff,ffffffff,0,c56563f0,...) at dofileread+0x96 kern_readv(c5543aa0,0,e9b1dc60,bfbfee4b,1,...) at kern_readv+0x58 read(c5543aa0,e9b1dcfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(e9b1dd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 930 tid 100065 td 0xc559d000 sched_switch(c559d000,0,1,180,ab249a30,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559d000,0,c0aa0842,19b,c538a810,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c538a810,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c538a810,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c538a800,c538a810,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c538a800,ebd39c60,0,ebd39b9c,c067a61d,...) at ttread+0x4e7 ttyread(c538e100,ebd39c60,0,ebd39bb8,c071e1c8,...) at ttyread+0x39 scread(c538e100,ebd39c60,0,168,0,...) at scread+0x2d giant_read(c538e100,ebd39c60,0,0,1,...) at giant_read+0x48 devfs_read_f(c55a41b0,ebd39c60,c50f4300,0,c559d000,...) at devfs_read_f+0x71 dofileread(ebd39c60,ffffffff,ffffffff,0,c55a41b0,...) at dofileread+0x96 kern_readv(c559d000,0,ebd39c60,bfbfee4b,1,...) at kern_readv+0x58 read(c559d000,ebd39cfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(ebd39d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 929 tid 100151 td 0xc5af4880 sched_switch(c5af4880,0,1,180,aa9a3df4,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5af4880,0,c0aa0842,19b,c538ac10,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c538ac10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c538ac10,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c538ac00,c538ac10,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c538ac00,ebec5c60,0,ebec5b9c,c067a61d,...) at ttread+0x4e7 ttyread(c5373900,ebec5c60,0,ebec5bb8,c071e1c8,...) at ttyread+0x39 scread(c5373900,ebec5c60,0,168,0,...) at scread+0x2d giant_read(c5373900,ebec5c60,0,0,1,...) at giant_read+0x48 devfs_read_f(c55a31f8,ebec5c60,c50f4300,0,c5af4880,...) at devfs_read_f+0x71 dofileread(ebec5c60,ffffffff,ffffffff,0,c55a31f8,...) at dofileread+0x96 kern_readv(c5af4880,0,ebec5c60,bfbfee4b,1,...) at kern_readv+0x58 read(c5af4880,ebec5cfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(ebec5d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 928 tid 100077 td 0xc5599440 sched_switch(c5599440,0,1,180,ab04c1b4,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5599440,0,c0aa0842,19b,c5389810,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c5389810,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c5389810,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c5389800,c5389810,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c5389800,ebd15c60,0,ebd15b9c,c067a61d,...) at ttread+0x4e7 ttyread(c5373a00,ebd15c60,0,ebd15bb8,c071e1c8,...) at ttyread+0x39 scread(c5373a00,ebd15c60,0,168,0,...) at scread+0x2d giant_read(c5373a00,ebd15c60,0,0,1,...) at giant_read+0x48 devfs_read_f(c55a4990,ebd15c60,c50f4300,0,c5599440,...) at devfs_read_f+0x71 dofileread(ebd15c60,ffffffff,ffffffff,0,c55a4990,...) at dofileread+0x96 kern_readv(c5599440,0,ebd15c60,bfbfee4b,1,...) at kern_readv+0x58 read(c5599440,ebd15cfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(ebd15d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 927 tid 100055 td 0xc5377220 sched_switch(c5377220,0,1,180,aae49548,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5377220,0,c0aa0842,19b,c538b410,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c538b410,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c538b410,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c538b400,c538b410,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c538b400,e9adac60,0,e9adab9c,c067a61d,...) at ttread+0x4e7 ttyread(c5373b00,e9adac60,0,e9adabb8,c071e1c8,...) at ttyread+0x39 scread(c5373b00,e9adac60,0,168,0,...) at scread+0x2d giant_read(c5373b00,e9adac60,0,0,1,...) at giant_read+0x48 devfs_read_f(c55a3480,e9adac60,c50f4300,0,c5377220,...) at devfs_read_f+0x71 dofileread(e9adac60,ffffffff,ffffffff,0,c55a3480,...) at dofileread+0x96 kern_readv(c5377220,0,e9adac60,bfbfee4b,1,...) at kern_readv+0x58 read(c5377220,e9adacfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(e9adad38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 926 tid 100108 td 0xc5543220 sched_switch(c5543220,0,1,180,aaf9cf5c,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5543220,0,c0aa0842,19b,c538c410,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c538c410,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c538c410,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c538c400,c538c410,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c538c400,e9b11c60,0,e9b11b9c,c067a61d,...) at ttread+0x4e7 ttyread(c537d700,e9b11c60,0,e9b11bb8,c071e1c8,...) at ttyread+0x39 scread(c537d700,e9b11c60,0,168,0,...) at scread+0x2d giant_read(c537d700,e9b11c60,0,0,1,...) at giant_read+0x48 devfs_read_f(c5617d38,e9b11c60,c50f4300,0,c5543220,...) at devfs_read_f+0x71 dofileread(e9b11c60,ffffffff,ffffffff,0,c5617d38,...) at dofileread+0x96 kern_readv(c5543220,0,e9b11c60,bfbfee4b,1,...) at kern_readv+0x58 read(c5543220,e9b11cfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(e9b11d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 925 tid 100120 td 0xc5863880 sched_switch(c5863880,0,1,180,ab30623c,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5863880,0,c0aa0842,19b,c538c010,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c538c010,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c538c010,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c538c000,c538c010,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c538c000,ebe3ac60,0,ebe3ab9c,c067a61d,...) at ttread+0x4e7 ttyread(c537d600,ebe3ac60,0,ebe3abb8,c071e1c8,...) at ttyread+0x39 scread(c537d600,ebe3ac60,0,168,0,...) at scread+0x2d giant_read(c537d600,ebe3ac60,0,0,1,...) at giant_read+0x48 devfs_read_f(c5853c60,ebe3ac60,c50f4300,0,c5863880,...) at devfs_read_f+0x71 dofileread(ebe3ac60,ffffffff,ffffffff,0,c5853c60,...) at dofileread+0x96 kern_readv(c5863880,0,ebe3ac60,bfbfee4b,1,...) at kern_readv+0x58 read(c5863880,ebe3acfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(ebe3ad38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 924 tid 100054 td 0xc5377440 sched_switch(c5377440,0,1,180,ab0373e0,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5377440,0,c0aa0842,19b,c537fc10,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c537fc10,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c537fc10,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c537fc00,c537fc10,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c537fc00,e9addc60,0,e9addb9c,c067a61d,...) at ttread+0x4e7 ttyread(c537d500,e9addc60,0,e9addbb8,c071e1c8,...) at ttyread+0x39 scread(c537d500,e9addc60,0,168,0,...) at scread+0x2d giant_read(c537d500,e9addc60,0,0,1,...) at giant_read+0x48 devfs_read_f(c55e7cf0,e9addc60,c50f4300,0,c5377440,...) at devfs_read_f+0x71 dofileread(e9addc60,ffffffff,ffffffff,0,c55e7cf0,...) at dofileread+0x96 kern_readv(c5377440,0,e9addc60,bfbfee4b,1,...) at kern_readv+0x58 read(c5377440,e9addcfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(e9addd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command getty pid 923 tid 100089 td 0xc55c4000 sched_switch(c55c4000,0,1,180,ab35ec44,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c55c4000,0,c0aa0842,19b,c5380410,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,59,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c5380410,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c5380410,0,159,c0aa41dd,0,...) at _sleep+0x359 ttysleep(c5380400,c5380410,159,c0aa41dd,0,...) at ttysleep+0x39 ttread(c5380400,ebd5fc60,0,ebd5fb9c,c067a61d,...) at ttread+0x4e7 ttyread(c537d400,ebd5fc60,0,ebd5fbb8,c071e1c8,...) at ttyread+0x39 scread(c537d400,ebd5fc60,0,168,0,...) at scread+0x2d giant_read(c537d400,ebd5fc60,0,0,1,...) at giant_read+0x48 devfs_read_f(c55a3678,ebd5fc60,c50f4300,0,c55c4000,...) at devfs_read_f+0x71 dofileread(ebd5fc60,ffffffff,ffffffff,0,c55a3678,...) at dofileread+0x96 kern_readv(c55c4000,0,ebd5fc60,bfbfee4b,1,...) at kern_readv+0x58 read(c55c4000,ebd5fcfc,c,c0acf182,c0b49ac8,...) at read+0x4f syscall(ebd5fd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28164443, esp = 0xbfbfee2c, ebp = 0xbfbfee58 --- Tracing command sleep pid 921 tid 100100 td 0xc56e9440 sched_switch(c56e9440,0,1,180,87825700,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c56e9440,0,c0aa0842,19b,c0bb0824,...) at sleepq_switch+0xf0 sleepq_catch_signals(ebda0bc4,c07423dd,c0bb8680,4,5c,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c0bb0824,0,c0a9d757,da,0,...) at sleepq_timedwait_sig+0x17 _sleep(c0bb0824,0,15c,c0a9df4b,ea61,...) at _sleep+0x331 kern_nanosleep(c56e9440,ebda0c64,ebda0c6c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c56e9440,ebda0cfc,8,ebda0d38,c0b4b100,...) at nanosleep+0x6f syscall(ebda0d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28135f2f, esp = 0xbfbfee5c, ebp = 0xbfbfee98 --- Tracing command logger pid 920 tid 100066 td 0xc559ccc0 sched_switch(c559ccc0,0,1,180,8553358a,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559ccc0,0,c0aa0842,19b,c586d948,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,4c,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c586d948,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c586d948,c586dab8,14c,c0aa2569,0,...) at _sleep+0x359 pipe_read(c58a2dc8,ebd36c60,c50f4300,0,c559ccc0,...) at pipe_read+0x3f6 dofileread(ebd36c60,ffffffff,ffffffff,0,c58a2dc8,...) at dofileread+0x96 kern_readv(c559ccc0,0,ebd36c60,28206000,1000,...) at kern_readv+0x58 read(c559ccc0,ebd36cfc,c,c0aa289e,c0b49ac8,...) at read+0x4f syscall(ebd36d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (3, FreeBSD ELF32, read), eip = 0x28154443, esp = 0xbfbfe99c, ebp = 0xbfbfe9b8 --- Tracing command sh pid 919 tid 100076 td 0xc5599660 sched_switch(c5599660,0,1,180,848ec352,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5599660,0,c0aa0842,19b,c559f2ac,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c559f2ac,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c559f2ac,c559f33c,15c,c0aa296f,0,...) at _sleep+0x359 kern_wait(c5599660,ffffffff,ebd18c2c,2,0,...) at kern_wait+0xa66 wait4(c5599660,ebd18cfc,10,c0aa294e,c0b49b28,...) at wait4+0x3b syscall(ebd18d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149d7b, esp = 0xbfbfd8fc, ebp = 0xbfbfd918 --- Tracing command sh pid 918 tid 100067 td 0xc559caa0 sched_switch(c559caa0,0,1,180,842dbd5e,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559caa0,0,c0aa0842,19b,c5598558,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c5598558,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c5598558,c55985e8,15c,c0aa296f,0,...) at _sleep+0x359 kern_wait(c559caa0,ffffffff,ebd33c2c,2,0,...) at kern_wait+0xa66 wait4(c559caa0,ebd33cfc,10,c0aa294e,c0b49b28,...) at wait4+0x3b syscall(ebd33d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149d7b, esp = 0xbfbfda9c, ebp = 0xbfbfdab8 --- Tracing command inetd pid 902 tid 100081 td 0xc55c7220 sched_switch(c55c7220,0,1,180,7daee5f4,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c55c7220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c7220,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,ebd77b18,...) at _cv_wait_sig+0x1fc kern_select(c55c7220,6,bfbfec8c,0,0,0,c55a44c8,bfbfee48) at kern_select+0x6cc select(c55c7220,ebd77cfc,14,c0aa29c2,c0b4a338,...) at select+0x5e syscall(ebd77d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281773c3, esp = 0xbfbfe08c, ebp = 0xbfbfee78 --- Tracing command moused pid 875 tid 100068 td 0xc559c880 sched_switch(c559c880,0,1,180,609d1204,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559c880,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c559c880,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,c065602f,...) at _cv_wait_sig+0x1fc kern_select(c559c880,400,bfbfeaac,0,0,0,4773a417,0) at kern_select+0x6cc select(c559c880,ebd30cfc,14,c0acf182,c0b4a338,...) at select+0x5e syscall(ebd30d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x2817c3c3, esp = 0xbfbfea2c, ebp = 0xbfbfebc8 --- Tracing command cron pid 859 tid 100118 td 0xc5863cc0 sched_switch(c5863cc0,0,1,180,b0b05d04,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5863cc0,0,c0aa0842,19b,c0bb0824,...) at sleepq_switch+0xf0 sleepq_catch_signals(ebe40bc4,c07423dd,c0bb8700,4,5c,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c0bb0824,0,c0a9d757,da,0,...) at sleepq_timedwait_sig+0x17 _sleep(c0bb0824,0,15c,c0a9df4b,ea61,...) at _sleep+0x331 kern_nanosleep(c5863cc0,ebe40c64,ebe40c6c,3c,0,...) at kern_nanosleep+0xc1 nanosleep(c5863cc0,ebe40cfc,8,c0aa2ab3,c0b4b100,...) at nanosleep+0x6f syscall(ebe40d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (240, FreeBSD ELF32, nanosleep), eip = 0x28150f2f, esp = 0xbfbfed0c, ebp = 0xbfbfed38 --- Tracing command sshd pid 852 tid 100082 td 0xc55c7000 sched_switch(c55c7000,0,1,180,86031ef8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c55c7000,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c7000,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,ebd74afc,...) at _cv_wait_sig+0x1fc kern_select(c55c7000,7,2850b0b8,0,0,0,c,ebd74c98) at kern_select+0x6cc select(c55c7000,ebd74cfc,14,ebd74d38,c0b4a338,...) at select+0x5e syscall(ebd74d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283e33c3, esp = 0xbfbfe6cc, ebp = 0xbfbfee98 --- Tracing command httpd pid 835 tid 100109 td 0xc5543000 sched_switch(c5543000,0,1,180,e4d583d8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5543000,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(e9b0ea84,c074278a,c0c01dc4,0,c5543000,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c0c01ddc,3e9,c0aba6ad,101,0,...) at sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c0c01ddc,c0c01dc4,3e9,30f,e9b0eafc,...) at _cv_timedwait_sig+0x20b kern_select(c5543000,0,0,0,0,e9b0ec70,1,0) at kern_select+0x6b4 select(c5543000,e9b0ecfc,14,c0aa278a,c0b4a338,...) at select+0x5e syscall(e9b0ed38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x283333c3, esp = 0xbfbfec1c, ebp = 0xbfbfec48 --- Tracing command mysqld pid 833 tid 100139 td 0xc591eaa0 sched_switch(c591eaa0,0,1,180,9e87e438,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c591eaa0,0,c0aa0842,19b,c561f2c0,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c561f2c0,c0bb14d0,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c561f2c0,c0bb14d0,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c591eaa0,ebe89cfc,ebe89d2c,c0a04743,c591eaa0,...) at __umtx_op_cv_wait+0x354 _umtx_op(c591eaa0,ebe89cfc,14,ebe89d38,c0b4c510,...) at _umtx_op+0x27 syscall(ebe89d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 0xbf1c5edc, ebp = 0xbf1c5ef8 --- Tracing command mysqld pid 833 tid 100138 td 0xc591ecc0 sched_switch(c591ecc0,0,1,180,6dc2a8fc,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c591ecc0,0,c0aa0842,19b,ebe8cbe0,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,68,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(ebe8cbe0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(ebe8cbe0,c55c833c,168,c0a99fde,0,...) at _sleep+0x359 kern_sigtimedwait(c591ecc0,26005,0,0,0,...) at kern_sigtimedwait+0x2af sigwait(c591ecc0,ebe8ccfc,8,c0aa2849,c0b4c2b8,...) at sigwait+0x77 syscall(ebe8cd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (429, FreeBSD ELF32, sigwait), eip = 0x2864e6bb, esp = 0xbf1f6f1c, ebp = 0xbf1f6f48 --- Tracing command mysqld pid 833 tid 100135 td 0xc591f000 sched_switch(c591f000,0,1,180,1453f504,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c591f000,0,c0aa0842,19b,c561f240,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c561f240,c0bb1440,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c561f240,c0bb1440,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c591f000,ebe8fcfc,ebe8fd2c,c0a04743,c591f000,...) at __umtx_op_cv_wait+0x354 _umtx_op(c591f000,ebe8fcfc,14,ebe8fd38,c0b4c510,...) at _umtx_op+0x27 syscall(ebe8fd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 0xbf2f7eac, ebp = 0xbf2f7ec8 --- Tracing command mysqld pid 833 tid 100137 td 0xc591f220 sched_switch(c591f220,0,1,180,801e49e8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c591f220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(ebe92a84,c074278a,c0c01dc4,0,c591f220,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c0c01ddc,7d1,c0aba6ad,101,0,...) at sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c0c01ddc,c0c01dc4,7d1,30f,0,...) at _cv_timedwait_sig+0x20b kern_select(c591f220,0,0,0,0,ebe92c70,2,0) at kern_select+0x6b4 select(c591f220,ebe92cfc,14,16,c0b4a338,...) at select+0x5e syscall(ebe92d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x286e03c3, esp = 0xbf3f8f0c, ebp = 0xbf3f8f38 --- Tracing command mysqld pid 833 tid 100136 td 0xc591f440 sched_switch(c591f440,0,1,180,4272d1ae,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c591f440,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(ebe95a84,c074278a,c0c01dc4,0,c591f440,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c0c01ddc,3e9,c0aba6ad,101,0,...) at sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c0c01ddc,c0c01dc4,3e9,30f,0,...) at _cv_timedwait_sig+0x20b kern_select(c591f440,0,0,0,0,ebe95c70,1,0) at kern_select+0x6b4 select(c591f440,ebe95cfc,14,16,c0b4a338,...) at select+0x5e syscall(ebe95d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x286e03c3, esp = 0xbf4f9f0c, ebp = 0xbf4f9f38 --- Tracing command mysqld pid 833 tid 100134 td 0xc591f880 sched_switch(c591f880,0,1,180,992f4b8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c591f880,0,c0aa0842,19b,c561f140,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c561f140,c0bb19b0,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c561f140,c0bb19b0,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c591f880,ebe9bcfc,ebe9bd2c,c0a04743,c591f880,...) at __umtx_op_cv_wait+0x354 _umtx_op(c591f880,ebe9bcfc,14,c07871f8,c0b4c510,...) at _umtx_op+0x27 syscall(ebe9bd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 0xbf6fbcfc, ebp = 0xbf6fbd18 --- Tracing command mysqld pid 833 tid 100133 td 0xc591faa0 sched_switch(c591faa0,0,1,180,46f7d224,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c591faa0,0,c0aa0842,19b,c561f100,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c561f100,c0bb1710,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c561f100,c0bb1710,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c591faa0,ebe9ecfc,ebe9ed2c,c0a04743,c591faa0,...) at __umtx_op_cv_wait+0x354 _umtx_op(c591faa0,ebe9ecfc,14,ebe9ed38,c0b4c510,...) at _umtx_op+0x27 syscall(ebe9ed38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 0xbf7fccfc, ebp = 0xbf7fcd18 --- Tracing command mysqld pid 833 tid 100132 td 0xc55c7660 sched_switch(c55c7660,0,1,180,40ae3094,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c55c7660,0,c0aa0842,19b,c55ae740,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c55ae740,c0bb1470,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c55ae740,c0bb1470,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c55c7660,ebd7dcfc,ebd7dd2c,c0a04743,c55c7660,...) at __umtx_op_cv_wait+0x354 _umtx_op(c55c7660,ebd7dcfc,14,c0aa363a,c0b4c510,...) at _umtx_op+0x27 syscall(ebd7dd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 0xbf8fdcfc, ebp = 0xbf8fdd18 --- Tracing command mysqld pid 833 tid 100131 td 0xc55c7880 sched_switch(c55c7880,0,1,180,9830e02,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c55c7880,0,c0aa0842,19b,c55ae700,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c55ae700,c0bb1200,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c55ae700,c0bb1200,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c55c7880,ebd80cfc,ebd80d2c,c0a04743,c55c7880,...) at __umtx_op_cv_wait+0x354 _umtx_op(c55c7880,ebd80cfc,14,c0aa363a,c0b4c510,...) at _umtx_op+0x27 syscall(ebd80d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x2864e39b, esp = 0xbf9fecfc, ebp = 0xbf9fed18 --- Tracing command mysqld pid 833 tid 100086 td 0xc55c4660 sched_switch(c55c4660,0,1,180,9e866198,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c55c4660,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c4660,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,246,...) at _cv_wait_sig+0x1fc kern_select(c55c4660,d,bfbfead8,0,0,0,0,2860772c) at kern_select+0x6cc select(c55c4660,ebd68cfc,14,c0aa29c2,c0b4a338,...) at select+0x5e syscall(ebd68d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x286e03c3, esp = 0xbfbfe6ac, ebp = 0xbfbfe6d8 --- Tracing command sh pid 808 tid 100114 td 0xc5542440 sched_switch(c5542440,0,1,180,88d8c3f6,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5542440,0,c0aa0842,19b,c585fab0,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,5c,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c585fab0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c585fab0,c585fb40,15c,c0aa296f,0,...) at _sleep+0x359 kern_wait(c5542440,ffffffff,e9affc2c,2,0,...) at kern_wait+0xa66 wait4(c5542440,e9affcfc,10,c0aa294e,c0b49b28,...) at wait4+0x3b syscall(e9affd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x28149d7b, esp = 0xbfbfe7fc, ebp = 0xbfbfe818 --- Tracing command qmgr pid 803 tid 100056 td 0xc5377000 sched_switch(c5377000,0,1,180,f72875dc,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5377000,0,c0aa0842,19b,c587da00,...) at sleepq_switch+0xf0 sleepq_catch_signals(e9ad7aac,c07423dd,c0bb8700,4,58,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c587da00,0,c0a9d757,da,0,...) at sleepq_timedwait_sig+0x17 _sleep(c587da00,c587da00,158,c0a99450,493e1,...) at _sleep+0x331 kern_kevent(c5377000,8,0,64,e9ad7c60,...) at kern_kevent+0x3c3 kevent(c5377000,e9ad7cfc,18,c0aa2ab3,c0b4bc88,...) at kevent+0x19b syscall(e9ad7d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2839c14b, esp = 0xbfbfe16c, ebp = 0xbfbfe978 --- Tracing command pickup pid 802 tid 100074 td 0xc5599aa0 sched_switch(c5599aa0,0,1,180,efe64654,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5599aa0,0,c0aa0842,19b,c55aab80,...) at sleepq_switch+0xf0 sleepq_catch_signals(ebd1eaac,c07423dd,c0bb8700,4,58,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c55aab80,0,c0a9d757,da,0,...) at sleepq_timedwait_sig+0x17 _sleep(c55aab80,c55aab80,158,c0a99450,186a1,...) at _sleep+0x331 kern_kevent(c5599aa0,8,0,64,ebd1ec60,...) at kern_kevent+0x3c3 kevent(c5599aa0,ebd1ecfc,18,c0aa2ab3,c0b4bc88,...) at kevent+0x19b syscall(ebd1ed38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2839014b, esp = 0xbfbfe19c, ebp = 0xbfbfe9a8 --- Tracing command master pid 795 tid 100079 td 0xc5599000 sched_switch(c5599000,0,1,180,f7267f80,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5599000,0,c0aa0842,19b,c5564080,...) at sleepq_switch+0xf0 sleepq_catch_signals(ebd0faac,c07423dd,c0bb8680,4,58,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c5564080,0,c0a9d757,da,0,...) at sleepq_timedwait_sig+0x17 _sleep(c5564080,c5564080,158,c0a99450,ea61,...) at _sleep+0x331 kern_kevent(c5599000,c,0,64,ebd0fc60,...) at kern_kevent+0x3c3 kevent(c5599000,ebd0fcfc,18,c0aa2ab3,c0b4bc88,...) at kevent+0x19b syscall(ebd0fd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (363, FreeBSD ELF32, kevent), eip = 0x2838314b, esp = 0xbfbfe3ec, ebp = 0xbfbfebf8 --- Tracing command ntpd pid 728 tid 100112 td 0xc5542880 sched_switch(c5542880,0,1,180,3ff48692,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5542880,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c5542880,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,0,...) at _cv_wait_sig+0x1fc kern_select(c5542880,a,bfbfed7c,0,0,0,246,0) at kern_select+0x6cc select(c5542880,e9b05cfc,14,16,c0b4a338,...) at select+0x5e syscall(e9b05d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x2830d3c3, esp = 0xbfbfed5c, ebp = 0xbfbfee18 --- Tracing command named pid 643 tid 100130 td 0xc559d220 sched_switch(c559d220,0,1,180,149d8050,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559d220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c559d220,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,0,...) at _cv_wait_sig+0x1fc kern_select(c559d220,1a,bf4f9f10,bf4f9e90,0,0,8,2836572c) at kern_select+0x6cc select(c559d220,ebd3ccfc,14,c0a5633b,c0b4a338,...) at select+0x5e syscall(ebd3cd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x2843e3c3, esp = 0xbf4f9cfc, ebp = 0xbf4f9d28 --- Tracing command named pid 643 tid 100129 td 0xc559d440 sched_switch(c559d440,0,1,180,532312a8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559d440,0,c0aa0842,19b,c555d240,...) at sleepq_switch+0xf0 sleepq_catch_signals(c077f359,c559d5d4,7530,c077e7d0,0,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c555d240,7530,c0a9e334,100,0,...) at sleepq_timedwait_sig+0x17 _sleep(c555d240,c0bb0900,100,c0a9e334,7530,...) at _sleep+0x331 __umtx_op_cv_wait(c559d440,ebd3fcfc,ebd3fd2c,c0a04743,c559d440,...) at __umtx_op_cv_wait+0x430 _umtx_op(c559d440,ebd3fcfc,14,c0aa2df1,c0b4c510,...) at _umtx_op+0x27 syscall(ebd3fd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 0xbf5fae0c, ebp = 0xbf5fae28 --- Tracing command named pid 643 tid 100128 td 0xc559d660 sched_switch(c559d660,0,1,180,52c4f6a4,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559d660,0,c0aa0842,19b,c555d200,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c555d200,c0bb0ea0,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c555d200,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c559d660,ebd42cfc,ebd42d2c,c0a04743,c559d660,...) at __umtx_op_cv_wait+0x354 _umtx_op(c559d660,ebd42cfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27 syscall(ebd42d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 0xbf6fbecc, ebp = 0xbf6fbee8 --- Tracing command named pid 643 tid 100127 td 0xc559d880 sched_switch(c559d880,0,1,180,532425b8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559d880,0,c0aa0842,19b,c555d1c0,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c555d1c0,c0bb0ea0,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c555d1c0,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c559d880,ebd45cfc,ebd45d2c,c0a04743,c559d880,...) at __umtx_op_cv_wait+0x354 _umtx_op(c559d880,ebd45cfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27 syscall(ebd45d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 0xbf7fcecc, ebp = 0xbf7fcee8 --- Tracing command named pid 643 tid 100126 td 0xc559daa0 sched_switch(c559daa0,0,1,180,14a409e8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559daa0,0,c0aa0842,19b,c555d180,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c555d180,c0bb0ea0,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c555d180,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c559daa0,ebd48cfc,ebd48d2c,c0a04743,c559daa0,...) at __umtx_op_cv_wait+0x354 _umtx_op(c559daa0,ebd48cfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27 syscall(ebd48d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 0xbf8fdecc, ebp = 0xbf8fdee8 --- Tracing command named pid 643 tid 100125 td 0xc559dcc0 sched_switch(c559dcc0,0,1,180,1497db68,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c559dcc0,0,c0aa0842,19b,c555d140,...) at sleepq_switch+0xf0 sleepq_catch_signals(c0aa0842,155,0,100,0,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c555d140,c0bb0ea0,c0a9e334,100,0,...) at sleepq_wait_sig+0x14 _sleep(c555d140,c0bb0ea0,100,c0a9e334,0,...) at _sleep+0x359 __umtx_op_cv_wait(c559dcc0,ebd4bcfc,ebd4bd2c,c0a04743,c559dcc0,...) at __umtx_op_cv_wait+0x354 _umtx_op(c559dcc0,ebd4bcfc,14,c0aa2ab3,c0b4c510,...) at _umtx_op+0x27 syscall(ebd4bd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (454, FreeBSD ELF32, _umtx_op), eip = 0x283ac39b, esp = 0xbf9feecc, ebp = 0xbf9feee8 --- Tracing command named pid 643 tid 100107 td 0xc5543440 sched_switch(c5543440,0,1,180,ff77ecb0,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5543440,0,c0aa0842,19b,e9b14be0,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8700,4,c0a9bb4b,68,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(e9b14be0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(e9b14be0,c585e5e8,168,c0a99fde,0,...) at _sleep+0x359 kern_sigtimedwait(c5543440,4003,0,0,0,...) at kern_sigtimedwait+0x2af sigwait(c5543440,e9b14cfc,8,c0aa363a,c0b4c2b8,...) at sigwait+0x77 syscall(e9b14d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (429, FreeBSD ELF32, sigwait), eip = 0x283ac6bb, esp = 0xbfbfecdc, ebp = 0xbfbfed08 --- Tracing command syslogd pid 578 tid 100116 td 0xc5542000 sched_switch(c5542000,0,1,180,b40d7874,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5542000,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c5542000,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,d9191108,...) at _cv_wait_sig+0x1fc kern_select(c5542000,b,282200b8,0,0,0,c56f2000,28204000) at kern_select+0x6cc select(c5542000,e9af9cfc,14,c0abbe3e,c0b4a338,...) at select+0x5e syscall(e9af9d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x281673c3, esp = 0xbfbfdefc, ebp = 0xbfbfee48 --- Tracing command devd pid 517 tid 100099 td 0xc56e9660 sched_switch(c56e9660,0,1,180,ff905800,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c56e9660,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c56e9660,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,30f,ebda3afc,...) at _cv_wait_sig+0x1fc kern_select(c56e9660,5,bfbfedf8,0,0,0,0,bfbfe99c) at kern_select+0x6cc select(c56e9660,ebda3cfc,14,c0aa2cb6,c0b4a338,...) at select+0x5e syscall(ebda3d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (93, FreeBSD ELF32, select), eip = 0x807f96f, esp = 0xbfbfe9bc, ebp = 0xbfbfee98 --- Tracing command dhclient pid 321 tid 100058 td 0xc5265aa0 sched_switch(c5265aa0,0,1,180,386c9404,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5265aa0,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(e580ead8,c074278a,c0c01dc4,0,c5265aa0,...) at sleepq_catch_signals+0x21e sleepq_timedwait_sig(c0c01ddc,362f19,c0aba6ad,101,0,...) at sleepq_timedwait_sig+0x17 _cv_timedwait_sig(c0c01ddc,c0c01dc4,362f19,3bb,e580eb48,...) at _cv_timedwait_sig+0x20b poll(c5265aa0,e580ecfc,c,c0a5633b,c0b4ae18,...) at poll+0x554 syscall(e580ed38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2810f8df, esp = 0xbfbfee1c, ebp = 0xbfbfee48 --- Tracing command dhclient pid 301 tid 100088 td 0xc55c4220 sched_switch(c55c4220,0,1,180,83d8d1d0,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c55c4220,0,c0aa0842,19b,c0c01ddc,...) at sleepq_switch+0xf0 sleepq_catch_signals(c074278a,c0c01dc4,0,c0a9bb4b,c55c4220,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c0c01ddc,c0c01dc4,c0aba6ad,101,0,...) at sleepq_wait_sig+0x14 _cv_wait_sig(c0c01ddc,c0c01dc4,c0aa23c8,3bb,c0aa2465,...) at _cv_wait_sig+0x1fc poll(c55c4220,ebd62cfc,c,c0a8d9fa,c0b4ae18,...) at poll+0x56a syscall(ebd62d38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (209, FreeBSD ELF32, poll), eip = 0x2810f8df, esp = 0xbfbfee1c, ebp = 0xbfbfee48 --- Tracing command softdepflush pid 53 tid 100061 td 0xc5265440 sched_switch(c5265440,0,1,180,dd9baf10,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,44,...) at mi_switch+0x217 sleepq_switch(c5265440,0,c0aa0842,240,44,...) at sleepq_switch+0xf0 sleepq_timedwait(c0c0db44,3e8,c0abc2bb,0,0,...) at sleepq_timedwait+0x68 _sleep(c0c0db44,c0c0db1c,44,c0abc2bb,3e8,...) at _sleep+0x345 softdep_flush(0,e5805d38,c0a99763,2ea,c55412ac,...) at softdep_flush+0x2d2 fork_exit(c0935c40,0,e5805d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5805d70, ebp = 0 --- Tracing command syncer pid 52 tid 100062 td 0xc5265220 sched_switch(c5265220,0,1,180,7e75a8c6,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5265220,...) at mi_switch+0x217 sleepq_switch(c5265220,0,c0aa0842,21d,68,...) at sleepq_switch+0xf0 sleepq_wait(c0bb064c,0,c0aa8d55,0,0,...) at sleepq_wait+0x60 _sleep(c0bb064c,c0c02524,68,c0aa8d55,0,...) at _sleep+0x369 sched_sync(0,e5802d38,c0a99763,2ea,c5541558,...) at sched_sync+0x742 fork_exit(c07cba20,0,e5802d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5802d70, ebp = 0 --- Tracing command vnlru pid 51 tid 100040 td 0xc5264aa0 sched_switch(c5264aa0,0,1,180,dd6b4bf0,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,50,...) at mi_switch+0x217 sleepq_switch(c5264aa0,0,c0aa0842,240,50,...) at sleepq_switch+0xf0 sleepq_timedwait(c5374000,3e8,c0aa8e62,0,0,...) at sleepq_timedwait+0x68 _sleep(c5374000,c0c024f8,250,c0aa8e62,3e8,...) at _sleep+0x345 vnlru_proc(0,e57f9d38,c0a99763,2ea,c5374000,...) at vnlru_proc+0x115 fork_exit(c07cd020,0,e57f9d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57f9d70, ebp = 0 --- Tracing command bufdaemon pid 50 tid 100041 td 0xc5264880 sched_switch(c5264880,0,1,180,dd6ac18a,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,44,...) at mi_switch+0x217 sleepq_switch(c5264880,0,c0aa0842,240,44,...) at sleepq_switch+0xf0 sleepq_timedwait(c0c02244,3e8,c0aa6a11,0,0,...) at sleepq_timedwait+0x68 _sleep(c0c02244,c0c02248,44,c0aa6a11,3e8,...) at _sleep+0x345 buf_daemon(0,e57f6d38,c0a99763,2ea,c53742ac,...) at buf_daemon+0x21e fork_exit(c07b7510,0,e57f6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57f6d70, ebp = 0 --- Tracing command pagezero pid 49 tid 100042 td 0xc5264660 sched_switch(c5264660,0,1,180,38314fb8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5264660,0,c0aa0842,240,0,...) at sleepq_switch+0xf0 sleepq_timedwait(c0c0e700,493e0,c0ac0e41,0,0,...) at sleepq_timedwait+0x68 _sleep(c0c0e700,c0c0e2a4,0,c0ac0e41,493e0,...) at _sleep+0x345 vm_pagezero(0,e57f3d38,c0a99763,2ea,c5374558,...) at vm_pagezero+0xdc fork_exit(c096f430,0,e57f3d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57f3d70, ebp = 0 --- Tracing command vmdaemon pid 48 tid 100043 td 0xc5264440 sched_switch(c5264440,0,1,180,3821bd24,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5264440,...) at mi_switch+0x217 sleepq_switch(c5264440,0,c0aa0842,21d,68,...) at sleepq_switch+0xf0 sleepq_wait(c0c0e318,0,c0a9d757,da,0,...) at sleepq_wait+0x60 _sleep(c0c0e318,c0c0e31c,68,c0aa6a11,0,...) at _sleep+0x369 vm_daemon(0,e57f0d38,c0a99763,2ea,c5374804,...) at vm_daemon+0x59 fork_exit(c096a870,0,e57f0d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57f0d70, ebp = 0 --- Tracing command pagedaemon pid 47 tid 100044 td 0xc5264220 sched_switch(c5264220,0,1,180,1f4cbb44,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,44,...) at mi_switch+0x217 sleepq_switch(c5264220,0,c0aa0842,240,44,...) at sleepq_switch+0xf0 sleepq_timedwait(c0c0e2e0,1388,c0aa6a11,0,0,...) at sleepq_timedwait+0x68 _sleep(c0c0e2e0,c0c0e2a4,44,c0aa6a11,1388,...) at _sleep+0x345 vm_pageout(0,e57edd38,c0a99763,2ea,c5374ab0,...) at vm_pageout+0x2bb fork_exit(c096b420,0,e57edd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57edd70, ebp = 0 --- Tracing command sctp_iterator pid 46 tid 100045 td 0xc5264000 sched_switch(c5264000,0,1,180,770887a0,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5264000,...) at mi_switch+0x217 sleepq_switch(c5264000,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0 sleepq_wait(c0c03fac,c0c03eb4,c0aafbe2,0,0,...) at sleepq_wait+0x60 _sleep(c0c03fac,c0c03eb4,0,c0aafbe2,0,...) at _sleep+0x369 sctp_iterator_thread(0,e9af6d38,c0a99763,2ea,c5375000,...) at sctp_iterator_thread+0x60 fork_exit(c0833a90,0,e9af6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe9af6d70, ebp = 0 --- Tracing command irq7: ppc0 pid 45 tid 100046 td 0xc5222cc0 fork_trampoline() at fork_trampoline Tracing command fdc0 pid 44 tid 100047 td 0xc5222aa0 sched_switch(c5222aa0,0,1,180,c9a4620e,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217 sleepq_switch(c5222aa0,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0 sleepq_timedwait(c531ce3c,3e8,c0a94778,0,0,...) at sleepq_timedwait+0x68 _sleep(c531ce3c,c531cef0,4c,c0a94778,3e8,...) at _sleep+0x345 fdc_thread(c531ce00,e57bed38,c0a99763,2ea,c5375558,...) at fdc_thread+0x2b8 fork_exit(c09c82c0,c531ce00,e57bed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57bed70, ebp = 0 --- Tracing command swi0: sio pid 43 tid 100048 td 0xc5222880 sched_switch(c5222880,0,1,180,da52d0f0,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c535cd64,...) at mi_switch+0x217 ithread_loop(c536d440,e57bbd38,c0a99763,2ea,c5375804,...) at ithread_loop+0x34c fork_exit(c07328a0,c536d440,e57bbd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57bbd70, ebp = 0 --- Tracing command irq12: psm0 pid 42 tid 100049 td 0xc5222660 fork_trampoline() at fork_trampoline Tracing command irq1: atkbd0 pid 41 tid 100050 td 0xc5377cc0 sched_switch(c5377cc0,0,1,180,c36742f4,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c5111464,...) at mi_switch+0x217 ithread_loop(c536d820,e9ae9d38,c0a99763,2ea,c5376000,...) at ithread_loop+0x34c fork_exit(c07328a0,c536d820,e9ae9d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe9ae9d70, ebp = 0 --- Tracing command irq15: ata1 pid 40 tid 100030 td 0xc5221cc0 sched_switch(c5221cc0,0,1,180,3a187a8c,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c5110c64,...) at mi_switch+0x217 ithread_loop(c53582e0,e57acd38,c0a99763,2ea,c5262000,...) at ithread_loop+0x34c fork_exit(c07328a0,c53582e0,e57acd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57acd70, ebp = 0 --- Tracing command irq14: ata0 pid 39 tid 100031 td 0xc5221aa0 fork_trampoline() at fork_trampoline Tracing command aic_recovery1 pid 38 tid 100032 td 0xc5221880 sched_switch(c5221880,0,1,180,6ffb8158,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5221880,...) at mi_switch+0x217 sleepq_switch(c5221880,0,c0aa0842,21d,a0,...) at sleepq_switch+0xf0 sleepq_wait(c531e000,0,c0a9d757,da,0,...) at sleepq_wait+0x60 _sleep(c531e000,c531b634,a0,c0a5fa6c,0,...) at _sleep+0x369 ahd_recovery_thread(c531e000,e57a6d38,c0a99763,2ea,c5262558,...) at ahd_recovery_thread+0x72 fork_exit(c04c4080,c531e000,e57a6d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57a6d70, ebp = 0 --- Tracing command irq22: ahd1 pid 37 tid 100033 td 0xc5221660 sched_switch(c5221660,0,1,180,34633be0,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c515e4e4,...) at mi_switch+0x217 ithread_loop(c534e950,e57a3d38,c0a99763,2ea,c5262804,...) at ithread_loop+0x34c fork_exit(c07328a0,c534e950,e57a3d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57a3d70, ebp = 0 --- Tracing command aic_recovery0 pid 36 tid 100034 td 0xc5221440 sched_switch(c5221440,0,1,180,6ffb359c,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5221440,...) at mi_switch+0x217 sleepq_switch(c5221440,0,c0aa0842,21d,a0,...) at sleepq_switch+0xf0 sleepq_wait(c52d5000,0,c0a9d757,da,0,...) at sleepq_wait+0x60 _sleep(c52d5000,c52696b4,a0,c0a5fa6c,0,...) at _sleep+0x369 ahd_recovery_thread(c52d5000,e57a0d38,c0a99763,2ea,c5262ab0,...) at ahd_recovery_thread+0x72 fork_exit(c04c4080,c52d5000,e57a0d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57a0d70, ebp = 0 --- Tracing command aac0aif pid 35 tid 100035 td 0xc5221220 sched_switch(c5221220,0,1,180,405b856,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217 sleepq_switch(c5221220,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0 sleepq_timedwait(c5263000,4e20,c0a5af1f,0,0,...) at sleepq_timedwait+0x68 _sleep(c5263000,c5277124,4c,c0a5af1f,4e20,...) at _sleep+0x345 aac_command_thread(c5277000,e579dd38,c0a99763,2ea,c5263000,...) at aac_command_thread+0x8d fork_exit(c0496410,c5277000,e579dd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe579dd70, ebp = 0 --- Tracing command irq18: aac0 pid 34 tid 100036 td 0xc5221000 sched_switch(c5221000,0,1,180,b40a4750,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c515e6e4,...) at mi_switch+0x217 ithread_loop(c5276230,e579ad38,c0a99763,2ea,c52632ac,...) at ithread_loop+0x34c fork_exit(c07328a0,c5276230,e579ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe579ad70, ebp = 0 --- Tracing command irq21: bge0 pid 33 tid 100037 td 0xc5214cc0 kdb_enter(c0a8e6e4,c536f8f8,c0c1b0e8,8,c536f88c,...) at kdb_enter+0x32 siointr1(c0c1b0e8,0,c0ac563d,56f,e57979dc,...) at siointr1+0x11c siointr(c536f800,c0bb0844,2,c5214cc0,c5111280,...) at siointr+0x32 intr_execute_handlers(c5107cd0,e57979fc,e5797a58,c09eb844,3b,...) at intr_execute_handlers+0xec lapic_handle_intr(3b,e57979fc) at lapic_handle_intr+0x3f Xapic_isr1() at Xapic_isr1+0x34 --- interrupt, eip = 0xc07429c2, esp = 0xe5797a3c, ebp = 0xe5797a58 --- _mtx_lock_sleep(c538d07c,c5214cc0,0,c0aaaa4b,93,...) at _mtx_lock_sleep+0xa2 _mtx_lock_flags(c538d07c,0,c0aaaa4b,93,c0a9bb4b,...) at _mtx_lock_flags+0xef rtalloc1(e5797bac,1,0,e5797b20,c07426fc,...) at rtalloc1+0x63 rtredirect(e5797bbc,e5797bac,0,26,e5797b9c,...) at rtredirect+0x35e icmp_input(c57d1700,14,246,c0b9aca0,e5797c0c,...) at icmp_input+0x50f ip_input(c57d1700,14e,800,c5250400,800,...) at ip_input+0x650 netisr_dispatch(2,c57d1700,10,3,0,...) at netisr_dispatch+0x73 ether_demux(c5250400,c57d1700,3,0,3,...) at ether_demux+0x1f1 ether_input(c5250400,c57d1700,c0a6b43a,bc4,c5250400,...) at ether_input+0x37f bge_intr(c526d000,0,c0a999ef,471,c515e564,...) at bge_intr+0x7da ithread_loop(c5268640,e5797d38,c0a99763,2ea,c5263558,...) at ithread_loop+0x1b5 fork_exit(c07328a0,c5268640,e5797d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5797d70, ebp = 0 --- Tracing command usb2 pid 32 tid 100038 td 0xc5265000 sched_switch(c5265000,0,1,180,204964e,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,5c,...) at mi_switch+0x217 sleepq_switch(c5265000,0,c0aa0842,240,5c,...) at sleepq_switch+0xf0 sleepq_timedwait(c5210210,ea60,c0a916e6,0,0,...) at sleepq_timedwait+0x68 _sleep(c5210210,0,5c,c0a916e6,ea60,...) at _sleep+0x345 usb_event_thread(c523f800,e57ffd38,c0a99763,2ea,c5263804,...) at usb_event_thread+0xca fork_exit(c06cc3b0,c523f800,e57ffd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57ffd70, ebp = 0 --- Tracing command irq23: ehci0 pid 31 tid 100039 td 0xc5264cc0 fork_trampoline() at fork_trampoline Tracing command usb1 pid 30 tid 100021 td 0xc5214220 sched_switch(c5214220,0,1,180,235540a,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,5c,...) at mi_switch+0x217 sleepq_switch(c5214220,0,c0aa0842,240,5c,...) at sleepq_switch+0xf0 sleepq_timedwait(c5245210,ea60,c0a916e6,0,0,...) at sleepq_timedwait+0x68 _sleep(c5245210,0,5c,c0a916e6,ea60,...) at _sleep+0x345 usb_event_thread(c5253800,e5788d38,c0a99763,2ea,c52152ac,...) at usb_event_thread+0xca fork_exit(c06cc3b0,c5253800,e5788d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5788d70, ebp = 0 --- Tracing command irq19: uhci1 ahd0 pid 29 tid 100022 td 0xc5214000 sched_switch(c5214000,0,1,180,1d89dcfc,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c515e664,...) at mi_switch+0x217 ithread_loop(c524f1b0,e5785d38,c0a99763,2ea,c5215558,...) at ithread_loop+0x34c fork_exit(c07328a0,c524f1b0,e5785d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5785d70, ebp = 0 --- Tracing command usbtask-dr pid 28 tid 100023 td 0xc5119cc0 sched_switch(c5119cc0,0,1,180,6ff99adc,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5119cc0,...) at mi_switch+0x217 sleepq_switch(c5119cc0,0,c0aa0842,21d,5c,...) at sleepq_switch+0xf0 sleepq_wait(c0bae2f4,0,c0a9d757,da,0,...) at sleepq_wait+0x60 _sleep(c0bae2f4,0,5c,c0a91723,0,...) at _sleep+0x369 usb_task_thread(c0bae2f4,e3d5fd38,c0a99763,2ea,c5215804,...) at usb_task_thread+0x62 fork_exit(c06ccb30,c0bae2f4,e3d5fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d5fd70, ebp = 0 --- Tracing command usbtask-hc pid 27 tid 100024 td 0xc5119aa0 sched_switch(c5119aa0,0,1,180,6ff957ac,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5119aa0,...) at mi_switch+0x217 sleepq_switch(c5119aa0,0,c0aa0842,21d,5c,...) at sleepq_switch+0xf0 sleepq_wait(c0bae2e0,0,c0a9d757,da,0,...) at sleepq_wait+0x60 _sleep(c0bae2e0,0,5c,c0a91723,0,...) at _sleep+0x369 usb_task_thread(c0bae2e0,e3d5cd38,c0a99763,2ea,c5215ab0,...) at usb_task_thread+0x62 fork_exit(c06ccb30,c0bae2e0,e3d5cd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d5cd70, ebp = 0 --- Tracing command usb0 pid 26 tid 100025 td 0xc5119880 sched_switch(c5119880,0,1,180,235937a,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,5c,...) at mi_switch+0x217 sleepq_switch(c5119880,0,c0aa0842,240,5c,...) at sleepq_switch+0xf0 sleepq_timedwait(c524b210,ea60,c0a916e6,0,0,...) at sleepq_timedwait+0x68 _sleep(c524b210,0,5c,c0a916e6,ea60,...) at _sleep+0x345 usb_event_thread(c5254280,e3d59d38,c0a99763,2ea,c5220000,...) at usb_event_thread+0xca fork_exit(c06cc3b0,c5254280,e3d59d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d59d70, ebp = 0 --- Tracing command irq16: uhci0 pid 25 tid 100026 td 0xc5119660 fork_trampoline() at fork_trampoline Tracing command irq9: acpi0 pid 24 tid 100027 td 0xc5222440 fork_trampoline() at fork_trampoline Tracing command kqueue taskq pid 23 tid 100028 td 0xc5222220 sched_switch(c5222220,0,1,180,70473070,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5222220,...) at mi_switch+0x217 sleepq_switch(c5222220,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0 sleepq_wait(c5212600,c521261c,c0a94778,0,0,...) at sleepq_wait+0x60 _sleep(c5212600,c521261c,0,c0a94778,0,...) at _sleep+0x369 taskqueue_thread_loop(c0baeec4,e57b2d38,c0a99763,2ea,c5220804,...) at taskqueue_thread_loop+0xb4 fork_exit(c0780c30,c0baeec4,e57b2d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57b2d70, ebp = 0 --- Tracing command swi6: task queue pid 22 tid 100029 td 0xc5222000 sched_switch(c5222000,0,1,180,cc82e284,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c52126e4,...) at mi_switch+0x217 ithread_loop(c5206120,e57afd38,c0a99763,2ea,c5220ab0,...) at ithread_loop+0x34c fork_exit(c07328a0,c5206120,e57afd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe57afd70, ebp = 0 --- Tracing command swi6: Giant taskq pid 21 tid 100013 td 0xc5117880 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 9 tid 100014 td 0xc5117660 sched_switch(c5117660,0,1,180,348742c8,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5117660,...) at mi_switch+0x217 sleepq_switch(c5117660,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0 sleepq_wait(c5212880,c521289c,c0a94778,0,0,...) at sleepq_wait+0x60 _sleep(c5212880,c521289c,0,c0a94778,0,...) at _sleep+0x369 taskqueue_thread_loop(c0bba7d8,e3d41d38,c0a99763,2ea,c515fab0,...) at taskqueue_thread_loop+0xb4 fork_exit(c0780c30,c0bba7d8,e3d41d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d41d70, ebp = 0 --- Tracing command acpi_task_2 pid 8 tid 100015 td 0xc5117440 sched_switch(c5117440,0,1,180,700a22a0,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5117440,...) at mi_switch+0x217 sleepq_switch(c5117440,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0 sleepq_wait(c5212900,c521291c,c0a94778,0,0,...) at sleepq_wait+0x60 _sleep(c5212900,c521291c,0,c0a94778,0,...) at _sleep+0x369 taskqueue_thread_loop(c0d7eebc,e3d3ed38,c0a99763,2ea,c5213000,...) at taskqueue_thread_loop+0xb4 fork_exit(c0780c30,c0d7eebc,e3d3ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d3ed70, ebp = 0 --- Tracing command acpi_task_1 pid 7 tid 100016 td 0xc5117220 sched_switch(c5117220,0,1,180,7009e0a0,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5117220,...) at mi_switch+0x217 sleepq_switch(c5117220,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0 sleepq_wait(c5212900,c521291c,c0a94778,0,0,...) at sleepq_wait+0x60 _sleep(c5212900,c521291c,0,c0a94778,0,...) at _sleep+0x369 taskqueue_thread_loop(c0d7eebc,e3d3bd38,c0a99763,2ea,c52132ac,...) at taskqueue_thread_loop+0xb4 fork_exit(c0780c30,c0d7eebc,e3d3bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d3bd70, ebp = 0 --- Tracing command acpi_task_0 pid 6 tid 100017 td 0xc5214aa0 sched_switch(c5214aa0,0,1,180,70099d0c,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5214aa0,...) at mi_switch+0x217 sleepq_switch(c5214aa0,0,c0aa0842,21d,0,...) at sleepq_switch+0xf0 sleepq_wait(c5212900,c521291c,c0a94778,0,0,...) at sleepq_wait+0x60 _sleep(c5212900,c521291c,0,c0a94778,0,...) at _sleep+0x369 taskqueue_thread_loop(c0d7eebc,e5794d38,c0a99763,2ea,c5213558,...) at taskqueue_thread_loop+0xb4 fork_exit(c0780c30,c0d7eebc,e5794d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe5794d70, ebp = 0 --- Tracing command swi5: + pid 20 tid 100018 td 0xc5214880 fork_trampoline() at fork_trampoline Tracing command swi2: cambio pid 19 tid 100019 td 0xc5214660 sched_switch(c5214660,0,1,180,34663108,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c5212ae4,...) at mi_switch+0x217 ithread_loop(c5206170,e578ed38,c0a99763,2ea,c5213ab0,...) at ithread_loop+0x34c fork_exit(c07328a0,c5206170,e578ed38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe578ed70, ebp = 0 --- Tracing command xpt_thrd pid 5 tid 100020 td 0xc5214440 sched_switch(c5214440,0,1,180,6ff8aad4,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5214440,...) at mi_switch+0x217 sleepq_switch(c5214440,0,c0aa0842,21d,4c,...) at sleepq_switch+0xf0 sleepq_wait(c0b7f514,0,c0a9d757,da,0,...) at sleepq_wait+0x60 _sleep(c0b7f514,c0b7f52c,4c,c0a535da,0,...) at _sleep+0x369 xpt_scanner_thread(0,e578bd38,c0a99763,2ea,c5215000,...) at xpt_scanner_thread+0x41 fork_exit(c0468360,0,e578bd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe578bd70, ebp = 0 --- Tracing command yarrow pid 18 tid 100006 td 0xc5115220 sched_switch(c5115220,0,1,180,79be0e56,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5115220,0,c0aa0842,240,0,...) at sleepq_switch+0xf0 sleepq_timedwait(c0bb0654,64,c0a94778,2,0,...) at sleepq_timedwait+0x68 _sleep(c0bb0654,0,0,c0a94778,64,...) at _sleep+0x345 pause(c0a94778,64,c0a88f43,112,0,...) at pause+0x47 random_kthread(0,e3d26d38,c0a99763,2ea,c51182ac,...) at random_kthread+0x1d9 fork_exit(c0656d20,0,e3d26d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d26d70, ebp = 0 --- Tracing command g_down pid 4 tid 100007 td 0xc5115000 sched_switch(c5115000,0,1,180,7c36f0fe,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217 sleepq_switch(c5115000,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0 sleepq_timedwait(c0bae6cc,64,c0a94778,0,0,...) at sleepq_timedwait+0x68 _sleep(c0bae6cc,c0bae5e8,24c,c0a94778,64,...) at _sleep+0x345 g_io_schedule_down(c5115000,0,c0a95db7,74,0,...) at g_io_schedule_down+0x6b g_down_procbody(0,e3d23d38,c0a99763,2ea,c5118558,...) at g_down_procbody+0x8d fork_exit(c07019c0,0,e3d23d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d23d70, ebp = 0 --- Tracing command g_up pid 3 tid 100008 td 0xc5119440 sched_switch(c5119440,0,1,180,7c372a56,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217 sleepq_switch(c5119440,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0 sleepq_timedwait(c0bae6c8,64,c0a94778,0,0,...) at sleepq_timedwait+0x68 _sleep(c0bae6c8,c0bae628,24c,c0a94778,64,...) at _sleep+0x345 g_io_schedule_up(c5119440,0,c0a95db7,5d,0,...) at g_io_schedule_up+0x133 g_up_procbody(0,e3d53d38,c0a99763,2ea,c5118804,...) at g_up_procbody+0x8d fork_exit(c0701a50,0,e3d53d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d53d70, ebp = 0 --- Tracing command g_event pid 2 tid 100009 td 0xc5119220 sched_switch(c5119220,0,1,180,79eea34a,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,4c,...) at mi_switch+0x217 sleepq_switch(c5119220,0,c0aa0842,240,4c,...) at sleepq_switch+0xf0 sleepq_timedwait(c0bae6c0,64,c0a94778,0,0,...) at sleepq_timedwait+0x68 _sleep(c0bae6c0,0,4c,c0a94778,64,...) at _sleep+0x345 g_event_procbody(0,e3d50d38,c0a99763,2ea,c5118ab0,...) at g_event_procbody+0xcb fork_exit(c0701ae0,0,e3d50d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d50d70, ebp = 0 --- Tracing command swi3: vm pid 17 tid 100010 td 0xc5119000 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 16 tid 100011 td 0xc5117cc0 cpustop_handler(8,e3d4abd0,c0a04a50,38,c0aa17fa,...) at cpustop_handler+0x32 ipi_nmi_handler(38,c0aa17fa,6ce,c0c01c00,13,...) at ipi_nmi_handler+0x2f trap(e3d4abdc) at trap+0x30 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc07429df, esp = 0xe3d4ac1c, ebp = 0xe3d4ac38 --- _mtx_lock_sleep(c58887a4,c5117cc0,0,c0ab3291,1b6,...) at _mtx_lock_sleep+0xbf _mtx_lock_flags(c58887a4,0,c0ab3291,1b6,10207,...) at _mtx_lock_flags+0xef tcp_timer_rexmt(c588a570,0,c0a9e01d,ef,0,...) at tcp_timer_rexmt+0x96 softclock(0,0,c0a999ef,471,c515e364,...) at softclock+0x266 ithread_loop(c5113260,e3d4ad38,c0a99763,2ea,c515f2ac,...) at ithread_loop+0x1b5 fork_exit(c07328a0,c5113260,e3d4ad38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d4ad70, ebp = 0 --- Tracing command swi1: net pid 15 tid 100012 td 0xc5117aa0 sched_switch(c5117aa0,0,1,180,386ad10c,...) at sched_switch+0x49b mi_switch(1,0,c0a999ef,46b,c515e3e4,...) at mi_switch+0x217 ithread_loop(c5113270,e3d47d38,c0a99763,2ea,c515f558,...) at ithread_loop+0x34c fork_exit(c07328a0,c5113270,e3d47d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d47d70, ebp = 0 --- Tracing command idle: cpu0 pid 14 tid 100000 td 0xc5117000 sched_switch(c5117000,0,6,17b,385804f8,...) at sched_switch+0x49b mi_switch(6,0,c0a9ec6c,b6,0,...) at mi_switch+0x217 critical_exit(c515e500,0,2,c5117000,c515e500,...) at critical_exit+0x99 intr_execute_handlers(c5107f34,e3d38c50,e3d38c90,c09eb844,34,...) at intr_execute_handlers+0x14b lapic_handle_intr(34,e3d38c50) at lapic_handle_intr+0x3f Xapic_isr1() at Xapic_isr1+0x34 --- interrupt, eip = 0xc0d6e4a5, esp = 0xe3d38c90, ebp = 0xe3d38c90 --- acpi_cpu_c1(c0b4fdf8,c5117000,e3d38cac,0,c0bb8680,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(e3d38cf8,c076c949,c0bb8680,0,c0a9e7d2,...) at acpi_cpu_idle+0x18c cpu_idle(c0bb8680,0,c0a9e7d2,322,c0bb2280,...) at cpu_idle+0x28 sched_idletd(0,e3d38d38,c0a99763,2ea,c5114000,...) at sched_idletd+0x249 fork_exit(c076c700,0,e3d38d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d38d70, ebp = 0 --- Tracing command idle: cpu1 pid 13 tid 100001 td 0xc5115cc0 sched_switch(c5115cc0,0,1,17b,37c397f4,...) at sched_switch+0x49b mi_switch(1,0,c0a9e7d2,320,c0bb28c0,...) at mi_switch+0x217 sched_idletd(0,e3d35d38,c0a99763,2ea,c51142ac,...) at sched_idletd+0x217 fork_exit(c076c700,0,e3d35d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d35d70, ebp = 0 --- Tracing command idle: cpu2 pid 12 tid 100002 td 0xc5115aa0 cpustop_handler(4,e3d32c44,c0a04a50,38,c0aa17fa,...) at cpustop_handler+0x32 ipi_nmi_handler(38,c0aa17fa,6ce,c0c01c00,13,...) at ipi_nmi_handler+0x2f trap(e3d32c50) at trap+0x30 calltrap() at calltrap+0x6 --- trap 0x13, eip = 0xc0d6e4a5, esp = 0xe3d32c90, ebp = 0xe3d32c90 --- acpi_cpu_c1(c0b4fdf8,c5115aa0,e3d32cac,2,c0bb8700,...) at acpi_cpu_c1+0x5 acpi_cpu_idle(e3d32cf8,c076c949,c0bb8700,0,c0a9e7d2,...) at acpi_cpu_idle+0x18c cpu_idle(c0bb8700,0,c0a9e7d2,322,c0bb2f00,...) at cpu_idle+0x28 sched_idletd(0,e3d32d38,c0a99763,2ea,c5114558,...) at sched_idletd+0x249 fork_exit(c076c700,0,e3d32d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d32d70, ebp = 0 --- Tracing command idle: cpu3 pid 11 tid 100003 td 0xc5115880 sched_switch(c5115880,0,1,17b,879e466e,...) at sched_switch+0x49b mi_switch(1,0,c0a9e7d2,320,c0bb3540,...) at mi_switch+0x217 sched_idletd(0,e3d2fd38,c0a99763,2ea,c5114804,...) at sched_idletd+0x217 fork_exit(c076c700,0,e3d2fd38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d2fd70, ebp = 0 --- Tracing command init pid 1 tid 100004 td 0xc5115660 sched_switch(c5115660,0,1,180,93649798,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,0,...) at mi_switch+0x217 sleepq_switch(c5115660,0,c0aa0842,19b,c5114ab0,...) at sleepq_switch+0xf0 sleepq_catch_signals(c07423dd,c0bb8680,4,c0a9bb4b,5c,...) at sleepq_catch_signals+0x21e sleepq_wait_sig(c5114ab0,0,c0a9d757,da,0,...) at sleepq_wait_sig+0x14 _sleep(c5114ab0,c5114b40,15c,c0aa296f,0,...) at _sleep+0x359 kern_wait(c5115660,ffffffff,e3d2cc2c,0,0,...) at kern_wait+0xa66 wait4(c5115660,e3d2ccfc,10,e3d2cd38,c0b49b28,...) at wait4+0x3b syscall(e3d2cd38) at syscall+0x2b3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (7, FreeBSD ELF32, wait4), eip = 0x805455f, esp = 0xbfbfe97c, ebp = 0xbfbfe998 --- Tracing command audit pid 10 tid 100005 td 0xc5115440 sched_switch(c5115440,0,1,180,6fef90bc,...) at sched_switch+0x49b mi_switch(1,0,c0aa0842,1bf,c5115440,...) at mi_switch+0x217 sleepq_switch(c5115440,0,c0aa0842,21d,c5115440,...) at sleepq_switch+0xf0 sleepq_wait(c0c0d5b4,c0c0d594,c0ab9dd3,1,0,...) at sleepq_wait+0x60 _cv_wait(c0c0d5b4,c0c0d594,c0aba790,1c2,0,...) at _cv_wait+0x1fc audit_worker(0,e3d29d38,c0a99763,2ea,c5118000,...) at audit_worker+0x99 fork_exit(c091b750,0,e3d29d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe3d29d70, ebp = 0 --- Tracing command swapper pid 0 tid 0 td 0xc0baea30 sched_switch(c0baea30,0,1,180,3991d04c,...) at sched_switch+0x49b mi_switch(1,0,c0abea20,2d9,c0ac973d,...) at mi_switch+0x217 scheduler(0,101ec00,101ec00,101e000,1028000,...) at scheduler+0x2f2 mi_startup() at mi_startup+0x96 begin() at begin+0x2c db> show allocks No such command db> show alllocks Process 943 (sshd) thread 0xc5861440 (100144) exclusive sleep mutex radix node head r = 0 (0xc538d07c) locked @ /usr/src/sys/net/route.c:147 exclusive sleep mutex inp (tcpinp) r = 0 (0xc58887a4) locked @ /usr/src/sys/netinet/tcp_usrreq.c:779 exclusive sx so_snd_sx r = 0 (0xc5903430) locked @ /usr/src/sys/kern/uipc_sockbuf.c:145 Process 940 (sshd) thread 0xc5861220 (100145) exclusive sx so_rcv_sx r = 0 (0xc58a8530) locked @ /usr/src/sys/kern/uipc_sockbuf.c:145 Process 33 (irq21: bge0) thread 0xc5214cc0 (100037) exclusive sleep mutex rtentry r = 0 (0xc5722c18) locked @ /usr/src/sys/net/route.c:1065 exclusive spin mutex sio r = 0 (0xc0c1b0e8) locked @ /usr/src/sys/dev/sio/sio.c:1391 Process 16 (swi4: clock sio) thread 0xc5117cc0 (100011) exclusive sleep mutex tcp r = 0 (0xc0c0472c) locked @ /usr/src/sys/netinet/tcp_timer.c:423 db> show lockedvnods Locked vnodes db> call boot(0) panic: blockable sleep lock (sleep mutex) eventhandler @ /usr/src/sys/kern/subr_eventhandler.c:212 cpuid = 0 KDB: enter: panic [thread pid 33 tid 100037 ] Stopped at kdb_enter+0x32: leave db> reboot