From owner-freebsd-stable@FreeBSD.ORG Wed Jan 11 15:02:46 2006 Return-Path: X-Original-To: freebsd-stable@FreeBSD.org Delivered-To: freebsd-stable@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C327816A41F; Wed, 11 Jan 2006 15:02:46 +0000 (GMT) (envelope-from dsh@vlink.ru) Received: from deliver.smtp.vlink.ru (vlink-1.avtlg.ru [83.239.142.33]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7A53643D46; Wed, 11 Jan 2006 15:02:44 +0000 (GMT) (envelope-from dsh@vlink.ru) Received: from smtp.smtp.vlink.ru (clamav.smtp.vlink.ru [192.168.4.1]) by deliver.smtp.vlink.ru (Postfix) with ESMTP id 699FEFED0C8; Wed, 11 Jan 2006 18:02:42 +0300 (MSK) Received: from neva.vlink.ru (neva.vlink.ru [217.107.252.66]) by smtp.smtp.vlink.ru (Postfix) with ESMTP id 5A9361009A9E; Wed, 11 Jan 2006 18:02:40 +0300 (MSK) Received: from neva.vlink.ru (localhost [127.0.0.1]) by neva.vlink.ru (8.13.4/8.13.4) with ESMTP id k0BF2UIJ011158; Wed, 11 Jan 2006 18:02:30 +0300 (MSK) (envelope-from dsh@vlink.ru) Received: (from dsh@localhost) by neva.vlink.ru (8.13.4/8.13.4/Submit) id k0BF2TDv011155; Wed, 11 Jan 2006 18:02:29 +0300 (MSK) (envelope-from dsh@vlink.ru) X-Comment-To: Don Lewis To: Don Lewis References: <200601052157.k05LvJV0015906@gw.catspoiler.org> From: Denis Shaposhnikov Date: Wed, 11 Jan 2006 18:02:29 +0300 In-Reply-To: <200601052157.k05LvJV0015906@gw.catspoiler.org> (Don Lewis's message of "Thu, 5 Jan 2006 13:57:19 -0800 (PST)") Message-ID: <87wth6byey.fsf@neva.vlink.ru> User-Agent: Gnus/5.1007 (Gnus v5.10.7) XEmacs/21.4.18 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-stable@FreeBSD.org Subject: Re: Recurring problem: processes block accessing UFS file system X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2006 15:02:46 -0000 Hi! >>>>> "Don" == Don Lewis writes: Don> Are you using any unusual file systems, such as nullfs or Don> unionfs? >> Yes, I'm use a lots of nullfs. This is a host system for about 20 >> jails with nullfs mounted ro system: Don> That would be my guess as to the cause of the problem. Hopefully Don> DEBUG_VFS_LOCKS will help pinpoint the bug. I've got the problem again. Now I have debug kernel and crash dump. That is an output from the kdb. Do you need any additional information which I can get from the crash dump? Script started on Wed Jan 11 16:19:51 2006 Connected ~KDB: enter: Line break on console [thread pid 10 tid 100006 ] Stopped at kdb_enter+0x32: leave db> ps pid proc uid ppid pgrp flag stat wmesg wchan cmd 33279 c8fb6684 0 1 1 0000000 [SLPQ ufs 0xc690d454][SLP] sh 33017 c7f998b0 1008 1 2692 0000000 [SLPQ ufs 0xc690d300][SLP] httpd 33016 c9261000 1008 1 2692 0004000 [SLPQ ufs 0xc6bed454][SLP] parser3.cgi 33015 c8e1fd08 1008 1 2692 0004000 [SLPQ ufs 0xc6bed454][SLP] perl5.8.7 33013 c9796458 1008 1 2692 0004000 [SLPQ ufs 0xc690d454][SLP] parser3.cgi 30627 c8f2fd08 1012 21055 30627 0004002 [SLPQ ufs 0xc690d454][SLP] perl5.8.7 30210 c86e3684 1010 1 30210 0004000 [SLPQ ufs 0xc690d454][SLP] sftp-server 21225 c858522c 1012 1 21225 0004000 [SLPQ ufs 0xc6bf9300][SLP] sftp-server 21055 c6c6d684 1012 1 21055 0015002 [SLPQ ufs 0xc690d454][SLP] zsh 21031 c62978b0 1012 1 21031 0004002 [SLPQ ufs 0xc690d454][SLP] zsh 21003 c9269684 0 2728 21003 0004100 [SLPQ ufs 0xc690d454][SLP] sshd 5004 c941eadc 1010 1 5004 0004000 [SLPQ ufs 0xc690d454][SLP] sftp-server 2735 c8007684 0 1 2735 0000000 [SLPQ ufs 0xc690d454][SLP] cron 2728 c7ca1000 0 1 2728 0000100 [SLPQ ufs 0xc690d454][SLP] sshd 2705 c8006d08 1008 1 2703 0004002 [SLPQ ufs 0xc690d454][SLP] perl5.8.7 2639 c8006adc 0 1 2639 0000000 [SLPQ ufs 0xc690d454][SLP] syslogd 546 c6a10684 80 1 540 0000100 [SLPQ ufs 0xc6c07c4c][SLP] nginx 36 c60d722c 0 0 0 0000204 [SLPQ - 0xe4c45cf0][SLP] schedcpu 35 c60d7458 0 0 0 0000204 [SLPQ ktsusp 0xc60d74e0][SLP] syncer 34 c60d7684 0 0 0 0000204 [SLPQ ktsusp 0xc60d770c][SLP] vnlru 33 c60d78b0 0 0 0 0000204 [SLPQ ktsusp 0xc60d7938][SLP] bufdaemon 32 c60d7adc 0 0 0 000020c [SLPQ pgzero 0xc06d0b14][SLP] pagezero 31 c60d7d08 0 0 0 0000204 [SLPQ psleep 0xc06c8a48][SLP] vmdaemon 30 c60d8000 0 0 0 0000204 [SLPQ psleep 0xc06c8a08][SLP] pagedaemon 29 c60d822c 0 0 0 0000204 [IWAIT] irq1: atkbd0 28 c60d8458 0 0 0 0000204 [IWAIT] swi0: sio 27 c602d684 0 0 0 0000204 [IWAIT] irq18: atapci1 26 c602d8b0 0 0 0 0000204 [IWAIT] irq15: ata1 25 c602dadc 0 0 0 0000204 [IWAIT] irq14: ata0 24 c602dd08 0 0 0 0000204 [IWAIT] irq27: em1 23 c6070000 0 0 0 0000204 [IWAIT] irq26: em0 22 c607022c 0 0 0 0000204 [IWAIT] irq9: acpi0 21 c6070458 0 0 0 0000204 [IWAIT] swi2: cambio 20 c6070684 0 0 0 0000204 [IWAIT] swi6: task queue 9 c60708b0 0 0 0 0000204 [SLPQ - 0xc6064e00][SLP] acpi_task2 8 c6070adc 0 0 0 0000204 [SLPQ - 0xc6064e00][SLP] acpi_task1 7 c5fd822c 0 0 0 0000204 [SLPQ - 0xc6064e00][SLP] acpi_task0 19 c5fd8458 0 0 0 0000204 [IWAIT] swi6: Giant taskq 6 c5fd8684 0 0 0 0000204 [SLPQ - 0xc6065000][SLP] thread taskq 18 c5fd88b0 0 0 0 0000204 [IWAIT] swi5: Fast taskq 5 c5fd8adc 0 0 0 0000204 [SLPQ - 0xc60650c0][SLP] kqueue taskq 17 c5fd8d08 0 0 0 0000204 [SLPQ - 0xc06b8460][SLP] yarrow 4 c602d000 0 0 0 0000204 [SLPQ - 0xc06b8d00][SLP] g_down 3 c602d22c 0 0 0 0000204 [SLPQ - 0xc06b8cfc][SLP] g_up 2 c602d458 0 0 0 0000204 [SLPQ - 0xc06b8cf4][SLP] g_event 16 c5fd3000 0 0 0 0000204 [IWAIT] swi1: net 15 c5fd322c 0 0 0 0000204 [IWAIT] swi3: vm 14 c5fd3458 0 0 0 000020c [IWAIT] swi4: clock sio 13 c5fd3684 0 0 0 000020c [CPU 0] idle: cpu0 12 c5fd38b0 0 0 0 000020c [CPU 1] idle: cpu1 11 c5fd3adc 0 0 0 000020c [CPU 2] idle: cpu2 10 c5fd3d08 0 0 0 000020c [CPU 3] idle: cpu3 1 c5fd8000 0 0 1 0004200 [SLPQ ufs 0xc690d5a8][SLP] init 0 c06b8e00 0 0 0 0000200 [IWAIT] swapper 21030 c8e1f8b0 1012 21003 21003 0002100 zomb[INACTIVE] sshd 5005 c93f8684 0 2728 5005 0006100 zomb[INACTIVE] sshd 5000 c83d1d08 0 2728 5000 0006100 zomb[INACTIVE] sshd 30211 c7907d08 0 2728 30211 0006100 zomb[INACTIVE] sshd 4997 c6a10adc 0 2728 4997 0006100 zomb[INACTIVE] sshd 6404 c8558684 0 2728 6404 0006100 zomb[INACTIVE] sshd db> show lockedvnods Locked vnodes 0xc690d550: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xc9413934 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc8f84000 (pid 33279) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6 #1 0xc053d780 at vop_stdlock+0x32 #2 0xc0655075 at VOP_LOCK_APV+0x3b #3 0xc05db299 at ffs_lock+0x19 #4 0xc0655075 at VOP_LOCK_APV+0x3b #5 0xc0557513 at vn_lock+0x6c #6 0xc054a320 at vget+0x87 #7 0xc053a1f0 at cache_lookup+0xde #8 0xc053ac93 at vfs_cache_lookup+0xad #9 0xc0652eaf at VOP_LOOKUP_APV+0x3b #10 0xc053f786 at lookup+0x419 #11 0xc0540262 at namei+0x2e7 #12 0xc0558bb2 at vn_open_cred+0x3d #13 0xc055918a at vn_open+0x33 #14 0xc054e6d3 at kern_open+0xd2 #15 0xc054f055 at open+0x36 #16 0xc0648a47 at syscall+0x23a #17 0xc06326ff at Xint0x80_syscall+0x1f ino 2072602, on dev ad4s1g 0xc690d3fc: tag ufs, type VDIR usecount 48, writecount 0, refcount 49 mountedhere 0 flags () v_object 0xc810c6c8 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc7f9c000 (pid 33017) with 12 pending#0 0xc04cd7ad at lockmgr+0x3c6 #1 0xc053d780 at vop_stdlock+0x32 #2 0xc0655075 at VOP_LOCK_APV+0x3b #3 0xc05db299 at ffs_lock+0x19 #4 0xc0655075 at VOP_LOCK_APV+0x3b #5 0xc0557513 at vn_lock+0x6c #6 0xc053f430 at lookup+0xc3 #7 0xc0540262 at namei+0x2e7 #8 0xc0558d37 at vn_open_cred+0x1c2 #9 0xc055918a at vn_open+0x33 #10 0xc054e6d3 at kern_open+0xd2 #11 0xc054f055 at open+0x36 #12 0xc0648a47 at syscall+0x23a #13 0xc06326ff at Xint0x80_syscall+0x1f ino 2072603, on dev ad4s1g 0xc690d2a8: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xc839a8b8 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc8584300 (pid 21225) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6 #1 0xc053d780 at vop_stdlock+0x32 #2 0xc0655075 at VOP_LOCK_APV+0x3b #3 0xc05db299 at ffs_lock+0x19 #4 0xc0655075 at VOP_LOCK_APV+0x3b #5 0xc0557513 at vn_lock+0x6c #6 0xc054a320 at vget+0x87 #7 0xc053a1f0 at cache_lookup+0xde #8 0xc053ac93 at vfs_cache_lookup+0xad #9 0xc0652eaf at VOP_LOOKUP_APV+0x3b #10 0xc053f786 at lookup+0x419 #11 0xc0540262 at namei+0x2e7 #12 0xc0558d37 at vn_open_cred+0x1c2 #13 0xc055918a at vn_open+0x33 #14 0xc054e6d3 at kern_open+0xd2 #15 0xc054f055 at open+0x36 #16 0xc0648a47 at syscall+0x23a #17 0xc06326ff at Xint0x80_syscall+0x1f ino 2072615, on dev ad4s1g 0xc6bf92a8: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xc8a18174 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc9429780 (pid 33015) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6 #1 0xc053d780 at vop_stdlock+0x32 #2 0xc0655075 at VOP_LOCK_APV+0x3b #3 0xc05db299 at ffs_lock+0x19 #4 0xc0655075 at VOP_LOCK_APV+0x3b #5 0xc0557513 at vn_lock+0x6c #6 0xc054a320 at vget+0x87 #7 0xc053a1f0 at cache_lookup+0xde #8 0xc053ac93 at vfs_cache_lookup+0xad #9 0xc0652eaf at VOP_LOOKUP_APV+0x3b #10 0xc053f786 at lookup+0x419 #11 0xc0540262 at namei+0x2e7 #12 0xc05513e6 at kern_stat+0x3d #13 0xc05514e8 at stat+0x2f #14 0xc0648a47 at syscall+0x23a #15 0xc06326ff at Xint0x80_syscall+0x1f ino 2072680, on dev ad4s1g 0xc6bed3fc: tag ufs, type VDIR usecount 3, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xc84fce0c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc69d1d80 (pid 546) with 2 pending#0 0xc04cd7ad at lockmgr+0x3c6 #1 0xc053d780 at vop_stdlock+0x32 #2 0xc0655075 at VOP_LOCK_APV+0x3b #3 0xc05db299 at ffs_lock+0x19 #4 0xc0655075 at VOP_LOCK_APV+0x3b #5 0xc0557513 at vn_lock+0x6c #6 0xc054a320 at vget+0x87 #7 0xc053a1f0 at cache_lookup+0xde #8 0xc053ac93 at vfs_cache_lookup+0xad #9 0xc0652eaf at VOP_LOOKUP_APV+0x3b #10 0xc053f786 at lookup+0x419 #11 0xc0540262 at namei+0x2e7 #12 0xc0558d37 at vn_open_cred+0x1c2 #13 0xc055918a at vn_open+0x33 #14 0xc054e6d3 at kern_open+0xd2 #15 0xc054f055 at open+0x36 #16 0xc0648a47 at syscall+0x23a #17 0xc06326ff at Xint0x80_syscall+0x1f ino 2072795, on dev ad4s1g 0xc6c07bf4: tag ufs, type VDIR usecount 2, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xc940a744 ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc9146c00 (pid 33016) with 1 pending#0 0xc04cd7ad at lockmgr+0x3c6 #1 0xc053d780 at vop_stdlock+0x32 #2 0xc0655075 at VOP_LOCK_APV+0x3b #3 0xc05db299 at ffs_lock+0x19 #4 0xc0655075 at VOP_LOCK_APV+0x3b #5 0xc0557513 at vn_lock+0x6c #6 0xc054a320 at vget+0x87 #7 0xc053a1f0 at cache_lookup+0xde #8 0xc053ac93 at vfs_cache_lookup+0xad #9 0xc0652eaf at VOP_LOOKUP_APV+0x3b #10 0xc053f786 at lookup+0x419 #11 0xc0540262 at namei+0x2e7 #12 0xc0554134 at kern_rmdir+0x98 #13 0xc055436e at rmdir+0x22 #14 0xc0648a47 at syscall+0x23a #15 0xc06326ff at Xint0x80_syscall+0x1f ino 2072767, on dev ad4s1g db> show lockedbufs db> db> alltrace Tracing command sh pid 33279 tid 100567 td 0xc8f84000 sched_switch(c8f84000,0,1,3ef33c46,367ee305) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e91b8670,c04f425b) at mi_switch+0x150 sleepq_switch(e91b86a4,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,1) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,c0655075,c05db299,c0655075,c0557513) at acquire+0x7a lockmgr(c690d454,2002,c690d4c4,c8f84000,e91b878c) at lockmgr+0x4ce vop_stdlock(e91b87e4,c06a4f20,c06a4f20,e91b87e4,e91b879c) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e91b87e4,e91b87b4,c0655075,e91b87e4) at VOP_LOCK_APV+0x3b ffs_lock(e91b87e4,c0673faf,2,c690d3fc,e91b8800) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e91b87e4,e91b8814,c0504340,e91b8814) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,2002,c8f84000,c05e52a9,c690d550) at vn_lock+0x6c vget(c690d3fc,2002,c8f84000,0,e91b88a8) at vget+0x87 cache_lookup(c690d550,e91b8b80,e91b8b94,c8f84000,c5fd1780) at cache_lookup+0xde vfs_cache_lookup(e91b8940,e91b88ec,c690d550,e91b8b94,e91b895c) at vfs_cache_lookup+0xad VOP_LOOKUP_APV(c06a4f20,e91b8940,e91b892c,c8f84000,2) at VOP_LOOKUP_APV+0x3b lookup(e91b8b6c,c8d8b800,400,e91b8b88,c053d7b7) at lookup+0x419 namei(e91b8b6c,c678a618,c8f84000,e91b89f8,c0655118) at namei+0x2e7 vn_open_cred(e91b8b6c,e91b8c6c,1a4,c5fd1780,3) at vn_open_cred+0x3d vn_open(e91b8b6c,e91b8c6c,1a4,3,0) at vn_open+0x33 kern_open(c8f84000,8084768,0,20a,1b6) at kern_open+0xd2 open(c8f84000,e91b8d04,c,3,e91b8d38) at open+0x36 syscall(3b,3b,3b,8084768,1) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x281a6d6b, esp = 0xbfbfd08c, ebp = 0xbfbfd128 --- Tracing command httpd pid 33017 tid 100300 td 0xc7f9c000 sched_switch(c7f9c000,0,1,9d63b87a,bfac35b1) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8d94670,c04f425b) at mi_switch+0x150 sleepq_switch(e8d946a4,c04e4cc6,c690d300,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d300,50,c0672a31,0,d) at sleepq_wait+0xb msleep(c690d300,c06b9cd0,50,c0672a31,0) at msleep+0x454 acquire(60000,c7f9c000,e8d94718,c064191c,dab8b000) at acquire+0x7a lockmgr(c690d300,2002,c690d370,c7f9c000,e8d9478c) at lockmgr+0x4ce vop_stdlock(e8d947e4,da08b520,c06a4f20,e8d947e4,e8d9479c) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8d947e4,e8d947b4,c0655075,e8d947e4) at VOP_LOCK_APV+0x3b ffs_lock(e8d947e4,4c,2,c690d2a8,e8d94800) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8d947e4,40,e8d94848,c04cd7ad) at VOP_LOCK_APV+0x3b vn_lock(c690d2a8,2002,c7f9c000,c05e52a9,c690d3fc) at vn_lock+0x6c vget(c690d2a8,2002,c7f9c000,0,e8d948a8) at vget+0x87 cache_lookup(c690d3fc,e8d94b80,e8d94b94,c7f9c000,c7fac400) at cache_lookup+0xde vfs_cache_lookup(e8d94940,e8d948ec,c690d3fc,e8d94b94,e8d9495c) at vfs_cache_lookup+0xad VOP_LOOKUP_APV(c06a4f20,e8d94940,c7f9c000,c9120050,2) at VOP_LOOKUP_APV+0x3b lookup(e8d94b6c,c9ac3c00,400,e8d94b88,c802407c) at lookup+0x419 namei(e8d94b6c,e8d949f4,c06316b2,e8d94a18,c06326aa) at namei+0x2e7 vn_open_cred(e8d94b6c,e8d94c6c,0,c7fac400,a) at vn_open_cred+0x1c2 vn_open(e8d94b6c,e8d94c6c,0,a,0) at vn_open+0x33 kern_open(c7f9c000,81cb068,0,2,0) at kern_open+0xd2 open(c7f9c000,e8d94d04,c,0,e8d94d38) at open+0x36 syscall(811003b,e4d003b,bfbf003b,8103358,1) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0xe6faf6b, esp = 0xbfbfea7c, ebp = 0xbfbfeaa8 --- Tracing command parser3.cgi pid 33016 tid 100652 td 0xc9146c00 sched_switch(c9146c00,0,1,5e57bc72,aaef377f) at sched_switch+0xd8 mi_switch(1,0,0,e93d86c4,c04f425b) at mi_switch+0x150 sleepq_switch(e93d86f8,c04e4cc6,c6bed454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c6bed454,50,c0672a31,0,c602a080) at sleepq_wait+0xb msleep(c6bed454,c06b9c98,50,c0672a31,0) at msleep+0x454 acquire(60000,da1dcb4c,c0504340,da1dcb4c,4c) at acquire+0x7a lockmgr(c6bed454,2002,c6bed4c4,c9146c00,e93d87e0) at lockmgr+0x4ce vop_stdlock(e93d8838,120,c06a4f20,e93d8838,e93d87f0) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e93d8838,e93d8808,c0655075,e93d8838) at VOP_LOCK_APV+0x3b ffs_lock(e93d8838,f,2,c6bed3fc,e93d8854) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e93d8838,c0653a6a,c053acd2,c0652eaf) at VOP_LOCK_APV+0x3b vn_lock(c6bed3fc,2002,c9146c00,c06326ff,6) at vn_lock+0x6c vget(c6bed3fc,2002,c9146c00,c0653a6a,c053acd2) at vget+0x87 vfs_hash_get(c6268c00,1fa0db,2,c9146c00,e93d89bc) at vfs_hash_get+0xf5 ffs_vget(c6268c00,1fa0db,2,e93d89bc,e93d89c0) at ffs_vget+0x49 ufs_lookup(e93d8a6c,c0685ac5,c6c07bf4,e93d8c34,e93d8aa8) at ufs_lookup+0x965 VOP_CACHEDLOOKUP_APV(c06a4f20,e93d8a6c,e93d8c34,c9146c00,c7fac400) at VOP_CACHEDLOOKUP_APV+0x59 vfs_cache_lookup(e93d8b14,e93d8ac0,c6c07bf4,e93d8c34,e93d8b30) at vfs_cache_lookup+0xec VOP_LOOKUP_APV(c06a4f20,e93d8b14,c9146c00,c065510a,c054a6f6) at VOP_LOOKUP_APV+0x3b lookup(e93d8c0c,c954a800,400,e93d8c28,0) at lookup+0x419 namei(e93d8c0c,ffffffdf,2,0,c6268c00) at namei+0x2e7 kern_rmdir(c9146c00,824b500,0,e93d8d30,c0648a47) at kern_rmdir+0x98 rmdir(c9146c00,e93d8d04,4,d,e93d8d38) at rmdir+0x22 syscall(804003b,e50003b,bfbf003b,8310560,8310560) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (137, FreeBSD ELF32, rmdir), eip = 0xe8201bf, esp = 0xbfbfbcfc, ebp = 0xbfbfbd18 --- Tracing command perl5.8.7 pid 33015 tid 100540 td 0xc9429780 sched_switch(c9429780,0,1,3e4e9c0e,ccaac762) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e90837c8,c04f425b) at mi_switch+0x150 sleepq_switch(e90837fc,c04e4cc6,c6bed454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c6bed454,50,c0672a31,0,c06326ff) at sleepq_wait+0xb msleep(c6bed454,c06b9c98,50,c0672a31,0) at msleep+0x454 acquire(60000,c0648a47,c06326ff,0,0) at acquire+0x7a lockmgr(c6bed454,2002,c6bed4c4,c9429780,e90838e4) at lockmgr+0x4ce vop_stdlock(e908393c,c06a4f20,c06a4f20,e908393c,e90838f4) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e908393c,e908390c,c0655075,e908393c) at VOP_LOCK_APV+0x3b ffs_lock(e908393c,c0673faf,2,c6bed3fc,e9083958) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e908393c,e908396c,c0504340,e908396c) at VOP_LOCK_APV+0x3b vn_lock(c6bed3fc,2002,c9429780,c05e52a9,c6bf92a8) at vn_lock+0x6c vget(c6bed3fc,2002,c9429780,0,e9083a00) at vget+0x87 cache_lookup(c6bf92a8,e9083bb4,e9083bc8,c9429780,c7fac400) at cache_lookup+0xde vfs_cache_lookup(e9083a98,e9083a44,c6bf92a8,e9083bc8,e9083ab4) at vfs_cache_lookup+0xad VOP_LOOKUP_APV(c06a4f20,e9083a98,c9429780,c053838f,da2dbca8) at VOP_LOOKUP_APV+0x3b lookup(e9083ba0,c9890400,400,e9083bbc,c7ec7800) at lookup+0x419 namei(e9083ba0,c054aeb7,c0602f4b,e9083b74,c97a71f0) at namei+0x2e7 kern_stat(c9429780,86e80d4,0,e9083c14,c7d04494) at kern_stat+0x3d stat(c9429780,e9083d04,8,5,e9083d38) at stat+0x2f syscall(bfbf003b,bfbf003b,bfbf003b,86e80d4,86e80d4) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0xe696f0b, esp = 0xbfbf159c, ebp = 0xbfbf1618 --- Tracing command parser3.cgi pid 33013 tid 100514 td 0xc9662000 sched_switch(c9662000,0,1,ce8a74bc,fee529ff) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e910a978,c04f425b) at mi_switch+0x150 sleepq_switch(e910a9ac,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,c057cbf1,c67ebc1c,c882b700,e910aa28) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c9662000,e910aa94) at lockmgr+0x4ce vop_stdlock(e910aaec,0,c06a4f20,e910aaec,e910aaa4) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e910aaec,e910aabc,c0655075,e910aaec) at VOP_LOCK_APV+0x3b ffs_lock(e910aaec,c0face68,1002,c690d3fc,e910ab08) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e910aaec,0,e910ab18,0) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c9662000,c05f6b30,c0face68) at vn_lock+0x6c lookup(e910ac08,c8d9e800,400,e910ac24,3) at lookup+0xc3 namei(e910ac08,27f,0,0,c7f74a80) at namei+0x2e7 kern_access(c9662000,8164270,0,2,e910ad30) at kern_access+0x72 access(c9662000,e910ad04,8,1,e910ad38) at access+0x29 syscall(3b,3b,3b,8164270,8185c60) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (33, FreeBSD ELF32, access), eip = 0xe882a53, esp = 0xbfbfbd4c, ebp = 0xbfbfbd98 --- Tracing command perl5.8.7 pid 30627 tid 100547 td 0xc9429480 sched_switch(c9429480,0,1,b326927a,259d4869) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e907d8e0,c04f425b) at mi_switch+0x150 sleepq_switch(e907d914,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,c0547c81) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,4,4,e907d9cc,c053838f) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c9429480,e907d9fc) at lockmgr+0x4ce vop_stdlock(e907da54,c06316b2,c06a4f20,e907da54,e907da0c) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e907da54,e907da24,c0655075,e907da54) at VOP_LOCK_APV+0x3b ffs_lock(e907da54,c0655075,1002,c690d3fc,e907da70) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e907da54,c053d7b7,c065510a,c054a6f6) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c9429480,c9429480,e907daa0) at vn_lock+0x6c lookup(e907dc08,c9a55000,400,e907dc24,0) at lookup+0xc3 namei(e907dc08,c9429480,e907db80,c9429480,8d94000) at namei+0x2e7 kern_rename(c9429480,8602b40,86f7f40,0,e907dd30) at kern_rename+0x46 rename(c9429480,e907dd04,8,c06bc640,c5fd4d60) at rename+0x29 syscall(3b,895003b,bfbf003b,8602b40,8c34008) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (128, FreeBSD ELF32, rename), eip = 0xb42721f, esp = 0xbfbfeacc, ebp = 0xbfbfeaf8 --- Tracing command sftp-server pid 30210 tid 100409 td 0xc8583480 sched_switch(c8583480,0,1,8f67a44c,cabc55b8) at sched_switch+0xd8 mi_switch(1,0,0,e8f938b4,c04f425b) at mi_switch+0x150 sleepq_switch(e8f938e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,b) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,2169,d860e58a,ddcc828d,be7ecac0) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c8583480,e8f939d0) at lockmgr+0x4ce vop_stdlock(e8f93a28,c86e3684,c06a4f20,e8f93a28,e8f939e0) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8f93a28,e8f939f8,c0655075,e8f93a28) at VOP_LOCK_APV+0x3b ffs_lock(e8f93a28,c06bc640,1002,c690d3fc,e8f93a44) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8f93a28,c942b480,7fffffff,c86e3748) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c8583480,c8583480,c06bf160) at vn_lock+0x6c lookup(e8f93ba0,c9ab6c00,400,e8f93bbc,e8f93b48) at lookup+0xc3 namei(e8f93ba0,ffffffff,7fffffff,c97cac94,4) at namei+0x2e7 kern_stat(c8583480,804e080,0,e8f93c14,e8f93c40) at kern_stat+0x3d stat(c8583480,e8f93d04,8,2,e8f93d38) at stat+0x2f syscall(3b,3b,3b,804e080,bfbea620) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0xb497f0b, esp = 0xbfbea3fc, ebp = 0xbfbea698 --- Tracing command sftp-server pid 21225 tid 100402 td 0xc8584300 sched_switch(c8584300,0,1,fadc2abe,86dd987) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8fae670,c04f425b) at mi_switch+0x150 sleepq_switch(e8fae6a4,c04e4cc6,c6bf9300,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c6bf9300,50,c0672a31,0,b) at sleepq_wait+0xb msleep(c6bf9300,c06ba8f4,50,c0672a31,0) at msleep+0x454 acquire(60000,0,e8fae8fc,c0632c7e,e8fae724) at acquire+0x7a lockmgr(c6bf9300,2002,c6bf9370,c8584300,e8fae78c) at lockmgr+0x4ce vop_stdlock(e8fae7e4,c06a4f20,c06a4f20,e8fae7e4,e8fae79c) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8fae7e4,e8fae7b4,c0655075,e8fae7e4) at VOP_LOCK_APV+0x3b ffs_lock(e8fae7e4,c0673faf,2,c6bf92a8,e8fae800) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8fae7e4,e8fae814,c0504340,e8fae814) at VOP_LOCK_APV+0x3b vn_lock(c6bf92a8,2002,c8584300,c05e52a9,c690d2a8) at vn_lock+0x6c vget(c6bf92a8,2002,c8584300,0,e8fae8a8) at vget+0x87 cache_lookup(c690d2a8,e8faeb80,e8faeb94,c8584300,c8836180) at cache_lookup+0xde vfs_cache_lookup(e8fae940,e8fae8ec,c690d2a8,e8faeb94,e8fae95c) at vfs_cache_lookup+0xad VOP_LOOKUP_APV(c06a4f20,e8fae940,c8584300,c05589ca,c0558a6d) at VOP_LOOKUP_APV+0x3b lookup(e8faeb6c,c99c9800,400,e8faeb88,c21845d8) at lookup+0x419 namei(e8faeb6c,c7ca4d60,c06bc640,c858522c,e8fae9fc) at namei+0x2e7 vn_open_cred(e8faeb6c,e8faec6c,1a4,c8836180,5) at vn_open_cred+0x1c2 vn_open(e8faeb6c,e8faec6c,1a4,5,c80b9700) at vn_open+0x33 kern_open(c8584300,8052000,0,1,1b6) at kern_open+0xd2 open(c8584300,e8faed04,c,369e99,0) at open+0x36 syscall(804003b,804003b,bfbf003b,0,1) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0xb497f6b, esp = 0xbfbea6bc, ebp = 0xbfbfad08 --- Tracing command zsh pid 21055 tid 100144 td 0xc6a4c480 sched_switch(c6a4c480,0,1,2ef60aaa,c21bdf61) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8aab8b4,c04f425b) at mi_switch+0x150 sleepq_switch(e8aab8e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,0,80000,0,0) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c6a4c480,e8aab9d0) at lockmgr+0x4ce vop_stdlock(e8aaba28,c0557513,c06a4f20,e8aaba28,e8aab9e0) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8aaba28,e8aab9f8,c0655075,e8aaba28) at VOP_LOCK_APV+0x3b ffs_lock(e8aaba28,c84ab618,1002,c690d3fc,e8aaba44) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8aaba28,e8aaba18,46,e8aaba3c) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c6a4c480,8,28) at vn_lock+0x6c lookup(e8aabba0,c9a59000,400,e8aabbbc,c8683c8c) at lookup+0xc3 namei(e8aabba0,0,e8aabc04,e8aabbe7,e8aabbe8) at namei+0x2e7 kern_stat(c6a4c480,8054410,0,e8aabc14,0) at kern_stat+0x3d stat(c6a4c480,e8aabd04,8,3,e8aabd38) at stat+0x2f syscall(3b,3b,3b,bfbfe354,805441a) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0xb419f0b, esp = 0xbfbfe27c, ebp = 0xbfbfe338 --- Tracing command zsh pid 21031 tid 100053 td 0xc6294480 sched_switch(c6294480,0,1,c3ff51ac,c4792049) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e89478b4,c04f425b) at mi_switch+0x150 sleepq_switch(e89478e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,e89478e0) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,0,0,c62978b0,247) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c6294480,e89479d0) at lockmgr+0x4ce vop_stdlock(e8947a28,c0557513,c06a4f20,e8947a28,e89479e0) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8947a28,e89479f8,c0655075,e8947a28) at VOP_LOCK_APV+0x3b ffs_lock(e8947a28,c7f5b4c4,1002,c690d3fc,e8947a44) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8947a28,e8947a18,c053d7da,c6294480) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c6294480,0,0) at vn_lock+0x6c lookup(e8947ba0,c9547400,400,e8947bbc,c8aa8500) at lookup+0xc3 namei(e8947ba0,0,e8947c04,e8947be7,e8947be8) at namei+0x2e7 kern_stat(c6294480,b2f9db0,0,e8947c14,e8947c3c) at kern_stat+0x3d stat(c6294480,e8947d04,8,4,e8947d38) at stat+0x2f syscall(804003b,3b,bfbf003b,b2f9db0,8057e38) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0xb419f0b, esp = 0xbfbfeb6c, ebp = 0xbfbfec58 --- Tracing command sshd pid 21003 tid 100669 td 0xc94fd000 sched_switch(c94fd000,0,1,2131dd7e,bcba92fa) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e93a28b4,c04f425b) at mi_switch+0x150 sleepq_switch(e93a28e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,0,0,c9269684,283) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c94fd000,e93a29d0) at lockmgr+0x4ce vop_stdlock(e93a2a28,c0557513,c06a4f20,e93a2a28,e93a29e0) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e93a2a28,e93a29f8,c0655075,e93a2a28) at VOP_LOCK_APV+0x3b ffs_lock(e93a2a28,c7f45618,1002,c690d3fc,e93a2a44) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e93a2a28,e93a2a18,c053d7da,c94fd000) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c94fd000,0,0) at vn_lock+0x6c lookup(e93a2ba0,c9884400,400,e93a2bbc,4) at lookup+0xc3 namei(e93a2ba0,4,c0c6c468,e93a2b68,c05eea3a) at namei+0x2e7 kern_stat(c94fd000,2836ca34,0,e93a2c14,c94fd000) at kern_stat+0x3d stat(c94fd000,e93a2d04,8,2,e93a2d38) at stat+0x2f syscall(3b,3b,bfbf003b,0,0) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x28356f0b, esp = 0xbfbfdb9c, ebp = 0xbfbfdc88 --- Tracing command sftp-server pid 5004 tid 100468 td 0xc856d180 sched_switch(c856d180,0,1,8eac7b52,c9668742) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8f5175c,c04f425b) at mi_switch+0x150 sleepq_switch(e8f51790,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,da13433c) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,e8f517f0,c06316b2,e8f51814,c06326ff) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c856d180,e8f51878) at lockmgr+0x4ce vop_stdlock(e8f518d0,3,c06a4f20,e8f518d0,e8f51888) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8f518d0,e8f518a0,c0655075,e8f518d0) at VOP_LOCK_APV+0x3b ffs_lock(e8f518d0,e8f518dc,1002,c690d3fc,e8f518ec) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8f518d0,0,1,1) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c856d180,da7a1930,9afa) at vn_lock+0x6c lookup(e8f51b6c,c7bc9800,400,e8f51b88,c0652dda) at lookup+0xc3 namei(e8f51b6c,c8556ee0,c06bc640,c941eadc,e8f519fc) at namei+0x2e7 vn_open_cred(e8f51b6c,e8f51c6c,1a4,c8677800,5) at vn_open_cred+0x15c vn_open(e8f51b6c,e8f51c6c,1a4,5,c8430300) at vn_open+0x33 kern_open(c856d180,8052000,0,e02,1b6) at kern_open+0xd2 open(c856d180,e8f51d04,c,c602a080,e8f51cc8) at open+0x36 syscall(3b,3b,bfbe003b,e01,3a) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0xb497f6b, esp = 0xbfbea6bc, ebp = 0xbfbfad08 --- Tracing command cron pid 2735 tid 100313 td 0xc7f9aa80 sched_switch(c7f9aa80,0,1,a17a4cba,9e6ea105) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8d6d8b4,c04f425b) at mi_switch+0x150 sleepq_switch(e8d6d8e8,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,e8d6d8e0) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,e8d6d948,c06316b2,e8d6d96c,c06326ff) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c7f9aa80,e8d6d9d0) at lockmgr+0x4ce vop_stdlock(e8d6da28,c7f9aa80,c06a4f20,e8d6da28,e8d6d9e0) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8d6da28,e8d6d9f8,c0655075,e8d6da28) at VOP_LOCK_APV+0x3b ffs_lock(e8d6da28,c80ac3fc,1002,c690d3fc,e8d6da44) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8d6da28,4c,e8d6da28,c06316b2) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c7f9aa80,c065510a,c054a6f6) at vn_lock+0x6c lookup(e8d6dba0,c9426c00,400,e8d6dbbc,0) at lookup+0xc3 namei(e8d6dba0,e8d6db40,60,0,c7f9aa80) at namei+0x2e7 kern_stat(c7f9aa80,804e681,0,e8d6dc14,50) at kern_stat+0x3d stat(c7f9aa80,e8d6dd04,8,0,e8d6dd38) at stat+0x2f syscall(3b,3b,3b,bfbfedf0,3) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x28143f0b, esp = 0xbfbfe95c, ebp = 0xbfbfec68 --- Tracing command sshd pid 2728 tid 100238 td 0xc7ac4900 sched_switch(c7ac4900,0,1,c53f3824,ce10ace9) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8bfa7d4,c04f425b) at mi_switch+0x150 sleepq_switch(e8bfa808,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,c7ac4900) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,c0504340,c7df35cc,4c,e8bfa874) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c7ac4900,e8bfa8f0) at lockmgr+0x4ce vop_stdlock(e8bfa948,c07ca6e0,c06a4f20,e8bfa948,e8bfa900) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8bfa948,e8bfa918,c0655075,e8bfa948) at VOP_LOCK_APV+0x3b ffs_lock(e8bfa948,c7ac4900,1002,c690d3fc,e8bfa964) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8bfa948,0,0,0) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c7ac4900,c06326aa,3041) at vn_lock+0x6c lookup(e8bfab94,c9892400,400,e8bfabb0,c7f5b4c4) at lookup+0xc3 namei(e8bfab94,c069ba60,2,0,c7ac4900) at namei+0x2e7 unp_connect(c7ac4900,102,0,0,e8bfac20) at unp_connect+0xc9 uipc_connect(c97d4154,c7dfb100,c7ac4900) at uipc_connect+0x46 soconnect(c97d4154,c7dfb100,c7ac4900,6a,c052bf53) at soconnect+0x5d kern_connect(c7ac4900,4,c7dfb100,c7dfb100) at kern_connect+0xd6 connect(c7ac4900,e8bfad04,c,f,e8bfad38) at connect+0x45 syscall(3b,3b,bfbf003b,bfbfc850,bfbfc852) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (98, FreeBSD ELF32, connect), eip = 0x28344527, esp = 0xbfbfc83c, ebp = 0xbfbfc8d8 --- Tracing command perl5.8.7 pid 2705 tid 100309 td 0xc7f9b180 sched_switch(c7f9b180,0,1,444335ac,c86a47d5) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8d7975c,c04f425b) at mi_switch+0x150 sleepq_switch(e8d79790,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,c06b610c,c6782900,901f0000,c68f0714) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c7f9b180,e8d79878) at lockmgr+0x4ce vop_stdlock(e8d798d0,0,c06a4f20,e8d798d0,e8d79888) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8d798d0,e8d798a0,c0655075,e8d798d0) at VOP_LOCK_APV+0x3b ffs_lock(e8d798d0,c5fd5000,1002,c690d3fc,e8d798ec) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8d798d0,c5fd5000,4,bf49f752) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c7f9b180,c5fd5000,ff) at vn_lock+0x6c lookup(e8d79b6c,c6cf4400,400,e8d79b88,c0566fef) at lookup+0xc3 namei(e8d79b6c,0,0,e8d79a28,4) at namei+0x2e7 vn_open_cred(e8d79b6c,e8d79c6c,1a4,c7f42e80,3) at vn_open_cred+0x1c2 vn_open(e8d79b6c,e8d79c6c,1a4,3,246) at vn_open+0x33 kern_open(c7f9b180,e6a7d28,0,1,1b6) at kern_open+0xd2 open(c7f9b180,e8d79d04,c,369e99,0) at open+0x36 syscall(3b,81a003b,bfbf003b,4,e6c5f20) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0xe696f6b, esp = 0xbfbfeb1c, ebp = 0xbfbfeb48 --- Tracing command syslogd pid 2639 tid 100308 td 0xc7f9b300 sched_switch(c7f9b300,0,1,bfebc1b4,fce5836d) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8d7c950,c04f425b) at mi_switch+0x150 sleepq_switch(e8d7c984,c04e4cc6,c690d454,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d454,50,c0672a31,0,0) at sleepq_wait+0xb msleep(c690d454,c06b9cb4,50,c0672a31,0) at msleep+0x454 acquire(60000,1000,c13cbba8,3b,805b050) at acquire+0x7a lockmgr(c690d454,3002,c690d4c4,c7f9b300,e8d7ca6c) at lockmgr+0x4ce vop_stdlock(e8d7cac4,4000,c06a4f20,e8d7cac4,e8d7ca7c) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8d7cac4,e8d7ca94,c0655075,e8d7cac4) at VOP_LOCK_APV+0x3b ffs_lock(e8d7cac4,e8d7ca94,1002,c690d3fc,e8d7cae0) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8d7cac4,e8d7cac0,c06316b2,e8d7cae4) at VOP_LOCK_APV+0x3b vn_lock(c690d3fc,1002,c7f9b300,c055856a,c05080a7) at vn_lock+0x6c lookup(e8d7cc0c,c9ab9000,400,e8d7cc28,0) at lookup+0xc3 namei(e8d7cc0c,c06b2960,c7febd48,c7f9b300,c7f43900) at namei+0x2e7 kern_unlink(c7f9b300,804eb7b,0,e8d7cd30,c0648a47) at kern_unlink+0x9b unlink(c7f9b300,e8d7cd04,4,0,0) at unlink+0x22 syscall(3b,3b,3b,1,f) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (10, FreeBSD ELF32, unlink), eip = 0x28141827, esp = 0xbfbfd90c, ebp = 0xbfbfd9a8 --- Tracing command nginx pid 546 tid 100122 td 0xc69d1d80 sched_switch(c69d1d80,0,1,f002cf9e,6c43d35e) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e8a73670,c04f425b) at mi_switch+0x150 sleepq_switch(e8a736a4,c04e4cc6,c6c07c4c,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c6c07c4c,50,c0672a31,0,c044b82a) at sleepq_wait+0xb msleep(c6c07c4c,c06ba45c,50,c0672a31,0) at msleep+0x454 acquire(60000,c0655075,c05db299,c0655075,c0557513) at acquire+0x7a lockmgr(c6c07c4c,2002,c6c07cbc,c69d1d80,e8a7378c) at lockmgr+0x4ce vop_stdlock(e8a737e4,c06a4f20,c06a4f20,e8a737e4,e8a7379c) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e8a737e4,e8a737b4,c0655075,e8a737e4) at VOP_LOCK_APV+0x3b ffs_lock(e8a737e4,c0673faf,2,c6c07bf4,e8a73800) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e8a737e4,e8a73814,c0504340,e8a73814) at VOP_LOCK_APV+0x3b vn_lock(c6c07bf4,2002,c69d1d80,c05e52a9,c6bed3fc) at vn_lock+0x6c vget(c6c07bf4,2002,c69d1d80,0,e8a738a8) at vget+0x87 cache_lookup(c6bed3fc,e8a73b80,e8a73b94,c69d1d80,c6202900) at cache_lookup+0xde vfs_cache_lookup(e8a73940,e8a738ec,c6bed3fc,e8a73b94,e8a7395c) at vfs_cache_lookup+0xad VOP_LOOKUP_APV(c06a4f20,e8a73940,e8a7392c,c69d1d80,0) at VOP_LOOKUP_APV+0x3b lookup(e8a73b6c,c9630000,400,e8a73b88,c057cbf1) at lookup+0x419 namei(e8a73b6c,c0584a9f,c7ef4900,0,e8a739a8) at namei+0x2e7 vn_open_cred(e8a73b6c,e8a73c6c,1a4,c6202900,37) at vn_open_cred+0x1c2 vn_open(e8a73b6c,e8a73c6c,1a4,37,c83b0d48) at vn_open+0x33 kern_open(c69d1d80,80f0424,0,1,1a4) at kern_open+0xd2 open(c69d1d80,e8a73d04,c,c602a480,e8a73cc8) at open+0x36 syscall(80e003b,814003b,bfbf003b,80f0475,8151000) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x282f9d6b, esp = 0xbfbfea6c, ebp = 0xbfbfeb18 --- Tracing command schedcpu pid 36 tid 100029 td 0xc5fd9d80 sched_switch(c5fd9d80,0,1,deebc4ca,f168ca11) at sched_switch+0xd8 mi_switch(1,0,0,0,e4c45c8c) at mi_switch+0x150 sleepq_switch(3e8,e4c45cc0,c04e4e23,e4c45cf0,34) at sleepq_switch+0x115 sleepq_timedwait(e4c45cf0,34,c066a1ff,0,c7ca3900) at sleepq_timedwait+0xc msleep(e4c45cf0,0,34,c066a1ff,3e8) at msleep+0x5b1 schedcpu_thread(0,e4c45d38,ffffffff,ffffffff,ffffffff) at schedcpu_thread+0x22c fork_exit(c04f42d5,0,e4c45d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c45d6c, ebp = 0 --- Tracing command syncer pid 35 tid 100030 td 0xc5fd9c00 sched_switch(c5fd9c00,0,1,edd2ea48,455594e8) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e4c42c2c,c04f425b) at mi_switch+0x150 sleepq_switch(e4c42c60,c04e4cc6,c60d74e0,68,c066cfe3) at sleepq_switch+0x115 sleepq_wait(c60d74e0,68,c066cfe3,0,c60d74e0) at sleepq_wait+0xb msleep(c60d74e0,c60d74c0,68,c066cfe3,0) at msleep+0x454 kthread_suspend_check(c60d7458,0,68,c0673dfe,fa) at kthread_suspend_check+0x65 sched_sync(0,e4c42d38,ffffffff,ffffffff,ffffffff) at sched_sync+0x11f fork_exit(c054934e,0,e4c42d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c42d6c, ebp = 0 --- Tracing command vnlru pid 34 tid 100031 td 0xc5fd9a80 sched_switch(c5fd9a80,0,1,befcd2e8,5f176ae8) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e4c3fc40,c04f425b) at mi_switch+0x150 sleepq_switch(e4c3fc74,c04e4cc6,c60d770c,68,c066cfe3) at sleepq_switch+0x115 sleepq_wait(c60d770c,68,c066cfe3,0,c60d770c) at sleepq_wait+0xb msleep(c60d770c,c60d76ec,68,c066cfe3,0) at msleep+0x454 kthread_suspend_check(c60d7684,c06c17e0,250,c0673dcd,3e8) at kthread_suspend_check+0x65 vnlru_proc(0,e4c3fd38,ffffffff,ffffffff,ffffffff) at vnlru_proc+0x94 fork_exit(c0548aab,0,e4c3fd38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c3fd6c, ebp = 0 --- Tracing command bufdaemon pid 33 tid 100032 td 0xc5fd9900 sched_switch(c5fd9900,0,1,b29de270,bd7eb7ab) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e4c3cc84,c04f425b) at mi_switch+0x150 sleepq_switch(e4c3ccb8,c04e4cc6,c60d7938,68,c066cfe3) at sleepq_switch+0x115 sleepq_wait(c60d7938,68,c066cfe3,0,c60d7938) at sleepq_wait+0xb msleep(c60d7938,c60d7918,68,c066cfe3,0) at msleep+0x454 kthread_suspend_check(c60d78b0,c06c150c,44,c0672ce9,3e8) at kthread_suspend_check+0x65 buf_daemon(0,e4c3cd38,ffffffff,ffffffff,ffffffff) at buf_daemon+0x106 fork_exit(c0535c97,0,e4c3cd38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c3cd6c, ebp = 0 --- Tracing command pagezero pid 32 tid 100033 td 0xc5fd9780 sched_switch(c5fd9780,0,1,1aa738e4,41803987) at sched_switch+0xd8 mi_switch(1,0,4,0,e4c39ca0) at mi_switch+0x150 sleepq_switch(493e0,e4c39cd4,c04e4e23,c06d0b14,ff) at sleepq_switch+0x115 sleepq_timedwait(c06d0b14,ff,c067dae7,0,246) at sleepq_timedwait+0xc msleep(c06d0b14,c06c89e0,2ff,c067dae7,493e0) at msleep+0x5b1 vm_pagezero(0,e4c39d38,ffffffff,ffffffff,ffffffff) at vm_pagezero+0x8e fork_exit(c060a5a8,0,e4c39d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c39d6c, ebp = 0 --- Tracing command vmdaemon pid 31 tid 100034 td 0xc5fd9600 sched_switch(c5fd9600,0,1,68c14d46,5bf5a05f) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 sleepq_switch(e4c36cb0,c04e4cc6,c06c8a48,68,0) at sleepq_switch+0x115 sleepq_wait(c06c8a48,68,0,0,0) at sleepq_wait+0xb msleep(c06c8a48,0,68,c0672ce9,0) at msleep+0x454 vm_daemon(0,e4c36d38,ffffffff,ffffffff,ffffffff) at vm_daemon+0x7c fork_exit(c0608ca7,0,e4c36d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c36d6c, ebp = 0 --- Tracing command pagedaemon pid 30 tid 100035 td 0xc5fd9480 sched_switch(c5fd9480,0,1,16e08acc,9be9e077) at sched_switch+0xd8 mi_switch(1,0,4,0,e4c33be0) at mi_switch+0x150 sleepq_switch(1388,e4c33c14,c04e4e23,c06c8a08,44) at sleepq_switch+0x115 sleepq_timedwait(c06c8a08,44,c0672ce9,0,0) at sleepq_timedwait+0xc msleep(c06c8a08,c06c89e0,44,c0672ce9,1388) at msleep+0x5b1 vm_pageout(0,e4c33d38,ffffffff,ffffffff,ffffffff) at vm_pageout+0x329 fork_exit(c0606ed4,0,e4c33d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c33d6c, ebp = 0 --- Tracing command irq1: atkbd0 pid 29 tid 100036 td 0xc60d9d80 sched_switch(c60d9d80,0,1,653f5760,1aae6dca) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 ithread_loop(c60cd700,e674fd38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c60cd700,e674fd38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe674fd6c, ebp = 0 --- Tracing command swi0: sio pid 28 tid 100037 td 0xc60d9c00 sched_switch(c60d9c00,0,1,78e163aa,280d73cf) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 ithread_loop(c60cd760,e674cd38,6d206461,6c75646f,65622065) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c60cd760,e674cd38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe674cd6c, ebp = 0 --- Tracing command irq18: atapci1 pid 27 tid 100017 td 0xc5fd5900 sched_switch(c5fd5900,0,1,91f9c4ec,bbe83083) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 ithread_loop(c60bb1d0,e4c1ed38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c60bb1d0,e4c1ed38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c1ed6c, ebp = 0 --- Tracing command irq15: ata1 pid 26 tid 100018 td 0xc5fd5780 fork_trampoline() at fork_trampoline Tracing command irq14: ata0 pid 25 tid 100019 td 0xc5fd5600 sched_switch(c5fd5600,0,1,dcb75946,1d08a930) at sched_switch+0xd8 mi_switch(1,0,ffffffff,ffffffff,ffffffff) at mi_switch+0x150 ithread_loop(c60b2670,e4c18d38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c60b2670,e4c18d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c18d6c, ebp = 0 --- Tracing command irq27: em1 pid 24 tid 100020 td 0xc5fd5480 fork_trampoline() at fork_trampoline Tracing command irq26: em0 pid 23 tid 100021 td 0xc5fd5300 sched_switch(c5fd5300,0,1,10e80ad0,3614c13) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 ithread_loop(c609dc50,e4c12d38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c609dc50,e4c12d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c12d6c, ebp = 0 --- Tracing command irq9: acpi0 pid 22 tid 100022 td 0xc5fd5180 fork_trampoline() at fork_trampoline Tracing command swi2: cambio pid 21 tid 100023 td 0xc6071780 fork_trampoline() at fork_trampoline Tracing command swi6: task queue pid 20 tid 100024 td 0xc6071600 sched_switch(c6071600,0,1,61baa270,dfbf6a4b) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 ithread_loop(c5fb88d0,e6678d38,e8240489,ffffc5f2,5b0cc483) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c5fb88d0,e6678d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6678d6c, ebp = 0 --- Tracing command acpi_task2 pid 9 tid 100025 td 0xc6071480 sched_switch(c6071480,0,1,d88d1c6,a349b95) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e6675ca8,c04f425b) at mi_switch+0x150 sleepq_switch(e6675cdc,c04e4cc6,c6064e00,5c,c066a1ff) at sleepq_switch+0x115 sleepq_wait(c6064e00,5c,c066a1ff,0,ad7df67a) at sleepq_wait+0xb msleep(c6064e00,c6064e1c,5c,c066a1ff,0) at msleep+0x454 taskqueue_thread_loop(c07cc6b4,e6675d38,ebda3651,9850f34,c11001f6) at taskqueue_thread_loop+0x57 fork_exit(c0504e77,c07cc6b4,e6675d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6675d6c, ebp = 0 --- Tracing command acpi_task1 pid 8 tid 100026 td 0xc6071300 sched_switch(c6071300,0,1,af3a7fc6,a346356) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e6672ca8,c04f425b) at mi_switch+0x150 sleepq_switch(e6672cdc,c04e4cc6,c6064e00,5c,c066a1ff) at sleepq_switch+0x115 sleepq_wait(c6064e00,5c,c066a1ff,0,77598b73) at sleepq_wait+0xb msleep(c6064e00,c6064e1c,5c,c066a1ff,0) at msleep+0x454 taskqueue_thread_loop(c07cc6b4,e6672d38,c76de081,dce7e219,98eb29d) at taskqueue_thread_loop+0x57 fork_exit(c0504e77,c07cc6b4,e6672d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe6672d6c, ebp = 0 --- Tracing command acpi_task0 pid 7 tid 100008 td 0xc5fd4300 sched_switch(c5fd4300,0,1,73bafc46,a342668) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e4bf4ca8,c04f425b) at mi_switch+0x150 sleepq_switch(e4bf4cdc,c04e4cc6,c6064e00,5c,c066a1ff) at sleepq_switch+0x115 sleepq_wait(c6064e00,5c,c066a1ff,0,ffffffff) at sleepq_wait+0xb msleep(c6064e00,c6064e1c,5c,c066a1ff,0) at msleep+0x454 taskqueue_thread_loop(c07cc6b4,e4bf4d38,ffffffff,ffffffff,ffffffff) at taskqueue_thread_loop+0x57 fork_exit(c0504e77,c07cc6b4,e4bf4d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4bf4d6c, ebp = 0 --- Tracing command swi6: Giant taskq pid 19 tid 100009 td 0xc5fd4180 fork_trampoline() at fork_trampoline Tracing command thread taskq pid 6 tid 100010 td 0xc5fd4000 sched_switch(c5fd4000,0,1,bec3185e,4c4cc167) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e4beeca8,c04f425b) at mi_switch+0x150 sleepq_switch(e4beecdc,c04e4cc6,c6065000,5c,c066a1ff) at sleepq_switch+0x115 sleepq_wait(c6065000,5c,c066a1ff,0,ffffffff) at sleepq_wait+0xb msleep(c6065000,c606501c,5c,c066a1ff,0) at msleep+0x454 taskqueue_thread_loop(c06bffd4,e4beed38,ffffffff,ffffffff,ffffffff) at taskqueue_thread_loop+0x57 fork_exit(c0504e77,c06bffd4,e4beed38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4beed6c, ebp = 0 --- Tracing command swi5: Fast taskq pid 18 tid 100011 td 0xc5fd9300 fork_trampoline() at fork_trampoline Tracing command kqueue taskq pid 5 tid 100012 td 0xc5fd9180 sched_switch(c5fd9180,0,1,99f92146,a32a613) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e4c2dca8,c04f425b) at mi_switch+0x150 sleepq_switch(e4c2dcdc,c04e4cc6,c60650c0,5c,c066a1ff) at sleepq_switch+0x115 sleepq_wait(c60650c0,5c,c066a1ff,0,ffffffff) at sleepq_wait+0xb msleep(c60650c0,c60650dc,5c,c066a1ff,0) at msleep+0x454 taskqueue_thread_loop(c06b9580,e4c2dd38,ffffffff,ffffffff,ffffffff) at taskqueue_thread_loop+0x57 fork_exit(c0504e77,c06b9580,e4c2dd38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c2dd6c, ebp = 0 --- Tracing command yarrow pid 17 tid 100013 td 0xc5fd9000 sched_switch(c5fd9000,0,1,ed90b28e,e6cc3d64) at sched_switch+0xd8 mi_switch(1,0,0,0,e4c2ac9c) at mi_switch+0x150 sleepq_switch(64,e4c2acd0,c04e4e23,c06b8460,44) at sleepq_switch+0x115 sleepq_timedwait(c06b8460,44,c066a1ff,0,6) at sleepq_timedwait+0xc msleep(c06b8460,0,44,c066a1ff,64) at msleep+0x5b1 random_kthread(0,e4c2ad38,ffffffff,ffffffff,ffffffff) at random_kthread+0x1f0 fork_exit(c048ff8e,0,e4c2ad38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c2ad6c, ebp = 0 --- Tracing command g_down pid 4 tid 100014 td 0xc5fd5d80 sched_switch(c5fd5d80,0,1,acb067aa,6e026c0b) at sched_switch+0xd8 mi_switch(1,0,4,0,e4c27c68) at mi_switch+0x150 sleepq_switch(64,e4c27c9c,c04e4e23,c06b8d00,4c) at sleepq_switch+0x115 sleepq_timedwait(c06b8d00,4c,c066a1ff,0,4000) at sleepq_timedwait+0xc msleep(c06b8d00,c06b8be8,24c,c066a1ff,64) at msleep+0x5b1 g_io_schedule_down(c5fd5d80,4c,c5fd5d80,c049a065,e4c27d00) at g_io_schedule_down+0x11c g_down_procbody(0,e4c27d38,ffffffff,ffffffff,ffffffff) at g_down_procbody+0x97 fork_exit(c049a065,0,e4c27d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c27d6c, ebp = 0 --- Tracing command g_up pid 3 tid 100015 td 0xc5fd5c00 sched_switch(c5fd5c00,0,1,7297e676,fbc655b8) at sched_switch+0xd8 mi_switch(1,0,4,0,e4c24c88) at mi_switch+0x150 sleepq_switch(64,e4c24cbc,c04e4e23,c06b8cfc,4c) at sleepq_switch+0x115 sleepq_timedwait(c06b8cfc,4c,c066a1ff,0,c63313ac) at sleepq_timedwait+0xc msleep(c06b8cfc,c06b8c28,24c,c066a1ff,64) at msleep+0x5b1 g_io_schedule_up(c5fd5c00,4c,c5fd5c00,c0499fb0,e4c24d00) at g_io_schedule_up+0xc2 g_up_procbody(0,e4c24d38,ffffffff,ffffffff,ffffffff) at g_up_procbody+0x97 fork_exit(c0499fb0,0,e4c24d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c24d6c, ebp = 0 --- Tracing command g_event pid 2 tid 100016 td 0xc5fd5a80 sched_switch(c5fd5a80,0,1,41a6af52,24a002a3) at sched_switch+0xd8 mi_switch(1,0,0,0,e4c21ca8) at mi_switch+0x150 sleepq_switch(64,e4c21cdc,c04e4e23,c06b8cf4,4c) at sleepq_switch+0x115 sleepq_timedwait(c06b8cf4,4c,c066a1ff,0,0) at sleepq_timedwait+0xc msleep(c06b8cf4,0,4c,c066a1ff,64) at msleep+0x5b1 g_event_procbody(0,e4c21d38,ffffffff,ffffffff,ffffffff) at g_event_procbody+0xd9 fork_exit(c049a11a,0,e4c21d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c21d6c, ebp = 0 --- Tracing command swi1: net pid 16 tid 100000 td 0xc5fd5000 sched_switch(c5fd5000,0,1,b27eab48,150f90e0) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 ithread_loop(c5fb8840,e4c0cd38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c5fb8840,e4c0cd38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c0cd6c, ebp = 0 --- Tracing command swi3: vm pid 15 tid 100001 td 0xc5fd4d80 fork_trampoline() at fork_trampoline Tracing command swi4: clock sio pid 14 tid 100002 td 0xc5fd4c00 sched_switch(c5fd4c00,0,1,53e95da,fc43c0ce) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 ithread_loop(c5fb8860,e4c06d38,ffffffff,ffffffff,ffffffff) at ithread_loop+0x3b7 fork_exit(c04c1b0d,c5fb8860,e4c06d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c06d6c, ebp = 0 --- Tracing command idle: cpu0 pid 13 tid 100003 td 0xc5fd4a80 cpustop_handler(e4c03c64,c064844e,369e99,0,be1c6ca0) at cpustop_handler+0x2c ipi_nmi_handler(369e99,0,be1c6ca0,4af,a5f) at ipi_nmi_handler+0x2a trap(8,c04f0028,c5fd0028,c603a900,c603a918) at trap+0x30a calltrap() at calltrap+0x5 --- trap 0x13, eip = 0xc07bdc12, esp = 0xe4c03cac, ebp = 0xe4c03cac --- acpi_cpu_c1(caef575e,e4c03cd8,236b,e4c03cd0,c04f2fb4) at acpi_cpu_c1+0x5 acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a idle_proc(0,e4c03d38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149 fork_exit(c04c097a,0,e4c03d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c03d6c, ebp = 0 --- Tracing command idle: cpu1 pid 12 tid 100004 td 0xc5fd4900 cpustop_handler(e4c00c64,c064844e,ffffffff,ffffffff,ffffffff) at cpustop_handler+0x2c ipi_nmi_handler(ffffffff,ffffffff,ffffffff,ffffffff,c5fd4900) at ipi_nmi_handler+0x2a trap(8,c04f0028,c5fd0028,c603a700,c603a718) at trap+0x30a calltrap() at calltrap+0x5 --- trap 0x13, eip = 0xc07bdc12, esp = 0xe4c00cac, ebp = 0xe4c00cac --- acpi_cpu_c1(93b6c83c,e4c00cd8,236c,e4c00cd0,c04f2fb4) at acpi_cpu_c1+0x5 acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a idle_proc(0,e4c00d38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149 fork_exit(c04c097a,0,e4c00d38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4c00d6c, ebp = 0 --- Tracing command idle: cpu2 pid 11 tid 100005 td 0xc5fd4780 cpustop_handler(e4bfdc64,c064844e,ffffffff,ffffffff,ffffffff) at cpustop_handler+0x2c ipi_nmi_handler(ffffffff,ffffffff,ffffffff,ffffffff,c5fd4780) at ipi_nmi_handler+0x2a trap(8,c04f0028,c5fd0028,c603ab00,c603ab18) at trap+0x30a calltrap() at calltrap+0x5 --- trap 0x13, eip = 0xc07bdc12, esp = 0xe4bfdcac, ebp = 0xe4bfdcac --- acpi_cpu_c1(2a172b1b,e4bfdcd8,236d,e4bfdcd0,c04f2fb4) at acpi_cpu_c1+0x5 acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a idle_proc(0,e4bfdd38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149 fork_exit(c04c097a,0,e4bfdd38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4bfdd6c, ebp = 0 --- Tracing command idle: cpu3 pid 10 tid 100006 td 0xc5fd4600 kdb_enter(c067f27b,e4bfabfc,46,c5fd4600,c60d0c00) at kdb_enter+0x32 siointr1(e4bfac24,c04f4652,c5fd4600,c5fd4600,c5fd3d08) at siointr1+0x3de siointr(c60d0c00,0,c5fd4600,c5fd4600,0) at siointr+0x66 intr_execute_handlers(c5fcaca0,e4bfac6c,e4bfacac,c0632a63,37) at intr_execute_handlers+0xcc lapic_handle_intr(37) at lapic_handle_intr+0x2d Xapic_isr1() at Xapic_isr1+0x33 --- interrupt, eip = 0xc07bdc12, esp = 0xe4bfacac, ebp = 0xe4bfacac --- acpi_cpu_c1(0,0,0,e4bfacd0,c04f2fb4) at acpi_cpu_c1+0x5 acpi_cpu_idle(1,0,0,0,0) at acpi_cpu_idle+0x18a idle_proc(0,e4bfad38,ffffffff,ffffffff,ffffffff) at idle_proc+0x149 fork_exit(c04c097a,0,e4bfad38) at fork_exit+0x62 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xe4bfad6c, ebp = 0 --- Tracing command init pid 1 tid 100007 td 0xc5fd4480 sched_switch(c5fd4480,0,1,7942a4fc,85e8071) at sched_switch+0xd8 mi_switch(1,0,c0502e44,e4bf78f0,c04f425b) at mi_switch+0x150 sleepq_switch(e4bf7924,c04e4cc6,c690d5a8,50,c0672a31) at sleepq_switch+0x115 sleepq_wait(c690d5a8,50,c0672a31,0,c064191c) at sleepq_wait+0xb msleep(c690d5a8,c06b9c98,50,c0672a31,0) at msleep+0x454 acquire(60000,e4bf796c,c06b2ca0,c690d7f8,0) at acquire+0x7a lockmgr(c690d5a8,2002,c690d618,c5fd4480,e4bf7a0c) at lockmgr+0x4ce vop_stdlock(e4bf7a64,c065510a,c06a4f20,e4bf7a64,e4bf7a1c) at vop_stdlock+0x32 VOP_LOCK_APV(c06a55c0,e4bf7a64,e4bf7a34,c0655075,e4bf7a64) at VOP_LOCK_APV+0x3b ffs_lock(e4bf7a64,0,2,c690d550,e4bf7a80) at ffs_lock+0x19 VOP_LOCK_APV(c06a4f20,e4bf7a64,c0c5fa00,0,c690d6a4) at VOP_LOCK_APV+0x3b vn_lock(c690d550,2002,c5fd4480,7ffcaf32,369e99) at vn_lock+0x6c vflush(c6268c00,0,2,c5fd4480,4) at vflush+0x12a ffs_flushfiles(c6268c00,2,c5fd4480,c06b2ca0,c678f7f8) at ffs_flushfiles+0x93 ffs_unmount(c6268c00,80000,c5fd4480,c5fd4480,20) at ffs_unmount+0xea dounmount(c6268c00,80000,c5fd4480,da054aac,0) at dounmount+0x226 vfs_unmountall(c066ec26,0,0,0,0) at vfs_unmountall+0x3b boot(c5fd4480,e4bf7d38,bfbfe95c,c5fd8000,c5fd4480) at boot+0x80c reboot(c5fd4480,e4bf7d04,4,0,0) at reboot+0x3c syscall(3b,3b,3b,2,bfbfeef8) at syscall+0x23a Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (55, FreeBSD ELF32, reboot), eip = 0x8050703, esp = 0xbfbfeccc, ebp = 0xbfbfeda8 --- Tracing command swapper pid 0 tid 0 td 0xc06b9040 sched_switch(c06b9040,0,1,e7689d46,75507e12) at sched_switch+0xd8 mi_switch(1,0,0,0,0) at mi_switch+0x150 scheduler(0,0,c5fbf79c,81ec00,81e000) at scheduler+0x551 mi_startup() at mi_startup+0xb1 begin() at begin+0x2c db> show allpcpu Current CPU: 3 cpuid = 0 curthread = 0xc5fd4a80: pid 13 "idle: cpu0" curpcb = 0xe4c03d90 fpcurthread = none idlethread = 0xc5fd4a80: pid 13 "idle: cpu0" APIC ID = 0 currentldt = 0x50 cpuid = 1 curthread = 0xc5fd4900: pid 12 "idle: cpu1" curpcb = 0xe4c00d90 fpcurthread = none idlethread = 0xc5fd4900: pid 12 "idle: cpu1" APIC ID = 1 currentldt = 0x50 cpuid = 2 curthread = 0xc5fd4780: pid 11 "idle: cpu2" curpcb = 0xe4bfdd90 fpcurthread = none idlethread = 0xc5fd4780: pid 11 "idle: cpu2" APIC ID = 6 currentldt = 0x50 cpuid = 3 curthread = 0xc5fd4600: pid 10 "idle: cpu3" curpcb = 0xe4bfad90 fpcurthread = none idlethread = 0xc5fd4600: pid 10 "idle: cpu3" APIC ID = 7 currentldt = 0x50 db> call doadump Dumping 2047 MB (2 chunks) chunk 0: 1MB (154 pages) ... ok chunk 1: 2047MB (523968 pages) 2031 2015 1999 1983 1967 1951 1935 1919 1903 1887 1871 1855 1839 1823 1807 1791 1775 1759 1743 1727 1711 1695 1679 1663 1647 1631 1615 1599 1583 1567 1551 1535 1519 1503 1487 1471 1455 1439 1423 1407 1391 1375 1359 1343 1327 1311 1295 1279 1263 1247 1231 1215 1199 1183 1167 1151 1135 1119 1103 1087 1071 1055 1039 1023 1007 991 975 959 943 927 911 895 879 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 ... ok Dump complete = 0xf db> reset cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 3 -- DSS5-RIPE DSS-RIPN 2:550/5068@fidonet 2:550/5069@fidonet mailto:dsh@vlink.ru http://neva.vlink.ru/~dsh/