From owner-freebsd-current@FreeBSD.ORG Fri Jun 17 03:45:13 2005 Return-Path: X-Original-To: freebsd-current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3F93E16A41C for ; Fri, 17 Jun 2005 03:45:13 +0000 (GMT) (envelope-from kuriyama@imgsrc.co.jp) Received: from black.imgsrc.co.jp (black.imgsrc.co.jp [210.226.20.147]) by mx1.FreeBSD.org (Postfix) with ESMTP id B6CE543D49 for ; Fri, 17 Jun 2005 03:45:12 +0000 (GMT) (envelope-from kuriyama@imgsrc.co.jp) Received: from localhost (localhost [127.0.0.1]) by black.imgsrc.co.jp (Postfix) with ESMTP id 2E68850D01; Fri, 17 Jun 2005 12:45:11 +0900 (JST) Received: from black.imgsrc.co.jp (black.imgsrc.co.jp [IPv6:2001:218:422:2::9999]) by black.imgsrc.co.jp (Postfix) with ESMTP id 839CA50D04; Fri, 17 Jun 2005 12:45:09 +0900 (JST) Date: Fri, 17 Jun 2005 12:45:09 +0900 Message-ID: <7mll59myt6.wl%kuriyama@imgsrc.co.jp> From: Jun Kuriyama To: Doug White In-Reply-To: <20050614194742.U24745@carver.gumbysoft.com> References: <7mpsupni5r.wl%kuriyama@imgsrc.co.jp> <20050614194742.U24745@carver.gumbysoft.com> User-Agent: Wanderlust/2.14.0 (Africa) SEMI/1.14.6 (Maruoka) FLIM/1.14.7 (=?ISO-8859-4?Q?Sanj=F2?=) APEL/10.6 Emacs/21.3 (i386--freebsd) MULE/5.0 (SAKAKI) MIME-Version: 1.0 (generated by SEMI 1.14.6 - "Maruoka") Content-Type: text/plain; charset=US-ASCII X-Virus-Scanned: by amavisd 0.1 Cc: Current Subject: Re: How to help debugging of lock-up X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jun 2005 03:45:13 -0000 At Tue, 14 Jun 2005 19:48:27 -0700 (PDT), Doug White wrote: > > I'm not sure which process I should suspect. Is there something I can > > provide to help debugging about this? > > The trace looks normal for something network- and disk-bound. Perhaps your > NIC's overloaded or hung? Where is the amanda backup going -- back to the > same system? Yes, it looks trace is normal. But even serial console, getty does not respond. Backup server is another box, dump is going through the network. And I got another lock-up with today's kernel (including jeff's r1.103 of vfs_cache.c). If noone sees this behavior, is there a possibility which my hardware is broken? (but old 2005-04-09 kernel works without problem) ----- KDB: enter: Break sequence on console [thread pid 11 tid 100006 ] Stopped at kdb_enter+0x2b: nop db> ps pid proc uid ppid pgrp flag stat wmesg wchan cmd 35659 c6569c00 103 35657 758 0004000 [SLPQ getblk 0xd7775df0][SLP] as 35657 c3939a00 103 35656 758 0004000 [SLPQ wait 0xc3939a00][SLP] c++ 35656 c8d33c00 103 35635 758 0004000 [SLPQ wait 0xc8d33c00][SLP] sh 35655 c3b3cc00 103 35654 758 0004000 [SLPQ biord 0xd766c638][SLP] cc1 35654 c393c200 103 35653 758 0004000 [SLPQ wait 0xc393c200][SLP] cc 35653 c8ebe600 103 35651 758 0004000 [SLPQ wait 0xc8ebe600][SLP] sh 35651 c3b3ca00 103 35650 758 0004000 [SLPQ wait 0xc3b3ca00][SLP] sh 35650 c656a800 103 35350 758 0004000 [SLPQ select 0xc075cb24][SLP] make 35648 c8ec0e00 0 35646 35648 0000110 [SLPQ kqread 0xc8fd1480][SLP] cron 35647 c8d30000 0 35644 35647 0000010 [SLPQ kqread 0xc8ffc880][SLP] cron 35646 c953fa00 0 588 588 0000000 [SLPQ ppwait 0xc953fa00][SLP] cron 35644 c656a400 0 588 588 0000000 [SLPQ ppwait 0xc656a400][SLP] cron 35635 c8ebee00 103 34347 758 0004000 [SLPQ select 0xc075cb24][SLP] make 35350 c93c1e00 103 35349 758 0004000 [SLPQ wait 0xc93c1e00][SLP] sh 35349 c93c1a00 103 35348 758 0004000 [SLPQ select 0xc075cb24][SLP] make 35348 c93bec00 103 33419 758 0004000 [SLPQ wait 0xc93bec00][SLP] sh 34347 c8d33800 103 34346 758 0004000 [SLPQ wait 0xc8d33800][SLP] sh 34346 c3b3c200 103 33426 758 0004000 [SLPQ select 0xc075cb24][SLP] make 33426 c6569000 103 33419 758 0004000 [SLPQ wait 0xc6569000][SLP] sh 33419 c38c3a00 103 18262 758 0004000 [SLPQ select 0xc075cb24][SLP] make 32618 c8ebe400 1021 702 32618 0004002 [SLPQ select 0xc075cb24][SLP] ssh 18262 c38c4a00 103 18255 758 0004000 [SLPQ wait 0xc38c4a00][SLP] sh 18255 c8ec0000 103 18254 758 0004000 [SLPQ select 0xc075cb24][SLP] make 18254 c8ec0a00 103 894 758 0004000 [SLPQ wait 0xc8ec0a00][SLP] sh 894 c656a200 103 892 758 0004000 [SLPQ select 0xc075cb24][SLP] make 892 c3939c00 103 811 758 0004000 [SLPQ wait 0xc3939c00][SLP] sh 811 c656a000 103 810 758 0004000 [SLPQ select 0xc075cb24][SLP] make 810 c6569e00 103 760 758 0004000 [SLPQ wait 0xc6569e00][SLP] lockf 760 c3939e00 103 759 758 0004000 [SLPQ wait 0xc3939e00][SLP] sh 759 c3b37800 103 758 758 0004000 [SLPQ wait 0xc3b37800][SLP] lockf 758 c3b37200 103 757 758 0004000 [SLPQ pause 0xc3b37234][SLP] csh 757 c393ce00 0 750 750 0004100 [SLPQ wait 0xc393ce00][SLP] su 750 c3ac2a00 0 747 750 0004000 [SLPQ wait 0xc3ac2a00][SLP] sh 747 c38c4c00 0 588 588 0000000 [SLPQ piperd 0xc3941480][SLP] cron 702 c3b37400 1021 701 702 0004002 [SLPQ pause 0xc3b37434][SLP] zsh 701 c3abe800 1021 699 699 0000100 [SLPQ select 0xc075cb24][SLP] sshd 699 c3b37a00 0 566 699 0004100 [SLPQ sbwait 0xc3b13334][SLP] sshd 698 c393c600 0 1 698 0004002 [SLPQ ttyin 0xc372d410][SLP] getty 697 c3b3c000 0 1 697 0004002 [SLPQ ttyin 0xc3744010][SLP] getty ... db> trace 35659 Tracing pid 35659 tid 100126 td 0xc359da80 sched_switch(c359da80,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(d7775df0,e6a72968,c050da15,d7775df0,0) at sleepq_switch+0xe0 sleepq_wait(d7775df0,0,0,c396329c,b5) at sleepq_wait+0x30 msleep(d7775df0,c070eb90,50,c06aeed5,0) at msleep+0x311 acquire(e6a729c0,120,60000,c359da80,0) at acquire+0x76 lockmgr(d7775df0,202122,c396329c,c359da80) at lockmgr+0x42a getblk(c3963220,a0cee0,0,4000,0) at getblk+0x12a breadn(c3963220,a0cee0,0,4000,0) at breadn+0x31 bread(c3963220,a0cee0,0,4000,0) at bread+0x20 ffs_update(c976c110,0,c976c110,c387dc00,1) at ffs_update+0x228 ufs_inactive(e6a72b10,c976c18c,c976c110,e6a72b28,c055b84a) at ufs_inactive+0x16c VOP_INACTIVE_APV(c06f7380,e6a72b10) at VOP_INACTIVE_APV+0x9b vinactive(c976c110,c359da80) at vinactive+0x8a vput(c976c110,c387dc00,c0704720,c976c110,3) at vput+0x160 vn_close(c976c110,3,c3a21480,c359da80,e6a72bd8) at vn_close+0x96 vn_closefile(c3938750,c359da80) at vn_closefile+0xca fdrop_locked(c3938750,c359da80,c34a0fac,0,c06a41df) at fdrop_locked+0x88 fdrop(c3938750,c359da80,6af,c0716380,0) at fdrop+0x24 closef(c3938750,c359da80,0,0,3) at closef+0x35f db> trace 35655 Tracing pid 35655 tid 100118 td 0xc3b38480 sched_switch(c3b38480,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(d766c638,ecf9eacc,c050da15,d766c638,0) at sleepq_switch+0xe0 sleepq_wait(d766c638,0,0,c06ae919,e52) at sleepq_wait+0x30 msleep(d766c638,c075d140,4c,c06af047,0) at msleep+0x311 bwait(d766c638,4c,c06af047) at bwait+0x47 bufwait(d766c638,1,0,0,ecf9ebb8) at bufwait+0x1a breadn(c82c4220,0,0,1000,0) at breadn+0x266 bread(c82c4220,0,0,1000,0) at bread+0x20 ffs_read(ecf9ec04,c39381f8,c82c4220,ecf9ec50,c0565a6a) at ffs_read+0x23f VOP_READ_APV(c06f7380,ecf9ec04) at VOP_READ_APV+0x9b vn_read(c39381f8,ecf9ec78,c3a21480,0,c3b38480) at vn_read+0x196 dofileread(c3b38480,c39381f8,3,845e000,a9e) at dofileread+0xad read(c3b38480,ecf9ed04,3,9,202) at read+0x3b syscall(3b,3b,3b,845e000,a9e) at syscall+0x22f Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (3, FreeBSD ELF32, read), eip = 0x82d05d3, esp = 0xbfbfe60c, ebp = 0xbfbfe638 --- db> trace 35648 Tracing pid 35648 tid 100167 td 0xc8d32300 sched_switch(c8d32300,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8fd1480,0,c8d32300,ed04bb68,c050d9d9) at sleepq_switch+0xe0 sleepq_timedwait_sig(c8fd1480,0,0,100,c06a45e3) at sleepq_timedwait_sig+0xd msleep(c8fd1480,c8fd1480,158,c06a46bf,1388) at msleep+0x2d5 kqueue_scan(c8fd1480,1,ed04bcc8,ed04bcc0,ed04bbf4) at kqueue_scan+0x221 kern_kevent(c8d32300,5,0,1,ed04bcc8) at kern_kevent+0x151 kevent(c8d32300,ed04bd04,6,10,292) at kevent+0x55 syscall(3b,3b,3b,805f000,bfbfe3b0) at syscall+0x22f Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (363, FreeBSD ELF32, kevent), eip = 0x280c9503, esp = 0xbfbfe31c, ebp = 0xbfbfe458 --- db> trace 35647 Tracing pid 35647 tid 100136 td 0xc3b38180 sched_switch(c3b38180,0,1) at sched_switch+0x177 mi_switch(1,0) at mi_switch+0x270 sleepq_switch(c8ffc880,0,c3b38180,ecf98b68,c050d9d9) at sleepq_switch+0xe0 sleepq_timedwait_sig(c8ffc880,0,0,100,c06a45e3) at sleepq_timedwait_sig+0xd msleep(c8ffc880,c8ffc880,158,c06a46bf,1389) at msleep+0x2d5 kqueue_scan(c8ffc880,1,ecf98cc8,ecf98cc0,ecf98bf4) at kqueue_scan+0x221 kern_kevent(c3b38180,5,0,1,ecf98cc8) at kern_kevent+0x151 kevent(c3b38180,ecf98d04,6,e,292) at kevent+0x55 syscall(3b,3b,3b,805f000,bfbfe3b0) at syscall+0x22f Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (363, FreeBSD ELF32, kevent), eip = 0x280c9503, esp = 0xbfbfe31c, ebp = 0xbfbfe458 --- -- Jun Kuriyama // IMG SRC, Inc. // FreeBSD Project