From owner-freebsd-current@FreeBSD.ORG Tue Aug 3 03:38:10 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D432416A4CE; Tue, 3 Aug 2004 03:38:10 +0000 (GMT) Received: from bloodwood.hunterlink.net.au (smtp-local.hunterlink.net.au [203.12.144.17]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D0A443D55; Tue, 3 Aug 2004 03:38:04 +0000 (GMT) (envelope-from boris@brooknet.com.au) Received: from [61.8.39.16] (ppp2710.dyn.pacific.net.au [61.8.39.16]) i733RVc7019895; Tue, 3 Aug 2004 13:27:33 +1000 From: Sam Lawrance To: Pawel Jakub Dawidek In-Reply-To: <20040802063851.GL39839@darkness.comp.waw.pl> References: <1091411071.922.20.camel@dirk.no.domain> <20040802063851.GL39839@darkness.comp.waw.pl> Content-Type: text/plain Message-Id: <1091504341.729.25.camel@dirk.no.domain> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Tue, 03 Aug 2004 13:39:02 +1000 Content-Transfer-Encoding: 7bit cc: freebsd-current@FreeBSD.org Subject: Re: processes freezing when writing to gstripe'd device X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Aug 2004 03:38:11 -0000 > +> I am observing processes performing operations on a gstripe device > +> freeze in state 'bufwait'. An 'rm' process is stuck right now. The rest > +> of the system is fine. > +> > +> What's the best way to look in to this? I can't attach to rm with gdb > +> (it just ends up waiting for something). I can drop to kdb, but have no > +> idea where to go from there. > > You could use 'ps' command from DDB to which processes are alseep. > Then you can run 'tr ' where is PID of sleeping process. > Look for processes related somehow to this problem. > > It'll be also great if you can provide exact procedure which will also > me to reproduce this problem. Okay, I updated to current as of yesterday and still seeing the same problem. I'm new to these bits of the kernel but it looks like a locking problem. This is what I am doing: dd if=/dev/zero of=sd0 count=20480 cp sd0 sd1 mdconfig -a -t vnode -f sd0 mdconfig -a -t vnode -f sd1 gstripe label bork md0 md1 newfs /dev/stripe/bork mkdir teststripe mount /dev/stripe/bork teststripe cd teststripe Now I repeatedly 'cvs checkout' and 'rm -rf' the FreeBSD src tree. Usually it freezes during the first checkout. Siginfo shows: load:1.14 cmd: cvs 801 [biowr] 0.33u 3.35s 14% 2840k A trace of the frozen cvs process 801 shows: KDB: enter: manual escape to debugger [thread 100006] Stopped at kdb_enter+0x2b: nop db> tr 801 sched_switch(c1a87580,0) at sched_switch+0x12b mi_switch(1,0) at mi_switch+0x24d sleepq_switch(c63ee500,d0c83814,c06030e9,c63ee500,0) at sleepq_switch+0xe0 sleepq_wait(c63ee500,0,0,0,c07f3ab7) at sleepq_wait+0xb msleep(c63ee500,c08ddd80,4c,c07f40d1,0) at msleep+0x375 bwait(c63ee500,4c,c07f40d1) at bwait+0x47 bufwait(c63ee500,c088f1a0,c1cd6318,c63ee500,0) at bufwait+0x2d ibwrite(c63ee500,d0c838d8,c071906e,c63ee500,a00) at ibwrite+0x3e2 bwrite(c63ee500,a00,0,ee,c19b1834) at bwrite+0x32 ffs_update(c19c3738,1,0,c19b808c,c19c3738) at ffs_update+0x302 ufs_makeinode(81a4,c199f840,d0c83bf8,d0c83c0c) at ufs_makeinode+0x3a3 ufs_create(d0c83a74,d0c83b30,c0655238,d0c83a74,c08b8c00) at ufs_create+0x26 ufs_vnoperate(d0c83a74) at ufs_vnoperate+0x13 vn_open_cred(d0c83be4,d0c83ce4,1a4,c1d47700,8) at vn_open_cred+0x174 vn_open(d0c83be4,d0c83ce4,1a4,8,c08ad240) at vn_open+0x1e kern_open(c1a87580,8199430,0,602,1b6) at kern_open+0xd2 open(c1a87580,d0c83d14,3,1be,292) at open+0x18 syscall(2f,bfbf002f,bfbf002f,8,2836c7f8) at syscall+0x217 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x282e2437, esp = 0xbfbfdd7c, ebp = 0xbfbfdda8 --- If I try to rm -rf src, that process also ends up waiting: sched_switch(c1a879a0,0) at sched_switch+0x12b mi_switch(1,0) at mi_switch+0x24d sleepq_switch(c199f8ec,d0c8ca58,c06030e9,c199f8ec,0) at sleepq_switch+0xe0 sleepq_wait(c199f8ec,0,0,0,c199f840) at sleepq_wait+0xb msleep(c199f8ec,c08ad898,50,c07f34e1,0) at msleep+0x375 acquire(d0c8cab0,1000040,600,c1a879a0,0) at acquire+0x9e lockmgr(c199f8ec,1010002,c199f840,c1a879a0,10002) at lockmgr+0x37e ufs_lock(d0c8cadc,d0c8caf8,c06563a5,d0c8cadc,c088f0a0) at ufs_lock+0x3c ufs_vnoperate(d0c8cadc) at ufs_vnoperate+0x13 vn_lock(c199f840,10002,c1a879a0,c1cda000,c1cda000) at vn_lock+0xc5 vget(c199f840,2,c1a879a0,139a,c1a879a0) at vget+0xc9 vfs_cache_lookup(d0c8cbb4,d0c8cbd0,c0646613,d0c8cbb4,c1a879a0) at vfs_cache_lookup+0x1c9 ufs_vnoperate(d0c8cbb4) at ufs_vnoperate+0x13 lookup(d0c8cc30,c1a879a0,c197d948,0,c07f51e9) at lookup+0x2cf namei(d0c8cc30,8051aa8,0,0,c16e2c60) at namei+0x1f0 lstat(c1a879a0,d0c8cd14,2,7,292) at lstat+0x4a syscall(2f,2f,2f,8051a00,8051a48) at syscall+0x217 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (190, FreeBSD ELF32, lstat), eip = 0x280cb757, esp = 0xbfbfe74c, ebp = 0xbfbfe7e8 --- I'll keep poking around - if you have any further suggestions or need other information, fire away. Regards Sam.