From owner-freebsd-stable@FreeBSD.ORG Tue May 27 11:27:43 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id ABE7B37B401 for ; Tue, 27 May 2003 11:27:43 -0700 (PDT) Received: from hawk.mail.pas.earthlink.net (hawk.mail.pas.earthlink.net [207.217.120.22]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1A8EC43FA3 for ; Tue, 27 May 2003 11:27:43 -0700 (PDT) (envelope-from mvh@ix.netcom.com) Received: from lsanca1-ar6-4-62-205-062.lsanca1.elnk.dsl.genuity.net ([4.62.205.62] helo=netcom1.netcom.com) by hawk.mail.pas.earthlink.net with esmtp (Exim 3.33 #1) id 19KjAZ-0005uG-00; Tue, 27 May 2003 11:27:31 -0700 Received: from [127.0.0.1] (localhost [127.0.0.1]) by netcom1.netcom.com (Postfix) with ESMTP id 9C2A952FB; Tue, 27 May 2003 11:27:30 -0700 (PDT) From: Mike Harding To: Matthew Dillon In-Reply-To: <200305271809.h4RI9ssD066960@apollo.backplane.com> References: <20030521171941.364325314@netcom1.netcom.com> <20030524190051.R598@hub.org><20030526123617.D56519@hub.org> <1053964809.7831.6.camel@netcom1.netcom.com> <20030526130556.G56519@hub.org> <200305271809.h4RI9ssD066960@apollo.backplane.com> Content-Type: text/plain Organization: Message-Id: <1054060050.640.35.camel@netcom1.netcom.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.2.4 Date: 27 May 2003 11:27:30 -0700 Content-Transfer-Encoding: 7bit cc: stable@freebsd.org Subject: Re: system slowdown - vnode related X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2003 18:27:44 -0000 I'll try this if I can tickle the bug again. I may have just run out of freevnodes - I only have about 1-2000 free right now. I was just surprised because I have never seen a reference to tuning this sysctl. - Mike H. On Tue, 2003-05-27 at 11:09, Matthew Dillon wrote: > I'm a little confused. What state is the vnlru kernel thread in? It > sounds like vnlru must be stuck. > > Note that you can gdb the live kernel and get a stack backtrace of any > stuck process. > > gdb -k /kernel.debug /dev/mem (or whatever) > proc N (e.g. vnlru's pid) > back > > All the processes stuck in 'inode' are likely associated with the > problem, but if that is what is causing vnlru to be stuck I would expect > vnlru itself to be stuck in 'inode'. > > unionfs is probably responsible. I would not be surprised at all if > unionfs is causing a deadlock somewhere which is creating a chain of > processes stuck in 'inode' which is in turn causing vnlru to get stuck. > > -Matt > Matthew Dillon > > > : > :On Mon, 26 May 2003, Mike Harding wrote: > : > :> Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4? I > :> thought it was the other way around - that 4_8 only got _some_ of the > :> changes to RELENG_4... > : > :Ack, my fault ... sorry, wasn't thinking :( RELENG_4 is correct ... I > :should have confirmed my settings before blathering on ... > : > :One of the scripts I used extensively while debugging this ... a quite > :simple one .. was: > : > :#!/bin/tcsh > :while ( 1 ) > : echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'` > : sleep 10 > :end > : > :which outputs this: > : > :debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 - vlruwt > : > :I have my maxvnodes set to 512k right now ... now, when the server "hung", > :the output would look something like (this would be with 'default' vnodes): > : > :debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup > : > :with the critical bit being the vlruwt -> vlrup change ... > : > :with unionfs, you are using two vnodes per file, instead of one in > :non-union mode, which is why I went to 512k vs the default of ~256k vnodes > :... it doesn't *fix* the problem, it only reduces its occurance ... > :_______________________________________________ > :freebsd-stable@freebsd.org mailing list > :http://lists.freebsd.org/mailman/listinfo/freebsd-stable > :To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > :