From owner-freebsd-stable@FreeBSD.ORG Tue May 27 11:09:56 2003 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 659AB37B401 for ; Tue, 27 May 2003 11:09:56 -0700 (PDT) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id BD79343FAF for ; Tue, 27 May 2003 11:09:55 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) by apollo.backplane.com (8.12.9/8.12.6) with ESMTP id h4RI9tVI066961; Tue, 27 May 2003 11:09:55 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9/8.12.6/Submit) id h4RI9ssD066960; Tue, 27 May 2003 11:09:54 -0700 (PDT) Date: Tue, 27 May 2003 11:09:54 -0700 (PDT) From: Matthew Dillon Message-Id: <200305271809.h4RI9ssD066960@apollo.backplane.com> To: "Marc G. Fournier" References: <20030521171941.364325314@netcom1.netcom.com> <20030524190051.R598@hub.org><20030526123617.D56519@hub.org> <20030526130556.G56519@hub.org> cc: stable@freebsd.org Subject: Re: system slowdown - vnode related X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 May 2003 18:09:56 -0000 I'm a little confused. What state is the vnlru kernel thread in? It sounds like vnlru must be stuck. Note that you can gdb the live kernel and get a stack backtrace of any stuck process. gdb -k /kernel.debug /dev/mem (or whatever) proc N (e.g. vnlru's pid) back All the processes stuck in 'inode' are likely associated with the problem, but if that is what is causing vnlru to be stuck I would expect vnlru itself to be stuck in 'inode'. unionfs is probably responsible. I would not be surprised at all if unionfs is causing a deadlock somewhere which is creating a chain of processes stuck in 'inode' which is in turn causing vnlru to get stuck. -Matt Matthew Dillon : :On Mon, 26 May 2003, Mike Harding wrote: : :> Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4? I :> thought it was the other way around - that 4_8 only got _some_ of the :> changes to RELENG_4... : :Ack, my fault ... sorry, wasn't thinking :( RELENG_4 is correct ... I :should have confirmed my settings before blathering on ... : :One of the scripts I used extensively while debugging this ... a quite :simple one .. was: : :#!/bin/tcsh :while ( 1 ) : echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'` : sleep 10 :end : :which outputs this: : :debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 - vlruwt : :I have my maxvnodes set to 512k right now ... now, when the server "hung", :the output would look something like (this would be with 'default' vnodes): : :debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup : :with the critical bit being the vlruwt -> vlrup change ... : :with unionfs, you are using two vnodes per file, instead of one in :non-union mode, which is why I went to 512k vs the default of ~256k vnodes :... it doesn't *fix* the problem, it only reduces its occurance ... :_______________________________________________ :freebsd-stable@freebsd.org mailing list :http://lists.freebsd.org/mailman/listinfo/freebsd-stable :To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" :