From owner-freebsd-stable@FreeBSD.ORG  Tue May 27 09:04:44 2003
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id C47D837B404
	for <stable@freebsd.org>; Tue, 27 May 2003 09:04:43 -0700 (PDT)
Received: from hub.org (hub.org [64.117.225.220])
	by mx1.FreeBSD.org (Postfix) with ESMTP id CF48343FB1
	for <stable@freebsd.org>; Tue, 27 May 2003 09:04:41 -0700 (PDT)
	(envelope-from scrappy@hub.org)
Received: from portoclub.gr (unknown [64.117.225.220])
	by hub.org (Postfix) with ESMTP
	id 57EEA6BC909; Mon, 26 May 2003 13:12:44 -0300 (ADT)
Date: Mon, 26 May 2003 13:12:44 -0300 (ADT)
From: "Marc G. Fournier" <scrappy@hub.org>
To: Mike Harding <mvh@ix.netcom.com>
In-Reply-To: <1053964809.7831.6.camel@netcom1.netcom.com>
Message-ID: <20030526130556.G56519@hub.org>
References: <20030521171941.364325314@netcom1.netcom.com>
	<20030524190051.R598@hub.org><20030526123617.D56519@hub.org>
	<1053964809.7831.6.camel@netcom1.netcom.com>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
cc: stable@freebsd.org
Subject: Re: system slowdown - vnode related
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Production branch of FreeBSD source code
	<freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 27 May 2003 16:04:44 -0000

On Mon, 26 May 2003, Mike Harding wrote:

> Er - are any changes made to RELENG_4_8 that aren't made to RELENG_4?  I
> thought it was the other way around - that 4_8 only got _some_ of the
> changes to RELENG_4...

Ack, my fault ... sorry, wasn't thinking :(  RELENG_4 is correct ... I
should have confirmed my settings before blathering on ...

One of the scripts I used extensively while debugging this ... a quite
simple one .. was:

#!/bin/tcsh
while ( 1 )
  echo `sysctl debug.numvnodes` - `sysctl debug.freevnodes` - `sysctl debug.vnlru_nowhere` - `ps auxl | grep vnlru | grep -v grep | awk '{print $20}'`
  sleep 10
end

which outputs this:

debug.numvnodes: 463421 - debug.freevnodes: 220349 - debug.vnlru_nowhere: 3 - vlruwt

I have my maxvnodes set to 512k right now ... now, when the server "hung",
the output would look something like (this would be with 'default' vnodes):

debug.numvnodes: 199252 - debug.freevnodes: 23 - debug.vnlru_nowhere: 12 - vlrup

with the critical bit being the vlruwt -> vlrup change ...

with unionfs, you are using two vnodes per file, instead of one in
non-union mode, which is why I went to 512k vs the default of ~256k vnodes
... it doesn't *fix* the problem, it only reduces its occurance ...