From owner-freebsd-stable@FreeBSD.ORG  Tue Jul 29 16:23:39 2003
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP
	id EC1EA37B401; Tue, 29 Jul 2003 16:23:39 -0700 (PDT)
Received: from mail.sandvine.com (sandvine.com [199.243.201.138])
	by mx1.FreeBSD.org (Postfix) with ESMTP
	id 2735C43FB1; Tue, 29 Jul 2003 16:23:39 -0700 (PDT)
	(envelope-from don@sandvine.com)
Received: by mail.sandvine.com with Internet Mail Service (5.5.2653.19)
	id <305LG0L2>; Tue, 29 Jul 2003 19:23:38 -0400
Message-ID: <FE045D4D9F7AED4CBFF1B3B813C8533702742067@mail.sandvine.com>
From: Don Bowman <don@sandvine.com>
To: Don Bowman <don@sandvine.com>,
	'Robert Watson' <rwatson@freebsd.org>,
	Dave Dolson <ddolson@sandvine.com>
Date: Tue, 29 Jul 2003 19:23:36 -0400
MIME-Version: 1.0
X-Mailer: Internet Mail Service (5.5.2653.19)
Content-Type: text/plain;
	charset="iso-8859-1"
cc: "'freebsd-stable@freebsd.org'" <freebsd-stable@freebsd.org>
Subject: RE: kernel deadlock
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Production branch of FreeBSD source code
	<freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
	<mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 29 Jul 2003 23:23:40 -0000

> From: Robert Watson [mailto:rwatson@freebsd.org]
> > On Tue, 29 Jul 2003, Dave Dolson wrote:
> > 
> > > To follow up, I've discovered that the system has 
> exhausted its "FFS
> > > node" malloc type. 
>  ...
> > 
> > Some problems with this have turned up in -CURRENT on large-memory
> > machines where some of the scaling factors have been off.  In 
> 
> We currently have kern.maxvnodes=70354 set (automatically 
> scaled). This
> is a 1GB box.
> 
> I will try re-running the test with less.
> 
> when it hits kern.maxvnodes, what will it do?

So I dropped kern.maxvnodes in half (to 35000). This 
has a 1GB of physical memory in a 2x xeon (w/ HTT enabled,
so 4 procs).

when it hit the limit, the system stopped switching amongst
processes. my vmstat blocked in 'vlruwk'.

I merged kern/52425 (kern/vfs_subr.c 1.249.2.30,
kern/vfs_syscalls.c 1.151.2.18, sys/mount.h 1.89.2.7)
which is supposed to address this, but it didn't.
[we're running 4.7].

after a long time, my ^C to the vmstat came through, and
my shell prompt came back, but then bash stopped in 'inode'.

In this case i'm not short of memory.

the test is doing this:

for (i = 0; i < 100000; i++) 
 mkdir dir.$i
 cd dir.$i

and while it was running i had:

while true
do
    vmstat -m | grep FFS
    sleep 1
done

running to watch it.

So it seems the problem may not be running out of memory
in the malloc pool, but in the vnode reclamation?

--don