From owner-freebsd-fs@FreeBSD.ORG  Fri Mar  5 15:56:51 2004
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id BAF5516A4CE
	for <freebsd-fs@FreeBSD.org>; Fri,  5 Mar 2004 15:56:51 -0800 (PST)
Received: from gw.catspoiler.org (217-ip-163.nccn.net [209.79.217.163])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 7069543D1D
	for <freebsd-fs@FreeBSD.org>; Fri,  5 Mar 2004 15:56:51 -0800 (PST)
	(envelope-from truckman@FreeBSD.org)
Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2])
	by gw.catspoiler.org (8.12.9p2/8.12.9) with ESMTP id i25Nuh7E008690;
	Fri, 5 Mar 2004 15:56:48 -0800 (PST)
	(envelope-from truckman@FreeBSD.org)
Message-Id: <200403052356.i25Nuh7E008690@gw.catspoiler.org>
Date: Fri, 5 Mar 2004 15:56:43 -0800 (PST)
From: Don Lewis <truckman@FreeBSD.org>
To: rick@snowhite.cis.uoguelph.ca
In-Reply-To: <200403052250.RAA68715@snowhite.cis.uoguelph.ca>
MIME-Version: 1.0
Content-Type: TEXT/plain; charset=us-ascii
cc: freebsd-fs@FreeBSD.org
Subject: Re: newnfsd's stuck on "ufs"
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.1
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 05 Mar 2004 23:56:51 -0000

On  5 Mar, rick@snowhite.cis.uoguelph.ca wrote:
> I'm working away at porting my NFSv4 server to FreeBSD5.2. It goes along
> ok for a while, but when doing several ops concurrently, most of the nfsd
> threads end up stuck sleeping on "ufs" as shown by the attached "ps axl".
> (The amusing part is that, once all but one thread is stuck, the last
>  thread works fine. In other words, the "fix" is to only run one newnfsd:-)
> 
> Anybody happen to know off the top of head, what I've screwed up?
> 
> Thanks for any hints, rick
> --- ps axl of newnfsd ---
>     0   523   522   0   4  0  1192  736 nfsd   I     ??    0:05.34 newnfsd: ser
>     0   524   522   0  -4  0  1192  736 ufs    D     ??    0:00.02 newnfsd: ser
>     0   525   522   0  -4  0  1192  736 ufs    D     ??    0:00.02 newnfsd: ser

Most likely a vnode locking bug.  Rebuild the kernel with
	options         DEBUG_VFS_LOCKS
	options         DDB

and see what it turns up.

There are some tuneable variables you can tweak in DDB to tune the
behaviour of the debugging code.

int vfs_badlock_ddb = 1;        /* Drop into debugger on violation. */
int vfs_badlock_mutex = 1;      /* Check for interlock across VOPs. */
int vfs_badlock_print = 1;      /* Print lock violations. */

The default is to drop into DDB every time a vnode lock assertion is
violated.  If you turn off vfs_badlock_ddb, the assertion violations
will just spam the console.