From owner-freebsd-fs@FreeBSD.ORG  Fri Oct  8 15:19:10 2010
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id B4EB9106566C
	for <freebsd-fs@freebsd.org>; Fri,  8 Oct 2010 15:19:10 +0000 (UTC)
	(envelope-from martin@lispworks.com)
Received: from lwfs1-cam.cam.lispworks.com (mail.lispworks.com
	[193.34.186.230])
	by mx1.freebsd.org (Postfix) with ESMTP id 50D368FC0A
	for <freebsd-fs@freebsd.org>; Fri,  8 Oct 2010 15:19:09 +0000 (UTC)
Received: from higson.cam.lispworks.com
	(IDENT:U2FsdGVkX1/fLTiF8p2ELvuc74FyKm2jD/prQbn4XXk@higson
	[192.168.1.7])
	by lwfs1-cam.cam.lispworks.com (8.14.3/8.14.3) with ESMTP id
	o98FJ76Z054742; Fri, 8 Oct 2010 16:19:07 +0100 (BST)
	(envelope-from martin@lispworks.com)
Received: from higson.cam.lispworks.com by higson.cam.lispworks.com (8.13.1)
	id o98FJ7s7026878; Fri, 8 Oct 2010 16:19:07 +0100
Received: (from martin@localhost)
	by higson.cam.lispworks.com (8.13.1/8.13.1/Submit) id o98FJ7qJ026875;
	Fri, 8 Oct 2010 16:19:07 +0100
Date: Fri, 8 Oct 2010 16:19:07 +0100
Message-Id: <201010081519.o98FJ7qJ026875@higson.cam.lispworks.com>
From: Martin Simmons <martin@lispworks.com>
To: freebsd-fs@freebsd.org
In-reply-to: <E5332812-379B-4EC1-A134-12176C718B2E@free.de> (message from Kai
	Gallasch on Wed, 6 Oct 2010 20:51:49 +0200)
References: <39F05641-4E46-4BE0-81CA-4DEB175A5FBE@free.de>
	<201010061732.o96HW2Vi005945@higson.cam.lispworks.com>
	<E5332812-379B-4EC1-A134-12176C718B2E@free.de>
Subject: Re: Locked up processes after upgrade to ZFS v15
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
	<mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 08 Oct 2010 15:19:10 -0000

>>>>> On Wed, 6 Oct 2010 20:51:49 +0200, Kai Gallasch said:
> 
> Am 06.10.2010 um 19:32 schrieb Martin Simmons:
> 
> >>>>>> On Wed, 6 Oct 2010 14:28:31 +0200, Kai Gallasch said:
> >> 
> >> How can I debug this and get further information?
> > 
> > procstat -k -k $pid will generate a backtrace (or replace $pid by -a for all
> > processes).
> 
> procstat for process 12111 (state: zfs)
> sonnenkraft:~ # procstat -k -k 12111
>   PID    TID COMM             TDNAME           KSTACK                       
> 12111 102385 httpd            -                mi_switch+0x21b sleepq_switch+0x123 sleepq_wait+0x4d __lockmgr_args+0x7ae vop_stdlock+0x39 VOP_LOCK1_APV+0x9b _vn_lock+0x57 vget+0x7b cache_lookup+0x4e0 vfs_cache_lookup+0xc0 VOP_LOOKUP_APV+0xb7 lookup+0x3d3 namei+0x457 vn_open_cred+0x1e3 kern_openat+0x181 syscall+0x102 Xfast_syscall+0xe2

It looks like this thread is waiting for a lock.

So the question is, which thread is holding the lock?  It might be possible to
see from procstat -k -k -a but otherwise I think you will need to run it with
kdb and look at the lock.

__Martin