From owner-freebsd-bugs@FreeBSD.ORG  Wed Oct 12 04:10:10 2011
Return-Path: <owner-freebsd-bugs@FreeBSD.ORG>
Delivered-To: freebsd-bugs@hub.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 2E1D8106566C
	for <freebsd-bugs@hub.freebsd.org>;
	Wed, 12 Oct 2011 04:10:10 +0000 (UTC)
	(envelope-from gnats@FreeBSD.org)
Received: from freefall.freebsd.org (freefall.freebsd.org
	[IPv6:2001:4f8:fff6::28])
	by mx1.freebsd.org (Postfix) with ESMTP id 05B728FC13
	for <freebsd-bugs@hub.freebsd.org>;
	Wed, 12 Oct 2011 04:10:10 +0000 (UTC)
Received: from freefall.freebsd.org (localhost [127.0.0.1])
	by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p9C4A9M9045594
	for <freebsd-bugs@freefall.freebsd.org>; Wed, 12 Oct 2011 04:10:09 GMT
	(envelope-from gnats@freefall.freebsd.org)
Received: (from gnats@localhost)
	by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p9C4A9l1045593;
	Wed, 12 Oct 2011 04:10:09 GMT (envelope-from gnats)
Resent-Date: Wed, 12 Oct 2011 04:10:09 GMT
Resent-Message-Id: <201110120410.p9C4A9l1045593@freefall.freebsd.org>
Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer)
Resent-To: freebsd-bugs@FreeBSD.org
Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org,
	Garrett Cooper <gcooper@ixsystems.com>
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 4211D106564A
	for <freebsd-gnats-submit@FreeBSD.org>;
	Wed, 12 Oct 2011 04:09:03 +0000 (UTC)
	(envelope-from nobody@FreeBSD.org)
Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22])
	by mx1.freebsd.org (Postfix) with ESMTP id 318D58FC14
	for <freebsd-gnats-submit@FreeBSD.org>;
	Wed, 12 Oct 2011 04:09:03 +0000 (UTC)
Received: from red.freebsd.org (localhost [127.0.0.1])
	by red.freebsd.org (8.14.4/8.14.4) with ESMTP id p9C493pA057114
	for <freebsd-gnats-submit@FreeBSD.org>; Wed, 12 Oct 2011 04:09:03 GMT
	(envelope-from nobody@red.freebsd.org)
Received: (from nobody@localhost)
	by red.freebsd.org (8.14.4/8.14.4/Submit) id p9C4929H057113;
	Wed, 12 Oct 2011 04:09:02 GMT (envelope-from nobody)
Message-Id: <201110120409.p9C4929H057113@red.freebsd.org>
Date: Wed, 12 Oct 2011 04:09:02 GMT
From: Garrett Cooper <gcooper@ixsystems.com>
To: freebsd-gnats-submit@FreeBSD.org
X-Send-Pr-Version: www-3.1
Cc: 
Subject: kern/161511: Filesystem deadlocks when using multiple unionfs
	mounts on top of single filesystem
X-BeenThere: freebsd-bugs@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Bug reports <freebsd-bugs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-bugs>,
	<mailto:freebsd-bugs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-bugs>
List-Post: <mailto:freebsd-bugs@freebsd.org>
List-Help: <mailto:freebsd-bugs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-bugs>,
	<mailto:freebsd-bugs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 12 Oct 2011 04:10:10 -0000


>Number:         161511
>Category:       kern
>Synopsis:       Filesystem deadlocks when using multiple unionfs mounts on top of single filesystem
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Oct 12 04:10:09 UTC 2011
>Closed-Date:
>Last-Modified:
>Originator:     Garrett Cooper
>Release:        10-CURRENT
>Organization:
iXsystems, Inc.
>Environment:
FreeBSD streetfighter.ixsystems.com 10.0-CURRENT FreeBSD 10.0-CURRENT #4 r226140M: Tue Oct 11 17:03:51 PDT 2011     root@streetfighter.ixsystems.com:/usr/obj/usr/src/sys/STREETFIGHTER  amd64
>Description:
Mounting a filesystem twice with unionfs like so causes filesystem deadlocks when files are modified on top of unionfs:

#!/bin/sh

mkdir pristine
cat > supfile <<EOF
*default host=cvsup1.freebsd.org
*default base=/FreeBSD/sup
*default prefix=$PWD/FreeBSD
*default release=cvs
*default delete use-rel-suffix
*default compress
EOF
csup supfile
mkdir touched union
mount -t unionfs pristine union
mount -t unionfs touched union
cd union
make buildworld -j4

=====

Any value of -j above -j1 causes filesystem deadlocks; processes like make will block when trying to access directory information -- it's worse for UFS than ZFS because it seems like some structures -- in particular the UFS_DIRHASH -- might be locked indefinitely causing the entire filesystem to become unusable. In either case I've seen the filesystem will prevent rebooting from completing cleanly as there are processes waiting on filesystem resources to become available.

This can also be stimulated via freenas builds like so:

sh build/do_build.sh -U

LORs have also been identified within unionfs around when the deadlock occurs, which might suggest that incorrect locking is preventing unionfs from releasing resources when a [soft] failure occurs. I don't have the LORs right now, but I'm hunting them down in my logs..
>How-To-Repeat:
See decomposed example above.
>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: