From owner-freebsd-fs@FreeBSD.ORG Tue Feb 1 14:53:16 2011 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0F0731065748 for ; Tue, 1 Feb 2011 14:53:16 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id C011F8FC1A for ; Tue, 1 Feb 2011 14:53:15 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 590F346B03; Tue, 1 Feb 2011 09:53:15 -0500 (EST) Received: from jhbbsd.localnet (unknown [209.249.190.10]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4F3778A027; Tue, 1 Feb 2011 09:53:14 -0500 (EST) From: John Baldwin To: freebsd-fs@freebsd.org Date: Tue, 1 Feb 2011 09:46:16 -0500 User-Agent: KMail/1.13.5 (FreeBSD/7.4-CBSD-20110107; KDE/4.4.5; amd64; ; ) References: <4D47D847.8080308@bytecamp.net> In-Reply-To: <4D47D847.8080308@bytecamp.net> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: quoted-printable Message-Id: <201102010946.16466.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.6 (bigwig.baldwin.cx); Tue, 01 Feb 2011 09:53:14 -0500 (EST) X-Virus-Scanned: clamav-milter 0.96.3 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=0.5 required=4.2 tests=BAYES_00,MAY_BE_FORGED, RDNS_DYNAMIC autolearn=no version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on bigwig.baldwin.cx Cc: Subject: Re: LOR in dirhash X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Feb 2011 14:53:16 -0000 On Tuesday, February 01, 2011 4:54:15 am Robert Schulze wrote: > We are currently having continuous problems with 8-STABLE. > Since we upgraded, we have "double faults" every three days or so,=20 > yesterday we built a debugging kernel to identify the problem. >=20 > Today I saw a LOR in /var/log/messages, the whole output: >=20 > 8<---------------------------------------------------------- >=20 > lock order reversal: > 1st 0xffffff81ef08ce98 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:26= 36 > 2nd 0xffffff00157b4600 dirhash (dirhash) @=20 > /usr/src/sys/ufs/ufs/ufs_dirhash.c:285 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > kdb_backtrace() at kdb_backtrace+0x37 > _witness_debugger() at _witness_debugger+0x49 > witness_checkorder() at witness_checkorder+0x7d3 > _sx_xlock() at _sx_xlock+0x4a > ufsdirhash_acquire() at ufsdirhash_acquire+0x3a > ufsdirhash_add() at > ufsdirhash_add+0x19 > ufs_diren > ter() at > ufs_direnter+0x876 > ufs_makeinode() at > ufs_makeinode+0x239 > VOP_CREATE_APV() at > VOP_CREATE_APV+0xb6 > vn_open_cred() at vn_open_cred+0x415 > kern_openat() at > kern_openat+0x165 > syscallenter() at syscallenter+0xe5 > syscall() at > syscall+0x55 > Xfast_syscall() at > Xfast_syscall+0xe2 > --- syscall (5, FreeBSD ELF64 > , open) > , rip =3D > 0x8009 > a4a7c, > rsp =3D 0x > 7ffffff > e918, r > bp =3D 0x1 --- >=20 > 8<---------------------------------------------------------- >=20 > maybe this is a seriuos one? =46rom the source code in ufs_dirhash.c: /* * Locking: * * The relationship between inode and dirhash is protected either by an * exclusive vnode lock or the vnode interlock where a shared vnode lock * may be used. The dirhash_mtx is acquired after the dirhash lock. To * handle teardown races, code wishing to lock the dirhash for an inode * when using a shared vnode lock must obtain a private reference on the * dirhash while holding the vnode interlock. They can drop it once they * have obtained the dirhash lock and verified that the dirhash wasn't * recycled while they waited for the dirhash lock. * * ufsdirhash_build() acquires a shared lock on the dirhash when it is * successful. This lock is released after a call to ufsdirhash_lookup(). * * Functions requiring exclusive access use ufsdirhash_acquire() which may * free a dirhash structure that was recycled by ufsdirhash_recycle(). * * The dirhash lock may be held across io operations. * * WITNESS reports a lock order reversal between the "bufwait" lock * and the "dirhash" lock. However, this specific reversal will not * cause a deadlock. To get a deadlock, one would have to lock a * buffer followed by the dirhash while a second thread locked a * buffer while holding the dirhash lock. The second order can happen * under a shared or exclusive vnode lock for the associated directory * in lookup(). The first order, however, can only happen under an * exclusive vnode lock (e.g. unlink(), rename(), etc.). Thus, for * a thread to be doing a "bufwait" -> "dirhash" order, it has to hold * an exclusive vnode lock. That exclusive vnode lock will prevent * any other threads from doing a "dirhash" -> "bufwait" order. */ =2D-=20 John Baldwin