From owner-freebsd-current@FreeBSD.ORG Fri Jan 18 07:12:08 2008 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5BD3816A41B for ; Fri, 18 Jan 2008 07:12:08 +0000 (UTC) (envelope-from swhetzel@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.158]) by mx1.freebsd.org (Postfix) with ESMTP id A19B613C46A for ; Fri, 18 Jan 2008 07:12:07 +0000 (UTC) (envelope-from swhetzel@gmail.com) Received: by fg-out-1718.google.com with SMTP id 16so1077453fgg.35 for ; Thu, 17 Jan 2008 23:12:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=AnD6DFYNcK52VV8VNfoxyhxTo99WtfFQi9Y5PqDik7Q=; b=ZY5+gAu9F54DmAm+F/q6LOaZgOjQRjLGvj0FmsNKT8LWe4/ulrfDbmT6RGnYj7NefqW5doIKxLcQ807h36UiQzS+Jj7FJc/5Rgg7a5tgy2NUeTFtKbdxUUDUSRvq4GiDpQ3XBO/tWuLoKd26M7BcnNkp9Dp8zkKRmkgq9U5ll1w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=NoVZw0/syh0OJnPesF0kTkFyaZ9VzfPFqInZV4EFVBnOTZLrzRrdtXU2bGABjygsfE4UMmwmTITzXrg57vl5+kNJubzXFeBLY9yrwgepEIJFgSRNtxRlEo+PiCHEkxxd7UdY3i+DNvXlK5nWf7DA/MGBMo683+ZHR7WYLsLQYWE= Received: by 10.86.100.7 with SMTP id x7mr2762986fgb.63.1200640326504; Thu, 17 Jan 2008 23:12:06 -0800 (PST) Received: by 10.86.70.11 with HTTP; Thu, 17 Jan 2008 23:12:06 -0800 (PST) Message-ID: <790a9fff0801172312o18d46a65x3be9c8f752f7845d@mail.gmail.com> Date: Fri, 18 Jan 2008 01:12:06 -0600 From: "Scot Hetzel" To: "Pawel Jakub Dawidek" In-Reply-To: <20080117114023.GF8820@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <790a9fff0801150552l542a4238ofc12efe5fdb45fc2@mail.gmail.com> <20080115143924.GB57756@deviant.kiev.zoral.com.ua> <790a9fff0801152301l5d50461aw3a253608506368e2@mail.gmail.com> <790a9fff0801161624g459857b8y6217e579337f2f0e@mail.gmail.com> <790a9fff0801161810j5a9953c4j5d2f3f10b3a1baf@mail.gmail.com> <20080117114023.GF8820@garage.freebsd.pl> Cc: freebsd-current@freebsd.org Subject: Re: panic: System call lstat returning with 1 locks held X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Jan 2008 07:12:08 -0000 On 1/17/08, Pawel Jakub Dawidek wrote: > On Wed, Jan 16, 2008 at 08:10:03PM -0600, Scot Hetzel wrote: > > The local CVS repository is on a ZFS filesystem. Is anyone seeing > > this problem on a UFS filesystem? > > WITNESS won't work on ZFS' lock by default. Please add: > > CFLAGS+=-DDEBUG > > to sys/modules/zfs/Makefile and recompile zfs kernel module. DEBUG > define tells ZFS not to add NOWITNESS flag at lock initialization time. > I rebuilt the zfs module as suggested. When I reboot, I am now seeing 4 different lock order reversals related to ZFS: 1. This lock order reversal occurs most often: lock order reversal: 1st 0xffffff0001b95838 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1866 2nd 0xffffff00017531c0 db->db_mtx (db->db_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1888 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x606 _sx_xlock() at _sx_xlock+0x52 dbuf_sync_list() at dbuf_sync_list+0x215 dbuf_sync_list() at dbuf_sync_list+0x194 dnode_sync() at dnode_sync+0x385 dmu_objset_sync() at dmu_objset_sync+0x116 dsl_pool_sync() at dsl_pool_sync+0x153 spa_sync() at spa_sync+0x39e txg_sync_thread() at txg_sync_thread+0x17d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffd72a3d30, rbp = 0 --- 2. This lock order reversal is similar to the one above: lock order reversal: 1st 0xffffff0001b00d38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1866 2nd 0xffffff0001a27760 db->db_mtx (db->db_mtx) @ /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1837 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a witness_checkorder() at witness_checkorder+0x606 _sx_xlock() at _sx_xlock+0x52 dbuf_sync_list() at dbuf_sync_list+0xaf dbuf_sync_list() at dbuf_sync_list+0x194 dnode_sync() at dnode_sync+0x385 dmu_objset_sync() at dmu_objset_sync+0x116 dsl_pool_sync() at dsl_pool_sync+0x72 spa_sync() at spa_sync+0x39e txg_sync_thread() at txg_sync_thread+0x17d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffffffd72a3d30, rbp = 0 --- 3. lock order reversal related to the write syscall lock order reversal: 1st 0xffffff00269f6500 dn->dn_mtx