From owner-freebsd-current@FreeBSD.ORG Fri Mar 13 03:57:29 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 71E83106566B; Fri, 13 Mar 2009 03:57:29 +0000 (UTC) (envelope-from john@baldwin.cx) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 3BD348FC12; Fri, 13 Mar 2009 03:57:29 +0000 (UTC) (envelope-from john@baldwin.cx) Received: from server.baldwin.cx (pool-98-109-39-197.nwrknj.fios.verizon.net [98.109.39.197]) by cyrus.watson.org (Postfix) with ESMTPSA id 9E6E746B3C; Thu, 12 Mar 2009 23:57:28 -0400 (EDT) Received: from [192.168.1.52] ([192.168.1.52]) (authenticated bits=0) by server.baldwin.cx (8.14.3/8.14.3) with ESMTP id n2D3vLme054759; Thu, 12 Mar 2009 23:57:21 -0400 (EDT) (envelope-from john@baldwin.cx) References: <20090312175345.Y80227@rust.salford.ac.uk> <20090312191333.GA97342@hyperion.scode.org> <49B97617.8010709@freebsd.org> <86r6124f2v.fsf@gmail.com> <3bbf2fe10903122035i20b2767cod2322c39c6f850ee@mail.gmail.com> Message-Id: <29C8FA04-D5B1-49B7-ACF0-4185537367B0@baldwin.cx> From: John Baldwin To: Attilio Rao In-Reply-To: <3bbf2fe10903122035i20b2767cod2322c39c6f850ee@mail.gmail.com> Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit X-Mailer: iPhone Mail (5H11) Mime-Version: 1.0 (iPhone Mail 5H11) Date: Thu, 12 Mar 2009 23:57:19 -0400 X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [192.168.1.2]); Thu, 12 Mar 2009 23:57:22 -0400 (EDT) X-Virus-Scanned: ClamAV 0.94.2/9102/Thu Mar 12 16:54:00 2009 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.6 required=4.2 tests=AWL,BAYES_00 autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx X-Mailman-Approved-At: Fri, 13 Mar 2009 05:11:45 +0000 Cc: Pawel Jakub Dawidek , "freebsd-current@freebsd.org" , Tim Kientzle , Mark Powell , Anonymous , Peter Schuller Subject: Re: repeatable ZFS panic: share->excl X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Mar 2009 03:57:29 -0000 This is similar to the patch I've asked lulf@ to test except that it is longer and I fix a bug where zfs_lookup() can leak a vnode lock if the access check fails. :-) The last one I sent to lulf@ is at www.FreeBSD.org/~jhb/patches/zfs_ea.patch . -- John Baldwin On Mar 12, 2009, at 11:35 PM, Attilio Rao wrote: > 2009/3/12, Anonymous : >> Tim Kientzle writes: >> >>> Peter Schuller wrote: >>>>> First problem I notice is a panic, which seems to occur every >>>>> time 'tar' is used. Might be unrelated to tar, but it definately >>>>> provokes it. Simply the tar command or a pkg_add causes it. >>>> >>>> See the "ZFS/extattr lockup"/"bsdtar lockup on current" threads >>>> from >>>> the past handful of days. >>> >>> I think this may be a different problem. >>> The earlier thread involved a ZFS bug that causes >>> it to lock up if it receives a request to enumerate >>> extended attributes on a file (via extattr_list_link >>> system call). Tar recently added support for >>> backing up extended attributes. I've disabled >>> that support until this particular ZFS problem can >>> be fixed. >> >> >> I guess you're wrong, it's same issue. Here is output from unmodified >> kernel (r189728) under qemu which looks exactly as on that >> screenshot. Perhaps, the panic is triggered by INVARIANTS. >> >> # lsextattr -h user foo >> shared lock of (lockmgr) zfs @ /usr/src/sys/kern/vfs_lookup.c:477 >> while exclusively locked from /usr/src/sys/modules/zfs/../../cddl/ >> contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:152 >> panic: share->excl >> cpuid = 0 >> KDB: enter: panic >> [thread pid 105 tid 100078 ] >> Stopped at kdb_enter+0x3d: movq $0,0x662538(%rip) >> >> db> bt >> Tracing pid 105 tid 100078 td 0xffffff00015bea80 >> kdb_enter() at kdb_enter+0x3d >> panic() at panic+0x17b >> witness_checkorder() at witness_checkorder+0x16e >> __lockmgr_args() at __lockmgr_args+0xd1b >> vop_stdlock() at vop_stdlock+0x39 >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b >> _vn_lock() at _vn_lock+0x57 >> lookup() at lookup+0xf4 >> namei() at namei+0x545 >> zfs_listextattr() at zfs_listextattr+0x18c >> VOP_LISTEXTATTR_APV() at VOP_LISTEXTATTR_APV+0xb5 >> extattr_list_vp() at extattr_list_vp+0x22a >> extattr_list_link() at extattr_list_link+0xc3 >> syscall() at syscall+0x1e7 >> Xfast_syscall() at Xfast_syscall+0xab >> --- syscall (439, FreeBSD ELF64, extattr_list_link), rip = >> 0x800692e4c, rsp = 0x7fffffffed08, rbp = 0x7fffffffede0 --- >> >> db> show all locks >> Process 105 (lsextattr) thread 0xffffff00015bea80 (100078) >> exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581578) locked @ / >> usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ >> fs/zfs/zfs_znode.c:152 >> exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581a58) locked @ / >> usr/src/sys/kern/vfs_extattr.c:668 >> >> db> show lockedvnods >> Locked vnodes >> >> 0xffffff00015819c0: tag zfs, type VREG >> usecount 1, writecount 0, refcount 1 mountedhere 0 >> flags () >> v_object 0xffffff0001574898 ref 0 pages 0 >> lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105) >> #0 0xffffffff80530568 at __lockmgr_args+0x758 >> #1 0xffffffff805c0fd9 at vop_stdlock+0x39 >> #2 0xffffffff808557db at VOP_LOCK1_APV+0x9b >> #3 0xffffffff805dd6a7 at _vn_lock+0x57 >> #4 0xffffffff805c2f20 at extattr_list_vp+0xb0 >> #5 0xffffffff805c3173 at extattr_list_link+0xc3 >> #6 0xffffffff8080f227 at syscall+0x1e7 >> #7 0xffffffff807ec39b at Xfast_syscall+0xab >> >> 0xffffff00015814e0: tag zfs, type VDIR >> usecount 1, writecount 0, refcount 1 mountedhere 0 >> flags () >> lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105) >> #0 0xffffffff80530568 at __lockmgr_args+0x758 >> #1 0xffffffff805c0fd9 at vop_stdlock+0x39 >> #2 0xffffffff808557db at VOP_LOCK1_APV+0x9b >> #3 0xffffffff805dd6a7 at _vn_lock+0x57 >> #4 0xffffffff8108405e at zfs_znode_cache_constructor+0x4e >> #5 0xffffffff81085029 at zfs_znode_alloc+0x39 >> #6 0xffffffff81085915 at zfs_mknode+0x205 >> #7 0xffffffff810948f5 at zfs_make_xattrdir+0x155 >> #8 0xffffffff81095bc3 at zfs_get_xattrdir+0xd3 >> #9 0xffffffff810a4dbf at zfs_lookup+0x11f >> #10 0xffffffff810a51d8 at zfs_listextattr+0x128 >> #11 0xffffffff80853255 at VOP_LISTEXTATTR_APV+0xb5 >> #12 0xffffffff805c309a at extattr_list_vp+0x22a >> #13 0xffffffff805c3173 at extattr_list_link+0xc3 >> #14 0xffffffff8080f227 at syscall+0x1e7 >> #15 0xffffffff807ec39b at Xfast_syscall+0xab > > Could you please re-enable the extended attributes on bsdtar, try this > patch and report to me?: > http://www.freebsd.org/~attilio/zfs_vnops.diff > > (sorry if it is untested but I just don't have any ZFS machine). > Actually, I think the problem is that zfs_lookup() doesn't get the > correct handover once the extended attributes directory vnode is > retrieved. > > Thanks, > Attilio > > > -- > Peace can only be achieved by understanding - A. Einstein