From owner-freebsd-current@FreeBSD.ORG Fri Mar 13 15:52:48 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 294BE106566C; Fri, 13 Mar 2009 15:52:48 +0000 (UTC) (envelope-from swell.k@gmail.com) Received: from mail-ew0-f166.google.com (mail-ew0-f166.google.com [209.85.219.166]) by mx1.freebsd.org (Postfix) with ESMTP id 9FCEE8FC13; Fri, 13 Mar 2009 15:52:46 +0000 (UTC) (envelope-from swell.k@gmail.com) Received: by ewy10 with SMTP id 10so2869769ewy.43 for ; Fri, 13 Mar 2009 08:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:from:to:cc:subject:references :date:in-reply-to:message-id:user-agent:mime-version:content-type; bh=csySFt5DDKumO7BVWG9LOpMZyep6xKJLg9KSv64czuc=; b=S9WlBBhxfDkL956gl0/QP7qFom+rVVejjybOe1kIKuNuG3zJeEHbrhqzpc61EtEI3R 8NhOoDYtvZZxj8w/RiFrZ/ikvyF09/xf730k9llZJeOBue69whVelu5uQFrflm0xjQvL tSI8C4ehq7XhZk1loNLYuoKBEp5RqHTFPfl38= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version:content-type; b=fJVyN1alv9iAqV0EEqWW1eEm2K2h0G8VUn8ZMdcPUbP3MNqsTC/NlbjMl52BoOFn3T FQepEs/Edy4TqkVbNQtQo6nAmehJNyoK0EmoLKk+7f8DfJ3yL8bx/AWnOWfUhLknqQ9m fASNQRoS8WAN1a7WdqAk7g8ptGvudqfY+xgMo= Received: by 10.216.29.66 with SMTP id h44mr774280wea.136.1236959565668; Fri, 13 Mar 2009 08:52:45 -0700 (PDT) Received: from localhost (95-24-212-58.broadband.corbina.ru [95.24.212.58]) by mx.google.com with ESMTPS id x6sm4011102gvf.24.2009.03.13.08.52.41 (version=TLSv1/SSLv3 cipher=RC4-MD5); Fri, 13 Mar 2009 08:52:43 -0700 (PDT) From: Anonymous To: John Baldwin References: <20090312175345.Y80227@rust.salford.ac.uk> <20090312191333.GA97342@hyperion.scode.org> <49B97617.8010709@freebsd.org> <86r6124f2v.fsf@gmail.com> <3bbf2fe10903122035i20b2767cod2322c39c6f850ee@mail.gmail.com> <29C8FA04-D5B1-49B7-ACF0-4185537367B0@baldwin.cx> Date: Fri, 13 Mar 2009 18:52:38 +0300 In-Reply-To: <29C8FA04-D5B1-49B7-ACF0-4185537367B0@baldwin.cx> (John Baldwin's message of "Thu, 12 Mar 2009 23:57:19 -0400") Message-ID: <86iqmdv2rd.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/23.0.91 (berkeley-unix) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Pawel Jakub Dawidek , "freebsd-current@freebsd.org" , Attilio Rao , Tim Kientzle , Mark Powell , Peter Schuller Subject: Re: repeatable ZFS panic: share->excl X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Mar 2009 15:52:48 -0000 John Baldwin writes: > On Mar 12, 2009, at 11:35 PM, Attilio Rao wrote: > >> 2009/3/12, Anonymous : >>> Tim Kientzle writes: >>> >>>> Peter Schuller wrote: >>>>>> First problem I notice is a panic, which seems to occur every >>>>>> time 'tar' is used. Might be unrelated to tar, but it definately >>>>>> provokes it. Simply the tar command or a pkg_add causes it. >>>>> >>>>> See the "ZFS/extattr lockup"/"bsdtar lockup on current" threads >>>>> from >>>>> the past handful of days. >>>> >>>> I think this may be a different problem. >>>> The earlier thread involved a ZFS bug that causes >>>> it to lock up if it receives a request to enumerate >>>> extended attributes on a file (via extattr_list_link >>>> system call). Tar recently added support for >>>> backing up extended attributes. I've disabled >>>> that support until this particular ZFS problem can >>>> be fixed. >>> >>> >>> I guess you're wrong, it's same issue. Here is output from unmodified >>> kernel (r189728) under qemu which looks exactly as on that >>> screenshot. Perhaps, the panic is triggered by INVARIANTS. >>> >>> # lsextattr -h user foo >>> shared lock of (lockmgr) zfs @ /usr/src/sys/kern/vfs_lookup.c:477 >>> while exclusively locked from /usr/src/sys/modules/zfs/../../cddl/ >>> contrib/opensolaris/uts/common/fs/zfs/zfs_znode.c:152 >>> panic: share->excl >>> cpuid = 0 >>> KDB: enter: panic >>> [thread pid 105 tid 100078 ] >>> Stopped at kdb_enter+0x3d: movq $0,0x662538(%rip) >>> >>> db> bt >>> Tracing pid 105 tid 100078 td 0xffffff00015bea80 >>> kdb_enter() at kdb_enter+0x3d >>> panic() at panic+0x17b >>> witness_checkorder() at witness_checkorder+0x16e >>> __lockmgr_args() at __lockmgr_args+0xd1b >>> vop_stdlock() at vop_stdlock+0x39 >>> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b >>> _vn_lock() at _vn_lock+0x57 >>> lookup() at lookup+0xf4 >>> namei() at namei+0x545 >>> zfs_listextattr() at zfs_listextattr+0x18c >>> VOP_LISTEXTATTR_APV() at VOP_LISTEXTATTR_APV+0xb5 >>> extattr_list_vp() at extattr_list_vp+0x22a >>> extattr_list_link() at extattr_list_link+0xc3 >>> syscall() at syscall+0x1e7 >>> Xfast_syscall() at Xfast_syscall+0xab >>> --- syscall (439, FreeBSD ELF64, extattr_list_link), rip = >>> 0x800692e4c, rsp = 0x7fffffffed08, rbp = 0x7fffffffede0 --- >>> >>> db> show all locks >>> Process 105 (lsextattr) thread 0xffffff00015bea80 (100078) >>> exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581578) locked @ / >>> usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/ >>> fs/zfs/zfs_znode.c:152 >>> exclusive lockmgr zfs (zfs) r = 0 (0xffffff0001581a58) locked @ / >>> usr/src/sys/kern/vfs_extattr.c:668 >>> >>> db> show lockedvnods >>> Locked vnodes >>> >>> 0xffffff00015819c0: tag zfs, type VREG >>> usecount 1, writecount 0, refcount 1 mountedhere 0 >>> flags () >>> v_object 0xffffff0001574898 ref 0 pages 0 >>> lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105) >>> #0 0xffffffff80530568 at __lockmgr_args+0x758 >>> #1 0xffffffff805c0fd9 at vop_stdlock+0x39 >>> #2 0xffffffff808557db at VOP_LOCK1_APV+0x9b >>> #3 0xffffffff805dd6a7 at _vn_lock+0x57 >>> #4 0xffffffff805c2f20 at extattr_list_vp+0xb0 >>> #5 0xffffffff805c3173 at extattr_list_link+0xc3 >>> #6 0xffffffff8080f227 at syscall+0x1e7 >>> #7 0xffffffff807ec39b at Xfast_syscall+0xab >>> >>> 0xffffff00015814e0: tag zfs, type VDIR >>> usecount 1, writecount 0, refcount 1 mountedhere 0 >>> flags () >>> lock type zfs: EXCL by thread 0xffffff00015bea80 (pid 105) >>> #0 0xffffffff80530568 at __lockmgr_args+0x758 >>> #1 0xffffffff805c0fd9 at vop_stdlock+0x39 >>> #2 0xffffffff808557db at VOP_LOCK1_APV+0x9b >>> #3 0xffffffff805dd6a7 at _vn_lock+0x57 >>> #4 0xffffffff8108405e at zfs_znode_cache_constructor+0x4e >>> #5 0xffffffff81085029 at zfs_znode_alloc+0x39 >>> #6 0xffffffff81085915 at zfs_mknode+0x205 >>> #7 0xffffffff810948f5 at zfs_make_xattrdir+0x155 >>> #8 0xffffffff81095bc3 at zfs_get_xattrdir+0xd3 >>> #9 0xffffffff810a4dbf at zfs_lookup+0x11f >>> #10 0xffffffff810a51d8 at zfs_listextattr+0x128 >>> #11 0xffffffff80853255 at VOP_LISTEXTATTR_APV+0xb5 >>> #12 0xffffffff805c309a at extattr_list_vp+0x22a >>> #13 0xffffffff805c3173 at extattr_list_link+0xc3 >>> #14 0xffffffff8080f227 at syscall+0x1e7 >>> #15 0xffffffff807ec39b at Xfast_syscall+0xab >> >> Could you please re-enable the extended attributes on bsdtar, try this >> patch and report to me?: >> http://www.freebsd.org/~attilio/zfs_vnops.diff Before that I'd like lsextattr(8) to at least not panic.. Because I get same panic with tar(1). Here is output from r189756 + your patch # lsextattr -h user foo vrele: negative ref count 0xffffff0001501270: tag zfs, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) VI_LOCKed lock type zfs: UNLOCKED #0 0xffffffff80531d8a at __lockmgr_args+0x51a #1 0xffffffff805c2909 at vop_stdlock+0x39 #2 0xffffffff808571eb at VOP_LOCK1_APV+0x9b #3 0xffffffff805defd7 at _vn_lock+0x57 #4 0xffffffff805c6e94 at lookup+0xf4 #5 0xffffffff805c81e5 at namei+0x545 #6 0xffffffff810a5d6d at zfs_listextattr+0x1fd #7 0xffffffff80854c65 at VOP_LISTEXTATTR_APV+0xb5 #8 0xffffffff805c49ca at extattr_list_vp+0x22a #9 0xffffffff805c4aa3 at extattr_list_link+0xc3 #10 0xffffffff80810c37 at syscall+0x1e7 #11 0xffffffff807eddab at Xfast_syscall+0xab panic: vrele: negative ref cnt cpuid = 0 KDB: enter: panic [thread pid 84 tid 100077 ] Stopped at kdb_enter+0x3d: movq $0,0x667d88(%rip) db> bt Tracing pid 84 tid 100077 td 0xffffff0001572000 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b vrele() at vrele+0x11a zfs_listextattr() at zfs_listextattr+0x464 VOP_LISTEXTATTR_APV() at VOP_LISTEXTATTR_APV+0xb5 extattr_list_vp() at extattr_list_vp+0x22a extattr_list_link() at extattr_list_link+0xc3 syscall() at syscall+0x1e7 Xfast_syscall() at Xfast_syscall+0xab --- syscall (439, FreeBSD ELF64, extattr_list_link), rip = 0x800692e9c, rsp = 0x7fffffffed08, rbp = 0x7fffffffede0 --- db> show all locks Process 84 (lsextattr) thread 0xffffff0001572000 (100077) exclusive lockmgr zfs (zfs) r = 0 (0xffffff00015017e8) locked @ /usr/src/sys/kern/vfs_extattr.c:668 db> show lockedvnods Locked vnodes 0xffffff0001501750: tag zfs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () v_object 0xffffff00014bd578 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff0001572000 (pid 84) #0 0xffffffff80531fc8 at __lockmgr_args+0x758 #1 0xffffffff805c2909 at vop_stdlock+0x39 #2 0xffffffff808571eb at VOP_LOCK1_APV+0x9b #3 0xffffffff805defd7 at _vn_lock+0x57 #4 0xffffffff805c4850 at extattr_list_vp+0xb0 #5 0xffffffff805c4aa3 at extattr_list_link+0xc3 #6 0xffffffff80810c37 at syscall+0x1e7 #7 0xffffffff807eddab at Xfast_syscall+0xab >> >> (sorry if it is untested but I just don't have any ZFS machine). >> Actually, I think the problem is that zfs_lookup() doesn't get the >> correct handover once the extended attributes directory vnode is >> retrieved. >> >> Thanks, >> Attilio > > This is similar to the patch I've asked lulf@ to test except that it > is longer and I fix a bug where zfs_lookup() can leak a vnode lock if > the access check fails. :-) The last one I sent to lulf@ is at > www.FreeBSD.org/~jhb/patches/zfs_ea.patch . Unless I'm doing smth stupid the panic from your patch looks same as one from patch by attilio@. %%% @@ -1,6 +1,6 @@ # lsextattr -h user foo vrele: negative ref count -0xffffff0001501270: tag zfs, type VDIR +0xffffff00016f3000: tag zfs, type VDIR usecount 0, writecount 0, refcount 0 mountedhere 0 flags (VI_FREE) VI_LOCKed lock type zfs: UNLOCKED @@ -10,7 +10,7 @@ #3 0xffffffff805defd7 at _vn_lock+0x57 #4 0xffffffff805c6e94 at lookup+0xf4 #5 0xffffffff805c81e5 at namei+0x545 -#6 0xffffffff810a5d6d at zfs_listextattr+0x1fd +#6 0xffffffff810a527c at zfs_listextattr+0x18c #7 0xffffffff80854c65 at VOP_LISTEXTATTR_APV+0xb5 #8 0xffffffff805c49ca at extattr_list_vp+0x22a #9 0xffffffff805c4aa3 at extattr_list_link+0xc3 @@ -19,14 +19,14 @@ panic: vrele: negative ref cnt cpuid = 0 KDB: enter: panic -[thread pid 84 tid 100077 ] +[thread pid 84 tid 100075 ] Stopped at kdb_enter+0x3d: movq $0,0x667d88(%rip) db> bt -Tracing pid 84 tid 100077 td 0xffffff0001572000 +Tracing pid 84 tid 100075 td 0xffffff0001578700 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b vrele() at vrele+0x11a -zfs_listextattr() at zfs_listextattr+0x464 +zfs_listextattr() at zfs_listextattr+0x3f4 VOP_LISTEXTATTR_APV() at VOP_LISTEXTATTR_APV+0xb5 extattr_list_vp() at extattr_list_vp+0x22a extattr_list_link() at extattr_list_link+0xc3 @@ -34,16 +34,16 @@ Xfast_syscall() at Xfast_syscall+0xab --- syscall (439, FreeBSD ELF64, extattr_list_link), rip = 0x800692e9c, rsp = 0x7fffffffed08, rbp = 0x7fffffffede0 --- db> show all locks -Process 84 (lsextattr) thread 0xffffff0001572000 (100077) -exclusive lockmgr zfs (zfs) r = 0 (0xffffff00015017e8) locked @ /usr/src/sys/kern/vfs_extattr.c:668 +Process 84 (lsextattr) thread 0xffffff0001578700 (100075) +exclusive lockmgr zfs (zfs) r = 0 (0xffffff00016f3578) locked @ /usr/src/sys/kern/vfs_extattr.c:668 db> show lockedvnods Locked vnodes -0xffffff0001501750: tag zfs, type VREG +0xffffff00016f34e0: tag zfs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () - v_object 0xffffff00014bd578 ref 0 pages 0 - lock type zfs: EXCL by thread 0xffffff0001572000 (pid 84) + v_object 0xffffff0001563640 ref 0 pages 0 + lock type zfs: EXCL by thread 0xffffff0001578700 (pid 84) #0 0xffffffff80531fc8 at __lockmgr_args+0x758 #1 0xffffffff805c2909 at vop_stdlock+0x39 #2 0xffffffff808571eb at VOP_LOCK1_APV+0x9b %%% > > -- > John Baldwin