From owner-freebsd-current@FreeBSD.ORG Wed Sep 9 19:41:04 2009 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C86961065695; Wed, 9 Sep 2009 19:41:04 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 96A5D8FC17; Wed, 9 Sep 2009 19:41:04 +0000 (UTC) Received: from bigwig.baldwin.cx (66.111.2.69.static.nyinternet.net [66.111.2.69]) by cyrus.watson.org (Postfix) with ESMTPSA id 45CFE46B03; Wed, 9 Sep 2009 15:41:04 -0400 (EDT) Received: from jhbbsd.hudson-trading.com (unknown [209.249.190.8]) by bigwig.baldwin.cx (Postfix) with ESMTPA id 62B3B8A01B; Wed, 9 Sep 2009 15:41:03 -0400 (EDT) From: John Baldwin To: pluknet Date: Wed, 9 Sep 2009 15:40:57 -0400 User-Agent: KMail/1.9.7 References: <200909091455.42039.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200909091540.57890.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (bigwig.baldwin.cx); Wed, 09 Sep 2009 15:41:03 -0400 (EDT) X-Virus-Scanned: clamav-milter 0.95.1 at bigwig.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.5 required=4.2 tests=AWL,BAYES_00,RDNS_NONE autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on bigwig.baldwin.cx Cc: Attilio Rao , Kostik Belousov , freebsd-emulation@freebsd.org, freebsd-current@freebsd.org Subject: Re: acquiring duplicate lock of same type: "ftlk" X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Sep 2009 19:41:04 -0000 On Wednesday 09 September 2009 3:25:50 pm pluknet wrote: > 2009/9/9 John Baldwin : > > On Tuesday 08 September 2009 3:42:13 pm pluknet wrote: > >> 2009/9/8 Attilio Rao : > >> > 2009/9/8 Kostik Belousov : > >> >> On Mon, Sep 07, 2009 at 10:15:48PM +0400, pluknet wrote: > >> >>> lock order reversal: > >> >>> =A01st 0xc75365b8 pseudofs (pseudofs) @ /usr/src/sys/kern/vfs_look= up.c:497 > >> >>> =A02nd 0xc088ea3c allproc (allproc) @ /usr/src/sys/kern/kern_proc.= c:292 > >> >>> KDB: stack backtrace: > >> >>> db_trace_self_wrapper(c07fd8ea,e82148e4,c060a145,c05fac1b,c08008bf= ,...) > >> >>> at db_trace_self_wrapper+0x26 > >> >>> kdb_backtrace(c05fac1b,c08008bf,c58eabe8,c58e30d0,e8214940,...) at > >> >>> kdb_backtrace+0x29 > >> >>> _witness_debugger(c08008bf,c088ea3c,c07f981d,c58e30d0,c07f96f0,...= ) at > >> >>> _witness_debugger+0x25 > >> >>> witness_checkorder(c088ea3c,1,c07f96f0,124,0,...) at > > witness_checkorder+0x839 > >> >>> _sx_slock(c088ea3c,0,c07f96f0,124,c73c4980,...) at _sx_slock+0x85 > >> >>> pfind(514,c72ba1a0,4,c07f8d78,c5fe1b40,...) at pfind+0x2f > >> >>> pfs_visible(0,0,c07f042d,7c,c7536560,...) at pfs_visible+0x3a > >> >>> pfs_lookup(e8214a40,c082715e,c7536560,c7536560,e8214bf8,...) at > > pfs_lookup+0x3dd > >> >>> VOP_CACHEDLOOKUP_APV(c0843960,e8214a40,e8214bf8,e8214be4,c73c4e80,= =2E..) > >> >>> at VOP_CACHEDLOOKUP_APV+0xc5 > >> >>> vfs_cache_lookup(e8214acc,c08087d0,c0875a00,200000,e8214bcc,...) at > >> >>> vfs_cache_lookup+0xd6 > >> >>> VOP_LOOKUP_APV(c0843960,e8214acc,e8214bf8,1f1,e8214be4,...) at > >> >>> VOP_LOOKUP_APV+0xe5 > >> >>> lookup(e8214bcc,c5fd1800,0,c5,c5ef77f8,...) at lookup+0x63b > >> >>> namei(e8214bcc,c5c1500d,3f3,e8214c20,c5c1500d,...) at namei+0x57f > >> >>> kern_alternate_path(c5fe1b40,c0b4921c,2879f478,0,e8214c74,...) at > > kern_alternate > >> >>> _path+0x1cd > >> >>> linux_emul_convpath(c5fe1b40,2879f478,0,e8214c74,0,...) at > >> >>> linux_emul_convpath+0x3c > >> >>> linux_open(c5fe1b40,e8214cf8,e8214d18,e8214d1c,c0b4b58c,...) at > > linux_open+0x41 > >> >>> syscall(e8214d38) at syscall+0x2b4 > >> >>> Xint0x80_syscall() at Xint0x80_syscall+0x20 > >> >>> --- syscall (5, Linux ELF, linux_open), eip =3D 0x2889115e, esp =3D > >> >>> 0xbfbfbd1c, ebp =3D 0xbfbfbd6c --- > >> >>> acquiring duplicate lock of same type: "ftlk" > >> >>> [...] > >> >> > >> >> The second LOR actually exposes the right order. It would be intere= sting > >> >> to look up the point where the other order is established. > >> > > >> > You would manually patch the witness static table with this order and > >> > the opposite will show, when happening. > >> > > >> > >> I've patched witness order table, and still no opposite case, > >> nor any pseudofs related LORs at all. > >> =A0 =A0 =A0 =A0 { "pseudofs", &lock_class_lockmgr }, > >> =A0 =A0 =A0 =A0 { "allproc", &lock_class_sx }, > >> =A0 =A0 =A0 =A0 { NULL, NULL }, > >> > >> Seen orders with pseudofs: > >> "ufs","pseudofs" > >> "pseudofs","allproc" > >> "pseudofs","process lock" > >> "pseudofs","vnode interlock" > >> "pseudofs","struct mount mtx" > >> "pseudofs","UMA zone" > >> "pseudofs","sleep mtxpool" > >> "pseudofs","Name Cache" > >> "pseudofs","vnode_free_list" > >> "pseudofs","pfs_node" > >> "pseudofs","pfs_vncache" > >> > >> Something else? > > > > What are the seen orders with "allproc"? > > >=20 > sysctl debug.witness.fullgraph | grep allproc > "proctree","allproc" > "allproc","allprison" > "allproc","process lock" > "allproc","user map" > "allproc","fdesc" > "allproc","filedesc structure" > "sysctl lock","allproc" > "pseudofs","allproc" Ok, there is probably an order of "allproc" -> "filedesc" -> "ufs" -> "psuedofs". There might even be a "filedesc" -> "psuedofs" that could cause this. > Also, if it matters, I saw this LOR today: >=20 > lock order reversal: > 1st 0xc7f01164 ufs (ufs) @ /usr/RELENG_8/src/sys/kern/vfs_mount.c:1054 > 2nd 0xc7fb5058 pseudofs (pseudofs) @ > /usr/RELENG_8/src/sys/fs/pseudofs/pseudofs_vncache.c:193 I think this backs up my theory. I think the LOR is probably harmless, but there's not an easy way to shut it up I believe. =2D-=20 John Baldwin