Date: Wed, 9 Sep 2009 15:40:57 -0400 From: John Baldwin <jhb@freebsd.org> To: pluknet <pluknet@gmail.com> Cc: Attilio Rao <attilio@freebsd.org>, freebsd-emulation@freebsd.org, freebsd-current@freebsd.org Subject: Re: acquiring duplicate lock of same type: "ftlk" Message-ID: <200909091540.57890.jhb@freebsd.org> In-Reply-To: <a31046fc0909091225n2fe7e921m93071937004f08b8@mail.gmail.com> References: <a31046fc0908270158l63b103a7v6c9fd1b7be54d3ed@mail.gmail.com> <200909091455.42039.jhb@freebsd.org> <a31046fc0909091225n2fe7e921m93071937004f08b8@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wednesday 09 September 2009 3:25:50 pm pluknet wrote:
> 2009/9/9 John Baldwin <jhb@freebsd.org>:
> > On Tuesday 08 September 2009 3:42:13 pm pluknet wrote:
> >> 2009/9/8 Attilio Rao <attilio@freebsd.org>:
> >> > 2009/9/8 Kostik Belousov <kostikbel@gmail.com>:
> >> >> On Mon, Sep 07, 2009 at 10:15:48PM +0400, pluknet wrote:
> >> >>> lock order reversal:
> >> >>> =A01st 0xc75365b8 pseudofs (pseudofs) @ /usr/src/sys/kern/vfs_look=
up.c:497
> >> >>> =A02nd 0xc088ea3c allproc (allproc) @ /usr/src/sys/kern/kern_proc.=
c:292
> >> >>> KDB: stack backtrace:
> >> >>> db_trace_self_wrapper(c07fd8ea,e82148e4,c060a145,c05fac1b,c08008bf=
,...)
> >> >>> at db_trace_self_wrapper+0x26
> >> >>> kdb_backtrace(c05fac1b,c08008bf,c58eabe8,c58e30d0,e8214940,...) at
> >> >>> kdb_backtrace+0x29
> >> >>> _witness_debugger(c08008bf,c088ea3c,c07f981d,c58e30d0,c07f96f0,...=
) at
> >> >>> _witness_debugger+0x25
> >> >>> witness_checkorder(c088ea3c,1,c07f96f0,124,0,...) at
> > witness_checkorder+0x839
> >> >>> _sx_slock(c088ea3c,0,c07f96f0,124,c73c4980,...) at _sx_slock+0x85
> >> >>> pfind(514,c72ba1a0,4,c07f8d78,c5fe1b40,...) at pfind+0x2f
> >> >>> pfs_visible(0,0,c07f042d,7c,c7536560,...) at pfs_visible+0x3a
> >> >>> pfs_lookup(e8214a40,c082715e,c7536560,c7536560,e8214bf8,...) at
> > pfs_lookup+0x3dd
> >> >>> VOP_CACHEDLOOKUP_APV(c0843960,e8214a40,e8214bf8,e8214be4,c73c4e80,=
=2E..)
> >> >>> at VOP_CACHEDLOOKUP_APV+0xc5
> >> >>> vfs_cache_lookup(e8214acc,c08087d0,c0875a00,200000,e8214bcc,...) at
> >> >>> vfs_cache_lookup+0xd6
> >> >>> VOP_LOOKUP_APV(c0843960,e8214acc,e8214bf8,1f1,e8214be4,...) at
> >> >>> VOP_LOOKUP_APV+0xe5
> >> >>> lookup(e8214bcc,c5fd1800,0,c5,c5ef77f8,...) at lookup+0x63b
> >> >>> namei(e8214bcc,c5c1500d,3f3,e8214c20,c5c1500d,...) at namei+0x57f
> >> >>> kern_alternate_path(c5fe1b40,c0b4921c,2879f478,0,e8214c74,...) at
> > kern_alternate
> >> >>> _path+0x1cd
> >> >>> linux_emul_convpath(c5fe1b40,2879f478,0,e8214c74,0,...) at
> >> >>> linux_emul_convpath+0x3c
> >> >>> linux_open(c5fe1b40,e8214cf8,e8214d18,e8214d1c,c0b4b58c,...) at
> > linux_open+0x41
> >> >>> syscall(e8214d38) at syscall+0x2b4
> >> >>> Xint0x80_syscall() at Xint0x80_syscall+0x20
> >> >>> --- syscall (5, Linux ELF, linux_open), eip =3D 0x2889115e, esp =3D
> >> >>> 0xbfbfbd1c, ebp =3D 0xbfbfbd6c ---
> >> >>> acquiring duplicate lock of same type: "ftlk"
> >> >>> [...]
> >> >>
> >> >> The second LOR actually exposes the right order. It would be intere=
sting
> >> >> to look up the point where the other order is established.
> >> >
> >> > You would manually patch the witness static table with this order and
> >> > the opposite will show, when happening.
> >> >
> >>
> >> I've patched witness order table, and still no opposite case,
> >> nor any pseudofs related LORs at all.
> >> =A0 =A0 =A0 =A0 { "pseudofs", &lock_class_lockmgr },
> >> =A0 =A0 =A0 =A0 { "allproc", &lock_class_sx },
> >> =A0 =A0 =A0 =A0 { NULL, NULL },
> >>
> >> Seen orders with pseudofs:
> >> "ufs","pseudofs"
> >> "pseudofs","allproc"
> >> "pseudofs","process lock"
> >> "pseudofs","vnode interlock"
> >> "pseudofs","struct mount mtx"
> >> "pseudofs","UMA zone"
> >> "pseudofs","sleep mtxpool"
> >> "pseudofs","Name Cache"
> >> "pseudofs","vnode_free_list"
> >> "pseudofs","pfs_node"
> >> "pseudofs","pfs_vncache"
> >>
> >> Something else?
> >
> > What are the seen orders with "allproc"?
> >
>=20
> sysctl debug.witness.fullgraph | grep allproc
> "proctree","allproc"
> "allproc","allprison"
> "allproc","process lock"
> "allproc","user map"
> "allproc","fdesc"
> "allproc","filedesc structure"
> "sysctl lock","allproc"
> "pseudofs","allproc"
Ok, there is probably an order of "allproc" -> "filedesc" -> "ufs" ->
"psuedofs". There might even be a "filedesc" -> "psuedofs" that could
cause this.
> Also, if it matters, I saw this LOR today:
>=20
> lock order reversal:
> 1st 0xc7f01164 ufs (ufs) @ /usr/RELENG_8/src/sys/kern/vfs_mount.c:1054
> 2nd 0xc7fb5058 pseudofs (pseudofs) @
> /usr/RELENG_8/src/sys/fs/pseudofs/pseudofs_vncache.c:193
I think this backs up my theory. I think the LOR is probably harmless,
but there's not an easy way to shut it up I believe.
=2D-=20
John Baldwin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200909091540.57890.jhb>