Date: Wed, 18 Aug 2004 13:45:24 -0700 From: Kris Kennaway <kris@obsecurity.org> To: Kris Kennaway <kris@obsecurity.org> Cc: tjr@FreeBSD.org Subject: Re: vnode deadlock with nullfs? Message-ID: <20040818204524.GA47843@xor.obsecurity.org> In-Reply-To: <20040818185637.GA41627@xor.obsecurity.org> References: <20040818185637.GA41627@xor.obsecurity.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--0F1p//8PRICkK4MW
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Wed, Aug 18, 2004 at 11:56:37AM -0700, Kris Kennaway wrote:
> I turned on debug.mpsafevm on a SMP package build machine, and it
> didn't last long before grinding to a halt:
>=20
> db> trace 10328
> sched_switch(c5b32c60,0,15d,c072db39,c5b32c60) at sched_switch+0xec
> mi_switch(1,0,c072fd06,198,c053c3c3) at mi_switch+0x2e4
> sleepq_switch(c5abbd0c,1,c5b32c60,c5abbd0c,f121e928) at sleepq_switch+0x1=
35
> sleepq_wait(c5abbd0c,0,c072da37,e7,0) at sleepq_wait+0x11
> msleep(c5abbd0c,c07a0814,50,c0733c07,0) at msleep+0x4a7
> acquire(f121e9b0,1000040,200,e7,c5abbd0c) at acquire+0xf7
> lockmgr(c5abbd0c,1010002,c5abbc60,c5b32c60) at lockmgr+0x45e
> ufs_lock(f121ea00,f121ea1c,c05b5233,f121ea00,c067e47a) at ufs_lock+0x4a
> ufs_vnoperate(f121ea00,c067e47a,c5b1a2bc,1010002,c5b1a210) at ufs_vnopera=
te+0x18
> vn_lock(c5abbc60,10002,c5b32c60,0,2) at vn_lock+0x11a
> vget(c5abbc60,10002,c5b32c60,7d,c5b32c60) at vget+0x102
> ufs_ihashget(c59f0e00,2,2,f121eb00,1000000) at ufs_ihashget+0xd8
> ffs_vget(c585f800,2,2,f121eb00,0) at ffs_vget+0x44
> ufs_root(c585f800,f121eb54,c5b32c60,c071eb1f,c5b1a210) at ufs_root+0x29
> lookup(f121ec10,0,c07353b7,a4,1) at lookup+0x60c
> namei(f121ec10,0,d8,f121ec44,c053c100) at namei+0x2a1
> stat(c5b32c60,f121ed14,c0748ea3,3e2,c5b32c60) at stat+0x52
> syscall(2f,2f,2f,8083e12,8083e12) at syscall+0x2c8
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (188, FreeBSD ELF32, stat), eip =3D 0x8067157, esp =3D 0xbfbf=
dfbc, ebp =3D 0xbfbfecc0 ---
> db> show lockedvnods
> Locked vnodes
> 0xc5abbc60: tag ufs, type VDIR, usecount 16, writecount 0, refcount 1, fl=
ags (VV_ROOT|VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc5a7c420=
(pid 10304) with 15 pending
> ino 2, on dev da0s1e (4, 17)
> 0xc5d3a318: tag ufs, type VDIR, usecount 2, writecount 0, refcount 2, fla=
gs (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc5a1dc60 (pid 103=
10) with 1 pending
> ino 376832, on dev da0s1e (4, 17)
> 0xc5d4a420: tag ufs, type VDIR, usecount 3, writecount 0, refcount 2, fla=
gs (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc578d840 (pid 103=
09) with 1 pending
> ino 376833, on dev da0s1e (4, 17)
> 0xc5f24738: tag ufs, type VDIR, usecount 4, writecount 0, refcount 2, fla=
gs (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc5a7b9a0 (pid 103=
01) with 1 pending
> ino 377018, on dev da0s1e (4, 17)
> 0xc5fb5840: tag ufs, type VDIR, usecount 6, writecount 0, refcount 2, fla=
gs (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc5d4eb00 (pid 983=
1) with 1 pending
> ino 400386, on dev da0s1e (4, 17)
> 0xc6263000: tag ufs, type VDIR, usecount 5, writecount 0, refcount 2, fla=
gs (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc578db00 (pid 980=
1) with 2 pending
> ino 424617, on dev da0s1e (4, 17)
> 0xc9465a50: tag ufs, type VREG, usecount 3, writecount 0, refcount 0, loc=
k type ufs: EXCL (count 1) by thread 0xc5b319a0 (pid 9802) with 1 pending
> ino 425034, on dev da0s1e (4, 17)
> 0xc8fcb738: tag null, type VDIR, usecount 1, writecount 0, refcount 0, fl=
ags (VV_ROOT|VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc5d4eb00=
(pid 9831) with 1 pending
> vp=3D0xc8fcb738, lowervp=3D0xc5fb5840
> 0xc8bee318: tag null, type VDIR, usecount 0, writecount 0, refcount 0, lo=
ck type ufs: EXCL (count 1) by thread 0xc578db00 (pid 9801) with 2 pending
> vp=3D0xc8bee318, lowervp=3D0xc6263000
> 0xc8b09318: tag null, type VREG, usecount 1, writecount 0, refcount 0, lo=
ck type ufs: EXCL (count 1) by thread 0xc5b319a0 (pid 9802) with 1 pending
> vp=3D0xc8b09318, lowervp=3D0xc9465a50
> 0xc90ebc60: tag null, type VDIR, usecount 1, writecount 0, refcount 0, fl=
ags (VV_ROOT|VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xc5d4eb00=
(pid 9831) with 1 pending
> vp=3D0xc90ebc60, lowervp=3D0xc5fb5840
> 0xc8925e70: tag null, type VDIR, usecount 1, writecount 0, refcount 0, lo=
ck type ufs: EXCL (count 1) by thread 0xc578db00 (pid 9801) with 2 pending
> vp=3D0xc8925e70, lowervp=3D0xc6263000
> db>
>=20
> I'm going to try and repeat this with DEBUG_VFS_LOCKS and WITNESS.
They didn't give any debugging messages when it happened again:
db> show lockedvnods
Locked vnodes
0xc5d5c318: tag ufs, type VDIR, usecount 27, writecount 0, refcount 2, flag=
s (VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246=
) with 10 pending
ino 400386, on dev da0s1e (4, 17)
0xc657fb58: tag ufs, type VDIR, usecount 5, writecount 0, refcount 1, flags=
(VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xcace6dc0 (pid 4247)=
with 2 pending
ino 424617, on dev da0s1e (4, 17)
0xc644be70: tag ufs, type VREG, usecount 3, writecount 0, refcount 1, flags=
(VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xcad44160 (pid 4255)=
with 1 pending
ino 425034, on dev da0s1e (4, 17)
0xc6cdb528: tag null, type VDIR, usecount 1, writecount 0, refcount 0, flag=
s (VV_ROOT|VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xcace5840 (=
pid 4246) with 10 pending
vp=3D0xc6cdb528, lowervp=3D0xc5d5c318
0xc75ada50: tag null, type VDIR, usecount 0, writecount 0, refcount 0, lock=
type ufs: EXCL (count 1) by thread 0xcace6dc0 (pid 4247) with 2 pending
vp=3D0xc75ada50, lowervp=3D0xc657fb58
0xc5de5948: tag null, type VREG, usecount 1, writecount 0, refcount 0, lock=
type ufs: EXCL (count 1) by thread 0xcad44160 (pid 4255) with 1 pending
vp=3D0xc5de5948, lowervp=3D0xc644be70
0xc5f05318: tag null, type VDIR, usecount 1, writecount 0, refcount 0, flag=
s (VV_ROOT|VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xcace5840 (=
pid 4246) with 10 pending
vp=3D0xc5f05318, lowervp=3D0xc5d5c318
0xc6c3c210: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT|VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xcace5840 (=
pid 4246) with 10 pending
vp=3D0xc6c3c210, lowervp=3D0xc5d5c318
0xc9774630: tag null, type VDIR, usecount 1, writecount 0, refcount 0, flag=
s (VV_ROOT|VV_OBJBUF), lock type ufs: EXCL (count 1) by thread 0xcace5840 (=
pid 4246) with 10 pending
vp=3D0xc9774630, lowervp=3D0xc5d5c318
0xc759f738: tag null, type VDIR, usecount 1, writecount 0, refcount 0, lock=
type ufs: EXCL (count 1) by thread 0xcace6dc0 (pid 4247) with 2 pending
vp=3D0xc759f738, lowervp=3D0xc657fb58
0xc6121738: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc6121738, lowervp=3D0xc5d5c318
0xc6d68528: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc6d68528, lowervp=3D0xc5d5c318
0xc6d05d68: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc6d05d68, lowervp=3D0xc5d5c318
0xc6cf9528: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc6cf9528, lowervp=3D0xc5d5c318
0xc7c32e70: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc7c32e70, lowervp=3D0xc5d5c318
0xc7c64000: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc7c64000, lowervp=3D0xc5d5c318
0xc78b2a50: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc78b2a50, lowervp=3D0xc5d5c318
0xc780d210: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc780d210, lowervp=3D0xc5d5c318
0xc780b318: tag null, type VDIR, usecount 2, writecount 0, refcount 0, flag=
s (VV_ROOT), lock type ufs: EXCL (count 1) by thread 0xcace5840 (pid 4246) =
with 10 pending
vp=3D0xc780b318, lowervp=3D0xc5d5c318
db> show locks
db>
--0F1p//8PRICkK4MW
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (FreeBSD)
iD8DBQFBI7/kWry0BWjoQKURAswaAJ0Typs1v6+jLMjHQpVchH8lwPXdigCaAu9J
SmJCDyvAb+t8SqMswcJVRmk=
=Nb55
-----END PGP SIGNATURE-----
--0F1p//8PRICkK4MW--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20040818204524.GA47843>