Date: Mon, 29 May 2006 11:29:52 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: V??clav Haisman <V.Haisman@sh.cvut.cz> Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: LOR in vnode interlock and system map Message-ID: <20060529082951.GU54541@deviant.kiev.zoral.com.ua> In-Reply-To: <447A1F94.7020809@sh.cvut.cz> References: <447A1F94.7020809@sh.cvut.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
--7gQyIpR7q4QSXYu+
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Mon, May 29, 2006 at 12:09:24AM +0200, V??clav Haisman wrote:
> See attached file. The kernel is todays CVS 6.1.
>=20
> --
> Vaclav Haisman
>=20
>=20
> May 28 19:16:55 logout kernel: lock order reversal:
> May 28 19:16:55 logout kernel: 1st 0xc3d4bb1c vnode interlock (vnode inte=
rlock) @ /usr/src/sys/kern/vfs_subr.c:2218
> May 28 19:16:55 logout kernel: 2nd 0xc1043144 system map (system map) @ /=
usr/src/sys/vm/vm_kern.c:295
> May 28 19:16:55 logout kernel: KDB: stack backtrace:
> May 28 19:16:55 logout kernel: kdb_backtrace(c081a0c7,c1043144,c0819bfa,c=
0819bfa,c082f5af) at kdb_backtrace+0x2e
> May 28 19:16:55 logout kernel: witness_checkorder(c1043144,9,c082f5af,127=
,c08bdbc0) at witness_checkorder+0x5ef
> May 28 19:16:55 logout kernel: _mtx_lock_flags(c1043144,0,c082f5af,127,c1=
04e460) at _mtx_lock_flags+0x32
> May 28 19:16:55 logout kernel: _vm_map_lock(c10430c0,c082f5af,127,c337590=
0,c3375900) at _vm_map_lock+0x37
> May 28 19:16:55 logout kernel: kmem_malloc(c10430c0,1000,101,d5688afc,c07=
66493) at kmem_malloc+0x3a
> May 28 19:16:55 logout kernel: page_alloc(c104d300,1000,d5688aef,101,c05e=
0ed6) at page_alloc+0x27
> May 28 19:16:55 logout kernel: slab_zalloc(c104d300,101,c082ed67,8a2,8024=
6) at slab_zalloc+0xd3
> May 28 19:16:55 logout kernel: uma_zone_slab(c104d300,1,c082ed67,8a2,c082=
ed67) at uma_zone_slab+0x112
> May 28 19:16:55 logout kernel: uma_zalloc_internal(c104d300,0,1,0,d5688b9=
4) at uma_zalloc_internal+0x41
> May 28 19:16:55 logout kernel: bucket_alloc(80,1,c082ed67,95e,105) at buc=
ket_alloc+0x3b
> May 28 19:16:55 logout kernel: uma_zfree_arg(c104dc00,c389712c,0,d5688bbc=
,c072e964) at uma_zfree_arg+0x253
> May 28 19:16:55 logout kernel: mac_labelzone_free(c389712c) at mac_labelz=
one_free+0x22
> May 28 19:16:55 logout kernel: mac_vnode_label_free(c389712c,c3d4baa0,d56=
88be8,c061aa00,c3d4baa0) at mac_vnode_label_free+0x94
> May 28 19:16:55 logout kernel: mac_destroy_vnode(c3d4baa0,0,c081e172,30a,=
c3d4baa0) at mac_destroy_vnode+0x18
> May 28 19:16:55 logout kernel: vdestroy(c3d4baa0,d5688c1c,0,c3d4e9cc,c3d4=
e9cc) at vdestroy+0x60
> May 28 19:16:55 logout kernel: vdropl(c3d4baa0,d5688c1c,c081e172,835,c08b=
1440) at vdropl+0x50
> May 28 19:16:55 logout kernel: vput(c3d4baa0,0,c082c8a2,df7,c09117a0) at =
vput+0x1ae
> May 28 19:16:55 logout kernel: handle_workitem_remove(c36de9e0,0,2,3b3,1)=
at handle_workitem_remove+0x15a
> May 28 19:16:55 logout kernel: process_worklist_item(c35a0400,0,c082c8a2,=
348,4479db07) at process_worklist_item+0x1e1
> May 28 19:16:55 logout kernel: softdep_process_worklist(c35a0400,0,c082c8=
a2,2f2,3e8) at softdep_process_worklist+0x73
> May 28 19:16:55 logout kernel: softdep_flush(0,d5688d38,c08150d4,31d,0) a=
t softdep_flush+0x193
> May 28 19:16:55 logout kernel: fork_exit(c0744170,0,d5688d38) at fork_exi=
t+0x78
> May 28 19:16:55 logout kernel: fork_trampoline() at fork_trampoline+0x8
> May 28 19:16:55 logout kernel: --- trap 0x1, eip =3D 0, esp =3D 0xd5688d6=
c, ebp =3D 0 ---
I already reported similar LOR to Jeff Roberson, when latest big pile of
VFS fixes where to be MFCed. Try the following patch (note, to trigger the
situation you need really high load, and, desirably, low amount
of memory installed).
My backtrace was almost identical, the difference is that your trace
coming from softdepflush daemon, and mine comes from vnlru.
lock order reversal:
1st 0xc1a018f0 vnode interlock (vnode interlock) @ /usr/home/kostik/work/b=
=3D
sd/sys/kern/vfs_subr.c:2449
2nd 0xc0c43144 system map (system map) @ /usr/home/kostik/work/bsd/sys/vm/=
=3D
vm_kern.c:295
KDB: stack backtrace:
kdb_backtrace(0,ffffffff,c06676b0,c0667700,c0636024) at 0xc049d3c9 =3D3D kd=
b_=3D
backtrace+0x29
witness_checkorder(c0c43144,9,c061fe28,127) at 0xc04a80c2 =3D3D witness_che=
ck=3D
order+0x582
_mtx_lock_flags(c0c43144,0,c061fe28,127) at 0xc047b998 =3D3D _mtx_lock_flag=
s+=3D
0x58
_vm_map_lock(c0c430c0,c061fe28,127) at 0xc059eb46 =3D3D _vm_map_lock+0x26
kmem_malloc(c0c430c0,1000,101,c819fbe0,c059679f) at 0xc059e0d2 =3D3D kmem_m=
al=3D
loc+0x32
page_alloc(c0c4d300,1000,c819fbd3,101,c06a3bf8) at 0xc0596bda =3D3D page_al=
lo=3D
c+0x1a
slab_zalloc(c0c4d300,101,c0c4d300,c0647a64,c0c4e460) at 0xc059679f =3D3D sl=
ab=3D
_zalloc+0x9f
uma_zone_slab(c0c4d300,1,c0c4e468,0,c061f05a,8a2) at 0xc0597dec =3D3D uma_z=
on=3D
e_slab+0xec
uma_zalloc_internal(c0c4d300,0,1,0,c0c4dc48) at 0xc0598129 =3D3D uma_zalloc=
_i=3D
nternal+0x29
bucket_alloc(80,1,c0c380a0,0,c19ab6a4) at 0xc0595eac =3D3D bucket_alloc+0x2c
uma_zfree_arg(c0c4dc00,c19ab6a4,0) at 0xc0598483 =3D3D uma_zfree_arg+0x283
mac_labelzone_free(c19ab6a4,c1a01828,e8,c819fc9c,c0565ad2) at 0xc055dab3 =3D
=3D3D mac_labelzone_free+0x13
mac_vnode_label_free(c19ab6a4,c1a01828,c819fcac,c04d8766,c1a01828) at 0xc05=
=3D
65aaa =3D3D mac_vnode_label_free+0x6a
mac_destroy_vnode(c1a01828) at 0xc0565ad2 =3D3D mac_destroy_vnode+0x12
vdestroy(c1a01828,c1a01828,c819fcec,c04d8142,c1a01828) at 0xc04d8766 =3D3D =
vd=3D
estroy+0x1c6
vdropl(c1a01828,7,a8,c0653ee0,c1a01828) at 0xc04dad1e =3D3D vdropl+0x3e
vlrureclaim(c15e8000,c1529000,c156f000,c04d8360,c156f000) at 0xc04d8142 =3D=
3D=3D
vlrureclaim+0x282
vnlru_proc(0,c819fd38,0,c04d8360,0) at 0xc04d84e3 =3D3D vnlru_proc+0x183
fork_exit(c04d8360,0,c819fd38) at 0xc046de7d =3D3D fork_exit+0x9d
fork_trampoline() at 0xc05d33bc =3D3D fork_trampoline+0x8
--- trap 0x1, eip =3D3D 0, esp =3D3D 0xc819fd6c, ebp =3D3D 0 ---
A patch to fix the LOR (seems to be relevant for CURRENT too, added -current
to CC:):
--- sys/kern/vfs_subr.c.orig Sat Mar 4 10:44:47 2006
+++ sys/kern/vfs_subr.c Sat Mar 4 10:45:21 2006
@@ -787,9 +787,6 @@
VNASSERT(bo->bo_dirty.bv_root =3D3D=3D3D NULL, vp, ("dirtyblkroot not NUL=
L"));
VNASSERT(TAILQ_EMPTY(&vp->v_cache_dst), vp, ("vp has namecache dst"));
VNASSERT(LIST_EMPTY(&vp->v_cache_src), vp, ("vp has namecache src"));
-#ifdef MAC
- mac_destroy_vnode(vp);
-#endif
if (vp->v_pollinfo !=3D3D NULL) {
knlist_destroy(&vp->v_pollinfo->vpi_selinfo.si_note);
mtx_destroy(&vp->v_pollinfo->vpi_lock);
@@ -801,6 +798,9 @@
#endif
lockdestroy(vp->v_vnlock);
mtx_destroy(&vp->v_interlock);
+#ifdef MAC
+ mac_destroy_vnode(vp);
+#endif
uma_zfree(vnode_zone, vp);
}
Patch assumes that no mac_destroy_vnode does not mess with vnode internals,
as it is the case now.
--7gQyIpR7q4QSXYu+
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (FreeBSD)
iD8DBQFEerD/C3+MBN1Mb4gRAoJOAJ41O46TSH40giFk1ka10BOhV7dYCQCg8djD
0zMSlTB9fsvrq8s+d6wPy0s=
=QSZi
-----END PGP SIGNATURE-----
--7gQyIpR7q4QSXYu+--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060529082951.GU54541>