From owner-freebsd-stable@FreeBSD.ORG Mon May 29 08:30:02 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 86ED216A4CB; Mon, 29 May 2006 08:30:02 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from fw.zoral.com.ua (ll-227.216.82.212.sovam.net.ua [212.82.216.227]) by mx1.FreeBSD.org (Postfix) with ESMTP id 2513443D5A; Mon, 29 May 2006 08:30:00 +0000 (GMT) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by fw.zoral.com.ua (8.13.4/8.13.4) with ESMTP id k4T8TsbA089308 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 29 May 2006 11:29:54 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.13.6/8.13.6) with ESMTP id k4T8TsfF047612; Mon, 29 May 2006 11:29:54 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.13.6/8.13.6/Submit) id k4T8TqHR047611; Mon, 29 May 2006 11:29:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 29 May 2006 11:29:52 +0300 From: Konstantin Belousov To: V??clav Haisman Message-ID: <20060529082951.GU54541@deviant.kiev.zoral.com.ua> References: <447A1F94.7020809@sh.cvut.cz> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7gQyIpR7q4QSXYu+" Content-Disposition: inline In-Reply-To: <447A1F94.7020809@sh.cvut.cz> User-Agent: Mutt/1.4.2.1i X-Virus-Scanned: ClamAV version 0.88.2, clamav-milter version 0.88.2 on fw.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED autolearn=failed version=3.1.1 X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on fw.zoral.com.ua Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: Re: LOR in vnode interlock and system map X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 May 2006 08:30:03 -0000 --7gQyIpR7q4QSXYu+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 29, 2006 at 12:09:24AM +0200, V??clav Haisman wrote: > See attached file. The kernel is todays CVS 6.1. >=20 > -- > Vaclav Haisman >=20 >=20 > May 28 19:16:55 logout kernel: lock order reversal: > May 28 19:16:55 logout kernel: 1st 0xc3d4bb1c vnode interlock (vnode inte= rlock) @ /usr/src/sys/kern/vfs_subr.c:2218 > May 28 19:16:55 logout kernel: 2nd 0xc1043144 system map (system map) @ /= usr/src/sys/vm/vm_kern.c:295 > May 28 19:16:55 logout kernel: KDB: stack backtrace: > May 28 19:16:55 logout kernel: kdb_backtrace(c081a0c7,c1043144,c0819bfa,c= 0819bfa,c082f5af) at kdb_backtrace+0x2e > May 28 19:16:55 logout kernel: witness_checkorder(c1043144,9,c082f5af,127= ,c08bdbc0) at witness_checkorder+0x5ef > May 28 19:16:55 logout kernel: _mtx_lock_flags(c1043144,0,c082f5af,127,c1= 04e460) at _mtx_lock_flags+0x32 > May 28 19:16:55 logout kernel: _vm_map_lock(c10430c0,c082f5af,127,c337590= 0,c3375900) at _vm_map_lock+0x37 > May 28 19:16:55 logout kernel: kmem_malloc(c10430c0,1000,101,d5688afc,c07= 66493) at kmem_malloc+0x3a > May 28 19:16:55 logout kernel: page_alloc(c104d300,1000,d5688aef,101,c05e= 0ed6) at page_alloc+0x27 > May 28 19:16:55 logout kernel: slab_zalloc(c104d300,101,c082ed67,8a2,8024= 6) at slab_zalloc+0xd3 > May 28 19:16:55 logout kernel: uma_zone_slab(c104d300,1,c082ed67,8a2,c082= ed67) at uma_zone_slab+0x112 > May 28 19:16:55 logout kernel: uma_zalloc_internal(c104d300,0,1,0,d5688b9= 4) at uma_zalloc_internal+0x41 > May 28 19:16:55 logout kernel: bucket_alloc(80,1,c082ed67,95e,105) at buc= ket_alloc+0x3b > May 28 19:16:55 logout kernel: uma_zfree_arg(c104dc00,c389712c,0,d5688bbc= ,c072e964) at uma_zfree_arg+0x253 > May 28 19:16:55 logout kernel: mac_labelzone_free(c389712c) at mac_labelz= one_free+0x22 > May 28 19:16:55 logout kernel: mac_vnode_label_free(c389712c,c3d4baa0,d56= 88be8,c061aa00,c3d4baa0) at mac_vnode_label_free+0x94 > May 28 19:16:55 logout kernel: mac_destroy_vnode(c3d4baa0,0,c081e172,30a,= c3d4baa0) at mac_destroy_vnode+0x18 > May 28 19:16:55 logout kernel: vdestroy(c3d4baa0,d5688c1c,0,c3d4e9cc,c3d4= e9cc) at vdestroy+0x60 > May 28 19:16:55 logout kernel: vdropl(c3d4baa0,d5688c1c,c081e172,835,c08b= 1440) at vdropl+0x50 > May 28 19:16:55 logout kernel: vput(c3d4baa0,0,c082c8a2,df7,c09117a0) at = vput+0x1ae > May 28 19:16:55 logout kernel: handle_workitem_remove(c36de9e0,0,2,3b3,1)= at handle_workitem_remove+0x15a > May 28 19:16:55 logout kernel: process_worklist_item(c35a0400,0,c082c8a2,= 348,4479db07) at process_worklist_item+0x1e1 > May 28 19:16:55 logout kernel: softdep_process_worklist(c35a0400,0,c082c8= a2,2f2,3e8) at softdep_process_worklist+0x73 > May 28 19:16:55 logout kernel: softdep_flush(0,d5688d38,c08150d4,31d,0) a= t softdep_flush+0x193 > May 28 19:16:55 logout kernel: fork_exit(c0744170,0,d5688d38) at fork_exi= t+0x78 > May 28 19:16:55 logout kernel: fork_trampoline() at fork_trampoline+0x8 > May 28 19:16:55 logout kernel: --- trap 0x1, eip =3D 0, esp =3D 0xd5688d6= c, ebp =3D 0 --- I already reported similar LOR to Jeff Roberson, when latest big pile of VFS fixes where to be MFCed. Try the following patch (note, to trigger the situation you need really high load, and, desirably, low amount of memory installed). My backtrace was almost identical, the difference is that your trace coming from softdepflush daemon, and mine comes from vnlru. lock order reversal: 1st 0xc1a018f0 vnode interlock (vnode interlock) @ /usr/home/kostik/work/b= =3D sd/sys/kern/vfs_subr.c:2449 2nd 0xc0c43144 system map (system map) @ /usr/home/kostik/work/bsd/sys/vm/= =3D vm_kern.c:295 KDB: stack backtrace: kdb_backtrace(0,ffffffff,c06676b0,c0667700,c0636024) at 0xc049d3c9 =3D3D kd= b_=3D backtrace+0x29 witness_checkorder(c0c43144,9,c061fe28,127) at 0xc04a80c2 =3D3D witness_che= ck=3D order+0x582 _mtx_lock_flags(c0c43144,0,c061fe28,127) at 0xc047b998 =3D3D _mtx_lock_flag= s+=3D 0x58 _vm_map_lock(c0c430c0,c061fe28,127) at 0xc059eb46 =3D3D _vm_map_lock+0x26 kmem_malloc(c0c430c0,1000,101,c819fbe0,c059679f) at 0xc059e0d2 =3D3D kmem_m= al=3D loc+0x32 page_alloc(c0c4d300,1000,c819fbd3,101,c06a3bf8) at 0xc0596bda =3D3D page_al= lo=3D c+0x1a slab_zalloc(c0c4d300,101,c0c4d300,c0647a64,c0c4e460) at 0xc059679f =3D3D sl= ab=3D _zalloc+0x9f uma_zone_slab(c0c4d300,1,c0c4e468,0,c061f05a,8a2) at 0xc0597dec =3D3D uma_z= on=3D e_slab+0xec uma_zalloc_internal(c0c4d300,0,1,0,c0c4dc48) at 0xc0598129 =3D3D uma_zalloc= _i=3D nternal+0x29 bucket_alloc(80,1,c0c380a0,0,c19ab6a4) at 0xc0595eac =3D3D bucket_alloc+0x2c uma_zfree_arg(c0c4dc00,c19ab6a4,0) at 0xc0598483 =3D3D uma_zfree_arg+0x283 mac_labelzone_free(c19ab6a4,c1a01828,e8,c819fc9c,c0565ad2) at 0xc055dab3 =3D =3D3D mac_labelzone_free+0x13 mac_vnode_label_free(c19ab6a4,c1a01828,c819fcac,c04d8766,c1a01828) at 0xc05= =3D 65aaa =3D3D mac_vnode_label_free+0x6a mac_destroy_vnode(c1a01828) at 0xc0565ad2 =3D3D mac_destroy_vnode+0x12 vdestroy(c1a01828,c1a01828,c819fcec,c04d8142,c1a01828) at 0xc04d8766 =3D3D = vd=3D estroy+0x1c6 vdropl(c1a01828,7,a8,c0653ee0,c1a01828) at 0xc04dad1e =3D3D vdropl+0x3e vlrureclaim(c15e8000,c1529000,c156f000,c04d8360,c156f000) at 0xc04d8142 =3D= 3D=3D vlrureclaim+0x282 vnlru_proc(0,c819fd38,0,c04d8360,0) at 0xc04d84e3 =3D3D vnlru_proc+0x183 fork_exit(c04d8360,0,c819fd38) at 0xc046de7d =3D3D fork_exit+0x9d fork_trampoline() at 0xc05d33bc =3D3D fork_trampoline+0x8 --- trap 0x1, eip =3D3D 0, esp =3D3D 0xc819fd6c, ebp =3D3D 0 --- A patch to fix the LOR (seems to be relevant for CURRENT too, added -current to CC:): --- sys/kern/vfs_subr.c.orig Sat Mar 4 10:44:47 2006 +++ sys/kern/vfs_subr.c Sat Mar 4 10:45:21 2006 @@ -787,9 +787,6 @@ VNASSERT(bo->bo_dirty.bv_root =3D3D=3D3D NULL, vp, ("dirtyblkroot not NUL= L")); VNASSERT(TAILQ_EMPTY(&vp->v_cache_dst), vp, ("vp has namecache dst")); VNASSERT(LIST_EMPTY(&vp->v_cache_src), vp, ("vp has namecache src")); -#ifdef MAC - mac_destroy_vnode(vp); -#endif if (vp->v_pollinfo !=3D3D NULL) { knlist_destroy(&vp->v_pollinfo->vpi_selinfo.si_note); mtx_destroy(&vp->v_pollinfo->vpi_lock); @@ -801,6 +798,9 @@ #endif lockdestroy(vp->v_vnlock); mtx_destroy(&vp->v_interlock); +#ifdef MAC + mac_destroy_vnode(vp); +#endif uma_zfree(vnode_zone, vp); } Patch assumes that no mac_destroy_vnode does not mess with vnode internals, as it is the case now. --7gQyIpR7q4QSXYu+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (FreeBSD) iD8DBQFEerD/C3+MBN1Mb4gRAoJOAJ41O46TSH40giFk1ka10BOhV7dYCQCg8djD 0zMSlTB9fsvrq8s+d6wPy0s= =QSZi -----END PGP SIGNATURE----- --7gQyIpR7q4QSXYu+--