From owner-freebsd-arm@FreeBSD.ORG Sun Sep 23 14:31:05 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 08CE31065672 for ; Sun, 23 Sep 2012 14:31:05 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 96AF88FC0A for ; Sun, 23 Sep 2012 14:31:04 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q8NEUx3K036881 for ; Sun, 23 Sep 2012 08:31:00 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q8NEUusT063006; Sun, 23 Sep 2012 08:30:56 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Alan Cox In-Reply-To: <505E5DF7.9050107@rice.edu> References: <1345315508.27688.260.camel@revolution.hippie.lan> <503D12AE.1050705@rice.edu> <1346350374.1140.525.camel@revolution.hippie.lan> <5045351F.6060201@rice.edu> <1346723041.1140.602.camel@revolution.hippie.lan> <504B85BE.3030101@rice.edu> <1347316458.1137.41.camel@revolution.hippie.lan> <504F8BAC.4040902@rice.edu> <20120915045040.GZ58312@funkthat.com> <5054D69B.40209@rice.edu> <20120917033308.GB58312@funkthat.com> <505DE03D.7050101@rice.edu> <1348333663.5548.24.camel@revolution.hippie.lan> <505E0739.8080802@rice.edu> <1348346711.15745.3.camel@revolution.hippie.lan> <505E5DF7.9050107@rice.edu> Content-Type: text/plain; charset="us-ascii" Date: Sun, 23 Sep 2012 08:30:56 -0600 Message-ID: <1348410656.15745.9.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: "arm@freebsd.org" Subject: Re: arm pmap locking X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Sep 2012 14:31:05 -0000 On Sat, 2012-09-22 at 19:55 -0500, Alan Cox wrote: > On 09/22/2012 15:45, Ian Lepore wrote: > > Yep. Looks like the first one (reversed) is very very early in the > > init. After this one, there were no other LORs during the boot until > > the one that you said is fixed now in -current. > > > > lock order reversal: > > 1st 0xc055bd08 4096 (UMA zone) @ /usr/src/sys/vm/uma_core.c:2011 > > 2nd 0xc043c198 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:3734 > > Ah. Notice that this is an "UMA zone" lock, but nonetheless a different > zone and lock from before, i.e., "4096" versus "PV ENTRY". By default, > witness treats all uma zone locks as having the same lock ordering. > This helps to keep the distinct lock types that witness must deal with > and consequently its run-time overhead down. However, sometimes uma > zone locks on zones that are used within the VM system need to be > handled specially. There is an argument to uma_zcreate() for doing > this: UMA_ZONE_MTXCLASS > > Try this now. Remove whatever change that you made to witness and apply > a change like this: > > Index: arm/arm/pmap.c > =================================================================== > --- arm/arm/pmap.c (revision 240803) > +++ arm/arm/pmap.c (working copy) > @@ -1821,7 +1821,7 @@ pmap_init(void) > * Initialize the PV entry allocator. > */ > pvzone = uma_zcreate("PV ENTRY", sizeof (struct pv_entry), > NULL, NULL, > - NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); > + NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_MTXCLASS | UMA_ZONE_VM); > TUNABLE_INT_FETCH("vm.pmap.shpgperproc", &shpgperproc); > pv_entry_max = shpgperproc * maxproc + cnt.v_page_count; > uma_zone_set_obj(pvzone, &pvzone_obj, pv_entry_max); > > I've also removed UMA_ZONE_NOFREE from the arguments because it's > redundant. uma_zone_set_obj() itself sets that flag. > > Alan > Did that, now the original LOR that happens at start_init() time involves pmap and the malloc 4096 uma zone rather than the PV ENTRY zone. Putting back in the preset order, it looks like a malloc() in mtx_pool_create() is now the first call with the wrong order, the bottom of the call stack looks like this: malloc() at malloc+0x10 scp=0xc00afe98 rlv=0xc00b28c8 (mtx_pool_create+0x54) rsp=0xc04ddeb4 rfp=0xc04dded0 r10=0x204fe888 r9=0x00000019 r8=0x20479a10 r7=0x00000000 r6=0xc0254a10 r5=0x00000080 r4=0xc0276a40 mtx_pool_create() at mtx_pool_create+0x10 scp=0xc00b2884 rlv=0xc00b290c (mtx_pool_setup_dynamic+0x1c) rsp=0xc04dded4 rfp=0xc04ddee0 r7=0x20000124 r6=0x00000004 r5=0x20000130 r4=0xc0276a40 mtx_pool_setup_dynamic() at mtx_pool_setup_dynamic+0x10 scp=0xc00b2900 rlv=0xc0082654 (mi_startup+0xf8) rsp=0xc04ddee4 rfp=0xc04ddef4 mi_startup() at mi_startup+0x10 scp=0xc008256c rlv=0xc00001cc (virt_done+0x18) rsp=0xc04ddef8 rfp=0x00000000 r4=0x2000020c If you need complete backtraces or other details let me know. -- Ian