From owner-freebsd-arm@FreeBSD.ORG Sun Sep 23 17:10:41 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D41DC106566B for ; Sun, 23 Sep 2012 17:10:41 +0000 (UTC) (envelope-from alc@rice.edu) Received: from mh10.mail.rice.edu (mh10.mail.rice.edu [128.42.201.30]) by mx1.freebsd.org (Postfix) with ESMTP id 9E7838FC0C for ; Sun, 23 Sep 2012 17:10:41 +0000 (UTC) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id 4FDEA604CA; Sun, 23 Sep 2012 12:10:40 -0500 (CDT) Received: from mh10.mail.rice.edu (localhost.localdomain [127.0.0.1]) by mh10.mail.rice.edu (Postfix) with ESMTP id 44DFC604C8; Sun, 23 Sep 2012 12:10:40 -0500 (CDT) X-Virus-Scanned: by amavis-2.7.0 at mh10.mail.rice.edu, auth channel Received: from mh10.mail.rice.edu ([127.0.0.1]) by mh10.mail.rice.edu (mh10.mail.rice.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id 0tz-Lu3Vbth2; Sun, 23 Sep 2012 12:10:40 -0500 (CDT) Received: from adsl-216-63-78-18.dsl.hstntx.swbell.net (adsl-216-63-78-18.dsl.hstntx.swbell.net [216.63.78.18]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh10.mail.rice.edu (Postfix) with ESMTPSA id DAB6A603DB; Sun, 23 Sep 2012 12:10:38 -0500 (CDT) Message-ID: <505F428D.9010807@rice.edu> Date: Sun, 23 Sep 2012 12:10:37 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:8.0) Gecko/20111113 Thunderbird/8.0 MIME-Version: 1.0 To: Ian Lepore References: <1345315508.27688.260.camel@revolution.hippie.lan> <503D12AE.1050705@rice.edu> <1346350374.1140.525.camel@revolution.hippie.lan> <5045351F.6060201@rice.edu> <1346723041.1140.602.camel@revolution.hippie.lan> <504B85BE.3030101@rice.edu> <1347316458.1137.41.camel@revolution.hippie.lan> <504F8BAC.4040902@rice.edu> <20120915045040.GZ58312@funkthat.com> <5054D69B.40209@rice.edu> <20120917033308.GB58312@funkthat.com> <505DE03D.7050101@rice.edu> <1348333663.5548.24.camel@revolution.hippie.lan> <505E0739.8080802@rice.edu> <1348346711.15745.3.camel@revolution.hippie.lan> <505E5DF7.9050107@rice.edu> <1348410656.15745.9.camel@revolution.hippie.lan> In-Reply-To: <1348410656.15745.9.camel@revolution.hippie.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "arm@freebsd.org" , Alan Cox Subject: Re: arm pmap locking X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Sep 2012 17:10:42 -0000 On 09/23/2012 09:30, Ian Lepore wrote: > On Sat, 2012-09-22 at 19:55 -0500, Alan Cox wrote: >> On 09/22/2012 15:45, Ian Lepore wrote: >>> Yep. Looks like the first one (reversed) is very very early in the >>> init. After this one, there were no other LORs during the boot until >>> the one that you said is fixed now in -current. >>> >>> lock order reversal: >>> 1st 0xc055bd08 4096 (UMA zone) @ /usr/src/sys/vm/uma_core.c:2011 >>> 2nd 0xc043c198 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:3734 >> Ah. Notice that this is an "UMA zone" lock, but nonetheless a different >> zone and lock from before, i.e., "4096" versus "PV ENTRY". By default, >> witness treats all uma zone locks as having the same lock ordering. >> This helps to keep the distinct lock types that witness must deal with >> and consequently its run-time overhead down. However, sometimes uma >> zone locks on zones that are used within the VM system need to be >> handled specially. There is an argument to uma_zcreate() for doing >> this: UMA_ZONE_MTXCLASS >> >> Try this now. Remove whatever change that you made to witness and apply >> a change like this: >> >> Index: arm/arm/pmap.c >> =================================================================== >> --- arm/arm/pmap.c (revision 240803) >> +++ arm/arm/pmap.c (working copy) >> @@ -1821,7 +1821,7 @@ pmap_init(void) >> * Initialize the PV entry allocator. >> */ >> pvzone = uma_zcreate("PV ENTRY", sizeof (struct pv_entry), >> NULL, NULL, >> - NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); >> + NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_MTXCLASS | UMA_ZONE_VM); >> TUNABLE_INT_FETCH("vm.pmap.shpgperproc",&shpgperproc); >> pv_entry_max = shpgperproc * maxproc + cnt.v_page_count; >> uma_zone_set_obj(pvzone,&pvzone_obj, pv_entry_max); >> >> I've also removed UMA_ZONE_NOFREE from the arguments because it's >> redundant. uma_zone_set_obj() itself sets that flag. >> >> Alan >> > Did that, now the original LOR that happens at start_init() time > involves pmap and the malloc 4096 uma zone rather than the PV ENTRY > zone. Putting back in the preset order, it looks like a malloc() in > mtx_pool_create() is now the first call with the wrong order, the bottom > of the call stack looks like this: > > malloc() at malloc+0x10 > scp=0xc00afe98 rlv=0xc00b28c8 (mtx_pool_create+0x54) > rsp=0xc04ddeb4 rfp=0xc04dded0 > r10=0x204fe888 r9=0x00000019 > r8=0x20479a10 r7=0x00000000 r6=0xc0254a10 r5=0x00000080 > r4=0xc0276a40 > mtx_pool_create() at mtx_pool_create+0x10 > scp=0xc00b2884 rlv=0xc00b290c (mtx_pool_setup_dynamic+0x1c) > rsp=0xc04dded4 rfp=0xc04ddee0 > r7=0x20000124 r6=0x00000004 > r5=0x20000130 r4=0xc0276a40 > mtx_pool_setup_dynamic() at mtx_pool_setup_dynamic+0x10 > scp=0xc00b2900 rlv=0xc0082654 (mi_startup+0xf8) > rsp=0xc04ddee4 rfp=0xc04ddef4 > mi_startup() at mi_startup+0x10 > scp=0xc008256c rlv=0xc00001cc (virt_done+0x18) > rsp=0xc04ddef8 rfp=0x00000000 > r4=0x2000020c > > If you need complete backtraces or other details let me know. > Thanks. I'd like to see the stack trace without any preset lock order. Unless I explicitly say otherwise, assume that I want UMA_ZONE_MTXCLASS specified to the pvzone creation. In general, uma zone locks should come before the pmap lock in the expected lock order. However, any of the zones used by the pmap are likely to be exceptions to this rule. After you added UMA_ZONE_MTXCLASS to the PV zone, I was expecting the next reported LOR to be on the l2 zone or l2table zone. Does that make sense? Alan