Date: Sun, 23 Sep 2012 12:10:37 -0500 From: Alan Cox <alc@rice.edu> To: Ian Lepore <freebsd@damnhippie.dyndns.org> Cc: "arm@freebsd.org" <arm@freebsd.org>, Alan Cox <alc@rice.edu> Subject: Re: arm pmap locking Message-ID: <505F428D.9010807@rice.edu> In-Reply-To: <1348410656.15745.9.camel@revolution.hippie.lan> References: <1345315508.27688.260.camel@revolution.hippie.lan> <503D12AE.1050705@rice.edu> <1346350374.1140.525.camel@revolution.hippie.lan> <5045351F.6060201@rice.edu> <1346723041.1140.602.camel@revolution.hippie.lan> <504B85BE.3030101@rice.edu> <1347316458.1137.41.camel@revolution.hippie.lan> <504F8BAC.4040902@rice.edu> <20120915045040.GZ58312@funkthat.com> <5054D69B.40209@rice.edu> <20120917033308.GB58312@funkthat.com> <505DE03D.7050101@rice.edu> <1348333663.5548.24.camel@revolution.hippie.lan> <505E0739.8080802@rice.edu> <1348346711.15745.3.camel@revolution.hippie.lan> <505E5DF7.9050107@rice.edu> <1348410656.15745.9.camel@revolution.hippie.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
On 09/23/2012 09:30, Ian Lepore wrote:
> On Sat, 2012-09-22 at 19:55 -0500, Alan Cox wrote:
>> On 09/22/2012 15:45, Ian Lepore wrote:
>>> Yep. Looks like the first one (reversed) is very very early in the
>>> init. After this one, there were no other LORs during the boot until
>>> the one that you said is fixed now in -current.
>>>
>>> lock order reversal:
>>> 1st 0xc055bd08 4096 (UMA zone) @ /usr/src/sys/vm/uma_core.c:2011
>>> 2nd 0xc043c198 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:3734
>> Ah. Notice that this is an "UMA zone" lock, but nonetheless a different
>> zone and lock from before, i.e., "4096" versus "PV ENTRY". By default,
>> witness treats all uma zone locks as having the same lock ordering.
>> This helps to keep the distinct lock types that witness must deal with
>> and consequently its run-time overhead down. However, sometimes uma
>> zone locks on zones that are used within the VM system need to be
>> handled specially. There is an argument to uma_zcreate() for doing
>> this: UMA_ZONE_MTXCLASS
>>
>> Try this now. Remove whatever change that you made to witness and apply
>> a change like this:
>>
>> Index: arm/arm/pmap.c
>> ===================================================================
>> --- arm/arm/pmap.c (revision 240803)
>> +++ arm/arm/pmap.c (working copy)
>> @@ -1821,7 +1821,7 @@ pmap_init(void)
>> * Initialize the PV entry allocator.
>> */
>> pvzone = uma_zcreate("PV ENTRY", sizeof (struct pv_entry),
>> NULL, NULL,
>> - NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE);
>> + NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_MTXCLASS | UMA_ZONE_VM);
>> TUNABLE_INT_FETCH("vm.pmap.shpgperproc",&shpgperproc);
>> pv_entry_max = shpgperproc * maxproc + cnt.v_page_count;
>> uma_zone_set_obj(pvzone,&pvzone_obj, pv_entry_max);
>>
>> I've also removed UMA_ZONE_NOFREE from the arguments because it's
>> redundant. uma_zone_set_obj() itself sets that flag.
>>
>> Alan
>>
> Did that, now the original LOR that happens at start_init() time
> involves pmap and the malloc 4096 uma zone rather than the PV ENTRY
> zone. Putting back in the preset order, it looks like a malloc() in
> mtx_pool_create() is now the first call with the wrong order, the bottom
> of the call stack looks like this:
>
> malloc() at malloc+0x10
> scp=0xc00afe98 rlv=0xc00b28c8 (mtx_pool_create+0x54)
> rsp=0xc04ddeb4 rfp=0xc04dded0
> r10=0x204fe888 r9=0x00000019
> r8=0x20479a10 r7=0x00000000 r6=0xc0254a10 r5=0x00000080
> r4=0xc0276a40
> mtx_pool_create() at mtx_pool_create+0x10
> scp=0xc00b2884 rlv=0xc00b290c (mtx_pool_setup_dynamic+0x1c)
> rsp=0xc04dded4 rfp=0xc04ddee0
> r7=0x20000124 r6=0x00000004
> r5=0x20000130 r4=0xc0276a40
> mtx_pool_setup_dynamic() at mtx_pool_setup_dynamic+0x10
> scp=0xc00b2900 rlv=0xc0082654 (mi_startup+0xf8)
> rsp=0xc04ddee4 rfp=0xc04ddef4
> mi_startup() at mi_startup+0x10
> scp=0xc008256c rlv=0xc00001cc (virt_done+0x18)
> rsp=0xc04ddef8 rfp=0x00000000
> r4=0x2000020c
>
> If you need complete backtraces or other details let me know.
>
Thanks. I'd like to see the stack trace without any preset lock order.
Unless I explicitly say otherwise, assume that I want UMA_ZONE_MTXCLASS
specified to the pvzone creation.
In general, uma zone locks should come before the pmap lock in the
expected lock order. However, any of the zones used by the pmap are
likely to be exceptions to this rule. After you added UMA_ZONE_MTXCLASS
to the PV zone, I was expecting the next reported LOR to be on the l2
zone or l2table zone. Does that make sense?
Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?505F428D.9010807>