From owner-freebsd-arm@FreeBSD.ORG Sun Sep 23 19:35:43 2012 Return-Path: Delivered-To: arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC459106564A for ; Sun, 23 Sep 2012 19:35:43 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from duck.symmetricom.us (duck.symmetricom.us [206.168.13.214]) by mx1.freebsd.org (Postfix) with ESMTP id 7D68B8FC08 for ; Sun, 23 Sep 2012 19:35:43 +0000 (UTC) Received: from damnhippie.dyndns.org (daffy.symmetricom.us [206.168.13.218]) by duck.symmetricom.us (8.14.5/8.14.5) with ESMTP id q8NJZaR9045310 for ; Sun, 23 Sep 2012 13:35:36 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q8NJZNij063255; Sun, 23 Sep 2012 13:35:23 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Alan Cox In-Reply-To: <505F428D.9010807@rice.edu> References: <1345315508.27688.260.camel@revolution.hippie.lan> <503D12AE.1050705@rice.edu> <1346350374.1140.525.camel@revolution.hippie.lan> <5045351F.6060201@rice.edu> <1346723041.1140.602.camel@revolution.hippie.lan> <504B85BE.3030101@rice.edu> <1347316458.1137.41.camel@revolution.hippie.lan> <504F8BAC.4040902@rice.edu> <20120915045040.GZ58312@funkthat.com> <5054D69B.40209@rice.edu> <20120917033308.GB58312@funkthat.com> <505DE03D.7050101@rice.edu> <1348333663.5548.24.camel@revolution.hippie.lan> <505E0739.8080802@rice.edu> <1348346711.15745.3.camel@revolution.hippie.lan> <505E5DF7.9050107@rice.edu> <1348410656.15745.9.camel@revolution.hippie.lan> <505F428D.9010807@rice.edu> Content-Type: text/plain; charset="us-ascii" Date: Sun, 23 Sep 2012 13:35:23 -0600 Message-ID: <1348428923.15745.22.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: "arm@freebsd.org" Subject: Re: arm pmap locking X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Sep 2012 19:35:43 -0000 On Sun, 2012-09-23 at 12:10 -0500, Alan Cox wrote: > On 09/23/2012 09:30, Ian Lepore wrote: > > On Sat, 2012-09-22 at 19:55 -0500, Alan Cox wrote: > >> On 09/22/2012 15:45, Ian Lepore wrote: > >>> Yep. Looks like the first one (reversed) is very very early in the > >>> init. After this one, there were no other LORs during the boot until > >>> the one that you said is fixed now in -current. > >>> > >>> lock order reversal: > >>> 1st 0xc055bd08 4096 (UMA zone) @ /usr/src/sys/vm/uma_core.c:2011 > >>> 2nd 0xc043c198 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:3734 > >> Ah. Notice that this is an "UMA zone" lock, but nonetheless a different > >> zone and lock from before, i.e., "4096" versus "PV ENTRY". By default, > >> witness treats all uma zone locks as having the same lock ordering. > >> This helps to keep the distinct lock types that witness must deal with > >> and consequently its run-time overhead down. However, sometimes uma > >> zone locks on zones that are used within the VM system need to be > >> handled specially. There is an argument to uma_zcreate() for doing > >> this: UMA_ZONE_MTXCLASS > >> > >> Try this now. Remove whatever change that you made to witness and apply > >> a change like this: > >> > >> Index: arm/arm/pmap.c > >> =================================================================== > >> --- arm/arm/pmap.c (revision 240803) > >> +++ arm/arm/pmap.c (working copy) > >> @@ -1821,7 +1821,7 @@ pmap_init(void) > >> * Initialize the PV entry allocator. > >> */ > >> pvzone = uma_zcreate("PV ENTRY", sizeof (struct pv_entry), > >> NULL, NULL, > >> - NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_VM | UMA_ZONE_NOFREE); > >> + NULL, NULL, UMA_ALIGN_PTR, UMA_ZONE_MTXCLASS | UMA_ZONE_VM); > >> TUNABLE_INT_FETCH("vm.pmap.shpgperproc",&shpgperproc); > >> pv_entry_max = shpgperproc * maxproc + cnt.v_page_count; > >> uma_zone_set_obj(pvzone,&pvzone_obj, pv_entry_max); > >> > >> I've also removed UMA_ZONE_NOFREE from the arguments because it's > >> redundant. uma_zone_set_obj() itself sets that flag. > >> > >> Alan > >> > > Did that, now the original LOR that happens at start_init() time > > involves pmap and the malloc 4096 uma zone rather than the PV ENTRY > > zone. Putting back in the preset order, it looks like a malloc() in > > mtx_pool_create() is now the first call with the wrong order, the bottom > > of the call stack looks like this: > > > > malloc() at malloc+0x10 > > scp=0xc00afe98 rlv=0xc00b28c8 (mtx_pool_create+0x54) > > rsp=0xc04ddeb4 rfp=0xc04dded0 > > r10=0x204fe888 r9=0x00000019 > > r8=0x20479a10 r7=0x00000000 r6=0xc0254a10 r5=0x00000080 > > r4=0xc0276a40 > > mtx_pool_create() at mtx_pool_create+0x10 > > scp=0xc00b2884 rlv=0xc00b290c (mtx_pool_setup_dynamic+0x1c) > > rsp=0xc04dded4 rfp=0xc04ddee0 > > r7=0x20000124 r6=0x00000004 > > r5=0x20000130 r4=0xc0276a40 > > mtx_pool_setup_dynamic() at mtx_pool_setup_dynamic+0x10 > > scp=0xc00b2900 rlv=0xc0082654 (mi_startup+0xf8) > > rsp=0xc04ddee4 rfp=0xc04ddef4 > > mi_startup() at mi_startup+0x10 > > scp=0xc008256c rlv=0xc00001cc (virt_done+0x18) > > rsp=0xc04ddef8 rfp=0x00000000 > > r4=0x2000020c > > > > If you need complete backtraces or other details let me know. > > > > Thanks. I'd like to see the stack trace without any preset lock order. > > Unless I explicitly say otherwise, assume that I want UMA_ZONE_MTXCLASS > specified to the pvzone creation. > > In general, uma zone locks should come before the pmap lock in the > expected lock order. However, any of the zones used by the pmap are > likely to be exceptions to this rule. After you added UMA_ZONE_MTXCLASS > to the PV zone, I was expecting the next reported LOR to be on the l2 > zone or l2table zone. Does that make sense? > > Alan > Without presetting the order... warning: no time-of-day clock registered, system time will not be set accurately lock order reversal: 1st 0xc0a8a0b0 pmap (pmap) @ /usr/src/sys/arm/arm/pmap.c:971 2nd 0xc055b388 128 Bucket (UMA zone) @ /usr/src/sys/vm/uma_core.c:2485 KDB: stack backtrace: db_trace_thread() at db_trace_thread+0x10 scp=0xc0215d7c rlv=0xc0215f80 (db_trace_self+0x1c) rsp=0xc2e9d90c rfp=0xc2e9d918 r10=0x00000000 r9=0xc0a4c4b8 r8=0xc040aec4 r7=0xffffffff r6=0xc0a4c6c0 r5=0xc023a21c r4=0xc2e9d924 db_trace_self() at db_trace_self+0x10 scp=0xc0215f74 rlv=0xc001ee0c (db_trace_self_wrapper+0x30) rsp=0xc2e9d91c rfp=0xc2e9da38 db_trace_self_wrapper() at db_trace_self_wrapper+0x10 scp=0xc001edec rlv=0xc00ef4f8 (kdb_backtrace+0x3c) rsp=0xc2e9da3c rfp=0xc2e9da4c r4=0xc02dbf44 kdb_backtrace() at kdb_backtrace+0x10 scp=0xc00ef4cc rlv=0xc0102c48 (_witness_debugger+0x2c) rsp=0xc2e9da50 rfp=0xc2e9da64 r4=0x00000001 _witness_debugger() at _witness_debugger+0x10 scp=0xc0102c2c rlv=0xc0103468 (witness_checkorder+0x7e8) rsp=0xc2e9da68 rfp=0xc2e9dab4 r5=0x00000000 r4=0xc055b388 witness_checkorder() at witness_checkorder+0x10 scp=0xc0102c90 rlv=0xc00b37fc (_mtx_lock_flags+0xa4) rsp=0xc2e9dab8 rfp=0xc2e9dadc r10=0x00000000 r9=0xc043c5ac r8=0x00000000 r7=0x000009b5 r6=0xc026e6b4 r5=0x00000000 r4=0xc055b388 _mtx_lock_flags() at _mtx_lock_flags+0x10 scp=0xc00b3768 rlv=0xc01f2e20 (zone_alloc_item+0x34) rsp=0xc2e9dae0 rfp=0xc2e9db08 r8=0xc026e6b4 r7=0x00000201 r6=0x00000000 r5=0xc055a3c0 r4=0x00000201 zone_alloc_item() at zone_alloc_item+0x10 scp=0xc01f2dfc rlv=0xc01f46a4 (bucket_alloc+0x3c) rsp=0xc2e9db0c rfp=0xc2e9db20 r10=0x00000000 r8=0xc055a6e0 r7=0xc055a740 r6=0x00000000 r5=0xc02cd428 r4=0x00000201 bucket_alloc() at bucket_alloc+0x10 scp=0xc01f4678 rlv=0xc01f4f8c (uma_zalloc_arg+0x450) rsp=0xc2e9db24 rfp=0xc2e9db60 r5=0x00000201 r4=0xc026e6b4 uma_zalloc_arg() at uma_zalloc_arg+0x10 scp=0xc01f4b4c rlv=0xc021b4d0 (pmap_get_pv_entry+0x40) rsp=0xc2e9db64 rfp=0xc2e9db70 r10=0x00000000 r9=0xc043c5ac r8=0x2078e55e r7=0xc0a8a0b0 r6=0xc0272f84 r5=0xc05370b8 r4=0x00000000 pmap_get_pv_entry() at pmap_get_pv_entry+0x10 scp=0xc021b4a0 rlv=0xc02201a0 (pmap_enter_locked+0xcfc) rsp=0xc2e9db74 rfp=0xc2e9dbf0 pmap_enter_locked() at pmap_enter_locked+0x10 scp=0xc021f4b4 rlv=0xc0220704 (pmap_enter+0x74) rsp=0xc2e9dbf4 rfp=0xc2e9dc20 r10=0x00000007 r9=0xc2e9dde4 r8=0xc056dea0 r7=0xbffff000 r6=0xc043a07c r5=0xc0a8a0b0 r4=0xc0272f84 pmap_enter() at pmap_enter+0x10 scp=0xc02206a0 rlv=0xc01f80e0 (vm_fault+0x1688) rsp=0xc2e9dc24 rfp=0xc2e9dd3c r10=0x00000002 r8=0xbffff000 r7=0xc0a88000 r6=0x00000000 r5=0xc0273420 r4=0xc056dea0 vm_fault() at vm_fault+0x10 scp=0xc01f6a68 rlv=0xc0224078 (data_abort_handler+0x360) rsp=0xc2e9dd40 rfp=0xc2e9dde0 r10=0x00000002 r9=0xc2e9dde4 r8=0xbffff000 r7=0xc0a88000 r6=0x00000000 r5=0xc0273420 r4=0xc0a86088 data_abort_handler() at data_abort_handler+0x10 scp=0xc0223d28 rlv=0xc0217784 (exception_exit) rsp=0xc2e9dde4 rfp=0xc2e9de84 r10=0xc02b7d0a r9=0xc0a86000 r8=0xc2e9deac r7=0xc02b7d0a r6=0xc02b7d00 r5=0xffff1004 r4=0xffffffff start_init() at start_init+0x10 scp=0xc0082244 rlv=0xc009c460 (fork_exit+0x84) rsp=0xc2e9de88 rfp=0xc2e9dea8 r10=0x00000000 r9=0x00000000 r8=0xc2e9deac r7=0x00000000 r6=0xc0082234 r5=0xc0a86000 r4=0xc0a88000 fork_exit() at fork_exit+0x10 scp=0xc009c3ec rlv=0xc022354c (fork_trampoline+0x14) rsp=0xc2e9deac rfp=0x00000000 r8=0x00000000 r7=0x6c6c6568 r6=0x735f6365 r5=0x00000000 r4=0xc0082234 -- Ian