From owner-freebsd-ppc@FreeBSD.ORG Sat Sep 27 06:57:10 2014 Return-Path: Delivered-To: freebsd-ppc@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 40F62336 for ; Sat, 27 Sep 2014 06:57:10 +0000 (UTC) Received: from asp.reflexion.net (outbound-242.asp.reflexion.net [69.84.129.242]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8337777F for ; Sat, 27 Sep 2014 06:57:08 +0000 (UTC) Received: (qmail 26742 invoked from network); 27 Sep 2014 06:57:07 -0000 Received: from unknown (HELO mail-cs-01.app.dca.reflexion.local) (10.81.19.1) by 0 (rfx-qmail) with SMTP; 27 Sep 2014 06:57:07 -0000 Received: by mail-cs-01.app.dca.reflexion.local (Reflexion email security v7.30.7) with SMTP; Sat, 27 Sep 2014 02:57:07 -0400 (EDT) Received: (qmail 17465 invoked from network); 27 Sep 2014 06:56:00 -0000 Received: from unknown (HELO iron2.pdx.net) (69.64.224.71) by 0 (rfx-qmail) with (DHE-RSA-AES256-SHA encrypted) SMTP; 27 Sep 2014 06:56:00 -0000 X-No-Relay: not in my network X-No-Relay: not in my network X-No-Relay: not in my network Received: from [192.168.1.8] (c-98-246-178-138.hsd1.or.comcast.net [98.246.178.138]) by iron2.pdx.net (Postfix) with ESMTPSA id C922A1C402B; Fri, 26 Sep 2014 23:55:59 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: lr=u_trap+0x10 and srr0=k_trap+0x28 for "stopped at 0 illegal instruction 0" before-copyright hang on PowerMac G5's From: Mark Millard In-Reply-To: <7008CDAA-2DA2-419F-9BEC-AD823ECBFCCC@dsl-only.net> Date: Fri, 26 Sep 2014 23:55:59 -0700 Message-Id: <2E98A886-36FF-4B68-B729-F2143339E1DE@dsl-only.net> References: <1118046C-0FF7-49FC-82DA-DB9A7A310991@dsl-only.net> <2ED3DB50-B985-4382-8FF2-3B44E7E65453@dsl-only.net> <6D729F43-662A-429E-9503-0148EC3250B1@dsl-only.net> <72535F89-3942-45A6-B351-7F746209ED9F@dsl-only.net> <0703EF26-6E33-4446-9273-BBFD0CB72893@dsl-only.net> <37575F94-763C-43BF-8DD9-F648F4A7C09F@dsl-only.net> <5422E513.6010806@freebsd.org> <1C02D0D4-14B8-465F-B493-4D3A64E4C35C@dsl-only.net> <0DF8A9EC-C81C-4E15-9420-6831BA7D5F8E@dsl-only.net> <54248467.4050900@freebsd.org> <34AA4542-56A7-453E-A00E-868EE352C96C@dsl-only.net> <7008CDAA-2DA2-419F-9BEC-AD823ECBFCCC@dsl-only.net> To: Nathan Whitehorn X-Mailer: Apple Mail (2.1878.6) Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Justin Hibbits , FreeBSD PowerPC ML X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Sep 2014 06:57:10 -0000 According to my adjusted dumping: At the "before = Copyright"/ofwcall-for-peer crash ofw_real_mode=3D=3D0. And that does turn off exception vector save/restore: __inline void ofw_save_trap_vec(char *save_trap_vec) { if (!ofw_real_mode) return; bcopy((void *)EXC_RST, save_trap_vec, EXC_LAST - EXC_RST); } static __inline void ofw_restore_trap_vec(char *restore_trap_vec) { if (!ofw_real_mode) return; bcopy(restore_trap_vec, (void *)EXC_RST, EXC_LAST - EXC_RST); __syncicache(EXC_RSVD, EXC_LAST - EXC_RSVD); } So now it is clear to me how FreeBSD's exception vectors could be = involved in a context that does not have FreeBSD's environment in place. = (Finally!) For powerpc64/GENERIC64 it should also then establish OFW_STD_32BIT: boolean_t OF_bootstrap() { boolean_t status =3D FALSE; =20 if (openfirmware_entry !=3D NULL) { if (ofw_real_mode) { status =3D OF_install(OFW_STD_REAL, 0); } else { #ifdef __powerpc64__ status =3D OF_install(OFW_STD_32BIT, 0); #else status =3D OF_install(OFW_STD_DIRECT, 0); #endif } This seems to be like OFW_STD_REAL in what it sets up: ofw_real_methods. static ofw_def_t ofw_real =3D { OFW_STD_REAL, ofw_real_methods, 0 }; OFW_DEF(ofw_real); static ofw_def_t ofw_32bit =3D { OFW_STD_32BIT, ofw_real_methods, 0 }; OFW_DEF(ofw_32bit); ofw_real_mode is used to figure out the context when it matters from = what I can tell so far. Just to experiment to be sure I temporarily hacked in ignoring = ofw_real_mode in ofw_save_trap_vec and ofw_restore_trap_vec so they = would be effective at exception vector swapping. As I guessed it still hangs before the copyright notice. (Without = getting to DDB so no dump information is displayed.) =3D=3D=3D Mark Millard markmi at dsl-only.net On Sep 26, 2014, at 10:18 PM, Mark Millard = wrote: The first send of this was big enough for the moderator to be involved. = So I canceled and am sending with less history included. [I'll note that I seem to have trouble typing 0xdbb290 vs. 0xbdd290. The = actual value is 0xdbb290. The references to the incorrect typing should = say 0xbdd290, which is the wrong value. But I've had both types of = references listing the wrong text... in various notes.] =3D=3D=3D Mark Millard markmi@dsl-only.net On Sep 26, 2014, at 10:11 PM, Mark Millard wrote: The openfirmware peer crash (i.e., the before Copyright notice crash) = happens during/just-after the MMU setup and the peer pfwcall is the = first ofwcall where pmap_bootstrapped is non-zero at the time. In other = words: the very first ofwcall in the new context fails. And this failure involves some of the same code area that I got a = backtrace for and reported as a separate crash (with the trace listed). = As a reminder for that backtrace that has a difference failure point: .pvo_vaddr_compare+0x14, instruction ld r0, r4, 0x58 [or ld r0,88(r4) in = an alternate notation] .pvo_tree_RB_FIND+0x38 .moea64_dev_direct_mapped_0x90 .pmap_dev_direct_mapped+0x84 ("_dev" was missing in earlier note) .bs_remap_earlyboot_0x6c .moea64_late_bootstrap+0x178 .moea64_bootstrap_native+0x120 .pmap_bootstrap+0xac .powerpc_init+0x514 btext+0xa8 As for the sequence of ofwcall's that I reported: starting at the last = OF_finddevice before the OF_instance_to_package that I reported in the = sequence of ofwcall's from quiesce until the crash... moea64_late_bootstrap does chosen =3D OF_finddevice("/chosen"); if (chosen !=3D -1 && OF_getprop(chosen, "mmu", &mmui, 4) !=3D = -1) { mmu =3D OF_instance_to_package(mmui); if (mmu =3D=3D -1 || (sz =3D OF_getproplen(mmu, = "translations")) =3D=3D -1) sz =3D 0; if (sz > 6144 /* tmpstksz - 2 KB headroom */) panic("moea64_bootstrap: too many ofw translations"); =20 if (sz > 0) moea64_add_ofw_mappings(mmup, mmu, sz); } with moea64_add_ofw_mappings called. Then... moea64_add_ofw_mappings does... bzero(translations, sz); OF_getprop(OF_finddevice("/"), "#address-cells", &acells, sizeof(acells)); if (OF_getprop(mmu, "translations", trans_cells, sz) =3D=3D -1) panic("moea64_bootstrap: can't get ofw translations"); And it is the next ofwcall after that last OF_getprop that fails. (It = happens to be a peer request.) Adding a dump of the pmap_bootstrapped = value with the ofwcall name in my hack for reporting things about the = crash confirmed that peer ofwcall as the first with pmap_bootstrapped = non-zero. I will note here that it is somewhat later than the above code that = pvo_vaddr_compare ends up executing via bs_remap_earlyboot. That earlier = moea64_late_bootstrap code continues after the } from the first if above = with: /* * Calculate the last available physical address. */ for (i =3D 0; phys_avail[i + 2] !=3D 0; i +=3D 2) ; Maxmem =3D powerpc_btop(phys_avail[i + 1]); /* * Initialize MMU and remap early physical mappings */ MMU_CPU_BOOTSTRAP(mmup,0); mtmsr(mfmsr() | PSL_DR | PSL_IR); pmap_bootstrapped++; bs_remap_earlyboot(); (and more). I've not found the peer call yet but it may well be after = the pvo_vaddr_compare shown above as far as execution order goes. =3D=3D=3D Mark Millard markmi at dsl-only.net On Sep 25, 2014, at 2:41 PM, Mark Millard = wrote: The first boot after make -8 kernel without quiesce also died during = peer, I'd guess the same one. Looks like quiesce does not matter for the issue. (But it is handy for = identifying which peer fails.) =3D=3D=3D Mark Millard markmi at dsl-only.net On Sep 25, 2014, at 2:08 PM, Nathan Whitehorn wrote: Can you comment out the call to quiesce? It may not be necessary on your = system. -Nathan On 09/25/14 13:17, Mark Millard wrote: > The "before copyright" hang/exception is during the first openfirmware = "peer" after "quiesce". The ofw_restore_trap_vec(save_trap_init) = completes fine, the ofwcall(args) is made but it does not return = normally. >=20 > Ignoring the ofwcall's from before quiesce, the sequence of ofwcall's = is: >=20 > quiesce > finddevice > parent > getprop > getprop > getprop > finddevice > getprop > instance-to-package > getproplen > finddevice > getprop > getprop > peer >=20 > And when the boot fails before the copyright that ofwcall for peer = ends up resulting in the register dump with no register pointing to the = kernel's normal stack area. >=20 > I still have no clue what is happening during peer. = ofw_restore_trap_vec(save_trap_init) is being called and is returning = before ofwcall is used. For all I know some uses of peer could require = not being quiesce'd in order for peer to be reliable. >=20 > In the form of my display indicating what executed the text reported = ends in: >=20 > ^ >=20 > where the ^ indicates the stage that last completed in the call = sequence inside openfirmware_core. This information is displayed by the >=20 > x/s ofw_name_history >=20 > in the automatically created default script for DDB. I read the = sequence backwards from the end marker (here ^), following the = wraparound if there is that much text and if I care to go back that far. >=20 > FreeBSD FBSDG5M1 10.1-BETA2 FreeBSD 10.1-BETA2 #11 r271944M: Thu Sep = 25 12:14:05 PDT 2014 root@FBSDG5M1:/usr/obj/usr/src/sys/GENERIC64 = powerpc >=20 > My current hacks to get this information are: >=20 > Index: /usr/src/sys/ddb/db_script.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/ddb/db_script.c (revision 271944) > +++ /usr/src/sys/ddb/db_script.c (working copy) > @@ -319,10 +319,25 @@ > { > char scriptname[DB_MAXSCRIPTNAME]; > =20 > + /* HACK!!! : Additional lines to force a basic default script to = exist. > + * Will dump information even if ddb input is not available for = early crash. > + * Used to get more information about PowerMac G5 "before Copyright" = hangs. > + */ > + struct ddb_script *dsp =3D = db_script_lookup(DB_SCRIPT_KDBENTER_DEFAULT); > + if (!dsp) db_script_set(DB_SCRIPT_KDBENTER_DEFAULT, "show registers; = bt; x/s ofw_name_history"); > + > snprintf(scriptname, sizeof(scriptname), "%s.%s", > DB_SCRIPT_KDBENTER_PREFIX, eventname); > if (db_script_exec(scriptname, 0) =3D=3D ENOENT) > (void)db_script_exec(DB_SCRIPT_KDBENTER_DEFAULT, 0); > + > + /* HACK!!! : Additional lines to always use the default script, > + * even if scriptname existed and was executed. > + * Will dump information even if ddb input is not available for = early crash. > + * Used to get more information about PowerMac G5 "before Copyright" = hangs. > + */ > + else > + (void)db_script_exec(DB_SCRIPT_KDBENTER_DEFAULT, 0); > } > =20 > /*- > Index: /usr/src/sys/powerpc/conf/GENERIC64 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/powerpc/conf/GENERIC64 (revision 271944) > +++ /usr/src/sys/powerpc/conf/GENERIC64 (working copy) > @@ -76,6 +76,8 @@ > # Debugging support. Always need this: > options KDB # Enable kernel debugger support. > options KDB_TRACE # Print a stack trace for a panic. > +options DDB > +options GDB > =20 > # Make an SMP-capable kernel by default > options SMP # Symmetric MultiProcessor Kernel > Index: /usr/src/sys/powerpc/ofw/ofw_machdep.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- /usr/src/sys/powerpc/ofw/ofw_machdep.c (revision 271944) > +++ /usr/src/sys/powerpc/ofw/ofw_machdep.c (working copy) > @@ -324,6 +324,12 @@ > openfirmware(&args); > } > =20 > +/* Part of HACK to have record of ofw call names */ > +#define ofw_name_history_record_size 256 > +char ofw_name_history[ofw_name_history_record_size+1] =3D {}; /* = Initially: automatically '\0' filled */ > +char * ofw_name_history_pos =3D ofw_name_history; > +/* End Part of HACK */ > + > static int > openfirmware_core(void *args) > { > @@ -330,6 +336,42 @@ > int result; > register_t oldmsr; > =20 > + { /* HACK to have record of ofw call names */ > + struct argtype_prefix { > + cell_t name; > + }; > + > + char *name =3D (char*) (uintptr_t) (((struct = argtype_prefix*)args)->name); > +=20 > + int i; > + > + *ofw_name_history_pos =3D '<'; > + > + for(i=3D0; (*name) && i!=3D20; i++) { > + ofw_name_history_pos++; > + if (ofw_name_history_pos =3D=3D = &ofw_name_history[ofw_name_history_record_size]) { > + ofw_name_history_pos =3D ofw_name_history; > + } > + *ofw_name_history_pos =3D *name; > + > + name++; > + } > + > + ofw_name_history_pos++; > + if (ofw_name_history_pos =3D=3D = &ofw_name_history[ofw_name_history_record_size]) { > + ofw_name_history_pos =3D ofw_name_history; > + } > + *ofw_name_history_pos =3D '>'; > + > + ofw_name_history_pos++; > + if (ofw_name_history_pos =3D=3D = &ofw_name_history[ofw_name_history_record_size]) { > + ofw_name_history_pos =3D ofw_name_history; > + } > + *ofw_name_history_pos =3D '@'; > + > + ofw_name_history[ofw_name_history_record_size] =3D '\0'; /* Paranoia = */ > + } /* HACK end */ > + > /* > * Turn off exceptions - we really don't want to end up > * anywhere unexpected with PCPU set to something strange > @@ -337,14 +379,22 @@ > */ > oldmsr =3D intr_disable(); > =20 > + *ofw_name_history_pos =3D '#'; /* HACK */ > + > ofw_sprg_prepare(); > =20 > + *ofw_name_history_pos =3D '$'; /* HACK */ > + > /* Save trap vectors */ > ofw_save_trap_vec(save_trap_of); > =20 > + *ofw_name_history_pos =3D '%'; /* HACK */ > + > /* Restore initially saved trap vectors */ > ofw_restore_trap_vec(save_trap_init); > =20 > + *ofw_name_history_pos =3D '^'; /* HACK */ > + > #if defined(AIM) && !defined(__powerpc64__) > /* > * Clear battable[] translations > @@ -357,13 +407,21 @@ > =20 > result =3D ofwcall(args); > =20 > + *ofw_name_history_pos =3D '&'; /* HACK */ > + > /* Restore trap vecotrs */ > ofw_restore_trap_vec(save_trap_of); > =20 > + *ofw_name_history_pos =3D '*'; /* HACK */ > + > ofw_sprg_restore(); > =20 > + *ofw_name_history_pos =3D '~'; /* HACK */ > + > intr_restore(oldmsr); > =20 > + *ofw_name_history_pos =3D '!'; /* HACK */ > + > return (result); > } >=20 >=20 >=20 >=20 >=20 > =3D=3D=3D > Mark Millard > markmi at dsl-only.net >=20