Date: Mon, 27 Jun 2016 02:28:41 -0700 From: Mark Millard <markmi@dsl-only.net> To: Nathan Whitehorn <nwhitehorn@freebsd.org>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: svn commit: r302214 - head/sys/powerpc/aim [my PowerMac G5 testing delayed] Message-ID: <A4F120A6-73B0-49F6-BBF5-509E4A1A85D8@dsl-only.net> In-Reply-To: <B2997643-4FB3-43AF-8942-61E6E4DB2ACF@dsl-only.net> References: <B2997643-4FB3-43AF-8942-61E6E4DB2ACF@dsl-only.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Nathan Whitehorn nwhitehorn at freebsd.org wrote on Sun Jun 26 23:38:36 = UTC 2016: > On 06/26/16 16:22, Mark Millard wrote: > > Quoting: > > > >> Author: nwhitehorn > >> Date: Sun Jun 26 18:43:42 2016 > >> New Revision: 302214 > >> URL: > >> https://svnweb.freebsd.org/changeset/base/302214 > >> > >> > >> Log: > >> Enter 64-bit mode as early as possible in the 64-bit PowerPC = boot sequence. > >> Most of the effect of setting MSR[SF] is that the CPU will stop = ignoring > >> the high 32 bits of registers containing addresses in load/store > >> instructions. As such, the kernel was setting it only when it = began to > >> need access to high memory. MSR[SF] also affects the operation = of some > >> conditional instructions, however, and so setting it at late = times could > >> subtly break code at very early times. This fixes use of the FDT = mode in > >> loader, and FDT boot more generally, on 64-bit PowerPC systems. > >> =20 > >> Hardware provided by: IBM LTC > >> Approved by: re (kib) > >> > >> Modified: > >> head/sys/powerpc/aim/aim_machdep.c > >> head/sys/powerpc/aim/locore64.S > > Usually I would test a couple of powerpc64 capable PowerMacs (a = so-called "Quad Core" one and a Dual Processor, single core per = processor, one) for if this changes the boot behavior noticeably for = them. But it will likely be a couple of months or more before I again = have access to them. Hopefully others will happen to test such examples = before then. > > > > > > Context note for PowerMacs used under powerpc64 FreeBSD: > > > > When I boot a powerpc64 PowerMac with a FreeBSD for powerpc64 I = normally run with a personal PowerMac specific change to allow reliable = booting, even when there is lots of RAM (8G, 12G, or 16G). The change is = tied to the transitions into and out of Apple's OpenFirmware and I = observed it to avoid memory trashing that I had observed otherwise. > > > > When I made this change I effectively eliminated just one = instruction from ofw_sprg_prepare(void): I went for the smallest = generated-code change that made the booting reliable on the PowerMac's = that I commonly have access to. > > > > The below just eliminates the pair: > > > > "mtsprg0 %1\n\t" > > and > > "r"(ofmsr[1]), >=20 > Those might be fixed with this change. Your patch (not restoring OF's=20= > SPRG0) will break a lot of Apple hardware. It's also not clear to me = why=20 > it would do anything: it replaces firmware's SPRG0 with FreeBSD's = SPRG0,=20 > trashing firmware state if it uses the register, in a situation=20 > (interrupts off) in which FreeBSD never uses SPRG0 either. As such, I=20= > would assume you are exposing another problem (maybe the one fixed in=20= > this commit). I'm nervous about the change because of an old memory about about the = initial instructions in Apple's openfirmware code. So I looked at at = some of my old messages, quoting one here: > the decoding of the first instructions for the openfirmware entry = point (0xff846d78 starting address on the G5 Quad-Core) is: >=20 > or r2,r0,r2, > addis r2,r0,-0x49 > ori r2,r2,0xf000 /* so %r2:=3D0xFFB7F000: fixed address (32-bit = mode). */ > std r1,r2,0x8, /* %r1 saved to have a special, separate copy = */ > std r0,r2,0x10, /* more saves to fixed locations */ > mfspr r0,lr > std r0,r2,0x120, /* more saves to fixed locations: return = address */ > mfmsr r1 > std r1,r2,0x108, /* more saves to fixed locations: msr */ > rldicl r1,r1,0,1 /* clears the most significant %r1 bit */ > 0x7c200164 (actually mtmsrd %r1) /* forces 32-bit mode */ > isync >=20 > Unfortunately for 64-bit mode at the start: %r2=3D0xFF...FFB7F000 and = std r1,r2,0x8, ends up rejecting the effective address. Apple does not = force 32-bit mode until a little later in the above. Thus ofwcall does = need to force 32-bit mode and return it back to normal, despite the msr = and other save/restore code that openfirmware has.=20 >=20 > (I've not explored trying to set up a mapping for the involved 64-bit = effective address range in order to allow the translation of the 64-bit = addresses in openfirmware's %r2 above. With that it might be that 64-bit = mode could be left in place.) >=20 > As far as I can tell this %r2 value and its use is the only reason = that FreeBSD needs to establish 32-bit mode before the call into = openfirmware. >=20 > %r2 needs to be restored after openfirmware returns since openfirmware = is using the Darwin ABI where %r2 is non-volatile and non-dedicated (no = use of TOCs), as can be seen above. (Apple's openfirmware uses Darwin's rule that %r2 is = volatile/non-dedicated --no TOC use-- and the above replaces r2's = value.) Now I may have been wrong back then about this code and its = implications. But at the moment I do not see any error in the above = quote. (Not that I have any reference materials with me beyond that = recorded in the list history. So I'm not that likely to find anything = wrong.) [If openfirmware aliases appropriate address ranges to the same memory = that might sidestep the above.] It sounds like a good experiment might be to compare and contrast = without and with my SPRG0 change after -r320214. I'd love to boot = reliably from a standard FeeBSD build. I fully agree that my change effectively switches to using FreeBSD's = SPRG0 value. It would be interesting to learn of an example type of PowerMac G5 = confirmed to fail someway with my hack present, at least if the failure = was different compared to normal FreeBSD code. Even more interesting = would be if the boot failures are repeatable on some PowerMac G5 type = when my change is present. (But I'd not have such a distinct type of G5 = to test with so I'd not be able to do much with the information.) [It is also true that in the two types of PowerMac G5's that I use I = also use only a limited range of devices: I only use the machine, RAM, = SATA SSDs, USB flash drives, Firewire hard drives, the internal Super = Drives, the video cards (a couple types of video cards, all of which = have worked), and the Ethernet ports. I only have access to the two = types of G5's.] > One thing it would be great to have some testing on after this change = is=20 > the FDT layer in loader. If you set usefdt=3D1 from the loader prompt,=20= > loader will distill the OF device tree into an FDT and then stop Open=20= > Firmware completely before transferring control to FreeBSD. This = should=20 > avoid any possible problems accessing Open Firmware from the kernel, = as=20 > well as making boot a little faster. > -Nathan Hopefully I'll remember this once I have access again. Feel free to send = me a reminder in a couple of months if you think about it. Side notes. . . Details related to FreeBSD's sprg0 use and my discoveries (other than = the code that I removed the assignment from): The evidence that I got for the crashes were tied to observed trashing = of FreeBSD memory as shown by being dumped out by DDB and detectably = incorrect %r1/%r3 values after open-firmware returned: %r1 should have = had no net change and %r3 has limited valid values. But the odd result = was: %r1=3D%r3=3D%r2+0x400, with %r2 having the before openfirware value = that is preserved to the after openfirmware context. As FreeBSD moved = things around in RAM in updates [such as by growing] the trashed area = moved around as well. The specific observed trashed areas for each = specific version that I tested stopped being trashed with the change and = the register values quit being bad as well. So something was using the only value that I changed (SPRG0) and also = touching both memory and the 2 registers with known correctness = properties that were being violated. [Note: I discovered the register value problems before I discovered the = memory trashing. I discovered the memory trashing before I discovered = the change in behavior based on SPRG0's value. That last was discovered = by cutting down a much larger change to powerpc64 specific FreeBSD code = that made things work as far as I could observe: it was the only part of = the bigger change that mattered to the behavior in question. Material = from all these stages are visible in the list history. But it s a mess = to go through my explorations.] Unfortunately I've got no good way to find the SPRG0 usage that might be = in Apple's OpenFirmware G5's and I'm not aware of any theory of = operation information available for Apple's implementations that could = be used to figure anything out about its SPRG0 use. [I sometimes wonder if interrupts are reenabled outside FreeBSD's = control sometimes during open-firmware code.] So my evidence is only the lack of the otherwise-observed trashing and = the lack of problems being noticed in the time that I've used my change. = I wish that I had better information. So far for builds with my change I've no evidence of any odd problems: = historically mostly 10.x STABLE starting when I isolated the change and = more recently mostly 11.0, many vintages of each tried. No standard = FreeBSD build that I've tried has booted the G5's reliably over the same = period and any time I've checked I've found the memory trashing and = register value oddities. (Not that I check such details for standard = builds often any more.) The sprg0 usage in FreeBSD based on "find sys -exec grep -i sprg0 {} \; = -print | more" and then looking at some of what it listed: #define GET_CPUINFO(r) \ mfsprg0 r . . . #define FRAME_SETUP(savearea) = \ /* Have to enable translation to allow access of kernel stack: */ = \ GET_CPUINFO(%r31); = \ mfsrr0 %r30; = \ std %r30,(savearea+CPUSAVE_SRR0)(%r31); /* save SRR0 */ = \ mfsrr1 %r30; = \ std %r30,(savearea+CPUSAVE_SRR1)(%r31); /* save SRR1 */ = \ . . . GET_CPUINFO(%r2); = \ ld %r27,(savearea+CPUSAVE_R27)(%r2); /* get saved r27 */ = \ ld %r28,(savearea+CPUSAVE_R28)(%r2); /* get saved r28 */ = \ ld %r29,(savearea+CPUSAVE_R29)(%r2); /* get saved r29 */ = \ ld %r30,(savearea+CPUSAVE_R30)(%r2); /* get saved r30 */ = \ ld %r31,(savearea+CPUSAVE_R31)(%r2); /* get saved r31 */ = \ . . . (from sys/powerpc/aim/trap_subr64.S) (The GET_CPUINFO macro has a lot of references. FreeBSD does store and = recall based on offsets from SPRG0 values, just in a very specific type = of context. I was never able to tell if this code was involved in the = memory trash or not.) sprgs[0] =3D mfspr(SPR_SPRG0); mtspr(SPR_SPRG0, sprgs[0]); (from sys/powerpc/aim/aim_machdep.c) #define SPR_USPRG0 0x100 /* 4.. User SPR General 0 */ #define SPR_SPRG0 0x110 /* 468 SPR General 0 */ (from sys/powerpc/include/spr.h) =3D=3D=3D Mark Millard markmi@dsl-only.net
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A4F120A6-73B0-49F6-BBF5-509E4A1A85D8>