Date: Mon, 27 Jun 2016 02:28:41 -0700 From: Mark Millard <markmi@dsl-only.net> To: Nathan Whitehorn <nwhitehorn@freebsd.org>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org> Subject: Re: svn commit: r302214 - head/sys/powerpc/aim [my PowerMac G5 testing delayed] Message-ID: <A4F120A6-73B0-49F6-BBF5-509E4A1A85D8@dsl-only.net> In-Reply-To: <B2997643-4FB3-43AF-8942-61E6E4DB2ACF@dsl-only.net> References: <B2997643-4FB3-43AF-8942-61E6E4DB2ACF@dsl-only.net>
index | next in thread | previous in thread | raw e-mail
Nathan Whitehorn nwhitehorn at freebsd.org wrote on Sun Jun 26 23:38:36 UTC 2016: > On 06/26/16 16:22, Mark Millard wrote: > > Quoting: > > > >> Author: nwhitehorn > >> Date: Sun Jun 26 18:43:42 2016 > >> New Revision: 302214 > >> URL: > >> https://svnweb.freebsd.org/changeset/base/302214 > >> > >> > >> Log: > >> Enter 64-bit mode as early as possible in the 64-bit PowerPC boot sequence. > >> Most of the effect of setting MSR[SF] is that the CPU will stop ignoring > >> the high 32 bits of registers containing addresses in load/store > >> instructions. As such, the kernel was setting it only when it began to > >> need access to high memory. MSR[SF] also affects the operation of some > >> conditional instructions, however, and so setting it at late times could > >> subtly break code at very early times. This fixes use of the FDT mode in > >> loader, and FDT boot more generally, on 64-bit PowerPC systems. > >> > >> Hardware provided by: IBM LTC > >> Approved by: re (kib) > >> > >> Modified: > >> head/sys/powerpc/aim/aim_machdep.c > >> head/sys/powerpc/aim/locore64.S > > Usually I would test a couple of powerpc64 capable PowerMacs (a so-called "Quad Core" one and a Dual Processor, single core per processor, one) for if this changes the boot behavior noticeably for them. But it will likely be a couple of months or more before I again have access to them. Hopefully others will happen to test such examples before then. > > > > > > Context note for PowerMacs used under powerpc64 FreeBSD: > > > > When I boot a powerpc64 PowerMac with a FreeBSD for powerpc64 I normally run with a personal PowerMac specific change to allow reliable booting, even when there is lots of RAM (8G, 12G, or 16G). The change is tied to the transitions into and out of Apple's OpenFirmware and I observed it to avoid memory trashing that I had observed otherwise. > > > > When I made this change I effectively eliminated just one instruction from ofw_sprg_prepare(void): I went for the smallest generated-code change that made the booting reliable on the PowerMac's that I commonly have access to. > > > > The below just eliminates the pair: > > > > "mtsprg0 %1\n\t" > > and > > "r"(ofmsr[1]), > > Those might be fixed with this change. Your patch (not restoring OF's > SPRG0) will break a lot of Apple hardware. It's also not clear to me why > it would do anything: it replaces firmware's SPRG0 with FreeBSD's SPRG0, > trashing firmware state if it uses the register, in a situation > (interrupts off) in which FreeBSD never uses SPRG0 either. As such, I > would assume you are exposing another problem (maybe the one fixed in > this commit). I'm nervous about the change because of an old memory about about the initial instructions in Apple's openfirmware code. So I looked at at some of my old messages, quoting one here: > the decoding of the first instructions for the openfirmware entry point (0xff846d78 starting address on the G5 Quad-Core) is: > > or r2,r0,r2, > addis r2,r0,-0x49 > ori r2,r2,0xf000 /* so %r2:=0xFFB7F000: fixed address (32-bit mode). */ > std r1,r2,0x8, /* %r1 saved to have a special, separate copy */ > std r0,r2,0x10, /* more saves to fixed locations */ > mfspr r0,lr > std r0,r2,0x120, /* more saves to fixed locations: return address */ > mfmsr r1 > std r1,r2,0x108, /* more saves to fixed locations: msr */ > rldicl r1,r1,0,1 /* clears the most significant %r1 bit */ > 0x7c200164 (actually mtmsrd %r1) /* forces 32-bit mode */ > isync > > Unfortunately for 64-bit mode at the start: %r2=0xFF...FFB7F000 and std r1,r2,0x8, ends up rejecting the effective address. Apple does not force 32-bit mode until a little later in the above. Thus ofwcall does need to force 32-bit mode and return it back to normal, despite the msr and other save/restore code that openfirmware has. > > (I've not explored trying to set up a mapping for the involved 64-bit effective address range in order to allow the translation of the 64-bit addresses in openfirmware's %r2 above. With that it might be that 64-bit mode could be left in place.) > > As far as I can tell this %r2 value and its use is the only reason that FreeBSD needs to establish 32-bit mode before the call into openfirmware. > > %r2 needs to be restored after openfirmware returns since openfirmware is using the Darwin ABI where %r2 is non-volatile and non-dedicated (no use of TOCs), as can be seen above. (Apple's openfirmware uses Darwin's rule that %r2 is volatile/non-dedicated --no TOC use-- and the above replaces r2's value.) Now I may have been wrong back then about this code and its implications. But at the moment I do not see any error in the above quote. (Not that I have any reference materials with me beyond that recorded in the list history. So I'm not that likely to find anything wrong.) [If openfirmware aliases appropriate address ranges to the same memory that might sidestep the above.] It sounds like a good experiment might be to compare and contrast without and with my SPRG0 change after -r320214. I'd love to boot reliably from a standard FeeBSD build. I fully agree that my change effectively switches to using FreeBSD's SPRG0 value. It would be interesting to learn of an example type of PowerMac G5 confirmed to fail someway with my hack present, at least if the failure was different compared to normal FreeBSD code. Even more interesting would be if the boot failures are repeatable on some PowerMac G5 type when my change is present. (But I'd not have such a distinct type of G5 to test with so I'd not be able to do much with the information.) [It is also true that in the two types of PowerMac G5's that I use I also use only a limited range of devices: I only use the machine, RAM, SATA SSDs, USB flash drives, Firewire hard drives, the internal Super Drives, the video cards (a couple types of video cards, all of which have worked), and the Ethernet ports. I only have access to the two types of G5's.] > One thing it would be great to have some testing on after this change is > the FDT layer in loader. If you set usefdt=1 from the loader prompt, > loader will distill the OF device tree into an FDT and then stop Open > Firmware completely before transferring control to FreeBSD. This should > avoid any possible problems accessing Open Firmware from the kernel, as > well as making boot a little faster. > -Nathan Hopefully I'll remember this once I have access again. Feel free to send me a reminder in a couple of months if you think about it. Side notes. . . Details related to FreeBSD's sprg0 use and my discoveries (other than the code that I removed the assignment from): The evidence that I got for the crashes were tied to observed trashing of FreeBSD memory as shown by being dumped out by DDB and detectably incorrect %r1/%r3 values after open-firmware returned: %r1 should have had no net change and %r3 has limited valid values. But the odd result was: %r1=%r3=%r2+0x400, with %r2 having the before openfirware value that is preserved to the after openfirmware context. As FreeBSD moved things around in RAM in updates [such as by growing] the trashed area moved around as well. The specific observed trashed areas for each specific version that I tested stopped being trashed with the change and the register values quit being bad as well. So something was using the only value that I changed (SPRG0) and also touching both memory and the 2 registers with known correctness properties that were being violated. [Note: I discovered the register value problems before I discovered the memory trashing. I discovered the memory trashing before I discovered the change in behavior based on SPRG0's value. That last was discovered by cutting down a much larger change to powerpc64 specific FreeBSD code that made things work as far as I could observe: it was the only part of the bigger change that mattered to the behavior in question. Material from all these stages are visible in the list history. But it s a mess to go through my explorations.] Unfortunately I've got no good way to find the SPRG0 usage that might be in Apple's OpenFirmware G5's and I'm not aware of any theory of operation information available for Apple's implementations that could be used to figure anything out about its SPRG0 use. [I sometimes wonder if interrupts are reenabled outside FreeBSD's control sometimes during open-firmware code.] So my evidence is only the lack of the otherwise-observed trashing and the lack of problems being noticed in the time that I've used my change. I wish that I had better information. So far for builds with my change I've no evidence of any odd problems: historically mostly 10.x STABLE starting when I isolated the change and more recently mostly 11.0, many vintages of each tried. No standard FreeBSD build that I've tried has booted the G5's reliably over the same period and any time I've checked I've found the memory trashing and register value oddities. (Not that I check such details for standard builds often any more.) The sprg0 usage in FreeBSD based on "find sys -exec grep -i sprg0 {} \; -print | more" and then looking at some of what it listed: #define GET_CPUINFO(r) \ mfsprg0 r . . . #define FRAME_SETUP(savearea) \ /* Have to enable translation to allow access of kernel stack: */ \ GET_CPUINFO(%r31); \ mfsrr0 %r30; \ std %r30,(savearea+CPUSAVE_SRR0)(%r31); /* save SRR0 */ \ mfsrr1 %r30; \ std %r30,(savearea+CPUSAVE_SRR1)(%r31); /* save SRR1 */ \ . . . GET_CPUINFO(%r2); \ ld %r27,(savearea+CPUSAVE_R27)(%r2); /* get saved r27 */ \ ld %r28,(savearea+CPUSAVE_R28)(%r2); /* get saved r28 */ \ ld %r29,(savearea+CPUSAVE_R29)(%r2); /* get saved r29 */ \ ld %r30,(savearea+CPUSAVE_R30)(%r2); /* get saved r30 */ \ ld %r31,(savearea+CPUSAVE_R31)(%r2); /* get saved r31 */ \ . . . (from sys/powerpc/aim/trap_subr64.S) (The GET_CPUINFO macro has a lot of references. FreeBSD does store and recall based on offsets from SPRG0 values, just in a very specific type of context. I was never able to tell if this code was involved in the memory trash or not.) sprgs[0] = mfspr(SPR_SPRG0); mtspr(SPR_SPRG0, sprgs[0]); (from sys/powerpc/aim/aim_machdep.c) #define SPR_USPRG0 0x100 /* 4.. User SPR General 0 */ #define SPR_SPRG0 0x110 /* 468 SPR General 0 */ (from sys/powerpc/include/spr.h) === Mark Millard markmi@dsl-only.nethelp
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A4F120A6-73B0-49F6-BBF5-509E4A1A85D8>