Date: Tue, 15 Apr 2008 12:58:18 -0500 From: Nathan Whitehorn <nathanw@uchicago.edu> To: grehan@freebsd.org Cc: freebsd-ppc@freebsd.org Subject: Re: G5 Bridge-mode MMU Message-ID: <4804ECBA.5030905@uchicago.edu> In-Reply-To: <4804C9E9.6010303@freebsd.org> References: <4804AE13.2060600@uchicago.edu> <4804C9E9.6010303@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Peter Grehan wrote: > Hi Nathan, > > Great to see you taking on this work ! > >> I'm trying to get G5 support going, and have run into an interesting >> problem while bootstrapping the MMU. Firmware loads the kernel in >> protected mode, and we have no BAT facility. Thus, the bootstrap >> allocator fails because low physical memory isn't 1:1 mapped. And we >> can't add this region to the page table, because the page table is >> what we are trying to allocate. And we can't even use the MMU's large >> page facility in bridge mode. > > You really can't use the existing MMU code. The G5 was the driving > factor to create the pmap indirection. Yeah, the bridge mode MMU is just different enough to be annoying. I'm using the existing code as a base, because (a) it's nice to have a template, and (b) most of the PTE handling stuff can be reused with s/struct pte/struct lpte and s/PTE/LPTE, and some changes to the hash function and PTE_EXEC bits. > My thoughts were to abandon the existing method of trying to keep the > OFW mappings the same as the kernel mappings, and completely switch > address spaces when calling back into OFW, rather than just changing > segment registers. This would require an indirection in the OFW code > to jumpto to a routine that switches stacks, but code similar to that > already exists in NetBSD for older G3's where OFW runs in real-mode. So you suggest saving OF's SDR1 and restoring it when we need to call in? I guess this would solve part of the problem (debugging output while initializing the MMU), but I'm sure sure I understand the real benefit to this. >> One solution is to drop back to data real mode while the page table >> is allocated and set up. This seems to work ok, but requires that the >> kernel stack is mapped 1:1, which seems to be the case on my iMac G5. >> I'm not sure how reliable this assumption is, though. It's also >> irritating because any open firmware calls from real mode (e.g. >> printf()) make the machine hang up. > > It's a good enough assumption to make in pmap_bootstrap(). Calls to > OFW usually copy the parameters into BSS, though there are some that > don't. OK. So it should be generally true that all the local variables and statically allocated things in pmap_bootstrap() should have virtual addresses equal to their physical address? If that is true then a brief trip to real mode seems to be the best option. >> The other solution is to prepare a temporary statically-allocated >> page table with enough mappings to map the kernel, OF, and the real >> page table, then switch to the dynamically allocated one. > > I think you want to allocate the real page table, put enough mappings > in there for the kernel text+data+bss (and the frame buffer if that is > being used for the console) and the page table itself, and modify OFW > so that there is a big context switch when calling into it. Use the > existing call into OFW to get it's mappings to make sure those pages > are excluded from kernel use. This isn't the real problem. The real problem is that we don't have the 1:1 BAT mapping, so you can't allocate memory (e.g. for the page table) without putting entries for it in the page table. Which you don't have yet. So you can either have a little, static page table that you can set up first or drop to real mode whenever you need to use dynamically allocated memory before the MMU is up. In the first case, you need to put 256 KB of page table on the stack, or globally allocate it, which means you've wasted a big chunk of memory that you will never need again and I'm not even sure there is that much stack available. In the second, if the rest of the kernel isn't 1:1 mapped, everything will instantly break. Also, going to real mode seems like kind of a hack, but I think it may be the best choice. It is also what OS X does. > There are some other gotchas that you may know about: > > - the OFW calls to determine the amount of memory don't use two cells > for addresses, required on a G5. That's a simple fix (which you might > already have :). You might wonder why I did that work for the loader > and never submitted it for the kernel :) Memory should be restricted > to <= 4G. There's no need for a PAE-style mode to get more memory. > Instead, that should be the target of a true 64-bit PPC port. Yeah, I put that in already, and not planning on going anywhere near a PAE-type mechanism. > - interrupts switch the CPU to 64-bit mode. You have to immediately > drop to 32-bit mode in the interrupt handler before any branch is > taken. My plan was to have a macro in the exception asm that did this, > and also expanded to 'rfid' for the return. The asm would then be > included into trap_subr.S and expanded to 32/64-bit, in a similar > fashion to how the kernel does elf32 and elf64 via macro expansion. > Installing the vectors would also then become CPU-model specific, but > that was inevitable to support other CPU models with different vector > requirements. This is kind of annoying. I haven't really thought about how best to handle it. Something to handle once the MMU is set up. > - the G3/4 pmap code relies on 1:1 BAT mapping for page-zeroing. The > G5 code will have to use an i386-style temporary mapping for this. Right. It looks like /dev/mem needs the BAT mapping too, along with a few other things. > - another 1:1 relic is the use of UMA_MD_SMALL_ALLOC. I talked with > Alan Cox a long time back about being able to dynamically determine > whether this could be used, though that is a fair amount of work. I > would like to see a single kernel for G3/4/5, but that may be asking > too much up front, especially given my lack of progress in achieving > this goal :) Well, once we have the page table set up, we can add a 1:1 mapping in certain places -- e.g. low memory -- by adding in a whole bunch of 4 KB pages. > - the G5 uses two cascaded OpenPIC interrupt controllers. The > interrupt code needs to be reworked to support this concept. This also goes on the list of things to worry about after I've seen the FreeBSD copyright notice... -Nathan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4804ECBA.5030905>