Date: Wed, 4 Aug 2010 01:14:42 +0530 From: "Jayachandran C." <c.jayachandran@gmail.com> To: Juli Mallett <jmallett@freebsd.org>, "M. Warner Losh" <imp@bsdimp.com>, freebsd-mips@freebsd.org Subject: Merged pmap.c - Re: Support for 64bit userspace. Message-ID: <AANLkTimOr-JGU_jxFZ5QP2QSM-jWKJDXRc7O3ZTaoALX@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
On Sat, Jul 31, 2010 at 5:17 AM, Jayachandran C.
<c.jayachandran@gmail.com>wrote:
> On Fri, Jul 30, 2010 at 2:07 PM, Juli Mallett <jmallett@freebsd.org>wrote:
>
>> On Fri, Jul 30, 2010 at 01:15, Jayachandran C. <c.jayachandran@gmail.com>
>> wrote:
>> > I will look at merging them without too many ifdefs. Another
>> > possibility is to have a pmap.c, pmap32.c and pmap64.c, so that the
>> > common code is not duplicated, but that too (as far as I can see) will
>> > require a config option.
>>
>> I'm not really sure what the aversion to ifdefs is — they kind of come
>> with the territory and this is exactly the part of the system where
>> you expect to have them to conditionalize exactly this kind of
>> behavior. If you were going to split out some subroutines for 32-bit
>> and 64-bit, there are two ways you can do it without a config option
>> (and I agree with you that avoiding one is very desirable — we already
>> use too many knobs to get the right ABI in kernel configs). You could
>> do:
>>
>> #ifndef __mips_n64
>> #include "pmap32.c"
>> #else
>> #include "pmap64.c"
>> #endif
>>
>> Or you could wrap the whole 32/64 files (modulo a "struct __hack;" or
>> whatever to avoid an empty translation unit) with ifdefs.
>>
>> Personally, I find having two implementations of a function a lot
>> harder to read and keep track of than a single implementation with
>> ifdefs in the body — there's very few places in the kernel where we do
>> the former. If you do split out functions into separate files I would
>> beg you to only split out the really big functions that are actually
>> doing things based on different levels of page tables and then it's
>> obvious why those things have two implementations and what needs to be
>> abstracted to merge them. Having two copies of trivial functions or
>> non-trivial functions that are mostly identical is a bad thing.
>>
>> You're doing the work, so at some point it's kind of your call, but I
>> feel like you're making style changes to things that I cleaned up and
>> tried to unify in pmap.c and that the way that they're done is pretty
>> significant given how intimidating people find pmap. If we were going
>> to support different kinds of MMUs (e.g. adding support for BAT-based
>> MMUs) I'd be in favor of splitting things up, but then I'd want to
>> follow PowerPC's lead. For now I think the appropriate thing to do is
>> to follow the lead of my 64-bit work in the MIPS pmap and the PAE work
>> in the i386 pmap and to use ifdefs and to try to abstract out the
>> page-table differences.
>>
>
> I had not looked at merging the files so far (mostly because it was easier
> to do it this way), so I will see how best this can be done.
>
> I'm really thrilled with the work you're doing and can't wait for it
>> to be in head — I'm quite eager to move everything on Octeon to n64 :)
>>
>
> I have checked in everything except the page table implementation code.
>
> If you like to try the current code, the attached patch has the rest of the
> changes. This has a few more issues fixed, n64 mostly works for me, except
> a csh crash I'm looking at.
>
> I'll have a look at merging pmap.c with pmap64.c, and have another look at
> the PAE.
>
Here's is the merged version of n64 changes. I had to change the existing
pmap.c style a bit, so that merging the 32 bit and 64 bit versions was
easier.
On the n64 compilation, I have done a few more fixes, and the current code
comes up multi-user and holds up pretty well on XLR.
I'm doing some stress testing on o32 to see that I have not broken anything
there during the changes.
Let me know your comments. Also note that PTEs are still 32 bit and KSEG0
addresses are still used for page table pages, but these can be added...
Thanks,
JC.
[-- Attachment #2 --]
Index: sys/mips/include/param.h
===================================================================
--- sys/mips/include/param.h (revision 210796)
+++ sys/mips/include/param.h (working copy)
@@ -107,8 +107,18 @@
#define NPTEPG (PAGE_SIZE/(sizeof (pt_entry_t)))
#define NPDEPG (PAGE_SIZE/(sizeof (pd_entry_t)))
+#if defined(__mips_n64)
+#define SEGSHIFT 31 /* LOG2(NBSEG) */
+#define NBSEG (1ul << SEGSHIFT) /* bytes/segment */
+#define PDRSHIFT 22 /* second level */
+#define PDRMASK ((1 << PDRSHIFT) - 1)
+#else
#define SEGSHIFT 22 /* LOG2(NBSEG) */
#define NBSEG (1 << SEGSHIFT) /* bytes/segment */
+#define PDRSHIFT SEGSHIFT /* alias for SEG in 32 bit */
+#define PDRMASK ((1 << PDRSHIFT) - 1)
+#endif
+#define NBPDR (1 << PDRSHIFT) /* bytes/pagedir */
#define SEGMASK (NBSEG-1) /* byte offset into segment */
#define MAXPAGESIZES 1 /* maximum number of supported page sizes */
@@ -119,7 +129,7 @@
/*
* The kernel stack needs to be aligned on a (PAGE_SIZE * 2) boundary.
*/
-#define KSTACK_PAGES 2 /* kernel stack*/
+#define KSTACK_PAGES 2 /* kernel stack */
#define KSTACK_GUARD_PAGES 2 /* pages of kstack guard; 0 disables */
#define UPAGES 2
Index: sys/mips/include/vmparam.h
===================================================================
--- sys/mips/include/vmparam.h (revision 210796)
+++ sys/mips/include/vmparam.h (working copy)
@@ -185,7 +185,7 @@
* allocations use HIGHMEM if available, and then DEFAULT.
* - HIGHMEM for other pages
*/
-#ifdef __mips_n64
+#if 0 /* Not yet, change n64 to use xkphys */
#define VM_NFREELIST 1
#define VM_FREELIST_DEFAULT 0
#define VM_FREELIST_DIRECT VM_FREELIST_DEFAULT
Index: sys/mips/mips/exception.S
===================================================================
--- sys/mips/mips/exception.S (revision 210796)
+++ sys/mips/mips/exception.S (working copy)
@@ -137,7 +137,15 @@
PTR_L k1, 0(k1) #08: k1=seg entry
MFC0 k0, MIPS_COP_0_BAD_VADDR #09: k0=bad address (again)
beq k1, zero, 2f #0a: ==0 -- no page table
- srl k0, PAGE_SHIFT - 2 #0b: k0=VPN (aka va>>10)
+#ifdef __mips_n64
+ PTR_SRL k0, PDRSHIFT - PTRSHIFT # k0=VPN
+ andi k0, k0, PTRMASK # k0=pde offset
+ PTR_ADDU k1, k0, k1 # k1=pde entry address
+ PTR_L k1, 0(k1) # k1=pde entry
+ MFC0 k0, MIPS_COP_0_BAD_VADDR # k0=bad address (again)
+ beq k1, zero, 2f # ==0 -- no page table
+#endif
+ PTR_SRL k0, PAGE_SHIFT - 2 #0b: k0=VPN (aka va>>10)
andi k0, k0, 0xff8 #0c: k0=page tab offset
PTR_ADDU k1, k1, k0 #0d: k1=pte address
lw k0, 0(k1) #0e: k0=lo0 pte
@@ -836,6 +844,18 @@
beqz k1, 3f
nop
+#ifdef __mips_n64
+ MFC0 k0, MIPS_COP_0_BAD_VADDR
+ PTR_SRL k0, PDRSHIFT - PTRSHIFT # k0=pde offset (almost)
+ beq k1, zero, MipsKernGenException # ==0 -- no pde tab
+ andi k0, k0, PTRMASK # k0=pde offset
+ PTR_ADDU k1, k0, k1 # k1=pde entry address
+ PTR_L k1, 0(k1) # k1=pde entry
+
+ /* Validate pde table pointer. */
+ beqz k1, 3f
+ nop
+#endif
MFC0 k0, MIPS_COP_0_BAD_VADDR # k0=bad address (again)
PTR_SRL k0, PAGE_SHIFT - 2 # k0=VPN
andi k0, k0, 0xffc # k0=page tab offset
@@ -996,6 +1016,14 @@
PTR_L k1, 0(k1) # k1=seg entry
MFC0 k0, MIPS_COP_0_BAD_VADDR # k0=bad address (again)
beq k1, zero, MipsKernGenException # ==0 -- no page table
+#ifdef __mips_n64
+ PTR_SRL k0, PDRSHIFT - PTRSHIFT # k0=VPN
+ andi k0, k0, PTRMASK # k0=pde offset
+ PTR_ADDU k1, k0, k1 # k1=pde entry address
+ PTR_L k1, 0(k1) # k1=pde entry
+ MFC0 k0, MIPS_COP_0_BAD_VADDR # k0=bad address (again)
+ beq k1, zero, MipsKernGenException # ==0 -- no page table
+#endif
PTR_SRL k0, PAGE_SHIFT - 2 # k0=VPN
andi k0, k0, 0xff8 # k0=page tab offset
PTR_ADDU k1, k1, k0 # k1=pte address
Index: sys/mips/mips/genassym.c
===================================================================
--- sys/mips/mips/genassym.c (revision 210796)
+++ sys/mips/mips/genassym.c (working copy)
@@ -93,6 +93,7 @@
ASSYM(PAGE_SHIFT, PAGE_SHIFT);
ASSYM(PAGE_SIZE, PAGE_SIZE);
ASSYM(PAGE_MASK, PAGE_MASK);
+ASSYM(PDRSHIFT, PDRSHIFT);
ASSYM(SEGSHIFT, SEGSHIFT);
ASSYM(NPTEPG, NPTEPG);
ASSYM(TDF_NEEDRESCHED, TDF_NEEDRESCHED);
Index: sys/mips/mips/pmap.c
===================================================================
--- sys/mips/mips/pmap.c (revision 210796)
+++ sys/mips/mips/pmap.c (working copy)
@@ -69,6 +69,8 @@
__FBSDID("$FreeBSD$");
#include "opt_msgbuf.h"
+#include "opt_ddb.h"
+
#include <sys/param.h>
#include <sys/systm.h>
#include <sys/proc.h>
@@ -76,6 +78,9 @@
#include <sys/vmmeter.h>
#include <sys/mman.h>
#include <sys/smp.h>
+#ifdef DDB
+#include <ddb/ddb.h>
+#endif
#include <vm/vm.h>
#include <vm/vm_param.h>
@@ -125,23 +130,21 @@
* is defined such that it ends immediately after NPDEPG*NPTEPG*PAGE_SIZE,
* so we end up getting NUSERPGTBLS of 0.
*/
-#define pmap_segshift(v) (((v) >> SEGSHIFT) & (NPDEPG - 1))
-#define segtab_pde(m, v) ((m)[pmap_segshift((v))])
+#define pmap_seg_index(v) (((v) >> SEGSHIFT) & (NPDEPG - 1))
+#define pmap_pde_index(v) (((v) >> PDRSHIFT) & (NPDEPG - 1))
+#define pmap_pte_index(v) (((v) >> PAGE_SHIFT) & (NPTEPG - 1))
+#define pmap_pde_pindex(v) ((v) >> PDRSHIFT) /* XXX TODO */
-#if defined(__mips_n64)
-#define NUSERPGTBLS (NPDEPG)
+#ifdef __mips_n64
+#define NUPDE (NPDEPG * NPDEPG)
+#define NUSERPGTBLS (NUPDE + NPDEPG)
#else
-#define NUSERPGTBLS (pmap_segshift(VM_MAXUSER_ADDRESS))
+#define NUPDE (NPDEPG)
+#define NUSERPGTBLS (NUPDE)
#endif
-#define mips_segtrunc(va) ((va) & ~SEGMASK)
+
#define is_kernel_pmap(x) ((x) == kernel_pmap)
-/*
- * Given a virtual address, get the offset of its PTE within its page
- * directory page.
- */
-#define PDE_OFFSET(va) (((vm_offset_t)(va) >> PAGE_SHIFT) & (NPTEPG - 1))
-
struct pmap kernel_pmap_store;
pd_entry_t *kernel_segmap;
@@ -151,10 +154,9 @@
static int nkpt;
unsigned pmap_max_asid; /* max ASID supported by the system */
-
#define PMAP_ASID_RESERVED 0
-vm_offset_t kernel_vm_end;
+vm_offset_t kernel_vm_end = VM_MIN_KERNEL_ADDRESS;
static void pmap_asid_alloc(pmap_t pmap);
@@ -179,11 +181,10 @@
static void pmap_remove_entry(struct pmap *pmap, vm_page_t m, vm_offset_t va);
static boolean_t pmap_try_insert_pv_entry(pmap_t pmap, vm_page_t mpte,
vm_offset_t va, vm_page_t m);
-static __inline void
-pmap_invalidate_page(pmap_t pmap, vm_offset_t va);
+static __inline void pmap_invalidate_page(pmap_t pmap, vm_offset_t va);
+static int _pmap_unwire_pte_hold(pmap_t pmap, vm_offset_t va, vm_page_t m);
static vm_page_t pmap_allocpte(pmap_t pmap, vm_offset_t va, int flags);
-
static vm_page_t _pmap_allocpte(pmap_t pmap, unsigned ptepindex, int flags);
static int pmap_unuse_pt(pmap_t, vm_offset_t, vm_page_t);
static int init_pte_prot(vm_offset_t va, vm_page_t m, vm_prot_t prot);
@@ -259,36 +260,71 @@
intr_restore(intr)
#endif
-static inline pt_entry_t *
+/*
+ * Page table entry lookup routines.
+ */
+static __inline pd_entry_t *
pmap_segmap(pmap_t pmap, vm_offset_t va)
{
- if (pmap->pm_segtab != NULL)
- return (segtab_pde(pmap->pm_segtab, va));
- else
+ return (&pmap->pm_segtab[pmap_seg_index(va)]);
+}
+
+#ifdef __mips_n64
+static __inline pd_entry_t *
+pmap_pdpe_to_pde(pd_entry_t *pdpe, vm_offset_t va)
+{
+ pd_entry_t *pde;
+
+ pde = (pd_entry_t *)*pdpe;
+ return (&pde[pmap_pde_index(va)]);
+}
+
+static __inline pd_entry_t *
+pmap_pde(pmap_t pmap, vm_offset_t va)
+{
+ pd_entry_t *pdpe;
+
+ pdpe = pmap_segmap(pmap, va);
+ if (pdpe == NULL || *pdpe == NULL)
return (NULL);
+
+ return (pmap_pdpe_to_pde(pdpe, va));
}
+#else
+static __inline pd_entry_t *
+pmap_pdpe_to_pde(pd_entry_t *pdpe, vm_offset_t va)
+{
+ return pdpe;
+}
-/*
- * Routine: pmap_pte
- * Function:
- * Extract the page table entry associated
- * with the given map/virtual_address pair.
- */
+static __inline
+pd_entry_t *pmap_pde(pmap_t pmap, vm_offset_t va)
+{
+ return pmap_segmap(pmap, va);
+}
+#endif
+
+static __inline pt_entry_t *
+pmap_pde_to_pte(pd_entry_t *pde, vm_offset_t va)
+{
+ pt_entry_t *pte;
+
+ pte = (pt_entry_t *)*pde;
+ return (&pte[pmap_pte_index(va)]);
+}
+
pt_entry_t *
pmap_pte(pmap_t pmap, vm_offset_t va)
{
- pt_entry_t *pdeaddr;
+ pd_entry_t *pde;
- if (pmap) {
- pdeaddr = pmap_segmap(pmap, va);
- if (pdeaddr) {
- return pdeaddr + PDE_OFFSET(va);
- }
- }
- return ((pt_entry_t *)0);
+ pde = pmap_pde(pmap, va);
+ if (pde == NULL || *pde == NULL)
+ return (NULL);
+
+ return (pmap_pde_to_pte(pde, va));
}
-
vm_offset_t
pmap_steal_memory(vm_size_t size)
{
@@ -326,12 +362,69 @@
* Bootstrap the system enough to run with virtual memory. This
* assumes that the phys_avail array has been initialized.
*/
+static void
+pmap_create_kernel_pagetable(void)
+{
+ int i, j;
+ vm_offset_t ptaddr;
+ pt_entry_t *pte;
+#ifdef __mips_n64
+ pd_entry_t *pde;
+ vm_offset_t pdaddr;
+ int npt, npde;
+#endif
+
+ /*
+ * Allocate segment table for the kernel
+ */
+ kernel_segmap = (pd_entry_t *)pmap_steal_memory(PAGE_SIZE);
+
+ /*
+ * Allocate second level page tables for the kernel
+ */
+#ifdef __mips_n64
+ npde = howmany(NKPT, NPDEPG);
+ pdaddr = pmap_steal_memory(PAGE_SIZE * npde);
+#endif
+ nkpt = NKPT;
+ ptaddr = pmap_steal_memory(PAGE_SIZE * nkpt);
+
+ /*
+ * The R[4-7]?00 stores only one copy of the Global bit in the
+ * translation lookaside buffer for each 2 page entry. Thus invalid
+ * entrys must have the Global bit set so when Entry LO and Entry HI
+ * G bits are anded together they will produce a global bit to store
+ * in the tlb.
+ */
+ for (i = 0, pte = (pt_entry_t *)ptaddr; i < (nkpt * NPTEPG); i++, pte++)
+ *pte = PTE_G;
+
+#ifdef __mips_n64
+ for (i = 0, npt = nkpt; npt > 0; i++) {
+ kernel_segmap[i] = (pd_entry_t)(pdaddr + i * PAGE_SIZE);
+ pde = (pd_entry_t *)kernel_segmap[i];
+
+ for (j = 0; j < NPDEPG && npt > 0; j++, npt--)
+ pde[j] = (pd_entry_t)(ptaddr + (i * NPDEPG + j) * PAGE_SIZE);
+ }
+#else
+ for (i = 0, j = pmap_seg_index(VM_MIN_KERNEL_ADDRESS); i < nkpt; i++, j++)
+ kernel_segmap[j] = (pd_entry_t)(ptaddr + (i * PAGE_SIZE));
+#endif
+
+ PMAP_LOCK_INIT(kernel_pmap);
+ kernel_pmap->pm_segtab = kernel_segmap;
+ kernel_pmap->pm_active = ~0;
+ TAILQ_INIT(&kernel_pmap->pm_pvlist);
+ kernel_pmap->pm_asid[0].asid = PMAP_ASID_RESERVED;
+ kernel_pmap->pm_asid[0].gen = 0;
+ kernel_vm_end += nkpt * NPTEPG * PAGE_SIZE;
+}
+
void
pmap_bootstrap(void)
{
- pt_entry_t *pgtab;
- pt_entry_t *pte;
- int i, j;
+ int i;
#if !defined(__mips_n64)
int memory_larger_than_512meg = 0;
#endif
@@ -440,66 +533,10 @@
}
}
#endif
-
- /*
- * Allocate segment table for the kernel
- */
- kernel_segmap = (pd_entry_t *)pmap_steal_memory(PAGE_SIZE);
-
- /*
- * Allocate second level page tables for the kernel
- */
- nkpt = NKPT;
-#if !defined(__mips_n64)
- if (memory_larger_than_512meg) {
- /*
- * If we have a large memory system we CANNOT afford to hit
- * pmap_growkernel() and allocate memory. Since we MAY end
- * up with a page that is NOT mappable. For that reason we
- * up front grab more. Normall NKPT is 120 (YMMV see pmap.h)
- * this gives us 480meg of kernel virtual addresses at the
- * cost of 120 pages (each page gets us 4 Meg). Since the
- * kernel starts at virtual_avail, we can use this to
- * calculate how many entris are left from there to the end
- * of the segmap, we want to allocate all of it, which would
- * be somewhere above 0xC0000000 - 0xFFFFFFFF which results
- * in about 256 entries or so instead of the 120.
- */
- nkpt = (PAGE_SIZE / sizeof(pd_entry_t)) - (virtual_avail >> SEGSHIFT);
- }
-#endif
- pgtab = (pt_entry_t *)pmap_steal_memory(PAGE_SIZE * nkpt);
-
- /*
- * The R[4-7]?00 stores only one copy of the Global bit in the
- * translation lookaside buffer for each 2 page entry. Thus invalid
- * entrys must have the Global bit set so when Entry LO and Entry HI
- * G bits are anded together they will produce a global bit to store
- * in the tlb.
- */
- for (i = 0, pte = pgtab; i < (nkpt * NPTEPG); i++, pte++)
- *pte = PTE_G;
-
- /*
- * The segment table contains the KVA of the pages in the second
- * level page table.
- */
- for (i = 0, j = (virtual_avail >> SEGSHIFT); i < nkpt; i++, j++)
- kernel_segmap[j] = (pd_entry_t)(pgtab + (i * NPTEPG));
-
- /*
- * The kernel's pmap is statically allocated so we don't have to use
- * pmap_create, which is unlikely to work correctly at this part of
- * the boot sequence (XXX and which no longer exists).
- */
- PMAP_LOCK_INIT(kernel_pmap);
- kernel_pmap->pm_segtab = kernel_segmap;
- kernel_pmap->pm_active = ~0;
- TAILQ_INIT(&kernel_pmap->pm_pvlist);
- kernel_pmap->pm_asid[0].asid = PMAP_ASID_RESERVED;
- kernel_pmap->pm_asid[0].gen = 0;
+ pmap_create_kernel_pagetable();
pmap_max_asid = VMNUM_PIDS;
mips_wr_entryhi(0);
+ mips_wr_pagemask(0);
}
/*
@@ -738,6 +775,8 @@
npte |= PTE_C_UNCACHED;
pte = pmap_pte(kernel_pmap, va);
+ if (pte == NULL)
+ printf("va %p segmap - %p seg_index %lx, pte_index %lx\n", (void *)va, kernel_pmap->pm_segtab, (u_long)pmap_seg_index(va),(u_long) pmap_pte_index(va));
opte = *pte;
*pte = npte;
@@ -858,19 +897,56 @@
* This routine unholds page table pages, and if the hold count
* drops to zero, then it decrements the wire count.
*/
+static PMAP_INLINE int
+pmap_unwire_pte_hold(pmap_t pmap, vm_offset_t va, vm_page_t m)
+{
+ --m->wire_count;
+ if (m->wire_count == 0)
+ return (_pmap_unwire_pte_hold(pmap, va, m));
+ else
+ return (0);
+}
+
static int
-_pmap_unwire_pte_hold(pmap_t pmap, vm_page_t m)
+_pmap_unwire_pte_hold(pmap_t pmap, vm_offset_t va, vm_page_t m)
{
+ pd_entry_t *pde;
+ PMAP_LOCK_ASSERT(pmap, MA_OWNED);
/*
* unmap the page table page
*/
- pmap->pm_segtab[m->pindex] = 0;
- --pmap->pm_stats.resident_count;
+#ifdef __mips_n64
+ if (m->pindex < NUPDE)
+ pde = pmap_pde(pmap, va);
+ else
+ pde = pmap_segmap(pmap, va);
+ *pde = 0;
+#else
+ pde = pmap_pde(pmap, va);
+#endif
+ *pde = 0;
+ pmap->pm_stats.resident_count--;
+#ifdef __mips_n64
+ if (m->pindex < NUPDE) {
+ pd_entry_t *pdp;
+ vm_page_t pdpg;
+
+ /*
+ * Recursively decrement next level pagetable refcount
+ */
+ if (pmap->pm_ptphint == m)
+ pmap->pm_ptphint = NULL;
+
+ pdp = (pd_entry_t *)*pmap_segmap(pmap, va);
+ pdpg = PHYS_TO_VM_PAGE(MIPS_KSEG0_TO_PHYS(pdp));
+ pmap_unwire_pte_hold(pmap, va, pdpg);
+ }
+#else
if (pmap->pm_ptphint == m)
pmap->pm_ptphint = NULL;
-
+#endif
/*
* If the page is finally unwired, simply free it.
*/
@@ -879,16 +955,6 @@
return (1);
}
-static PMAP_INLINE int
-pmap_unwire_pte_hold(pmap_t pmap, vm_page_t m)
-{
- --m->wire_count;
- if (m->wire_count == 0)
- return (_pmap_unwire_pte_hold(pmap, m));
- else
- return (0);
-}
-
/*
* After removing a page table entry, this routine is used to
* conditionally free the page, and manage the hold/wire counts.
@@ -903,17 +969,17 @@
return (0);
if (mpte == NULL) {
- ptepindex = pmap_segshift(va);
+ ptepindex = pmap_pde_pindex(va);
if (pmap->pm_ptphint &&
(pmap->pm_ptphint->pindex == ptepindex)) {
mpte = pmap->pm_ptphint;
} else {
- pteva = pmap_segmap(pmap, va);
+ pteva = *pmap_pde(pmap, va);
mpte = PHYS_TO_VM_PAGE(MIPS_KSEG0_TO_PHYS(pteva));
pmap->pm_ptphint = mpte;
}
}
- return pmap_unwire_pte_hold(pmap, mpte);
+ return pmap_unwire_pte_hold(pmap, va, mpte);
}
void
@@ -999,7 +1065,7 @@
static vm_page_t
_pmap_allocpte(pmap_t pmap, unsigned ptepindex, int flags)
{
- vm_offset_t pteva;
+ vm_offset_t pageva;
vm_page_t m;
KASSERT((flags & (M_NOWAIT | M_WAITOK)) == M_NOWAIT ||
@@ -1029,10 +1095,41 @@
* Map the pagetable page into the process address space, if it
* isn't already there.
*/
+ pageva = MIPS_PHYS_TO_KSEG0(VM_PAGE_TO_PHYS(m));
- pteva = MIPS_PHYS_TO_KSEG0(VM_PAGE_TO_PHYS(m));
+#ifdef __mips_n64
+ if (ptepindex >= NUPDE) {
+ pmap->pm_segtab[ptepindex - NUPDE] = (pd_entry_t)pageva;
+ } else {
+ pd_entry_t *pdep, *pde;
+ int segindex = ptepindex >> (SEGSHIFT - PDRSHIFT);
+ int pdeindex = ptepindex & (NPDEPG - 1);
+ vm_page_t pg;
+
+ pdep = &pmap->pm_segtab[segindex];
+ if (*pdep == NULL) {
+ /* recurse for allocating page dir */
+ if (_pmap_allocpte(pmap, NUPDE + segindex,
+ flags) == NULL) {
+ /* alloc failed, release current */
+ --m->wire_count;
+ atomic_subtract_int(&cnt.v_wire_count, 1);
+ vm_page_free_zero(m);
+ return (NULL);
+ }
+ } else {
+ pg = PHYS_TO_VM_PAGE(MIPS_KSEG0_TO_PHYS(*pdep));
+ pg->wire_count++;
+ }
+ /* Next level entry */
+ pde = (pd_entry_t *)*pdep;
+ pde[pdeindex] = (pd_entry_t)pageva;
+ pmap->pm_ptphint = m;
+ }
+#else
+ pmap->pm_segtab[ptepindex] = (pd_entry_t)pageva;
+#endif
pmap->pm_stats.resident_count++;
- pmap->pm_segtab[ptepindex] = (pd_entry_t)pteva;
/*
* Set the page table hint
@@ -1045,7 +1142,7 @@
pmap_allocpte(pmap_t pmap, vm_offset_t va, int flags)
{
unsigned ptepindex;
- vm_offset_t pteva;
+ pd_entry_t *pde;
vm_page_t m;
KASSERT((flags & (M_NOWAIT | M_WAITOK)) == M_NOWAIT ||
@@ -1055,18 +1152,18 @@
/*
* Calculate pagetable page index
*/
- ptepindex = pmap_segshift(va);
+ ptepindex = pmap_pde_pindex(va);
retry:
/*
* Get the page directory entry
*/
- pteva = (vm_offset_t)pmap->pm_segtab[ptepindex];
+ pde = pmap_pde(pmap, va);
/*
* If the page table page is mapped, we just increment the hold
* count, and activate it.
*/
- if (pteva) {
+ if (pde != NULL && *pde != NULL) {
/*
* In order to get the page table page, try the hint first.
*/
@@ -1074,7 +1171,7 @@
(pmap->pm_ptphint->pindex == ptepindex)) {
m = pmap->pm_ptphint;
} else {
- m = PHYS_TO_VM_PAGE(MIPS_KSEG0_TO_PHYS(pteva));
+ m = PHYS_TO_VM_PAGE(MIPS_KSEG0_TO_PHYS(*pde));
pmap->pm_ptphint = m;
}
m->wire_count++;
@@ -1087,7 +1184,7 @@
if (m == NULL && (flags & M_WAITOK))
goto retry;
}
- return m;
+ return (m);
}
@@ -1137,46 +1234,44 @@
pmap_growkernel(vm_offset_t addr)
{
vm_page_t nkpg;
+ pd_entry_t *pde, *pdpe;
pt_entry_t *pte;
int i;
mtx_assert(&kernel_map->system_mtx, MA_OWNED);
- if (kernel_vm_end == 0) {
- kernel_vm_end = VM_MIN_KERNEL_ADDRESS;
- nkpt = 0;
- while (segtab_pde(kernel_segmap, kernel_vm_end)) {
- kernel_vm_end = (kernel_vm_end + PAGE_SIZE * NPTEPG) &
- ~(PAGE_SIZE * NPTEPG - 1);
- nkpt++;
- if (kernel_vm_end - 1 >= kernel_map->max_offset) {
- kernel_vm_end = kernel_map->max_offset;
- break;
- }
- }
- }
- addr = (addr + PAGE_SIZE * NPTEPG) & ~(PAGE_SIZE * NPTEPG - 1);
+ addr = roundup2(addr, NBSEG);
if (addr - 1 >= kernel_map->max_offset)
addr = kernel_map->max_offset;
while (kernel_vm_end < addr) {
- if (segtab_pde(kernel_segmap, kernel_vm_end)) {
- kernel_vm_end = (kernel_vm_end + PAGE_SIZE * NPTEPG) &
- ~(PAGE_SIZE * NPTEPG - 1);
+ pdpe = pmap_segmap(kernel_pmap, kernel_vm_end);
+#ifdef __mips_n64
+ if (*pdpe == 0) {
+ /* new intermediate page table entry */
+ nkpg = pmap_alloc_pte_page(nkpt, VM_ALLOC_INTERRUPT);
+ if (nkpg == NULL)
+ panic("pmap_growkernel: no memory to grow kernel");
+ *pdpe = (pd_entry_t)MIPS_PHYS_TO_KSEG0(VM_PAGE_TO_PHYS(nkpg));
+ continue; /* try again */
+ }
+#endif
+ pde = pmap_pdpe_to_pde(pdpe, kernel_vm_end);
+ if (*pde != 0) {
+ kernel_vm_end = (kernel_vm_end + NBPDR) & ~PDRMASK;
if (kernel_vm_end - 1 >= kernel_map->max_offset) {
kernel_vm_end = kernel_map->max_offset;
break;
}
continue;
}
+
/*
* This index is bogus, but out of the way
*/
- nkpg = pmap_alloc_pte_page(nkpt, VM_ALLOC_INTERRUPT);
+ nkpg = pmap_alloc_pte_page(nkpt, VM_ALLOC_INTERRUPT);
if (!nkpg)
panic("pmap_growkernel: no memory to grow kernel");
-
nkpt++;
- pte = (pt_entry_t *)MIPS_PHYS_TO_KSEG0(VM_PAGE_TO_PHYS(nkpg));
- segtab_pde(kernel_segmap, kernel_vm_end) = (pd_entry_t)pte;
+ *pde = (pd_entry_t)MIPS_PHYS_TO_KSEG0(VM_PAGE_TO_PHYS(nkpg));
/*
* The R[4-7]?00 stores only one copy of the Global bit in
@@ -1185,11 +1280,11 @@
* Entry LO and Entry HI G bits are anded together they will
* produce a global bit to store in the tlb.
*/
- for (i = 0; i < NPTEPG; i++, pte++)
- *pte = PTE_G;
+ pte = (pt_entry_t *)*pde;
+ for (i = 0; i < NPTEPG; i++)
+ pte[i] = PTE_G;
- kernel_vm_end = (kernel_vm_end + PAGE_SIZE * NPTEPG) &
- ~(PAGE_SIZE * NPTEPG - 1);
+ kernel_vm_end = (kernel_vm_end + NBPDR) & ~PDRMASK;
if (kernel_vm_end - 1 >= kernel_map->max_offset) {
kernel_vm_end = kernel_map->max_offset;
break;
@@ -1480,7 +1575,9 @@
void
pmap_remove(struct pmap *pmap, vm_offset_t sva, vm_offset_t eva)
{
- vm_offset_t va, nva;
+ vm_offset_t va_next;
+ pd_entry_t *pde, *pdpe;
+ pt_entry_t *pte;
if (pmap == NULL)
return;
@@ -1499,15 +1596,30 @@
pmap_remove_page(pmap, sva);
goto out;
}
- for (va = sva; va < eva; va = nva) {
- if (pmap_segmap(pmap, va) == NULL) {
- nva = mips_segtrunc(va + NBSEG);
+ for (; sva < eva; sva = va_next) {
+ pdpe = pmap_segmap(pmap, sva);
+#ifdef __mips_n64
+ if (*pdpe == 0) {
+ va_next = (sva + NBSEG) & ~SEGMASK;
+ if (va_next < sva)
+ va_next = eva;
continue;
}
- pmap_remove_page(pmap, va);
- nva = va + PAGE_SIZE;
+#endif
+ va_next = (sva + NBPDR) & ~PDRMASK;
+ if (va_next < sva)
+ va_next = eva;
+
+ pde = pmap_pdpe_to_pde(pdpe, sva);
+ if (*pde == 0)
+ continue;
+ if (va_next > eva)
+ va_next = eva;
+ for (pte = pmap_pde_to_pte(pde, sva); sva != va_next;
+ pte++, sva += PAGE_SIZE) {
+ pmap_remove_page(pmap, sva);
+ }
}
-
out:
vm_page_unlock_queues();
PMAP_UNLOCK(pmap);
@@ -1596,6 +1708,8 @@
pmap_protect(pmap_t pmap, vm_offset_t sva, vm_offset_t eva, vm_prot_t prot)
{
pt_entry_t *pte;
+ pd_entry_t *pde, *pdpe;
+ vm_offset_t va_next;
if (pmap == NULL)
return;
@@ -1609,44 +1723,53 @@
vm_page_lock_queues();
PMAP_LOCK(pmap);
- while (sva < eva) {
- pt_entry_t pbits, obits;
+ for (; sva < eva; sva = va_next) {
+ pt_entry_t obits, pbits;
vm_page_t m;
- vm_offset_t pa;
+ vm_paddr_t pa;
- /*
- * If segment table entry is empty, skip this segment.
- */
- if (pmap_segmap(pmap, sva) == NULL) {
- sva = mips_segtrunc(sva + NBSEG);
+ pdpe = pmap_segmap(pmap, sva);
+#ifdef __mips_n64
+ if (*pdpe == 0) {
+ va_next = (sva + NBSEG) & ~SEGMASK;
+ if (va_next < sva)
+ va_next = eva;
continue;
}
- /*
- * If pte is invalid, skip this page
- */
- pte = pmap_pte(pmap, sva);
- if (!pte_test(pte, PTE_V)) {
- sva += PAGE_SIZE;
+#endif
+ va_next = (sva + NBPDR) & ~PDRMASK;
+ if (va_next < sva)
+ va_next = eva;
+
+ pde = pmap_pdpe_to_pde(pdpe, sva);
+ if (pde == NULL || *pde == NULL)
continue;
- }
-retry:
- obits = pbits = *pte;
- pa = TLBLO_PTE_TO_PA(pbits);
+ if (va_next > eva)
+ va_next = eva;
- if (page_is_managed(pa) && pte_test(&pbits, PTE_D)) {
- m = PHYS_TO_VM_PAGE(pa);
- vm_page_dirty(m);
- m->md.pv_flags &= ~PV_TABLE_MOD;
- }
- pte_clear(&pbits, PTE_D);
- pte_set(&pbits, PTE_RO);
+ for (pte = pmap_pde_to_pte(pde, sva); sva != va_next; pte++,
+ sva += PAGE_SIZE) {
- if (pbits != *pte) {
- if (!atomic_cmpset_int((u_int *)pte, obits, pbits))
- goto retry;
- pmap_update_page(pmap, sva, pbits);
+ /* Skip invalid PTEs */
+ if (!pte_test(pte, PTE_V))
+ continue;
+retry:
+ obits = pbits = *pte;
+ pa = TLBLO_PTE_TO_PA(pbits);
+ if (page_is_managed(pa) && pte_test(&pbits, PTE_D)) {
+ m = PHYS_TO_VM_PAGE(pa);
+ vm_page_dirty(m);
+ m->md.pv_flags &= ~PV_TABLE_MOD;
+ }
+ pte_clear(&pbits, PTE_D);
+ pte_set(&pbits, PTE_RO);
+
+ if (pbits != *pte) {
+ if (!atomic_cmpset_int((u_int *)pte, obits, pbits))
+ goto retry;
+ pmap_update_page(pmap, sva, pbits);
+ }
}
- sva += PAGE_SIZE;
}
vm_page_unlock_queues();
PMAP_UNLOCK(pmap);
@@ -1899,32 +2022,32 @@
* creating it here.
*/
if (va < VM_MAXUSER_ADDRESS) {
+ pd_entry_t *pde;
unsigned ptepindex;
- vm_offset_t pteva;
/*
* Calculate pagetable page index
*/
- ptepindex = pmap_segshift(va);
+ ptepindex = pmap_pde_pindex(va);
if (mpte && (mpte->pindex == ptepindex)) {
mpte->wire_count++;
} else {
/*
* Get the page directory entry
*/
- pteva = (vm_offset_t)pmap->pm_segtab[ptepindex];
+ pde = pmap_pde(pmap, va);
/*
* If the page table page is mapped, we just
* increment the hold count, and activate it.
*/
- if (pteva) {
+ if (pde && *pde != 0) {
if (pmap->pm_ptphint &&
(pmap->pm_ptphint->pindex == ptepindex)) {
mpte = pmap->pm_ptphint;
} else {
mpte = PHYS_TO_VM_PAGE(
- MIPS_KSEG0_TO_PHYS(pteva));
+ MIPS_KSEG0_TO_PHYS(*pde));
pmap->pm_ptphint = mpte;
}
mpte->wire_count++;
@@ -1954,7 +2077,7 @@
if ((m->flags & (PG_FICTITIOUS | PG_UNMANAGED)) == 0 &&
!pmap_try_insert_pv_entry(pmap, mpte, va, m)) {
if (mpte != NULL) {
- pmap_unwire_pte_hold(pmap, mpte);
+ pmap_unwire_pte_hold(pmap, va, mpte);
mpte = NULL;
}
return (mpte);
@@ -2506,21 +2629,19 @@
PMAP_LOCK(pv->pv_pmap);
pte = pmap_pte(pv->pv_pmap, pv->pv_va);
-
if (setem) {
- *(int *)pte |= bit;
+ *pte |= bit;
pmap_update_page(pv->pv_pmap, pv->pv_va, *pte);
} else {
- vm_offset_t pbits = *(vm_offset_t *)pte;
+ pt_entry_t pbits = *pte;
if (pbits & bit) {
if (bit == PTE_D) {
- if (pbits & PTE_D) {
+ if (pbits & PTE_D)
vm_page_dirty(m);
- }
- *(int *)pte = (pbits & ~PTE_D) | PTE_RO;
+ *pte = (pbits & ~PTE_D) | PTE_RO;
} else {
- *(int *)pte = pbits & ~bit;
+ *pte = pbits & ~bit;
}
pmap_update_page(pv->pv_pmap, pv->pv_va, *pte);
}
@@ -2658,13 +2779,17 @@
boolean_t
pmap_is_prefaultable(pmap_t pmap, vm_offset_t addr)
{
+ pd_entry_t *pde;
pt_entry_t *pte;
boolean_t rv;
rv = FALSE;
PMAP_LOCK(pmap);
- if (pmap_segmap(pmap, addr) != NULL) {
- pte = pmap_pte(pmap, addr);
+ pde = pmap_pde(pmap, addr);
+ if (pde != NULL && *pde != 0) {
+ pte = pmap_pde_to_pte(pde, addr);
+ if (pte == NULL)
+ printf("Fail addr %p pde %p, pte %p\n", (void *)addr, pde, pte);
rv = (*pte == 0);
}
PMAP_UNLOCK(pmap);
@@ -2927,74 +3052,65 @@
return;
}
-int pmap_pid_dump(int pid);
-
-int
-pmap_pid_dump(int pid)
+DB_SHOW_COMMAND(ptable, ddb_pid_dump)
{
pmap_t pmap;
+ struct thread *td = NULL;
struct proc *p;
- int npte = 0;
- int index;
+ int i, j, k;
+ vm_paddr_t pa;
+ vm_offset_t va;
- sx_slock(&allproc_lock);
- LIST_FOREACH(p, &allproc, p_list) {
- if (p->p_pid != pid)
- continue;
+ if (have_addr) {
+ td = db_lookup_thread(addr, TRUE);
+ if (td == NULL) {
+ db_printf("Invalid pid or tid");
+ return;
+ }
+ p = td->td_proc;
+ if (p->p_vmspace == NULL) {
+ db_printf("No vmspace for process");
+ return;
+ }
+ pmap = vmspace_pmap(p->p_vmspace);
+ } else
+ pmap = kernel_pmap;
- if (p->p_vmspace) {
- int i, j;
-
- printf("vmspace is %p\n",
- p->p_vmspace);
- index = 0;
- pmap = vmspace_pmap(p->p_vmspace);
- printf("pmap asid:%x generation:%x\n",
+ db_printf("pmap:%p segtab:%p asid:%x generation:%x\n",
+ pmap, pmap->pm_segtab,
pmap->pm_asid[0].asid,
pmap->pm_asid[0].gen);
- for (i = 0; i < NUSERPGTBLS; i++) {
- pd_entry_t *pde;
- pt_entry_t *pte;
- unsigned base = i << SEGSHIFT;
+ for (i = 0; i < NPDEPG; i++) {
+ pd_entry_t *pdpe;
+ pt_entry_t *pde;
+ pt_entry_t pte;
- pde = &pmap->pm_segtab[i];
- if (pde && *pde != 0) {
- for (j = 0; j < 1024; j++) {
- vm_offset_t va = base +
- (j << PAGE_SHIFT);
-
- pte = pmap_pte(pmap, va);
- if (pte && pte_test(pte, PTE_V)) {
- vm_offset_t pa;
- vm_page_t m;
-
- pa = TLBLO_PFN_TO_PA(*pte);
- m = PHYS_TO_VM_PAGE(pa);
- printf("va: %p, pt: %p, h: %d, w: %d, f: 0x%x",
- (void *)va,
- (void *)pa,
- m->hold_count,
- m->wire_count,
- m->flags);
- npte++;
- index++;
- if (index >= 2) {
- index = 0;
- printf("\n");
- } else {
- printf(" ");
- }
- }
- }
- }
+ pdpe = (pd_entry_t *)pmap->pm_segtab[i];
+ if (pdpe == NULL)
+ continue;
+ db_printf("[%4d] %p\n", i, pdpe);
+#ifdef __mips_n64
+ for (j = 0; j < NPDEPG; j++) {
+ pde = (pt_entry_t *)pdpe[j];
+ if (pde == NULL)
+ continue;
+ db_printf("\t[%4d] %p\n", j, pde);
+#else
+ {
+ j = 0;
+ pde = (pt_entry_t *)pdpe;
+#endif
+ for (k = 0; k < NPTEPG; k++) {
+ pte = pde[k];
+ if (pte == 0 || !pte_test(&pte, PTE_V))
+ continue;
+ pa = TLBLO_PTE_TO_PA(pte);
+ va = ((u_long)i << SEGSHIFT) | (j << PDRSHIFT) | (k << PAGE_SHIFT);
+ db_printf("\t\t[%04d] va: %p pte: %8x pa:%lx\n",
+ k, (void *)va, pte, (u_long)pa);
}
- } else {
- printf("Process pid:%d has no vm_space\n", pid);
}
- break;
}
- sx_sunlock(&allproc_lock);
- return npte;
}
@@ -3126,7 +3242,6 @@
PHYS_TO_VM_PAGE(pa)->md.pv_flags |= (PV_TABLE_REF | PV_TABLE_MOD);
}
-
/*
* Routine: pmap_kextract
* Function:
@@ -3145,7 +3260,6 @@
if (va >= MIPS_XKPHYS_START && va < MIPS_XKPHYS_END)
return (MIPS_XKPHYS_TO_PHYS(va));
#endif
-
if (va >= MIPS_KSEG0_START && va < MIPS_KSEG0_END)
return (MIPS_KSEG0_TO_PHYS(va));
@@ -3205,7 +3319,7 @@
if (m != NULL) {
for (pv = TAILQ_FIRST(&m->md.pv_list); pv;
- pv = TAILQ_NEXT(pv, pv_list)) {
+ pv = TAILQ_NEXT(pv, pv_list)) {
mips_dcache_wbinv_range_index(pv->pv_va, PAGE_SIZE);
}
}
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTimOr-JGU_jxFZ5QP2QSM-jWKJDXRc7O3ZTaoALX>