Skip site navigation (1)Skip section navigation (2) 
Date:      Tue, 30 Apr 2019 20:58:49 -0700
From:      Mark Millard <marklmi@yahoo.com>
To:        Justin Hibbits <chmeeedalf@gmail.com>, FreeBSD PowerPC ML <freebsd-ppc@freebsd.org>
Subject:   How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap
Message-ID:  <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com>
next in thread | raw e-mail | index | archive | help 
[At the end this note shows why the old VM_MAX_KERNEL_ADDRESS
lead to no slb-miss exceptions in cpudep_ap_bootstrap.]

There is code in moea64_late_bootstrap that looks like:

        virtual_avail = VM_MIN_KERNEL_ADDRESS;
        virtual_end = VM_MAX_SAFE_KERNEL_ADDRESS;

        /*
         * Map the entire KVA range into the SLB. We must not fault there.
         */
        #ifdef __powerpc64__
        for (va = virtual_avail; va < virtual_end; va += SEGMENT_LENGTH)
                moea64_bootstrap_slb_prefault(va, 0);
        #endif

where (modern):

#define VM_MIN_KERNEL_ADDRESS           0xe000000000000000UL
#define VM_MAX_SAFE_KERNEL_ADDRESS      VM_MAX_KERNEL_ADDRESS
#define VM_MAX_KERNEL_ADDRESS           0xe0000007ffffffffUL
#define       SEGMENT_LENGTH  0x10000000UL

So:

0xe000000000000000UL: VM_MIN_KERNEL_ADDRESS
0x0000000010000000UL: SEGMENT_LENGTH
0xe0000007ffffffffUL: VM_MAX_KERNEL_ADDRESS

So I see the loop as doing moea64_bootstrap_slb_prefault
128 times (decimal, 0x00..0x7f at the appropriate
byte in va).

(I do not see why this loop keeps going once the slb
kernel slots are all full. Nor is it obvious to me
why the larger va values should be the ones more
likely to still be covered. But I'm going a different
direction below.)

That also means that the code does random replacement (based
on mftb()%n_slbs, but avoiding USER_SLB_SLOT) 128-(64-1),
or 65 times. The slb_insert_kernel use in 
moea64_bootstrap_slb_prefault does that:

moea64_bootstrap_slb_prefault(vm_offset_t va, int large)
{
        struct slb *cache;
        struct slb entry;
        uint64_t esid, slbe;
        uint64_t i;

        cache = PCPU_GET(aim.slb);
        esid = va >> ADDR_SR_SHFT;
        slbe = (esid << SLBE_ESID_SHIFT) | SLBE_VALID;
        
        for (i = 0; i < 64; i++) {
                if (cache[i].slbe == (slbe | i))
                        return;
        }
        
        entry.slbe = slbe;
        entry.slbv = KERNEL_VSID(esid) << SLBV_VSID_SHIFT;
        if (large)
                entry.slbv |= SLBV_L;
        
        slb_insert_kernel(entry.slbe, entry.slbv);
}

where slb_insert_kernel is in turn has the code that
will do replacements:

void
slb_insert_kernel(uint64_t slbe, uint64_t slbv)
{
        struct slb *slbcache;
        int i;

        /* We don't want to be preempted while modifying the kernel map */
        critical_enter();

        slbcache = PCPU_GET(aim.slb);

        /* Check for an unused slot, abusing the user slot as a full flag */
        if (slbcache[USER_SLB_SLOT].slbe == 0) {
                for (i = 0; i < n_slbs; i++) {
                        if (i == USER_SLB_SLOT)
                                continue;
                        if (!(slbcache[i].slbe & SLBE_VALID))
                                goto fillkernslb;
                }

                if (i == n_slbs)
                        slbcache[USER_SLB_SLOT].slbe = 1;
        }

        i = mftb() % n_slbs;
        if (i == USER_SLB_SLOT)
                        i = (i+1) % n_slbs;

fillkernslb:
        KASSERT(i != USER_SLB_SLOT,
            ("Filling user SLB slot with a kernel mapping"));
        slbcache[i].slbv = slbv;
        slbcache[i].slbe = slbe | (uint64_t)i;

        /* If it is for this CPU, put it in the SLB right away */
        if (pmap_bootstrapped) {
                /* slbie not required */
                __asm __volatile ("slbmte %0, %1" ::
                    "r"(slbcache[i].slbv), "r"(slbcache[i].slbe));
        }

        critical_exit();
}

[The USER_SLB_SLOT handling makes selection of slot
USER_SLB_SLOT+1 for what to replace more likely than
the other kernel slots.]

I expect that the above explains the variability in
if cpudep_ap_bootstrap 's:

sp = pcpup->pc_curpcb->pcb_sp

gets a slb fault for dereferencing the pc_curpcb stage
of that vs. not.

I also expect that the old VM_MAX_KERNEL_ADDRESS value
explains the lack of slb-misses in old times:

0xe000000000000000UL: VM_MIN_KERNEL_ADDRESS
0x0000000010000000UL: SEGMENT_LENGTH
0xe0000001c7ffffffUL: VM_MAX_KERNEL_ADDRESS

So 0x00..0x1c is 29 alternatives (decimal). That
fits in 64-1 slots, or even 32-1 slots: no
random replacements happened above or elsewhere.
That, in turn meant no testing of the handling
of any slb-misses back then.


[Other list messages suggest missing context synchronizing
instructions for slbmte and related instructions. The
history is not evidence about that, given the lack of
slb-misses.]

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3C69CF7C-7F33-4C79-92C0-3493A1294996>