From owner-freebsd-ppc@freebsd.org Wed May 1 04:45:06 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89A8B1584D0C for ; Wed, 1 May 2019 04:45:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-15.consmr.mail.bf2.yahoo.com (sonic309-15.consmr.mail.bf2.yahoo.com [74.6.129.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9607876742 for ; Wed, 1 May 2019 04:45:05 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ZEGLB1IVM1nJTY8BwYrYFNsUVq_EJ86.grEGkeq85roXL88FjrOqvLYvt5By045 WptIKZbowFW1fvB8VOMvz_lvxBYzBjP8HHOVoBgIXcWjN8MeNOnUEu.8Z33Xk2v34I3uCRGFV9nf FY2xvHQvPJGK1FYgWYMK76OPWuX7iWmWEvdabNiBDpRimjq.H_KaI0tLF4J58fUVA0r7mf6d6tMP X8P8UWDkNHegQl3pLWop2ojZboxcFxmQfAztiBsFGDnlkgSX9p999rx_iuSAEjHtsa7VFyn_rqcC f1s0QIJBlKWIXSTP9KIKwojWGku7Pv3.e_14.EaeJeNA7sxakrkjTUKtz4B5PLuNuSlPZftqewOu BoN2lovMPILdLqmImcMm_ibMTrsByA43T3cD1GE0K1nOSLL4tS46PN_woyPuhW2SnhlsuugoDIFD 1wtXPNxKi6AR1UZrAfGjLDSDTMe5cxXks7sMBtPd5wwwhQgJuOsGeldAGYEGJ_JSmFxWQPXv3asw TFVDp5hGn8vDOLtMA.HhTuqmln.gW3MBghWUTiqNT6IS46qNoHHnQkcPSobHXQZM.jY9RtuVRABP hYfNRDlw4H9ST.bJQtJpWvESdm4mJYyYydvyPk_xNjI4GqHtpFxOa1_Pyln3shUkgIrv9U0nCxoy xPBZjmdVBtUCxeKYk03wrmWGAXnHZI4MZVuYlO8jumkrfKHjRVYG_m9xhdXRTi3AXY6HSVFc0Fpi MJDiF4KQBCDtCRQBhWxRnTxzA1.l2n1.SPfgtfOv2PLqvI7sKhpw3p6qxahsgsnFd5FsB8pyon_T IKNxQ0Y6xy2g136u1M3XKiioSFQ4u20Ot8CYx760Jz9O4PnLpoCRvFSuploCkPfO7nr709h_CzhQ i8CvqQNdwZCAnWle5JJEZxCu0bUP7ReJ4PAePV.7MojgI08Bsr68WobjmGtTekljsjiHkwfQPv2O 1ggUxPUMRNweKwKj2roWC.3e1m7puh7X1ni30GJxc7zvhsk9hlwhAQpvPF6HDmpIbDZzlnI8svZd 93Z8dL5hRT8hYIWlB6QVzzEkpaeIyqsL3cBk2HeRI4V5uWBwKQsu.TgjhKe0D.yd4GfPIsDObE5m 95VJHKz13RBOF.AeCbKQ1taPwCZ65H69OOkZB4WwR8MJdK91NThvFS4zQWRok7WaugPbxkdZl6nt 04Tw- Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.bf2.yahoo.com with HTTP; Wed, 1 May 2019 04:45:03 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp417.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 3173328dd38b23b98aa2b2ced3aeb107; Wed, 01 May 2019 04:45:02 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap Date: Tue, 30 Apr 2019 21:45:00 -0700 References: <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com> To: Justin Hibbits , FreeBSD PowerPC ML In-Reply-To: <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com> Message-Id: <6159F4A6-9431-4B99-AA62-451B8DF08A6E@yahoo.com> X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 9607876742 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.17 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:26101, ipnet:74.6.128.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.01)[-0.009,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.95)[0.947,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.48)[ip: (4.62), ipnet: 74.6.128.0/21(1.56), asn: 26101(1.25), country: US(-0.06)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.26)[0.264,0]; RCVD_IN_DNSWL_NONE(0.00)[125.129.6.74.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[125.129.6.74.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 04:45:06 -0000 [I realized another implication about a another point of potential slb-misses in cpudep_ap_bootstrap: the address in sprg0 on the cpu might end up not able to be dereferenced.] On 2019-Apr-30, at 20:58, Mark Millard wrote: > [At the end this note shows why the old VM_MAX_KERNEL_ADDRESS > lead to no slb-miss exceptions in cpudep_ap_bootstrap.] > > There is code in moea64_late_bootstrap that looks like: > > virtual_avail = VM_MIN_KERNEL_ADDRESS; > virtual_end = VM_MAX_SAFE_KERNEL_ADDRESS; > > /* > * Map the entire KVA range into the SLB. We must not fault there. > */ > #ifdef __powerpc64__ > for (va = virtual_avail; va < virtual_end; va += SEGMENT_LENGTH) > moea64_bootstrap_slb_prefault(va, 0); > #endif > > where (modern): > > #define VM_MIN_KERNEL_ADDRESS 0xe000000000000000UL > #define VM_MAX_SAFE_KERNEL_ADDRESS VM_MAX_KERNEL_ADDRESS > #define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffffUL > #define SEGMENT_LENGTH 0x10000000UL > > So: > > 0xe000000000000000UL: VM_MIN_KERNEL_ADDRESS > 0x0000000010000000UL: SEGMENT_LENGTH > 0xe0000007ffffffffUL: VM_MAX_KERNEL_ADDRESS > > So I see the loop as doing moea64_bootstrap_slb_prefault > 128 times (decimal, 0x00..0x7f at the appropriate > byte in va). > > (I do not see why this loop keeps going once the slb > kernel slots are all full. Nor is it obvious to me > why the larger va values should be the ones more > likely to still be covered. But I'm going a different > direction below.) > > That also means that the code does random replacement (based > on mftb()%n_slbs, but avoiding USER_SLB_SLOT) 128-(64-1), > or 65 times. The slb_insert_kernel use in > moea64_bootstrap_slb_prefault does that: > > moea64_bootstrap_slb_prefault(vm_offset_t va, int large) > { > struct slb *cache; > struct slb entry; > uint64_t esid, slbe; > uint64_t i; > > cache = PCPU_GET(aim.slb); > esid = va >> ADDR_SR_SHFT; > slbe = (esid << SLBE_ESID_SHIFT) | SLBE_VALID; > > for (i = 0; i < 64; i++) { > if (cache[i].slbe == (slbe | i)) > return; > } > > entry.slbe = slbe; > entry.slbv = KERNEL_VSID(esid) << SLBV_VSID_SHIFT; > if (large) > entry.slbv |= SLBV_L; > > slb_insert_kernel(entry.slbe, entry.slbv); > } > > where slb_insert_kernel is in turn has the code that > will do replacements: > > void > slb_insert_kernel(uint64_t slbe, uint64_t slbv) > { > struct slb *slbcache; > int i; > > /* We don't want to be preempted while modifying the kernel map */ > critical_enter(); > > slbcache = PCPU_GET(aim.slb); > > /* Check for an unused slot, abusing the user slot as a full flag */ > if (slbcache[USER_SLB_SLOT].slbe == 0) { > for (i = 0; i < n_slbs; i++) { > if (i == USER_SLB_SLOT) > continue; > if (!(slbcache[i].slbe & SLBE_VALID)) > goto fillkernslb; > } > > if (i == n_slbs) > slbcache[USER_SLB_SLOT].slbe = 1; > } > > i = mftb() % n_slbs; > if (i == USER_SLB_SLOT) > i = (i+1) % n_slbs; > > fillkernslb: > KASSERT(i != USER_SLB_SLOT, > ("Filling user SLB slot with a kernel mapping")); > slbcache[i].slbv = slbv; > slbcache[i].slbe = slbe | (uint64_t)i; > > /* If it is for this CPU, put it in the SLB right away */ > if (pmap_bootstrapped) { > /* slbie not required */ > __asm __volatile ("slbmte %0, %1" :: > "r"(slbcache[i].slbv), "r"(slbcache[i].slbe)); > } > > critical_exit(); > } > > [The USER_SLB_SLOT handling makes selection of slot > USER_SLB_SLOT+1 for what to replace more likely than > the other kernel slots.] > > I expect that the above explains the variability in > if cpudep_ap_bootstrap 's: > > sp = pcpup->pc_curpcb->pcb_sp > > gets a slb fault for dereferencing the pc_curpcb stage > of that vs. not. Note: the random replacements could also make dereferencing pcpup-> (aka (get_pcpu())->) end up with a slb-miss, where: static __inline struct pcpu * get_pcpu(void) { struct pcpu *ret; __asm __volatile("mfsprg %0, 0" : "=r"(ret)); return (ret); } If the slb entry covering address ranges accessed based on sprg0 is ever replaced, no code based on getting sprg0's value to find the matching pcpu information is going to work, *including in the slb spill trap code* [GET_CPUINFO(%r?)]. Does a kernel entry need to be reserved for the CPU that never is replaced so that sprg0 can always be used to find pcpu information via sprg0? This is something my hack did not deal with. And, in fact, trying to force 2 entries to exist at the same time, one for "dereferencing sprg0" and one for dereferencing the pc_curpcb so found is currently messy, given the way other things work. The lack of handing may explain the (rare) hangups with the existing hack present. > I also expect that the old VM_MAX_KERNEL_ADDRESS value > explains the lack of slb-misses in old times: > > 0xe000000000000000UL: VM_MIN_KERNEL_ADDRESS > 0x0000000010000000UL: SEGMENT_LENGTH > 0xe0000001c7ffffffUL: VM_MAX_KERNEL_ADDRESS > > So 0x00..0x1c is 29 alternatives (decimal). That > fits in 64-1 slots, or even 32-1 slots: no > random replacements happened above or elsewhere. > That, in turn meant no testing of the handling > of any slb-misses back then. > > > [Other list messages suggest missing context synchronizing > instructions for slbmte and related instructions. The > history is not evidence about that, given the lack of > slb-misses.] === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)