From owner-freebsd-ppc@freebsd.org Wed May 1 18:51:52 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17086159E2E3 for ; Wed, 1 May 2019 18:51:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-2.consmr.mail.bf2.yahoo.com (sonic306-2.consmr.mail.bf2.yahoo.com [74.6.132.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BFC50892B3 for ; Wed, 1 May 2019 18:51:49 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: YsF8UBUVM1mwpYrz9NREuYPSGasMBRdPl8Ig9RZrDjlWZe.iE_ZJ89xBv4faEw0 a1xQp.Nf3eMi4RFObWzW8548J8syjC6iJakY.G7VZvPbH2BgnFbmr_lc7vpxOMoZ5fgIfQZ8vKi. g7HjxpuFN1q6IiSVpkdnsgdlWmQ85aOJyPSiL9YWmYvpOkuq_gZrbRhP.YPHJotnBro7nIaOYE5P _3JeEAKUVBcvhoCY1BN1L9En2tdNZX3p0mcTEJgJ0VWGSSEA.c15d9Z_V0seMc1pNS3oQkEGKOtC XK9gWm.JuW27Wy511hshxS9uHV36oAW7Rm23P7aPe4GBh7g3jXtHNJMjoQjFWmGW0vxRT.31DmlE QJKv9rNR5xGjaDa165YS4O3pLhna1miMlDHiWcsp8Yf.XietYlveB.wUmz52uJxm4QYYARby84Li o8bJYbjEmyV1GOAD42o5uHh2dPpMpHydS5.ErsIHNFxTJUglP7K9miB_z9rF1hariWD9rG2gha04 vJkifPdco.NwEk_oM6Gh5kjNFlYkJhauNgBiu9M7IE0XxuZhflhzAkIK0mUfcoRex56B8wdFzGJg wYdNp_NTQLkLZTmuX322jqejCXLUS91d3rsdHDyQ9OIxrr2vrgj6R.9SOOhvzOUph6Pjz0sN1HX1 aHwmBqqSFaJeVmJrxLrXrguGPyXW_rqnn0yU_fiIRpl5Teue2akoJ3BQYz9GxsqvKICOdiVh3bfS MYinIY61BaWDDS73dl1Zs0ra7K1IKP6nwjwLkELnrEpFoLbq8ikGZ_JzRwyTvkS8MO1DQX3aILgo t9SCdN2BCvfEAwkYvuY4MkFjzZs0CgkCYX.hessTHhpjvnlKfu_xnGJXu7q8IkTH8nA4u9Dnog4G amxW5GhUvMJduJClAyuATrjMDKNx8caPuIA_X0C_aUd78eL36OwsucDffzUgfy6_oEOeJJDJduCZ wBtYIEeqVWiLseVj4FVKOdZU10CfI9dn8m_2kQwbQGshLW_crm5ajzu_3F9sKKZs0eU1PWRC.T21 mT.EWauFgHJW_wdpxR7zXPEmN6DV4xmQ.el050CwDuv0KdxtN3Jrc5C6F5H58bSPXCsnaMCe6Vv_ _1yhckhBshtTTYMVRpwq5O7RCsv1_27mDt9Yd2wihl_llmOEeyv38noT0W.2NfoiT8YCwXGC5hcp B9Mwf4Fc- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.bf2.yahoo.com with HTTP; Wed, 1 May 2019 18:51:48 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp432.mail.bf1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 97859ccef0ce9249b4063f00add64977; Wed, 01 May 2019 18:51:47 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap From: Mark Millard In-Reply-To: <20190501094029.542c5f46@titan.knownspace> Date: Wed, 1 May 2019 11:51:44 -0700 Cc: FreeBSD PowerPC ML Content-Transfer-Encoding: 7bit Message-Id: <212E50E5-7EB1-4381-A662-D5EACB1E5D46@yahoo.com> References: <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com> <6159F4A6-9431-4B99-AA62-451B8DF08A6E@yahoo.com> <20190501094029.542c5f46@titan.knownspace> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: BFC50892B3 X-Spamd-Bar: + X-Spamd-Result: default: False [1.13 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:26101, ipnet:74.6.128.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.07)[-0.072,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.83)[0.831,0]; NEURAL_HAM_LONG(-0.48)[-0.482,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.36)[ip: (4.05), ipnet: 74.6.128.0/21(1.56), asn: 26101(1.25), country: US(-0.06)]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[41.132.6.74.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 18:51:52 -0000 On 2019-May-1, at 07:40, Justin Hibbits wrote: > On Tue, 30 Apr 2019 21:45:00 -0700 > Mark Millard wrote: > >> [I realized another implication about a another point of >> potential slb-misses in cpudep_ap_bootstrap: the >> address in sprg0 on the cpu might end up not able to be >> dereferenced.] >> >> On 2019-Apr-30, at 20:58, Mark Millard wrote: >> >>> [At the end this note shows why the old VM_MAX_KERNEL_ADDRESS >>> lead to no slb-miss exceptions in cpudep_ap_bootstrap.] >>> >>> There is code in moea64_late_bootstrap that looks like: >>> >>> virtual_avail = VM_MIN_KERNEL_ADDRESS; >>> virtual_end = VM_MAX_SAFE_KERNEL_ADDRESS; >>> >>> /* >>> * Map the entire KVA range into the SLB. We must not fault >>> there. */ >>> #ifdef __powerpc64__ >>> for (va = virtual_avail; va < virtual_end; va += >>> SEGMENT_LENGTH) moea64_bootstrap_slb_prefault(va, 0); >>> #endif > > What happens if you revert all your patches, Most of the patches in Bugzilla 233863 are not for this issue at all and are not tied to starting the non-bsp cpus. (The one for improving how close the Time Base registers are is tied to starting these cpus.) Only the aim/mp_cpudep.c and aim/slb.c changes seem relevant. Are you worried about some form of interaction that means I need to avoid patches for other issues? Note: for now I'm staying at using head -r345758 as the basis for my experiments. > and change this loop to > stop at n_slb? So something more akin to: > > int i = 0; > > for (va = virtual_avail; va < virtual_end && i < n_slb - 1; va > += SEGMENT_LENGTH, i++); > ... > > If it reliably boots with that, then that's fine. We can prefault as > much as we can and leave the rest for on-demand. I'm happy to experiment with this loop without my hack for forcing the slb entry to exist in cpudep_ap_bootstrap. But, it seems to presume that the pc_curpcb's will all always point into the lower address range spanned when cpudep_ap_bootstrap is executing on the cpu. Does some known property limit the pc_curpcb-> references to such? Only that would be sure to avoid an slb-miss at that stage. Or is this just an alternate hack or a means of getting evidence, not a proposed solution? (Again, I'm happy to disable my hack that forces the slb entry and to try the loop suggested.) >>> >>> where (modern): >>> >>> #define VM_MIN_KERNEL_ADDRESS 0xe000000000000000UL >>> #define VM_MAX_SAFE_KERNEL_ADDRESS VM_MAX_KERNEL_ADDRESS >>> #define VM_MAX_KERNEL_ADDRESS 0xe0000007ffffffffUL >>> #define SEGMENT_LENGTH 0x10000000UL >>> >>> So: >>> >>> 0xe000000000000000UL: VM_MIN_KERNEL_ADDRESS >>> 0x0000000010000000UL: SEGMENT_LENGTH >>> 0xe0000007ffffffffUL: VM_MAX_KERNEL_ADDRESS >>> >>> So I see the loop as doing moea64_bootstrap_slb_prefault >>> 128 times (decimal, 0x00..0x7f at the appropriate >>> byte in va). >>> >>> (I do not see why this loop keeps going once the slb >>> kernel slots are all full. Nor is it obvious to me >>> why the larger va values should be the ones more >>> likely to still be covered. But I'm going a different >>> direction below.) >>> >>> That also means that the code does random replacement (based >>> on mftb()%n_slbs, but avoiding USER_SLB_SLOT) 128-(64-1), >>> or 65 times. The slb_insert_kernel use in >>> moea64_bootstrap_slb_prefault does that: >>> > ... >>> >>> I expect that the above explains the variability in >>> if cpudep_ap_bootstrap 's: >>> >>> sp = pcpup->pc_curpcb->pcb_sp >>> >>> gets a slb fault for dereferencing the pc_curpcb stage >>> of that vs. not. >> >> >> Note: the random replacements could also make >> dereferencing pcpup-> (aka (get_pcpu())->) end >> up with a slb-miss, where: >> >> static __inline struct pcpu * >> get_pcpu(void) >> { >> struct pcpu *ret; >> >> __asm __volatile("mfsprg %0, 0" : "=r"(ret)); >> >> return (ret); >> } >> >> If the slb entry covering address ranges accessed >> based on sprg0 is ever replaced, no code based on >> getting sprg0's value to find the matching pcpu >> information is going to work, *including in the >> slb spill trap code* [GET_CPUINFO(%r?)]. > > Keep in mind that the PCPU pointer is in the DMAP, since it's in the > kernel image. It's not in KVA. However, some structures pointed to by > pcpu are in KVA, and those are what are faulting. Ahh, the slb is not involved for the DMAP's address range. Good to know. Also, no virtual address range is set up to map to the same memory and then used. Also good to know. (As is probably clear, I'm figuring things out as I go. Some things are easier to figure out from what I see than others. Thanks for the notes!) >> >> Does a kernel entry need to be reserved for the >> CPU that never is replaced so that sprg0 can always >> be used to find pcpu information via sprg0? >> >> This is something my hack did not deal with. >> And, in fact, trying to force 2 entries to exist >> at the same time, one for "dereferencing sprg0" >> and one for dereferencing the pc_curpcb so found >> is currently messy, given the way other things >> work. > > It should not. The kernel SLB miss handler runs entirely in real > mode, and __pcpu[] is in the DMAP, so real addresses match up directly > with DMAP addresses (modulo 0xc000000000000000). Good to know the limited usage context (for example, DMAP only addresses in sprg0). Thanks again for the notes. They definately help. >> >> The lack of handing may explain the (rare) hangups >> with the existing hack present. >> >> >>> I also expect that the old VM_MAX_KERNEL_ADDRESS value >>> explains the lack of slb-misses in old times: >>> >>> 0xe000000000000000UL: VM_MIN_KERNEL_ADDRESS >>> 0x0000000010000000UL: SEGMENT_LENGTH >>> 0xe0000001c7ffffffUL: VM_MAX_KERNEL_ADDRESS >>> >>> So 0x00..0x1c is 29 alternatives (decimal). That >>> fits in 64-1 slots, or even 32-1 slots: no >>> random replacements happened above or elsewhere. >>> That, in turn meant no testing of the handling >>> of any slb-misses back then. >>> >>> >>> [Other list messages suggest missing context synchronizing >>> instructions for slbmte and related instructions. The >>> history is not evidence about that, given the lack of >>> slb-misses.] > > Possibly. However, we may want direct control of the slots at boot > time so as to make sure none of the range we're prefaulting gets > replaced by the pseudo-random slot chooser. Sounds like a potential direction. Thanks again. === Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)