From owner-freebsd-ppc@freebsd.org Wed May 1 21:36:08 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 764D915A150F for ; Wed, 1 May 2019 21:36:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-24.consmr.mail.ne1.yahoo.com (sonic311-24.consmr.mail.ne1.yahoo.com [66.163.188.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0ACB78E813 for ; Wed, 1 May 2019 21:36:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ppDRS8MVM1nOTyykYJfN5l_1EmbA1ZHCXDj15kEiHn2skesj3YqyTr6FTxTITy7 jOnQa7n3us6Aalh1LOf_thvNyRD6.v7ajQgfUjVHFNKLs_0e7ML9vBvxFeS.Oi7YC2W74gXCpSjz u_NDACLEdC82xlT7.5GvghwhHy1A4K.O4YTf5asdSwFBOwNkNnduMvKrwGv1az7ht_rG5REus05B meKX9MLYSKYLalp9TyJz1bcFJvGAkch3n9GvhDvKVIykTL8i__nLXYl2CJFi8mqp0uSLDE1QX3Tj dHx1LhXX85IBKRkLW2AFY0VW8AwL1n8tvi17cqT06aCnO6A4iydhGs5ZIfhJwzOmpWFDBuSoeiJN OJQqrPTyHzLp6WdGTXIbor1XcJMMxJix8wvafZCZNQSa9GtbzqWf8ZqBX1BQwQkp7_t2UnF1qaCW RfKJaoAj0UqvpoM69CJWq2fF9odgzlKdkY_Cqz4TcqfFok3uQQjx6ZAdYiJf5P95QPHwinTzotNu 2qHS_bBNom4_jrzrgGaPjv6oU37fl.26R8G.GvXOGGcrOFfap3tY3hN5zDhYwTsdRocpFG6ApmVv JDkUALKUIwvBi0wa.u4lGd8j_Wb0dpEuua0cGlXu8fxQ713Crb_geucTl.ykPfy0DeiYK046uvda jwVFdtHXo6.N.cCH.Z6ppWxea.Shh5uLQ5zF0XzvRapO467Ta0fu7rY7xNxudQ1ac3mjGvlYt8uC QG4KjrBPnilqyvM3cunLdamn2ezQ4bR2iLYQ1CZNjA9Xd4LZZJTRbXHZy6WbBFJnn4O0iu.S1Jv6 ep.MrGr06cUdCyyNeFKdHvrd2b4NsZjTRwV2RzDmZ7VogPcDNI7uAAHWkeTzDU.SvUgnAoOiBBeB nu34_vL2k8_Pd6CpRjVkM7jjjfINrt6yKmYYmrXKeKH0Mja63qMsAeh7K5Jg_6s0rhQibVcj1q.i 3YV5AAfYyDIOS8Wmy6KnYEkYzvBlUtxyM09dORMN6dOlu8Prytsj6rCOQcNFG39DeD03Xd0oYp_T apqR_OvaJza8mNYM.XYuWEASJ8S4g4FZWK5R9SYmMa526RmDOErGw1VjMu1Hx8.weYJpBoTSTw3C _W0z.dztkoLprAl3gQQ1EYA1nArwx1bwhLmOxL49c0XOn10o6bOCxIsT6sODQ3rfUcVejKuA0Utq AOaJw.4r3lLDN Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.ne1.yahoo.com with HTTP; Wed, 1 May 2019 21:35:59 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp402.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 7543420c27aee898f52a8c062f1736c8; Wed, 01 May 2019 21:35:58 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap From: Mark Millard In-Reply-To: <212E50E5-7EB1-4381-A662-D5EACB1E5D46@yahoo.com> Date: Wed, 1 May 2019 14:35:56 -0700 Cc: FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com> <6159F4A6-9431-4B99-AA62-451B8DF08A6E@yahoo.com> <20190501094029.542c5f46@titan.knownspace> <212E50E5-7EB1-4381-A662-D5EACB1E5D46@yahoo.com> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 0ACB78E813 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.83 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.88)[0.883,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(2.11)[ip: (8.11), ipnet: 66.163.184.0/21(1.39), asn: 36646(1.11), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.31)[0.306,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.04)[0.045,0]; RCVD_IN_DNSWL_NONE(0.00)[205.188.163.66.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 01 May 2019 21:36:08 -0000 [This just reports about the experiment, but not from an official head version or snapshot: preliminary information in the interest of time. It hangs, but in a different place/stage than cpudep_ap_bootstrap , matching Dennis Clarke's 2019-Feb-14 reports about hangups, from before my patches were available.] On 2019-May-1, at 11:51, Mark Millard wrote: > On 2019-May-1, at 07:40, Justin Hibbits = wrote: >=20 >> On Tue, 30 Apr 2019 21:45:00 -0700 >> Mark Millard wrote: >>=20 >>> [I realized another implication about a another point of >>> potential slb-misses in cpudep_ap_bootstrap: the >>> address in sprg0 on the cpu might end up not able to be >>> dereferenced.] >>>=20 >>> On 2019-Apr-30, at 20:58, Mark Millard wrote: >>>=20 >>>> [At the end this note shows why the old VM_MAX_KERNEL_ADDRESS >>>> lead to no slb-miss exceptions in cpudep_ap_bootstrap.] >>>>=20 >>>> There is code in moea64_late_bootstrap that looks like: >>>>=20 >>>> virtual_avail =3D VM_MIN_KERNEL_ADDRESS; >>>> virtual_end =3D VM_MAX_SAFE_KERNEL_ADDRESS; >>>>=20 >>>> /* >>>> * Map the entire KVA range into the SLB. We must not fault >>>> there. */ >>>> #ifdef __powerpc64__ >>>> for (va =3D virtual_avail; va < virtual_end; va +=3D >>>> SEGMENT_LENGTH) moea64_bootstrap_slb_prefault(va, 0); >>>> #endif >>=20 >> What happens if you revert all your patches, >=20 > Most of the patches in Bugzilla 233863 are not for this > issue at all and are not tied to starting the non-bsp > cpus. (The one for improving how close the Time Base > registers are is tied to starting these cpus.) Only the > aim/mp_cpudep.c and aim/slb.c changes seem relevant. >=20 > Are you worried about some form of interaction that means > I need to avoid patches for other issues? >=20 > Note: for now I'm staying at using head -r345758 as the > basis for my experiments. >=20 >> and change this loop to >> stop at n_slb? So something more akin to: >>=20 >> int i =3D 0; >>=20 >> for (va =3D virtual_avail; va < virtual_end && i < n_slb - 1; va >> +=3D SEGMENT_LENGTH, i++); >> ... >>=20 >> If it reliably boots with that, then that's fine. We can prefault as >> much as we can and leave the rest for on-demand. >=20 > I'm happy to experiment with this loop without my hack > for forcing the slb entry to exist in cpudep_ap_bootstrap. >=20 > But, it seems to presume that the pc_curpcb's will > all always point into the lower address range spanned > when cpudep_ap_bootstrap is executing on the cpu. > Does some known property limit the pc_curpcb-> > references to such? Only that would be sure to > avoid an slb-miss at that stage. Or is this just an > alternate hack or a means of getting evidence, not a > proposed solution? >=20 > (Again, I'm happy to disable my hack that forces the > slb entry and to try the loop suggested.) Note: I've not started any experiments for isync's related to instructions such as slbmte yet: that was all just inspection and reading about requirements so far. So to disable my slb-force-no-miss hack in cpudep_ap_bootstrap I reverted it: # svnlite revert /usr/src/sys/powerpc/aim/mp_cpudep.c = /usr/src/sys/powerpc/aim/slb.c Reverted 'sys/powerpc/aim/mp_cpudep.c' Reverted 'sys/powerpc/aim/slb.c' (hack_into_slb_if_needed(...) was implemented in mp_cpudep.c and used in slb.c before reverting.) And the patch for the loop looks like: virtual_end =3D VM_MAX_SAFE_KERNEL_ADDRESS;=20 =20 /* - * Map the entire KVA range into the SLB. We must not fault = there. + * Map the lower-address part of the KVA range into the SLB. We = must not fault there. */ #ifdef __powerpc64__ - for (va =3D virtual_avail; va < virtual_end; va +=3D = SEGMENT_LENGTH) + i =3D 0; + for (va =3D virtual_avail; va < virtual_end && i The idea with this is if you can test with stock -CURRENT (or > post-VM_KERNEL_MAXADDR change), to eliminate any other variables. = This > is *only* for testing that it brings up the APs, not that they're > properly synced. That will happen with other changes. This is a > proposed solution. =46rom my understanding, we typically allocate = from > low to high for KVA allocations, so keeping the low addresses in = memory > long enough to bring up the APs to sanity is the goal, so the commit > would be along the lines of "Prefault as much of KVA as we can fit = into > the SLB". This will have the sleep-gets-stuck problem, likely normally happening quickly after booting and logging in (presuming a boot). The resulting boots for such are not always all that useful after various threads hang up. Also, getting such a almost-exactly-head-revision variant set up without messing up my current context will take some time: I'm not set up for such. I currently have no access to a cross-build environment, the activity is self hosted on a 2-socket/2-cores-each G5. So I will have to build from a context that has patches (or is too old). Thus the preliminary results above that I could produce quickly that are not from the context that you asked for. But it also appears that "vt_upgrade(&vt_consdev). . ." would not be tied to cpudep_ap_bootstrap and evaluating: sp =3D pcpup->pc_curpcb->pcb_sp Still, I'll work on having a gcc-4.2.1-based just-head context built, not that it would install and boot in that state. So I will have to build from a context that has patches, using a different source tree for the "self-hosted cross build" to do your kind of experiment. But I'd then be ready for "self-hosted cross built" experiments. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)