From owner-freebsd-ppc@freebsd.org Thu May 2 06:21:56 2019 Return-Path: Delivered-To: freebsd-ppc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F8EC158A3E2 for ; Thu, 2 May 2019 06:21:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-19.consmr.mail.gq1.yahoo.com (sonic305-19.consmr.mail.gq1.yahoo.com [98.137.64.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B8B17756D for ; Thu, 2 May 2019 06:21:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: OJLlJ5kVM1lbX9FPk1.UqSjoiWWrIFhNnWoxOjx2WbrICPBugom.8DE1sNim3J9 2xVIioIOB9deV5ZbZvP_jVOWK_uztYS3qMcvQQabHZaWtw3EFvpZiwxw7tOqtpD7z4M7c97g.rEO KpjSUcBZxHfLG0OFRafJER.hyxG4_uC64KwdTZu9m.KsoKrZs12ZBDCU7YX9Fl9.KeidZXVXL410 muZttRmKyckAl8Q_yl4Nyw1MHlO0_TW8b2PPRIPMdlAr4HXi46FQrZXh6Q1FS.72hj5nDwUQ8_Wi tRUyEXDP3pnNzfUXSK2HZNwQMTqd_WaCdwcE6ow2Erh7QDq2MDB92jVeqTnpq4vF15saAWHKMeg9 wYrqVfr.S6ASDDvJ7XC6.w0.LnwSfZ3HKZg0fF99eJ.o7yzqEjV_xSI0SlvorVfYW5Q6YRth6FzR z36Jt4Qe3q86xzKsKshZFLjHRe8flKa2CaYpOY9nL6j.ebw6nqqX07zUcCKf1RY7MzGWiFM8IA62 CH4uGQF41KA6RA4o.MBRcB9NtgndUoDO01dwJ1D_KOGBGEXtqrbIWDmLqJ7HeFdDeH7Z4jv6CcQ2 _ZvNTVT4Y_iZGg0kIvuAdL2MgeXy_vucByXlzuloG3guEm5fgUtmOM5dJdBpg_QvO7vFDgbadIDf eg70UOdnhJM97nViuix7XTX9xNKBvyuH_SNh2LukvQq8nCCKKp0_rrXMZ5ROAvaeCIxhv0A7qHN. _Zf8dHc2VPn25zCP4Xh3vUoZcsTU0fSvN0ikmJIteXuBPuoN8382IGO94Eqgzq5fMC7QDFeK25cV jPESqHQ5DBQBtOSkA65zGLvm2um8ouL7MZ56aRVHIMD0LS7UxF7xzxUSzq1mb_tyZQEg2RMAExf6 hMvTMTq43xkM4gmpwjgpvEuvdf5WzQ8dTo7YTki1RbNflAwrsXbcc4igOZhtm8HYDCYLpblz_w6l lffrMQniUXQrcDDQlyCQ65pCvlaDOjbp.Dmyhv4s_WxVh7lXyUfUoP1VM63oYw2kXzAjRveKwN_7 NSt1uI0jro03ovmuJ61KsofiBVGyqaEwt7blwB2J9QeV_YmKNE6i5.c2c.DRHMvvW3OysaBPjvqh zoCi6E8zcGljN3AunnM0A69zIVampEsaI3_wG4AVquRyhgPGf9RGXhy5NUxWqWM14NJF3IZusxqB r Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Thu, 2 May 2019 06:21:46 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.103]) ([76.115.7.162]) by smtp402.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 45fdae4cde42bb0374e9baf2d6fa60b2; Thu, 02 May 2019 06:21:44 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: How many segments does it take to span from VM_MIN_KERNEL_ADDRESS through VM_MAX_SAFE_KERNEL_ADDRESS? 128 in moea64_late_bootstrap From: Mark Millard In-Reply-To: <1B8116F2-9749-4331-95BD-D528AA52A771@yahoo.com> Date: Wed, 1 May 2019 23:21:43 -0700 Cc: FreeBSD PowerPC ML Content-Transfer-Encoding: quoted-printable Message-Id: References: <3C69CF7C-7F33-4C79-92C0-3493A1294996@yahoo.com> <6159F4A6-9431-4B99-AA62-451B8DF08A6E@yahoo.com> <20190501094029.542c5f46@titan.knownspace> <212E50E5-7EB1-4381-A662-D5EACB1E5D46@yahoo.com> <20190501165403.7d8d1f8f@titan.knownspace> <1B8116F2-9749-4331-95BD-D528AA52A771@yahoo.com> To: Justin Hibbits X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 7B8B17756D X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.75 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[yahoo.com]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; NEURAL_SPAM_SHORT(0.92)[0.920,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(1.55)[ip: (6.21), ipnet: 98.137.64.0/21(0.88), asn: 36647(0.71), country: US(-0.06)]; NEURAL_SPAM_MEDIUM(0.59)[0.586,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.21)[0.206,0]; RCVD_IN_DNSWL_NONE(0.00)[82.64.137.98.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 May 2019 06:21:56 -0000 [Some results, mixed Im afraid.] On 2019-May-1, at 17:22, Mark Millard wrote: > On 2019-May-1, at 14:54, Justin Hibbits = wrote: >=20 >> On Wed, 1 May 2019 14:35:56 -0700 >> Mark Millard wrote: >>=20 >>>>> What happens if you revert all your patches, =20 >>>>=20 >>>> Most of the patches in Bugzilla 233863 are not for this >>>> issue at all and are not tied to starting the non-bsp >>>> cpus. (The one for improving how close the Time Base >>>> registers are is tied to starting these cpus.) Only the >>>> aim/mp_cpudep.c and aim/slb.c changes seem relevant. >>>>=20 >>>> Are you worried about some form of interaction that means >>>> I need to avoid patches for other issues? >>>>=20 >>>> Note: for now I'm staying at using head -r345758 as the >>>> basis for my experiments. >>>>=20 >>>>> and change this loop to >>>>> stop at n_slb? So something more akin to: >>>>>=20 >>>>> int i =3D 0; >>>>>=20 >>>>> for (va =3D virtual_avail; va < virtual_end && i < n_slb - >>>>> 1; va +=3D SEGMENT_LENGTH, i++); >>>>> ... >>>>>=20 >>>>> If it reliably boots with that, then that's fine. We can prefault >>>>> as much as we can and leave the rest for on-demand. =20 >>>>=20 >>>> I'm happy to experiment with this loop without my hack >>>> for forcing the slb entry to exist in cpudep_ap_bootstrap. >>>>=20 >>>> But, it seems to presume that the pc_curpcb's will >>>> all always point into the lower address range spanned >>>> when cpudep_ap_bootstrap is executing on the cpu. >>>> Does some known property limit the pc_curpcb-> >>>> references to such? Only that would be sure to >>>> avoid an slb-miss at that stage. Or is this just an >>>> alternate hack or a means of getting evidence, not a >>>> proposed solution? >>>>=20 >>>> (Again, I'm happy to disable my hack that forces the >>>> slb entry and to try the loop suggested.) =20 >> ... >>> And the patch for the loop looks like: >>>=20 >>> virtual_end =3D VM_MAX_SAFE_KERNEL_ADDRESS;=20 >>>=20 >>> /* >>> - * Map the entire KVA range into the SLB. We must not fault >>> there. >>> + * Map the lower-address part of the KVA range into the SLB. >>> We must not fault there. */ >>> #ifdef __powerpc64__ >>> - for (va =3D virtual_avail; va < virtual_end; va +=3D >>> SEGMENT_LENGTH) >>> + i =3D 0; >>> + for (va =3D virtual_avail; va < virtual_end && i>> +=3D SEGMENT_LENGTH, i++) moea64_bootstrap_slb_prefault(va, 0); >>> #endif >>>=20 >>=20 >> Yep, that's the patch I was going for. >>=20 >>>=20 >>> So I've built, installed, and have tested some: it did not go well >>> overall. >>>=20 >>> Using: >>>=20 >>> OK set debug.verbose_sysinit=3D1 >>>=20 >>> to show better context about where the hangs occur, shows: >>> (Typed from a screen picture.) >>>=20 >>> subsystem a800000 >>> boot_run_interrupt_driven_config_hooks(0)... >>> . . . (omitted) . . . >>> done. >>> vt_upgrade(&vt_consdev). . . >>>=20 >>> The "vt_upgrade(&vt_consdev). . ." never says done when booting >>> hangs with the above changes. >>>=20 >>> Trying to boot a bunch of times did produce one >>> completed boot, all 4 cpus working. Otherwise I'm >>> using kernel.old to manage to complete a boot. >>>=20 >>> I'll note that "vt_upgrade(&vt_consdev). . ." is where >>> Dennis Clarke reported for the hangups that he was >>> seeing, without any of my patches being available back >>> then: 2019-Feb-14. >>=20 >> Maybe try the commit that caused the problem back in July? r334498. >>=20 >=20 > I'd already started down the path of getting materials from: >=20 > = https://artifact.ci.freebsd.org/snapshot/head/r347003/powerpc/powerpc64/ >=20 > and putting them on a separate SSD that I sometimes use for = artifact.ci > or snapshot experiments. Also: checking out matching svn sources for > -r347003 and then doing a buildworld buildkernel with a bootstrap gcc > 4.2.1 compiler used. I'm verifying that I can build it before making > the source changes for the kernel. The build is of a debug kernel > (GENERIC64). >=20 > The test buildworld is still in process. >=20 > Let me know if this is insufficient for your purposes. I could revert > to: >=20 > = https://artifact.ci.freebsd.org/snapshot/head/r334594/powerpc/powerpc64/ >=20 > (There is no head/r334498/ and the first after that with a > powerpc64/ is head/r334594/ .) >=20 > For either head/r347003/ or head/r334594/ : >=20 > Use of artifact materials allows using officially built files for > every file but some specific file(s) that I replace. It also allows > comparison/contrast of the behavior of the official files vs. when > adjusted ones are substituted. >=20 > Use of artifact-version materials also means that I know I'm using > a vintage that actually built --and so I hope to avoid other problems > getting in the way. I present without-the-patch results before presenting with-the-patch results. The end result is mixed, I'm afraid. As for the results without any patch, just artifact materials . . . Note: "Add debug.verbose_sysinit tunable for VERBOSE_SYSINIT" was not checked-in until -r335458 . Trying to boot without any updates or rebuilds, just artifact materials shows variable stopping points: (For debug.verbose_sysinit=3D1 :) -r347003 stops sometimes at: vt_upgrade(&vt_consdev). . . -r347003 stops sometimes at: cpu_mp_unleash(0). . . -r334594 stops after: ada0 lines, VERBOSE_SYSINIT not built in So I had to build my own -r334594 kernel to see verbose_sysinit information about the stopping point. Again, no patch here, I just copied over my build of the /boot/kernel/kernel file: -r334594 stops sometimes at: vt_upgrade(&vt_consdev). . . -r334594 stops sometimes at: cpu_mp_unleash(0). . . Summary thus far: I did not find any obvious difference in how often each stops in either of the alternatives. So I'm seeing if the proposed patch changes the behavior of -r347003 . Later test of patched -r347003 . . . The patched kernel is based on: # svnlite diff /mnt/usr/src/ | more Index: /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c (revision 347003) +++ /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c (working copy) @@ -959,7 +959,8 @@ * Map the entire KVA range into the SLB. We must not fault = there. */ #ifdef __powerpc64__ - for (va =3D virtual_avail; va < virtual_end; va +=3D = SEGMENT_LENGTH) + i =3D 0; + for (va =3D virtual_avail; va < virtual_end && i prompt. When I enter "ufs:/dev/daa0s3" I get a panic for: panic: mtx_lock of spin mutex WWV @ = /mnt/usr/src/sys/powerpc/aim/mmu_oea64.c:2812 (it is a debug-kernel build) For reference, line 2812 is: PMAP_LOCK(pm); panic is reached via an interesting(?) call chain, showing the backtrace (typed from screen pictures): .__mtx_lock_flags+0xd4 .moea64_sync_icache+0x48 .pmap_sync_icache+0x90 .ppc_instr_emulate+0x1b4 .trap+0x10fc .powerpc_interrupt+0x2cc user PGM trap by 0x810053bb4: srr1=3D0x900000000008d032 r1=3D0x3ffffffffffffcc00 cr=3D0x20002024 xer=3D0 ctr=3D0x1 = r2=3D0x81007bdd0 frame=3D0xe000000070ca9810 It was thread pid 28 tid 100097 So far these details seem consistent. But I will note that openfirmware use via ofwdump -ap and the like causes system crashes going back to when the direct map base was moved to high memory addresses ( -r330610 and later ). This is one of the reasons I want to avoid openfirmware and use the conversion to fdt instead. (There is a -r330614 artifact to test such crashes with --or use a later one that otherwise boots.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)