From owner-freebsd-arm@freebsd.org Sat Feb 8 05:59:04 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 31C8C239366 for ; Sat, 8 Feb 2020 05:59:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-8.consmr.mail.gq1.yahoo.com (sonic307-8.consmr.mail.gq1.yahoo.com [98.137.64.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48F1h64Qp4z4Rt7 for ; Sat, 8 Feb 2020 05:59:02 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 3WdNNJ4VM1m..VaOfyCQNC0mChZhFrz3FTapH.YoDhvEQfKHRzx48B_eIJpazOj 5F9kVhX6D_6gcoN2P2KMebMm3prGfVUp1hW_KcAh64goPFIkbgDeJsUe.8ILul.oJPHzbUEDTAJN 3sv.z8UK7mZnPysZGQ6EzNLndeYMVKGVMktRHo6h7pKFUplHu_SRFh2ZtulpZ1JOETcc1vEgg10v fPxqxTyBa0z2NpKrSmA2tBkBgEhCudjuvmhIU8wpq8MutsUr5gYlBmspxijcxNtFtkC_9egE0TZC 9RvwrKag0o3t71JqUnB8TPI0xJbOjNWdI19bpliPgDFq6o.tj4TWqTOTqWvUQLgpj6JpHe6fz5o1 6sBHlyq9p1uyi9fInfEZpweBjre2gI1IJZseaN57a9c9.pWGQ1CGe5Fb4Rtn1rthud3eWjaWwMOX nKrj1lNmozui_X6DMR2LwTpTemOk.onfk7svuKeQkHt3zSdip77b8CAOQM5EhEy50LTnk8VkXHo5 m1NY9WmcJFtraBmTgojAGyzAU71IrHv3bOv_XNJx9c_HoYlWX_tifmqRbMAJgXkq3810J_1hvk95 DZc3DYq10Ep_Vu4LJgBFHoz24Tql0MVU7wie04r9GQ6w4wy2Pj0FkzPpWqnqsrISIRrizQkLsou2 XdMQwRDE4HYUeQOdx5xJBU5PvBCm5k_ghzrZcCWvFHalb8snTJO5BC2k6s1RlVSWnrm35oLHdAxc wBUPLJmyLe7soCjBaKHn3BzhNCtOEIyU9e4nsU_bnsBfeP6.YAPCK_3ZWmNtBSeRvkosP0sl3.lU yFLc2c0N.gxqNaXmL7GCU1.4.v.d7KwFeOrIG.pKttJzSj.BKJ6Tb5B.BM_WzPd7rVIbgnu3bD0j Dvwo9ZuBVcHeaFhDnetWOqIAb3kUf_ymnHFccaJO.Qn9TPS0QZHSAcX8Bjjk.MGtTRHUPhRYa7qq dNb3aOrW0dTHRtdv3OUSNuQH.7TAeqZUmFVlFKdOQlA5PS_TEh_NZArP3mzGYQXU8ULmjmeFNmEq sP40t6kX_0FAufxlUrg1GHcCgsutC0vjA4oI4culRIH6IABsPJ9kQpImRP5LOj9TEbtY9WKY_sVJ iM7iBp7SQqTmYgMT2Et7S4RpdEwNfkOXRaN09gO5odBZnLL9e2oLVdUEjv1i_7lmyyTQfuO0TNvF doXhHHegLJNsc2zjhJvxsfG9.BlvLPSPNA4S24ZoTl5k4wQXO0l8FF9cNlIcAUSN5uEpAKU3txFw lphdCP.dDbLS7ghiU.HOSZRPoqRxTUFkhVLZtMlq_0c_H38EOx2JCO8F1OuSUUHxrcP.460Novet 7uvtnSuF8JYqZi4qjdUGsEwgG7RDDTghyWP.E9ugEpTndxqca.WS9x5nHEQLOWkDoK6Ytn6PigLQ 8JCSSsjwWNdX3RjhM1.6oSFvCvC5U0K4szvnv7_Q1na8- Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Sat, 8 Feb 2020 05:59:00 +0000 Received: by smtp403.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID ae4148e2143e4a19eb73457d4d6aaa98; Sat, 08 Feb 2020 05:58:59 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: RPi3 not using SMP? From: Mark Millard In-Reply-To: Date: Fri, 7 Feb 2020 21:58:59 -0800 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <142E83E3-CD79-43B3-A86A-2AD45A778FB5@yahoo.com> References: <20200208011940.GA8570@www.zefox.net> <6B6CCB8F-B56A-4758-BEEC-6418718C95CB@yahoo.com> <9F1B762C-D1DA-40F6-A2D6-451B40A39E4A@yahoo.com> To: bob prohaska , "jeff@freebsd.org" X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48F1h64Qp4z4Rt7 X-Spamd-Bar: + X-Spamd-Result: default: False [1.05 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.03)[-0.026,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (5.87), ipnet: 98.137.64.0/21(0.83), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.58)[0.576,0]; RCVD_IN_DNSWL_NONE(0.00)[32.64.137.98.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Feb 2020 05:59:04 -0000 [A note on avoiding a bad interpretation of my evidence.] On 2020-Feb-7, at 20:12, Mark Millard wrote: > On 2020-Feb-7, at 19:10, Mark Millard wrote: >=20 >> [I now have some bounds on when PSCI_FN_VERSION >> gets a 0 result from smc #0 instead of the correct >> result: 2. Hopefully I can narrow the range more.] >>=20 >> On 2020-Feb-7, at 18:46, Mark Millard wrote: >>=20 >>=20 >>=20 >>> On 2020-Feb-7, at 17:19, bob prohaska wrote: >>>=20 >>>> For some weeks now an RPi3 running -current has seemed rather = slow.... >>>>=20 >>>> On looking at the early part of the boot message the processor >>>> attributes seem rather scant: >>>>=20 >>>> ...... >>>> elease APs...APs not started >>>> CPU 0: ARM Cortex-A53 r0p4 affinity: 0 >>>> Instruction Set Attributes 0 =3D >>>> Instruction Set Attributes 1 =3D <> >>>> Processor Features 0 =3D >>>> Processor Features 1 =3D <> >>>> Memory Model Features 0 =3D >>>> Memory Model Features 1 =3D <8bit VMID> >>>> Memory Model Features 2 =3D <32bit CCIDX,48bit VA> >>>> Debug Features 0 =3D <2 CTX BKPTs,4 Watchpoints,6 = Breakpoints,PMUv3,Debugv8> >>>> Debug Features 1 =3D <> >>>> Auxiliary Features 0 =3D <> >>>> Auxiliary Features 1 =3D <> >>>> CPU 1: (null) (null) r0p0 affinity: 0 >>>> CPU 2: (null) (null) r0p0 affinity: 0 >>>> CPU 3: (null) (null) r0p0 affinity: 0 >>>> ............ >>>> In a top window, STATE is reported as RUN, rather than the >>>> former CPUn. >>>>=20 >>>> Is a software switch now required to enable multiprocessing? >>>>=20 >>>> Or, could it be related to the lines: >>>> psci0: PSCI version number mismatched with DT=20 >>>> as pointed out by Mark M in reference to the cpu_reset failed >>>> problem, which is still manifest?=20 >>>>=20 >>>> The kernel is at r357644. >>>=20 >>> Head -r356767's kernel does not have this problem for RPi3/4 used as >>> aarch64 FreeBSD. >>>=20 >>> Head -r356776 and later all have this problem for both RPi3 and = RPi4. >>>=20 >>> Note: There are no head versions between those. >>>=20 >>> The console log shows evidence of the problem much earlier. >>> Instead of saying: >>>=20 >>> psci0: on ofwbus0 >>> psci0: PSCI version 0.2 compatible >>>=20 >>> (once) it says: >>>=20 >>> psci0: on ofwbus0 >>> psci0: PSCI version number 0 mismatched with DT, default 2 >>> device_attach: psci0 attach returned 6 >>>=20 >>> (and those 3 lines repeat in various places) for which none of >>> them show up for -r356767 . >>>=20 >>> Without identifying and using PSCI, the extra cores will not >>> start and the cpu(s) will not reset. (PSCI is the ARM interface >>> for doing such things.) >>>=20 >>> I've no clue why, but the version number it gets in my RPi4 >>> experiments is 0. My only guess is that at some point >>> memory important to ARM's PSCI operation has been touched >>> and is no longer valid for the PSCI operation. >>=20 >> I've been doing some crude printf-based information >> gathering for RPi4B boots (based on head -r357529 just >> because it is convenient in my context) and I have >> learned some about the staging of when PSCI_FN_VERSION >> works vs. when it is no longer working. >>=20 >> For the below text extraction from a boot . . . >> Lines with: "PSCI_FN_VERSION 2" are working cases. >> Lines with: "PSCI_FN_VERSION 0" are no-longer-working cases. >>=20 >> . . . >> FreeBSD clang version 9.0.1 (git@github.com:llvm/llvm-project.git = c1a0a213378a458fbea1a5c77b315c7dce08fd05) (based on LLVM 9.0.1) >> uma_startup1 start: PSCI_FN_VERSION 2 >> uma_startup1 end: PSCI_FN_VERSION 2 >> uma_startup2 start: PSCI_FN_VERSION 2 >> uma_startup2 end: PSCI_FN_VERSION 2 >> VT(efifb): resolution 1824x984 >> module firmware already present! >> psci_fdt_get_callfn: method 'smc' >> psci_fdt_get_callfn: arm_smccc_hvc '0xffff0000008663b0' >> psci_fdt_get_callfn: arm_smccc_smc '0xffff0000008663c8' >> psci_init: PSCI_FN_VERSION 0 >> Starting CPU 1 (1) >> Starting CPU 2 (2) >> Starting CPU 3 (3) >> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs >> random: unblocking device. >> uma_startup3 start: PSCI_FN_VERSION 0 >> uma_startup3 start: PSCI_FN_VERSION 0 >> . . . >>=20 >> So sometime after uma_startup2 ends but before psci_init >> sets up the normal use of arm_smccc_smc things are >> messed up. >>=20 >> My guess is that one or more of the kernel memory >> allocations ended up getting memory used by ARM's >> PSCI implementation and the content was invalidated >> for ARM's PSCI purposes. >>=20 >=20 > The transition from working to not is before the > debug.verbose_sysinit=3D1 output turns in to > symbolic notation: >=20 > . . . > subsystem 1800000 > mi_startup: PSCI_FN_VERSION 2 > 0xffff00000047e170(0)... done. > mi_startup: PSCI_FN_VERSION 2 > 0xffff0000007bda18(0)... done. > mi_startup: PSCI_FN_VERSION 2 > 0xffff0000004407e4(0)... done. > mi_startup: PSCI_FN_VERSION 2 > 0xffff0000004400c0(0xffff000000a75890)... done. > mi_startup: PSCI_FN_VERSION 2 > 0xffff0000004400c0(0xffff000000a76340)... done. > . . . > mi_startup: PSCI_FN_VERSION 2 > 0xffff0000004400c0(0xffff000000aca328)... done. > mi_startup: PSCI_FN_VERSION 2 > 0xffff0000004400c0(0xffff000000aca378)... done. > mi_startup: PSCI_FN_VERSION 0 > 0xffff0000004400c0(0xffff000000aca5f8)... done. > . . . > mi_startup: PSCI_FN_VERSION 0 > 0xffff0000004c62b0(0)... done. > mi_startup: PSCI_FN_VERSION 0 > 0xffff00000049e93c(0)... done. > mi_startup: PSCI_FN_VERSION 0 > linker_preload(0)... Preloaded elf kernel "/boot/kernel/kernel" at = 0xffff000001568000. > Preloaded elf module "/boot/kernel/umodem.ko" at 0xffff000001571020. > Preloaded elf module "/boot/kernel/ucom.ko" at 0xffff000001571838. > Preloaded boot_entropy_cache "/boot/entropy" at 0xffff000001572010. > module firmware already present! > done. > subsystem 1ffffff > mi_startup: PSCI_FN_VERSION 0 > ucom_init(0)... done. > subsystem 2000000 > mi_startup: PSCI_FN_VERSION 0 > procs_show_all_add(0)... done. > . . . >=20 >=20 > It turns out that 0xffff0000004400c0 is: malloc_init . >=20 > It turns out that 0xffff000000aca378 is for: M_IFMADDR : >=20 > . . . >=20 Do not take the above as an indication of a stable stage for the change to happen. As my activities change the memory allocation patterns and the memory layout some (and possibly other sources of variability are involved), where in the boot sequence moves around. For example, while the above was before the output turned to symbolic notation, that need not be the case in general. So it is just an example of where is possible and limits a kind of activity that is sufficient for the change in status to happen, since it is a fairly specific example context. The way things move around, I'm not likely to come up with a narrower type of activity spanning the status change. I have no evidence on if the Excluded Memory Regions are sufficient or respected. For reference, I'd used set debug.verbose_sysinit=3D1 boot -v for the above example and my own printf's were involved. And I based this testing on head -r357529 instead of directly on -r356776 . The kernel was a non-debug build (with symbols). =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)