From owner-freebsd-arm@freebsd.org Sat Feb 8 22:41:09 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 92ABF24DA71 for ; Sat, 8 Feb 2020 22:41:09 +0000 (UTC) (envelope-from ag@x86.ch) Received: from einstein.sui-inter.net (einstein.sui-inter.net [80.74.145.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.sui-inter.net", Issuer "DigiCert SHA2 Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48FRwN6KZ3z4NgV for ; Sat, 8 Feb 2020 22:41:08 +0000 (UTC) (envelope-from ag@x86.ch) Received: from einstein.sui-inter.net (localhost [127.0.0.1]) by einstein.sui-inter.net (Postfix) with ESMTPSA id 31575B1C032D for ; Sat, 8 Feb 2020 23:41:07 +0100 (CET) Received-SPF: pass (einstein.sui-inter.net: connection is authenticated) Received: from [213.55.225.147] ([213.55.225.147]) by webmail.x86.ch (Horde Framework) with HTTPS; Sat, 08 Feb 2020 23:41:07 +0100 Date: Sat, 08 Feb 2020 23:41:07 +0100 Message-ID: <20200208234107.Horde.v-weX3Sbiyakmx_Y3IwJlV0@webmail.x86.ch> From: ag@x86.ch To: freebsd-arm@freebsd.org Subject: Re: RPi3 not using SMP? References: <20200208011940.GA8570@www.zefox.net> <6B6CCB8F-B56A-4758-BEEC-6418718C95CB@yahoo.com> <9F1B762C-D1DA-40F6-A2D6-451B40A39E4A@yahoo.com> <142E83E3-CD79-43B3-A86A-2AD45A778FB5@yahoo.com> In-Reply-To: User-Agent: Horde Application Framework 5 MIME-Version: 1.0 X-Rspamd-Queue-Id: 48FRwN6KZ3z4NgV X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of ag@x86.ch has no SPF policy when checking 80.74.145.25) smtp.mailfrom=ag@x86.ch X-Spamd-Result: default: False [5.19 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[147.225.55.213.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11]; IP_SCORE(0.48)[ipnet: 80.74.144.0/20(1.49), asn: 21069(0.87), country: CH(0.04)]; RWL_MAILSPIKE_GOOD(0.00)[25.145.74.80.rep.mailspike.net : 127.0.0.18]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; TO_DN_NONE(0.00)[]; AUTH_NA(1.00)[]; URI_COUNT_ODD(1.00)[11]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[x86.ch]; NEURAL_SPAM_MEDIUM(0.91)[0.914,0]; NEURAL_SPAM_LONG(1.00)[0.996,0]; FROM_NO_DN(0.00)[]; R_SPF_NA(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[25.145.74.80.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:21069, ipnet:80.74.144.0/20, country:CH]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Transfer-Encoding: 8bit Content-Disposition: inline Content-Description: Textnachricht X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Feb 2020 22:41:09 -0000 thank you interesting Zitat von Mark Millard via freebsd-arm : > On 2020-Feb-7, at 21:58, Mark Millard wrote: > >> [A note on avoiding a bad interpretation of >> my evidence.] >> >> On 2020-Feb-7, at 20:12, Mark Millard wrote: >> >>> On 2020-Feb-7, at 19:10, Mark Millard wrote: >>> >>>> [I now have some bounds on when PSCI_FN_VERSION >>>> gets a 0 result from smc #0 instead of the correct >>>> result: 2. Hopefully I can narrow the range more.] >>>> >>>> On 2020-Feb-7, at 18:46, Mark Millard wrote: >>>> >>>>> On 2020-Feb-7, at 17:19, bob prohaska wrote: >>>>> >>>>>> For some weeks now an RPi3 running -current has seemed rather slow.... >>>>>> >>>>>> On looking at the early part of the boot message the processor >>>>>> attributes seem rather scant: >>>>>> >>>>>> ...... >>>>>> elease APs...APs not started >>>>>> CPU  0: ARM Cortex-A53 r0p4 affinity:  0 >>>>>> Instruction Set Attributes 0 = >>>>>> Instruction Set Attributes 1 = <> >>>>>>     Processor Features 0 = >>>>>>     Processor Features 1 = <> >>>>>> Memory Model Features 0 = >>>>> ASID,1TB PA> >>>>>> Memory Model Features 1 = <8bit VMID> >>>>>> Memory Model Features 2 = <32bit CCIDX,48bit VA> >>>>>>         Debug Features 0 = <2 CTX BKPTs,4 Watchpoints,6 >>>>>> Breakpoints,PMUv3,Debugv8> >>>>>>         Debug Features 1 = <> >>>>>>     Auxiliary Features 0 = <> >>>>>>     Auxiliary Features 1 = <> >>>>>> CPU  1: (null) (null) r0p0 affinity:  0 >>>>>> CPU  2: (null) (null) r0p0 affinity:  0 >>>>>> CPU  3: (null) (null) r0p0 affinity:  0 >>>>>> ............ >>>>>> In a top window, STATE is reported as RUN, rather than the >>>>>> former CPUn. >>>>>> >>>>>> Is a software switch now required to enable multiprocessing? >>>>>> >>>>>> Or, could it be related to  the lines: >>>>>> psci0: PSCI version number mismatched with DT >>>>>> as pointed out by Mark M in reference to the cpu_reset failed >>>>>> problem, which is still manifest? >>>>>> >>>>>> The kernel is at r357644. >>>>> >>>>> Head -r356767's kernel does not have this problem for RPi3/4 used as >>>>> aarch64 FreeBSD. >>>>> >>>>> Head -r356776 and later all have this problem for both RPi3 and RPi4. >>>>> >>>>> Note: There are no head versions between those. >>>>> >>>>> The console log shows evidence of the problem much earlier. >>>>> Instead of saying: >>>>> >>>>> psci0: on ofwbus0 >>>>> psci0: PSCI version 0.2 compatible >>>>> >>>>> (once) it says: >>>>> >>>>> psci0: on ofwbus0 >>>>> psci0: PSCI version number 0 mismatched with DT, default 2 >>>>> device_attach: psci0 attach returned 6 >>>>> >>>>> (and those 3 lines repeat in various places) for which none of >>>>> them show up for -r356767 . >>>>> >>>>> Without identifying and using PSCI, the extra cores will not >>>>> start and the cpu(s) will not reset. (PSCI is the ARM interface >>>>> for doing such things.) >>>>> >>>>> I've no clue why, but the version number it gets in my RPi4 >>>>> experiments is 0. My only guess is that at some point >>>>> memory important to ARM's PSCI operation has been touched >>>>> and is no longer valid for the PSCI operation. >>>> >>>> I've been doing some crude printf-based information >>>> gathering for RPi4B boots (based on head -r357529 just >>>> because it is convenient in my context) and I have >>>> learned some about the staging of when PSCI_FN_VERSION >>>> works vs. when it is no longer working. >>>> >>>> For the below text extraction from a boot . . . >>>> Lines with: "PSCI_FN_VERSION 2" are working cases. >>>> Lines with: "PSCI_FN_VERSION 0" are no-longer-working cases. >>>> >>>> . . . >>>> FreeBSD clang version 9.0.1 (git@github.com:llvm/llvm-project.git >>>> c1a0a213378a458fbea1a5c77b315c7dce08fd05) (based on LLVM 9.0.1) >>>> uma_startup1 start: PSCI_FN_VERSION 2 >>>> uma_startup1 end: PSCI_FN_VERSION 2 >>>> uma_startup2 start: PSCI_FN_VERSION 2 >>>> uma_startup2 end: PSCI_FN_VERSION 2 >>>> VT(efifb): resolution 1824x984 >>>> module firmware already present! >>>> psci_fdt_get_callfn: method 'smc' >>>> psci_fdt_get_callfn: arm_smccc_hvc '0xffff0000008663b0' >>>> psci_fdt_get_callfn: arm_smccc_smc '0xffff0000008663c8' >>>> psci_init: PSCI_FN_VERSION 0 >>>> Starting CPU 1 (1) >>>> Starting CPU 2 (2) >>>> Starting CPU 3 (3) >>>> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs >>>> random: unblocking device. >>>> uma_startup3 start: PSCI_FN_VERSION 0 >>>> uma_startup3 start: PSCI_FN_VERSION 0 >>>> . . . >>>> >>>> So sometime after uma_startup2 ends but before psci_init >>>> sets up the normal use of arm_smccc_smc things are >>>> messed up. >>>> >>>> My guess is that one or more of the kernel memory >>>> allocations ended up getting memory used by ARM's >>>> PSCI implementation and the content was invalidated >>>> for ARM's PSCI purposes. >>> >>> The transition from working to not is before the >>> debug.verbose_sysinit=1 output turns in to >>> symbolic notation: >>> >>> . . . >>> subsystem 1800000 >>> mi_startup: PSCI_FN_VERSION 2 >>> 0xffff00000047e170(0)... done. >>> mi_startup: PSCI_FN_VERSION 2 >>> 0xffff0000007bda18(0)... done. >>> mi_startup: PSCI_FN_VERSION 2 >>> 0xffff0000004407e4(0)... done. >>> mi_startup: PSCI_FN_VERSION 2 >>> 0xffff0000004400c0(0xffff000000a75890)... done. >>> mi_startup: PSCI_FN_VERSION 2 >>> 0xffff0000004400c0(0xffff000000a76340)... done. >>> . . . >>> mi_startup: PSCI_FN_VERSION 2 >>> 0xffff0000004400c0(0xffff000000aca328)... done. >>> mi_startup: PSCI_FN_VERSION 2 >>> 0xffff0000004400c0(0xffff000000aca378)... done. >>> mi_startup: PSCI_FN_VERSION 0 >>> 0xffff0000004400c0(0xffff000000aca5f8)... done. >>> . . . >>> mi_startup: PSCI_FN_VERSION 0 >>> 0xffff0000004c62b0(0)... done. >>> mi_startup: PSCI_FN_VERSION 0 >>> 0xffff00000049e93c(0)... done. >>> mi_startup: PSCI_FN_VERSION 0 >>> linker_preload(0)... Preloaded elf kernel "/boot/kernel/kernel" at >>> 0xffff000001568000. >>> Preloaded elf module "/boot/kernel/umodem.ko" at 0xffff000001571020. >>> Preloaded elf module "/boot/kernel/ucom.ko" at 0xffff000001571838. >>> Preloaded boot_entropy_cache "/boot/entropy" at 0xffff000001572010. >>> module firmware already present! >>> done. >>> subsystem 1ffffff >>> mi_startup: PSCI_FN_VERSION 0 >>> ucom_init(0)... done. >>> subsystem 2000000 >>> mi_startup: PSCI_FN_VERSION 0 >>> procs_show_all_add(0)... done. >>> . . . >>> >>> It turns out that 0xffff0000004400c0 is: malloc_init . >>> >>> It turns out that 0xffff000000aca378 is for: M_IFMADDR : >>> >>> . . . >> >> Do not take the above as an indication of a stable stage for the >> change to happen. As my activities change the memory allocation >> patterns and the memory layout some (and possibly other sources >> of variability are involved), where in the boot sequence moves >> around. >> >> For example, while the above was before the output >> turned to symbolic notation, that need not be the >> case in general. >> >> So it is just an example of where is possible and limits a >> kind of activity that is sufficient for the change in status >> to happen, since it is a fairly specific example context. >> >> The way things move around, I'm not likely to come up with >> a narrower type of activity spanning the status change. >> >> I have no evidence on if the Excluded Memory Regions >> are sufficient or respected. >> >> For reference, I'd used >> >> set debug.verbose_sysinit=1 >> boot -v >> >> for the above example and my own printf's were >> involved. And I based this testing on head >> -r357529 instead of directly on -r356776 . The >> kernel was a non-debug build (with symbols). > > FYI: I ran into a note on the web reporting that for the > RPi4: > > The SoC does not seem to feature a secure memory controller of any > kind, so portions of DRAM can’t be protected properly from the > Non-secure world. > > ( > https://trustedfirmware-a.readthedocs.io/en/latest/plat/rpi4.html#tf-a-port-design > ) > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-armTo > unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" thank you interesting Links: ------ [1] http://www.zefox.net