From owner-freebsd-arm@freebsd.org Sat Feb 8 04:12:55 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 63E1F2375CB for ; Sat, 8 Feb 2020 04:12:55 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic314-21.consmr.mail.gq1.yahoo.com (sonic314-21.consmr.mail.gq1.yahoo.com [98.137.69.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48DzKd5YFtz4Ms8 for ; Sat, 8 Feb 2020 04:12:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: K7kIokIVM1mrE2PZVFYg5qtioKwru7mqCpvKMfb8XhTkoY.LHYBM_NowGHIn2dl smnuqkH5ewRw2_awPrNtIAvju4AsUxSkqS.S.qUhGpjOdLYqVtWNAJmHY7k4bZAxqqcWIAkgaq_B 5xjmwEUsK.lLuucEhqc.7qdFqSHcY_XxGnkhu2DZHqgYkG3jsCGG.kLe..IOSpq.y8O3cuA1kBxI 41RUq9Zu2J7rtYkoVB.NB.UVYUIB93ZFiV.1CJJu8LZ6mIa7wGTzVHT_3ZogDwpsJrsyLuaMaUaE KyM0TlqKUmAQm8AI3YBKP4SThgDgdfrsRWJuYOB09M4QR7YeJ38M2G_j4ldtl7SqvLfTjQOTeWww vytVovfEAXbEOo9bZrrxNy9z4U5UevNHq41VpXW.MIdKWiTie027QRg.K0suTZUGSzhkYht2WxjA 0VIusaM9tJKh6IX.vlmA1wzUNfDgQlWO9s6B3dpwQUcisaJayTre0P.Z88OofWC1YZQseDV_Yrit Sz5nuxuaqRkV0TnqzVRsOpN2.u0bRdcwISHYCGRNxJHEAlKSZ8e29GQtBcYzM1sEmnQ9TN.5oqX5 zr9siF.xNApJ2tIfi7UGMJllIwljdlstp8Z8iySfWfBa0Oo__zBlllxHf1UTMiObJiU5GKMNxQTs DTz_kUZ_NQzyXDQIVowzvXbfxzxpTuClUFAxRlcCjNfkMn1b83j77XdWXAxZoazG5CAffDUkRG1I vcZvMx_rTBczeiDVHlXhfQsmDrDeUuNpRIwufQoKy6iNSxXm6dik7be8qvg06jHRIhQ4SIgj.3mj QZAf5cErW3wuDdtBls4FDzDP.0TgvsvvHXNq9hlzgPMCeug4Z_uztBpWPJSkA_HCFqwWfC4xUsGE Xjt_g1qjqT9rhVcc1ofiHvNHwGHHMJBUcwD3cXwhP1mof4yH1oimiKdzMjYSl5E5FMyc_4QmApH2 uuNrsHXvNeJ48RHXgYtenIjaUx9DzBIVwwMZasJtwDBvyeOsh6LMPQ5V8qPHBiAquOiqL4yWfPiJ TKKDm.TFeasXU39Ho5CEeAkEPp_gCDOil_.Dm7_wqh6owY5NnkVW1UrSKhXNCgcVkCAG36iRBx0e 1sXAyzGIWowDXrIU6852Rn13nCilOXsxJOp86q_F.BMzdxRhaLesCxQnMqWsxxYKyzRgRSkU49LD PSbx6AhBZzx6pA_Dk_jgyzIh8MVbNLGdmw05W5gTSGfCAPwcvOWAhXmgpp1oIqAYttou6Lp2QKJE .BoDPQGfsn4rgSo7yH8h9SEErbwxBhmIbIU.CfoPXGb2fW06StSzTK3E9ZRHVJLIWIvBoElUcQw. dcSqHaHZ3.0yiFumg3Xoroiwb3fzN4K7EiMxXki9rH2wJibUrzGqJs9hvSoVjza0QFK4Tb0MCEjY O2sm2kc.VyiGQJowXmLmIyuuV7oQWnGQEc03_hygbX_AID0k0NrnS9h2mVGCyT_MqHNiREQVXB3I zeoc- Received: from sonic.gate.mail.ne1.yahoo.com by sonic314.consmr.mail.gq1.yahoo.com with HTTP; Sat, 8 Feb 2020 04:12:51 +0000 Received: by smtp401.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID b426974927cf90b8413a75c81ec03993; Sat, 08 Feb 2020 04:12:49 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: RPi3 not using SMP? From: Mark Millard In-Reply-To: <9F1B762C-D1DA-40F6-A2D6-451B40A39E4A@yahoo.com> Date: Fri, 7 Feb 2020 20:12:47 -0800 Cc: freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20200208011940.GA8570@www.zefox.net> <6B6CCB8F-B56A-4758-BEEC-6418718C95CB@yahoo.com> <9F1B762C-D1DA-40F6-A2D6-451B40A39E4A@yahoo.com> To: bob prohaska , "jeff@freebsd.org" X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48DzKd5YFtz4Ms8 X-Spamd-Bar: / X-Spamd-Result: default: False [0.65 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.21)[-0.211,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (4.89), ipnet: 98.137.64.0/21(0.83), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.36)[0.364,0]; RCVD_IN_DNSWL_NONE(0.00)[84.69.137.98.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Feb 2020 04:12:55 -0000 On 2020-Feb-7, at 19:10, Mark Millard wrote: > [I now have some bounds on when PSCI_FN_VERSION > gets a 0 result from smc #0 instead of the correct > result: 2. Hopefully I can narrow the range more.] >=20 > On 2020-Feb-7, at 18:46, Mark Millard wrote: >=20 >=20 >=20 >> On 2020-Feb-7, at 17:19, bob prohaska wrote: >>=20 >>> For some weeks now an RPi3 running -current has seemed rather = slow.... >>>=20 >>> On looking at the early part of the boot message the processor >>> attributes seem rather scant: >>>=20 >>> ...... >>> elease APs...APs not started >>> CPU 0: ARM Cortex-A53 r0p4 affinity: 0 >>> Instruction Set Attributes 0 =3D >>> Instruction Set Attributes 1 =3D <> >>> Processor Features 0 =3D >>> Processor Features 1 =3D <> >>> Memory Model Features 0 =3D >>> Memory Model Features 1 =3D <8bit VMID> >>> Memory Model Features 2 =3D <32bit CCIDX,48bit VA> >>> Debug Features 0 =3D <2 CTX BKPTs,4 Watchpoints,6 = Breakpoints,PMUv3,Debugv8> >>> Debug Features 1 =3D <> >>> Auxiliary Features 0 =3D <> >>> Auxiliary Features 1 =3D <> >>> CPU 1: (null) (null) r0p0 affinity: 0 >>> CPU 2: (null) (null) r0p0 affinity: 0 >>> CPU 3: (null) (null) r0p0 affinity: 0 >>> ............ >>> In a top window, STATE is reported as RUN, rather than the >>> former CPUn. >>>=20 >>> Is a software switch now required to enable multiprocessing? >>>=20 >>> Or, could it be related to the lines: >>> psci0: PSCI version number mismatched with DT=20 >>> as pointed out by Mark M in reference to the cpu_reset failed >>> problem, which is still manifest?=20 >>>=20 >>> The kernel is at r357644. >>=20 >> Head -r356767's kernel does not have this problem for RPi3/4 used as >> aarch64 FreeBSD. >>=20 >> Head -r356776 and later all have this problem for both RPi3 and RPi4. >>=20 >> Note: There are no head versions between those. >>=20 >> The console log shows evidence of the problem much earlier. >> Instead of saying: >>=20 >> psci0: on ofwbus0 >> psci0: PSCI version 0.2 compatible >>=20 >> (once) it says: >>=20 >> psci0: on ofwbus0 >> psci0: PSCI version number 0 mismatched with DT, default 2 >> device_attach: psci0 attach returned 6 >>=20 >> (and those 3 lines repeat in various places) for which none of >> them show up for -r356767 . >>=20 >> Without identifying and using PSCI, the extra cores will not >> start and the cpu(s) will not reset. (PSCI is the ARM interface >> for doing such things.) >>=20 >> I've no clue why, but the version number it gets in my RPi4 >> experiments is 0. My only guess is that at some point >> memory important to ARM's PSCI operation has been touched >> and is no longer valid for the PSCI operation. >=20 > I've been doing some crude printf-based information > gathering for RPi4B boots (based on head -r357529 just > because it is convenient in my context) and I have > learned some about the staging of when PSCI_FN_VERSION > works vs. when it is no longer working. >=20 > For the below text extraction from a boot . . . > Lines with: "PSCI_FN_VERSION 2" are working cases. > Lines with: "PSCI_FN_VERSION 0" are no-longer-working cases. >=20 > . . . > FreeBSD clang version 9.0.1 (git@github.com:llvm/llvm-project.git = c1a0a213378a458fbea1a5c77b315c7dce08fd05) (based on LLVM 9.0.1) > uma_startup1 start: PSCI_FN_VERSION 2 > uma_startup1 end: PSCI_FN_VERSION 2 > uma_startup2 start: PSCI_FN_VERSION 2 > uma_startup2 end: PSCI_FN_VERSION 2 > VT(efifb): resolution 1824x984 > module firmware already present! > psci_fdt_get_callfn: method 'smc' > psci_fdt_get_callfn: arm_smccc_hvc '0xffff0000008663b0' > psci_fdt_get_callfn: arm_smccc_smc '0xffff0000008663c8' > psci_init: PSCI_FN_VERSION 0 > Starting CPU 1 (1) > Starting CPU 2 (2) > Starting CPU 3 (3) > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > random: unblocking device. > uma_startup3 start: PSCI_FN_VERSION 0 > uma_startup3 start: PSCI_FN_VERSION 0 > . . . >=20 > So sometime after uma_startup2 ends but before psci_init > sets up the normal use of arm_smccc_smc things are > messed up. >=20 > My guess is that one or more of the kernel memory > allocations ended up getting memory used by ARM's > PSCI implementation and the content was invalidated > for ARM's PSCI purposes. >=20 The transition from working to not is before the debug.verbose_sysinit=3D1 output turns in to symbolic notation: . . . subsystem 1800000 mi_startup: PSCI_FN_VERSION 2 0xffff00000047e170(0)... done. mi_startup: PSCI_FN_VERSION 2 0xffff0000007bda18(0)... done. mi_startup: PSCI_FN_VERSION 2 0xffff0000004407e4(0)... done. mi_startup: PSCI_FN_VERSION 2 0xffff0000004400c0(0xffff000000a75890)... done. mi_startup: PSCI_FN_VERSION 2 0xffff0000004400c0(0xffff000000a76340)... done. . . . mi_startup: PSCI_FN_VERSION 2 0xffff0000004400c0(0xffff000000aca328)... done. mi_startup: PSCI_FN_VERSION 2 0xffff0000004400c0(0xffff000000aca378)... done. mi_startup: PSCI_FN_VERSION 0 0xffff0000004400c0(0xffff000000aca5f8)... done. . . . mi_startup: PSCI_FN_VERSION 0 0xffff0000004c62b0(0)... done. mi_startup: PSCI_FN_VERSION 0 0xffff00000049e93c(0)... done. mi_startup: PSCI_FN_VERSION 0 linker_preload(0)... Preloaded elf kernel "/boot/kernel/kernel" at = 0xffff000001568000. Preloaded elf module "/boot/kernel/umodem.ko" at 0xffff000001571020. Preloaded elf module "/boot/kernel/ucom.ko" at 0xffff000001571838. Preloaded boot_entropy_cache "/boot/entropy" at 0xffff000001572010. module firmware already present! done. subsystem 1ffffff mi_startup: PSCI_FN_VERSION 0 ucom_init(0)... done. subsystem 2000000 mi_startup: PSCI_FN_VERSION 0 procs_show_all_add(0)... done. . . . It turns out that 0xffff0000004400c0 is: malloc_init . It turns out that 0xffff000000aca378 is for: M_IFMADDR : ffff000000aca328 g O .data 0000000000000020 M_IFADDR ffff000000aca348 l O .data 0000000000000018 M_IFADDR_init_sys_init ffff000000aca360 l O .data 0000000000000018 = M_IFADDR_uninit_sys_uninit ffff000000aca378 g O .data 0000000000000020 M_IFMADDR ffff000000aca398 l O .data 0000000000000018 M_IFMADDR_init_sys_init ffff000000aca3b0 l O .data 0000000000000018 = M_IFMADDR_uninit_sys_uninit ffff000000aca3c8 l O .data 0000000000000018 = vnet_init_vnet_if_init_sys_init ffff000000aca3e0 l O .data 0000000000000018 = vnet_init_vnet_if_init_sys_uninit ffff000000aca3f8 l O .data 0000000000000018 = vnet_uninit_vnet_if_uninit_sys_init ffff000000aca410 l O .data 0000000000000018 = vnet_uninit_vnet_if_uninit_sys_uninit ffff000000aca428 l O .data 0000000000000018 = vnet_uninit_vnet_if_return_sys_init ffff000000aca440 l O .data 0000000000000018 = vnet_uninit_vnet_if_return_sys_uninit ffff000000aca458 l O .data 0000000000000020 M_IFNET ffff000000aca478 l O .data 0000000000000018 ifepochalloc_sys_init ffff000000aca490 l O .data 0000000000000018 domainifattach_sys_init ffff000000aca4a8 l O .data 0000000000000004 ifdescr_maxlen ffff000000aca4b0 l O .data 0000000000000020 M_IFDESCR ffff000000aca4d0 l O .data 0000000000000004 log_promisc_mode_change ffff000000aca4d4 l O .data 0000000000000004 log_link_state_change ffff000000aca4d8 l O .data 0000000000000018 ifdescr_sx_args ffff000000aca4f0 l O .data 0000000000000018 ifnet_rw_args ffff000000aca508 l O .data 0000000000000018 ifnet_sx_args ffff000000aca520 l O .data 0000000000000028 vnet_if_init_vnet_init ffff000000aca548 l O .data 0000000000000028 = vnet_if_uninit_vnet_uninit ffff000000aca570 l O .data 0000000000000028 = vnet_if_return_vnet_uninit ffff000000aca598 l O .data 0000000000000018 = if_cloners_lock_mtx_sysinit_sys_init ffff000000aca5b0 l O .data 0000000000000018 = if_cloners_lock_mtx_sysuninit_sys_uninit ffff000000aca5c8 l O .data 0000000000000018 M_CLONE_init_sys_init ffff000000aca5e0 l O .data 0000000000000018 = M_CLONE_uninit_sys_uninit ffff000000aca5f8 l O .data 0000000000000020 M_CLONE =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)