From owner-freebsd-arm@freebsd.org Thu Feb 13 17:55:37 2020 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E99F91CEA88 for ; Thu, 13 Feb 2020 17:55:37 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-55.consmr.mail.gq1.yahoo.com (sonic308-55.consmr.mail.gq1.yahoo.com [98.137.68.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48JPLc5PPDz4H8V for ; Thu, 13 Feb 2020 17:55:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: ySHFVl8VM1l_pVzKOBhhEgUywBJWXSuF_V1r_kqqCYx_kgiuKDlOsZFX74vYA.u oB3iKkd97x_YEq3pVUYuNpFxB1ReuEgYubZ0zzTyFiSuoptu05k_1J7ow80i2UsrlHFp7b4GJv1m JEYrubWxeIn6Vc.UUp.AtrGjnNJJhnGGzpKwrfhNwiAc8WbaY1z8rMWnLP_OKv2b1nMsR33aA9Rp W5SKtFqQfA5JHu7YAfy.6jUgw4ndYBQV_WDjfg8uLroN4fL8EEtbazg3hb7MCKTm6Kaxd8xSO_SQ x2IBbIvneC7HFSOC.eGagQa0TWHKei14h7r6u.O.pbHx2aUHkB1RKh9TvaCLK6hh3vSufJLVOybs j5Z1WSi9dZDtXxoKIRWnOv6PD4UrUIVHtWa.Tw6A5521JwENwWAbv.J665oI2SRzWZAOedfM6M3q QlrOLlaFMyZNNWzc4znJmnlRMWzKV0eUE3jEg9agGQeDJuSZXa6nyif4csbbVc6RssbuyI6x0w9S w9zDpnHsTpHS3k3yjjO_cdFXNCSxcFkOFlfQk1q2SADoeMKisAxcW0a_4LeTEfXxTesaW5TomxJ2 hkPUznZs8VmgEeW5ZURvQjP2zXeKBCTD5Q6joc3mdmNB_N1EPgFslCntH9X93yNru9wn0AkEaz.V soApoEpPkOcRN7tslWxhvXJDY3i0jrLx9LIgSRSTi.gUJYw7mkXcTNihNVFnx1aDZPCyhlE1R2w6 SkH8TfBJ0Zwf9J4KNJxchTnyyrHzrtEa6XFiI25_phhwpdaPd5.1Hz9AWZObL9qLpyAlJfBBteYP GySRlPiuBoeBSBl08EYbb5YzbEosZ_ssKnplTLyjULqIbQ_Ljbz47V5eVdWE0O04FuZfsld1cw2s NB.9JYjZYMZxtXN92UdbJVlJ4_cGk1C3Zyd9VdvwdKGCn74O_n9P2tpNCtl.1VldMMzPPW3Ta3GF eks40jCOKPv5GI9cuoG7K9kE1hiWHnp5llYx2yNY.4uPEECqMeQaezc2ZYThl7p61pH2qGdeBYRZ qmDWBD7RDar.cRCSM.jiItZthARaX3UfeLei0rJ.T304OT6WmBbsufijnnPbxKaVTs1hyV5SSzOW pbTw5jOo49FsQ.rBE_AGDvN9uWeVfNgG9HD0nAWDw57DO4rQDI8xPDqhZ1olHF1.8IbQ.61Ef4kN wUBA3LFpVa3vErh5KNDsQtVIvrbYD1LVTdmz.1_kyt1Jhe9YLAAH2yalITAFFZK7nsRLdofkNHV9 wq8B3ifWzKk3ZaRIuQLeKSSM1lmm0zHmrRofiKUZPHE4i0d64ToCj4SgPiSELNU3cOosU2BL6WqO j55IuBCgC.Pm99wp3o0o8nXyCHJzyQbplAkYG5SI8LVO2ZE.0z3M.hsEKbsw2M9X3UjTfsmDTej8 Bivdne1wZ7z3D4BPJ6S869DstplXuxswnZjQDyJaNZx7pBE1C27_niJmcJ1uTbK.fzR7IVzQgMNV P0g-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Thu, 13 Feb 2020 17:55:34 +0000 Received: by smtp425.mail.ne1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID 3400c3b6cf3686a70ac32c9f29c3688d; Thu, 13 Feb 2020 17:55:29 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: A investigative hack that makes (for example) head -r356529 boot and operate normally an RPi4B (finally!): protect all armstub8-gic.bin's loaded content from replacement by the kernel From: Mark Millard In-Reply-To: <36CF6E4B-5607-4752-B2DF-C265BCFB95BA@yahoo.com> Date: Thu, 13 Feb 2020 09:55:27 -0800 Cc: Ralf Wenk , Andrew Turner , Oleksandr Tymoshenko , freebsd-arm , Emmanuel Vadot Content-Transfer-Encoding: quoted-printable Message-Id: <1BE59567-E669-4A88-8389-2E321B0AC1AE@yahoo.com> References: <7E7605DC-021D-448A-8459-8EC26BA9836D.ref@yahoo.com> <7E7605DC-021D-448A-8459-8EC26BA9836D@yahoo.com> <36CF6E4B-5607-4752-B2DF-C265BCFB95BA@yahoo.com> To: Kyle Evans X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Rspamd-Queue-Id: 48JPLc5PPDz4H8V X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.35 / 15.00]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; RCPT_COUNT_FIVE(0.00)[6]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; SUBJECT_HAS_EXCLAIM(0.00)[]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SH_EMAIL_ZRD(0.00)[0.0.0.0,0.0.0.2,0.0.0.3,0.0.0.1]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.887,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[0.0.0.0,0.0.0.1,0.0.0.2,0.0.0.3]; NEURAL_HAM_LONG(-0.97)[-0.967,0]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[31.68.137.98.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (1.55), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Feb 2020 17:55:38 -0000 On 2020-Feb-13, at 09:36, Mark Millard wrote: > On 2020-Feb-13, at 08:50, Mark Millard wrote: >=20 >> On 2020-Feb-13, at 07:22, Kyle Evans wrote: >>=20 >>> On Thu, Feb 13, 2020 at 9:05 AM Ralf Wenk = wrote: >>>>=20 >>>> On 2020-02-13 at 15:26 +0100 Ralf Wenk wrote: >>>>> On 2020-02-13 at 7:49 -0600 Kyle Evans wrote: >>>>>> On Thu, Feb 13, 2020 at 7:43 AM Ralf Wenk = wrote: >>>>>>>=20 >>>>>>> On 2020-02-12 at 18:00 -0800 Mark Millard wrote via freebsd-arm: >>>>>>>> [...] >>>>>>>>=20 >>>>>>>> # svnlite diff /usr/src/sys/dev/fdt/fdt_common.c >>>>>>>> Index: /usr/src/sys/dev/fdt/fdt_common.c >>>>>>>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>>>>> --- /usr/src/sys/dev/fdt/fdt_common.c (revision 357529) >>>>>>>> +++ /usr/src/sys/dev/fdt/fdt_common.c (working copy) >>>>>>>> @@ -485,7 +485,18 @@ >>>>>>>>=20 >>>>>>>> tuples =3D res_len / tuple_size; >>>>>>>> reservep =3D (pcell_t *)&reserve; >>>>>>>> +#ifdef __aarch64__ >>>>>>>> + //HACK!!! >>>>>>>> + // Reserve the first few pages, for example to >>>>>>>> + // preserve armstub8-gic.bin or armstub.bin >>>>>>>> + // content. >>>>>>>> + mr[0].mr_start=3D 0; >>>>>>>> + mr[0].mr_size=3D 2*4096; >>>>>>>> + tuples++; >>>>>>>> + for (i =3D 1; i < tuples; i++) { >>>>>>>> +#else >>>>>>>> for (i =3D 0; i < tuples; i++) { >>>>>>>> +#endif >>>>>>>>=20 >>>>>>>> rv =3D fdt_data_to_res(reservep, addr_cells, = size_cells, >>>>>>>> (u_long *)&mr[i].mr_start, (u_long = *)&mr[i].mr_size); >>>>>>>> @@ -512,6 +523,11 @@ >>>>>>>>=20 >>>>>>>> root =3D OF_finddevice("/reserved-memory"); >>>>>>>> if (root =3D=3D -1) { >>>>>>>> + // Fail over to checking for and handling = memreserve, >>>>>>>> + // such as for a RPi4B. >>>>>>>> + if (0 =3D=3D = fdt_get_reserved_regions(reserved,mreserved)) >>>>>>>> + return (0); >>>>>>>> + >>>>>>>> return (ENXIO); >>>>>>>> } >>>>>>>>=20 >>>>>>>=20 >>>>>>> I can confirm that with your patch(es) my RPi3 does not freeze = any more >>>>>>> when loading mac_ntpd.ko. The patches are applied against = r357853M. >>>>=20 >>>> An reboot is working again too. >>>>=20 >>>>>> Have you tested the RPi3 with just this second hunk of patch to >>>>>> fallover to memreserve, or is the first hunk definitely required = as >>>>>> well? >>>>>=20 >>>>> Good question. I tested both hunks together. >>>>> Will try what happens when just applying the second and report = back. >>>>=20 >>>> Here it is: >>>> Without the first hunk the system freezes again when loading = mac_ntpd.ko. >>>> Also the CPU information during boot for CPUs 1 to 3 looks strange = again. >>>>=20 >>>=20 >>> Yeah- I see it now; both armstubs are about 5k. I've raised an >>> issue[0] with upstream for armstub/rpi bits to work out the proper >>> solution, because I don't necessarily want to commit the workaround. >>> I'll throw up the second hunk on phabricator for review by = #arm/#arm64 >>> folks, because that seems to me the proper fallback. >>>=20 >>> I also discovered some issues when trying to read /memreserve/ with >>> our dtc and filed a PR[1] to fix those. >>>=20 >>> Thanks, >>>=20 >>> Kyle Evans >>>=20 >>> [0] https://github.com/raspberrypi/tools/issues/107 >>> [1] https://github.com/davidchisnall/dtc/pull/59 >>=20 >> The DTB information below is from: >>=20 >> U-Boot> fdt addr 0x7ef2000=20 >> U-Boot> fdt print / =20 >>=20 >> on a RPi4B 4 GiByte. >>=20 >> On at least the RPi4B memreserve is not what causes >> the first page to be excluded: >>=20 >> memreserve =3D <0x3b400000 0x04c00000>; >>=20 >> Nor is memory@0 the cause: >>=20 >> memory@0 { >> device_type =3D "memory"; >> reg =3D <0x00000000 0x00000000 0x3b400000 0x00000000 = 0x40000000 0xbc000000>; >> }; >>=20 >> (That also skips the memreserve area.) >>=20 >> I do not find anything in the DTB that indicates >> to exclude the first page. >>=20 >> My hypothesis is that the FreeBSD code excludes >> the page based on some less obvious relationship >> that I'm not identifying. >>=20 >> There is the cpu-rlease-addr information that seems >> to refer to some 1st memory page content: >>=20 >> cpus { >> #address-cells =3D <0x00000001>; >> #size-cells =3D <0x00000000>; >> enable-method =3D "brcm,bcm2836-smp"; >> phandle =3D <0x000000be>; >> cpu@0 { >> device_type =3D "cpu"; >> compatible =3D "arm,cortex-a72"; >> reg =3D <0x00000000>; >> enable-method =3D "spin-table"; >> cpu-release-addr =3D <0x00000000 0x000000d8>; >> phandle =3D <0x0000001d>; >> }; >> cpu@1 { >> device_type =3D "cpu"; >> compatible =3D "arm,cortex-a72"; >> reg =3D <0x00000001>; >> enable-method =3D "spin-table"; >> cpu-release-addr =3D <0x00000000 0x000000e0>; >> phandle =3D <0x0000001e>; >> }; >> cpu@2 { >> device_type =3D "cpu"; >> compatible =3D "arm,cortex-a72"; >> reg =3D <0x00000002>; >> enable-method =3D "spin-table"; >> cpu-release-addr =3D <0x00000000 0x000000e8>; >> phandle =3D <0x0000001f>; >> }; >> cpu@3 { >> device_type =3D "cpu"; >> compatible =3D "arm,cortex-a72"; >> reg =3D <0x00000003>; >> enable-method =3D "spin-table"; >> cpu-release-addr =3D <0x00000000 0x000000f0>; >> phandle =3D <0x00000020>; >> }; >> }; >=20 >=20 >=20 >=20 > Looking at the code there is: >=20 > /* Load the physical memory ranges */ > efihdr =3D (struct efi_map_header *)preload_search_info(kmdp, > MODINFO_METADATA | MODINFOMD_EFI_MAP); > if (efihdr !=3D NULL) > add_efi_map_entries(efihdr); > #ifdef FDT > else { > /* Grab physical memory regions information from device = tree. */ > if (fdt_get_mem_regions(mem_regions, &mem_regions_sz, > NULL) !=3D 0) > panic("Cannot get physical memory regions"); > arm_physmem_hardware_regions(mem_regions, = mem_regions_sz); > } > if (fdt_get_reserved_mem(mem_regions, &mem_regions_sz) =3D=3D = 0) > arm_physmem_exclude_regions(mem_regions, = mem_regions_sz, > EXFLAG_NODUMP | EXFLAG_NOALLOC); > #endif >=20 > /* Exclude the EFI framebuffer from our view of physical = memory. */ > efifb =3D (struct efi_fb *)preload_search_info(kmdp, > MODINFO_METADATA | MODINFOMD_EFI_FB); > if (efifb !=3D NULL) > arm_physmem_exclude_region(efifb->fb_addr, = efifb->fb_size, > EXFLAG_NOALLOC); > . . . > if (boothowto & RB_VERBOSE) { > print_efi_map_entries(efihdr); > arm_physmem_print_tables(); > } >=20 >=20 > It looks to me like the boot -v text: >=20 > Type Physical Virtual #Pages Attr > Reserved 000000000000 0 00000001 WB=20 > ConventionalMemory 000000001000 1000 00007ef1 WB=20 > BootServicesData 000007ef2000 7ef2000 0000001c WB=20 > ConventionalMemory 000007f0e000 7f0e000 00029f93 WB=20 > BootServicesData 000031ea1000 31ea1000 00000001 WB=20 > LoaderData 000031ea2000 31ea2000 00008001 WB=20 > LoaderCode 000039ea3000 39ea3000 000000a6 WB=20 > Reserved 000039f49000 39f49000 00000007 WB=20 > BootServicesData 000039f50000 39f50000 00000001 WB=20 > Reserved 000039f51000 39f51000 00000002 WB=20 > RuntimeServicesData 000039f53000 39f53000 00000001 WB RUNTIME > Reserved 000039f54000 39f54000 00000001 WB=20 > BootServicesData 000039f55000 39f55000 00000002 WB=20 > RuntimeServicesData 000039f57000 39f57000 00000001 WB RUNTIME > LoaderData 000039f58000 39f58000 00001408 WB=20 > RuntimeServicesCode 00003b360000 3b360000 00000010 WB RUNTIME > LoaderData 00003b370000 3b370000 00000090 WB=20 > BootServicesData 000040000000 40000000 000bc000 WB=20 > MemoryMappedIO 0000fe100000 fe100000 00000001 RUNTIME >=20 > is from print_efi_map_entries via the efihdr instead > of from the FreeBSD FDT code and the DTB. >=20 > So is it u-boot that provides the efihdr for which > add_efi_map_entries generated those regions? >=20 > That might explain why I do not find matching DTB > material for all of it. Looks like the efi memory map traces back to the loader and its use of GetMemoryMap (as far as FreeBSD goes): # grep -r "GetMemoryMap" /usr/src/sys/ | more /usr/src/sys/amd64/amd64/machdep.c: * Memory map data provided by = UEFI via the GetMemoryMap /usr/src/sys/arm64/arm64/machdep.c: * Memory map data provided by = UEFI via the GetMemoryMap /usr/src/sys/arm/arm/machdep_boot.c: * Memory map data provided by = UEFI via the GetMemoryMap /usr/src/sys/contrib/edk2/Include/Uefi/UefiSpec.h: EFI_GET_MEMORY_MAP = GetMemoryMap; # grep -r "GetMemoryMap" /usr/src/stand/ | more /usr/src/stand/efi/loader/copy.c: status =3D = BS->GetMemoryMap(&sz, map, &key, &dsz, &dver); /usr/src/stand/efi/loader/main.c: status =3D BS->GetMemoryMap(&sz, = 0, &key, &dsz, &dver); /usr/src/stand/efi/loader/main.c: status =3D BS->GetMemoryMap(&sz, = map, &key, &dsz, &dver); /usr/src/stand/efi/loader/bootinfo.c: status =3D = BS->GetMemoryMap(&sz, mm, &efi_mapkey, &dsz, &mmver); /usr/src/stand/efi/loader/bootinfo.c: = printf("%s: GetMemoryMap error %lu\n", __func__, /usr/src/stand/efi/include/efiapi.h: EFI_GET_MEMORY_MAP = GetMemoryMap; =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)