Date: Thu, 13 Feb 2020 10:27:51 -0800 From: Mark Millard <marklmi@yahoo.com> To: Kyle Evans <kevans@freebsd.org> Cc: Ralf Wenk <iz-rpi03@hs-karlsruhe.de>, Andrew Turner <andrew@freebsd.org>, Oleksandr Tymoshenko <gonzo@freebsd.org>, freebsd-arm <freebsd-arm@freebsd.org>, Emmanuel Vadot <manu@freebsd.org> Subject: Re: A investigative hack that makes (for example) head -r356529 boot and operate normally an RPi4B (finally!): protect all armstub8-gic.bin's loaded content from replacement by the kernel Message-ID: <27BE7BAF-FD2C-41C6-B270-4BAF77D1FB0C@yahoo.com> In-Reply-To: <1BE59567-E669-4A88-8389-2E321B0AC1AE@yahoo.com> References: <7E7605DC-021D-448A-8459-8EC26BA9836D.ref@yahoo.com> <7E7605DC-021D-448A-8459-8EC26BA9836D@yahoo.com> <E1j2EmR-000Mwm-ID@iz-wera-new.HS-Karlsruhe.DE> <CACNAnaHVYeuoK=4f7XzokpAVjcqd6ovD58-0LdP_vPGjiXQgdw@mail.gmail.com> <E1j2FRX-000MyR-1O@iz-wera-new.HS-Karlsruhe.DE> <E1j2G3j-000Mzo-Dw@iz-wera-new.HS-Karlsruhe.DE> <CACNAnaHUwu0GmqW9QZRhS2sMK0HpsjdYRz3YouL5FMBdWBCW-Q@mail.gmail.com> <B64C6065-D804-4D6F-8660-F8DBB9946DEB@yahoo.com> <36CF6E4B-5607-4752-B2DF-C265BCFB95BA@yahoo.com> <1BE59567-E669-4A88-8389-2E321B0AC1AE@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2020-Feb-13, at 09:55, Mark Millard <marklmi at yahoo.com> wrote:
On 2020-Feb-13, at 09:36, Mark Millard <marklmi at yahoo.com> wrote:
>=20
>> On 2020-Feb-13, at 08:50, Mark Millard <marklmi at yahoo.com> wrote:
>>=20
>>> On 2020-Feb-13, at 07:22, Kyle Evans <kevans at freebsd.org> wrote:
>>>=20
>>>> On Thu, Feb 13, 2020 at 9:05 AM Ralf Wenk =
<iz-rpi03@hs-karlsruhe.de> wrote:
>>>>>=20
>>>>> On 2020-02-13 at 15:26 +0100 Ralf Wenk wrote:
>>>>>> On 2020-02-13 at 7:49 -0600 Kyle Evans wrote:
>>>>>>> On Thu, Feb 13, 2020 at 7:43 AM Ralf Wenk =
<iz-rpi03@hs-karlsruhe.de> wrote:
>>>>>>>>=20
>>>>>>>> On 2020-02-12 at 18:00 -0800 Mark Millard wrote via =
freebsd-arm:
>>>>>>>>> [...]
>>>>>>>>>=20
>>>>>>>>> # svnlite diff /usr/src/sys/dev/fdt/fdt_common.c
>>>>>>>>> Index: /usr/src/sys/dev/fdt/fdt_common.c
>>>>>>>>> =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>>>>>>>> --- /usr/src/sys/dev/fdt/fdt_common.c (revision 357529)
>>>>>>>>> +++ /usr/src/sys/dev/fdt/fdt_common.c (working copy)
>>>>>>>>> @@ -485,7 +485,18 @@
>>>>>>>>>=20
>>>>>>>>> tuples =3D res_len / tuple_size;
>>>>>>>>> reservep =3D (pcell_t *)&reserve;
>>>>>>>>> +#ifdef __aarch64__
>>>>>>>>> + //HACK!!!
>>>>>>>>> + // Reserve the first few pages, for example to
>>>>>>>>> + // preserve armstub8-gic.bin or armstub.bin
>>>>>>>>> + // content.
>>>>>>>>> + mr[0].mr_start=3D 0;
>>>>>>>>> + mr[0].mr_size=3D 2*4096;
>>>>>>>>> + tuples++;
>>>>>>>>> + for (i =3D 1; i < tuples; i++) {
>>>>>>>>> +#else
>>>>>>>>> for (i =3D 0; i < tuples; i++) {
>>>>>>>>> +#endif
>>>>>>>>>=20
>>>>>>>>> rv =3D fdt_data_to_res(reservep, addr_cells, =
size_cells,
>>>>>>>>> (u_long *)&mr[i].mr_start, (u_long =
*)&mr[i].mr_size);
>>>>>>>>> @@ -512,6 +523,11 @@
>>>>>>>>>=20
>>>>>>>>> root =3D OF_finddevice("/reserved-memory");
>>>>>>>>> if (root =3D=3D -1) {
>>>>>>>>> + // Fail over to checking for and handling =
memreserve,
>>>>>>>>> + // such as for a RPi4B.
>>>>>>>>> + if (0 =3D=3D =
fdt_get_reserved_regions(reserved,mreserved))
>>>>>>>>> + return (0);
>>>>>>>>> +
>>>>>>>>> return (ENXIO);
>>>>>>>>> }
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> I can confirm that with your patch(es) my RPi3 does not freeze =
any more
>>>>>>>> when loading mac_ntpd.ko. The patches are applied against =
r357853M.
>>>>>=20
>>>>> An reboot is working again too.
>>>>>=20
>>>>>>> Have you tested the RPi3 with just this second hunk of patch to
>>>>>>> fallover to memreserve, or is the first hunk definitely required =
as
>>>>>>> well?
>>>>>>=20
>>>>>> Good question. I tested both hunks together.
>>>>>> Will try what happens when just applying the second and report =
back.
>>>>>=20
>>>>> Here it is:
>>>>> Without the first hunk the system freezes again when loading =
mac_ntpd.ko.
>>>>> Also the CPU information during boot for CPUs 1 to 3 looks strange =
again.
>>>>>=20
>>>>=20
>>>> Yeah- I see it now; both armstubs are about 5k. I've raised an
>>>> issue[0] with upstream for armstub/rpi bits to work out the proper
>>>> solution, because I don't necessarily want to commit the =
workaround.
>>>> I'll throw up the second hunk on phabricator for review by =
#arm/#arm64
>>>> folks, because that seems to me the proper fallback.
>>>>=20
>>>> I also discovered some issues when trying to read /memreserve/ with
>>>> our dtc and filed a PR[1] to fix those.
>>>>=20
>>>> Thanks,
>>>>=20
>>>> Kyle Evans
>>>>=20
>>>> [0] https://github.com/raspberrypi/tools/issues/107
>>>> [1] https://github.com/davidchisnall/dtc/pull/59
>>>=20
>>> The DTB information below is from:
>>>=20
>>> U-Boot> fdt addr 0x7ef2000=20
>>> U-Boot> fdt print / =20
>>>=20
>>> on a RPi4B 4 GiByte.
>>>=20
>>> On at least the RPi4B memreserve is not what causes
>>> the first page to be excluded:
>>>=20
>>> memreserve =3D <0x3b400000 0x04c00000>;
>>>=20
>>> Nor is memory@0 the cause:
>>>=20
>>> memory@0 {
>>> device_type =3D "memory";
>>> reg =3D <0x00000000 0x00000000 0x3b400000 0x00000000 =
0x40000000 0xbc000000>;
>>> };
>>>=20
>>> (That also skips the memreserve area.)
>>>=20
>>> I do not find anything in the DTB that indicates
>>> to exclude the first page.
>>>=20
>>> My hypothesis is that the FreeBSD code excludes
>>> the page based on some less obvious relationship
>>> that I'm not identifying.
>>>=20
>>> There is the cpu-rlease-addr information that seems
>>> to refer to some 1st memory page content:
>>>=20
>>> cpus {
>>> #address-cells =3D <0x00000001>;
>>> #size-cells =3D <0x00000000>;
>>> enable-method =3D "brcm,bcm2836-smp";
>>> phandle =3D <0x000000be>;
>>> cpu@0 {
>>> device_type =3D "cpu";
>>> compatible =3D "arm,cortex-a72";
>>> reg =3D <0x00000000>;
>>> enable-method =3D "spin-table";
>>> cpu-release-addr =3D <0x00000000 0x000000d8>;
>>> phandle =3D <0x0000001d>;
>>> };
>>> cpu@1 {
>>> device_type =3D "cpu";
>>> compatible =3D "arm,cortex-a72";
>>> reg =3D <0x00000001>;
>>> enable-method =3D "spin-table";
>>> cpu-release-addr =3D <0x00000000 0x000000e0>;
>>> phandle =3D <0x0000001e>;
>>> };
>>> cpu@2 {
>>> device_type =3D "cpu";
>>> compatible =3D "arm,cortex-a72";
>>> reg =3D <0x00000002>;
>>> enable-method =3D "spin-table";
>>> cpu-release-addr =3D <0x00000000 0x000000e8>;
>>> phandle =3D <0x0000001f>;
>>> };
>>> cpu@3 {
>>> device_type =3D "cpu";
>>> compatible =3D "arm,cortex-a72";
>>> reg =3D <0x00000003>;
>>> enable-method =3D "spin-table";
>>> cpu-release-addr =3D <0x00000000 0x000000f0>;
>>> phandle =3D <0x00000020>;
>>> };
>>> };
>>=20
>>=20
>>=20
>>=20
>> Looking at the code there is:
>>=20
>> /* Load the physical memory ranges */
>> efihdr =3D (struct efi_map_header *)preload_search_info(kmdp,
>> MODINFO_METADATA | MODINFOMD_EFI_MAP);
>> if (efihdr !=3D NULL)
>> add_efi_map_entries(efihdr);
>> #ifdef FDT
>> else {
>> /* Grab physical memory regions information from device =
tree. */
>> if (fdt_get_mem_regions(mem_regions, &mem_regions_sz,
>> NULL) !=3D 0)
>> panic("Cannot get physical memory regions");
>> arm_physmem_hardware_regions(mem_regions, =
mem_regions_sz);
>> }
>> if (fdt_get_reserved_mem(mem_regions, &mem_regions_sz) =3D=3D =
0)
>> arm_physmem_exclude_regions(mem_regions, =
mem_regions_sz,
>> EXFLAG_NODUMP | EXFLAG_NOALLOC);
>> #endif
>>=20
>> /* Exclude the EFI framebuffer from our view of physical =
memory. */
>> efifb =3D (struct efi_fb *)preload_search_info(kmdp,
>> MODINFO_METADATA | MODINFOMD_EFI_FB);
>> if (efifb !=3D NULL)
>> arm_physmem_exclude_region(efifb->fb_addr, =
efifb->fb_size,
>> EXFLAG_NOALLOC);
>> . . .
>> if (boothowto & RB_VERBOSE) {
>> print_efi_map_entries(efihdr);
>> arm_physmem_print_tables();
>> }
>>=20
>>=20
>> It looks to me like the boot -v text:
>>=20
>> Type Physical Virtual #Pages Attr
>> Reserved 000000000000 0 00000001 WB=20
>> ConventionalMemory 000000001000 1000 00007ef1 WB=20
>> BootServicesData 000007ef2000 7ef2000 0000001c WB=20
>> ConventionalMemory 000007f0e000 7f0e000 00029f93 WB=20
>> BootServicesData 000031ea1000 31ea1000 00000001 WB=20
>> LoaderData 000031ea2000 31ea2000 00008001 WB=20
>> LoaderCode 000039ea3000 39ea3000 000000a6 WB=20
>> Reserved 000039f49000 39f49000 00000007 WB=20
>> BootServicesData 000039f50000 39f50000 00000001 WB=20
>> Reserved 000039f51000 39f51000 00000002 WB=20
>> RuntimeServicesData 000039f53000 39f53000 00000001 WB RUNTIME
>> Reserved 000039f54000 39f54000 00000001 WB=20
>> BootServicesData 000039f55000 39f55000 00000002 WB=20
>> RuntimeServicesData 000039f57000 39f57000 00000001 WB RUNTIME
>> LoaderData 000039f58000 39f58000 00001408 WB=20
>> RuntimeServicesCode 00003b360000 3b360000 00000010 WB RUNTIME
>> LoaderData 00003b370000 3b370000 00000090 WB=20
>> BootServicesData 000040000000 40000000 000bc000 WB=20
>> MemoryMappedIO 0000fe100000 fe100000 00000001 RUNTIME
>>=20
>> is from print_efi_map_entries via the efihdr instead
>> of from the FreeBSD FDT code and the DTB.
>>=20
>> So is it u-boot that provides the efihdr for which
>> add_efi_map_entries generated those regions?
>>=20
>> That might explain why I do not find matching DTB
>> material for all of it.
>=20
>=20
> Looks like the efi memory map traces back to the loader
> and its use of GetMemoryMap (as far as FreeBSD goes):
>=20
> # grep -r "GetMemoryMap" /usr/src/sys/ | more
> /usr/src/sys/amd64/amd64/machdep.c: * Memory map data provided by =
UEFI via the GetMemoryMap
> /usr/src/sys/arm64/arm64/machdep.c: * Memory map data provided by =
UEFI via the GetMemoryMap
> /usr/src/sys/arm/arm/machdep_boot.c: * Memory map data provided by =
UEFI via the GetMemoryMap
> /usr/src/sys/contrib/edk2/Include/Uefi/UefiSpec.h: EFI_GET_MEMORY_MAP =
GetMemoryMap;
>=20
> # grep -r "GetMemoryMap" /usr/src/stand/ | more
> /usr/src/stand/efi/loader/copy.c: status =3D =
BS->GetMemoryMap(&sz, map, &key, &dsz, &dver);
> /usr/src/stand/efi/loader/main.c: status =3D =
BS->GetMemoryMap(&sz, 0, &key, &dsz, &dver);
> /usr/src/stand/efi/loader/main.c: status =3D =
BS->GetMemoryMap(&sz, map, &key, &dsz, &dver);
> /usr/src/stand/efi/loader/bootinfo.c: status =3D =
BS->GetMemoryMap(&sz, mm, &efi_mapkey, &dsz, &mmver);
> /usr/src/stand/efi/loader/bootinfo.c: =
printf("%s: GetMemoryMap error %lu\n", __func__,
> /usr/src/stand/efi/include/efiapi.h: EFI_GET_MEMORY_MAP =
GetMemoryMap;
Here is the memmap from the loader that is just based on EFI's
GetMemoryMap from what I can tell from a wuick look:
Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel] in 8 seconds...=20
Type '?' for a list of commands, 'help' for more detailed help.
OK memmap=20
Type Physical Virtual #Pages Attr
Reserved 000000000000 000000000000 00000001 WB=20
ConventionalMemory 000000001000 000000001000 00007ef1 WB=20
BootServicesData 000007ef2000 000007ef2000 0000001c WB=20
ConventionalMemory 000007f0e000 000007f0e000 00029f93 WB=20
BootServicesData 000031ea1000 000031ea1000 00000001 WB=20
LoaderData 000031ea2000 000031ea2000 00004000 WB=20
BootServicesData 000035ea2000 000035ea2000 00000001 WB=20
LoaderData 000035ea3000 000035ea3000 00004000 WB=20
LoaderCode 000039ea3000 000039ea3000 000000a6 WB=20
Reserved 000039f49000 000039f49000 00000007 WB=20
BootServicesData 000039f50000 000039f50000 00000001 WB=20
Reserved 000039f51000 000039f51000 00000002 WB=20
RuntimeServicesData 000039f53000 000039f53000 00000001 WB=20
Reserved 000039f54000 000039f54000 00000001 WB=20
BootServicesData 000039f55000 000039f55000 00000002 WB=20
RuntimeServicesData 000039f57000 000039f57000 00000001 WB=20
LoaderData 000039f58000 000039f58000 00001408 WB=20
RuntimeServicesCode 00003b360000 00003b360000 00000010 WB=20
LoaderData 00003b370000 00003b370000 00000090 WB=20
BootServicesData 000040000000 000040000000 000bc000 WB=20
MemoryMappedIO 0000fe100000 0000fe100000 00000001=20
OK=20
So at the loader time frame, the 2nd page of memory is not
Reserved (in EFI terms).
It would appear to me that avoiding messing up the
armstub8*.bin content should apply this early as well.
So it may be sysutils/u-boot-rpi{3,4} that needs to
arrange sufficient room to prevent messing up such.
(Unless armstub8*.bin can adjust something that
u-boot's EFI interface is based on for that initial
"Reserved" area?)
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?27BE7BAF-FD2C-41C6-B270-4BAF77D1FB0C>