Date: Sat, 7 Jan 2023 02:18:37 -0800 From: Mark Millard <marklmi@yahoo.com> To: =?utf-8?Q?Klaus_K=C3=BCchemann?= <maciphone2@googlemail.com> Cc: freebsd-arm@freebsd.org Subject: Re: How to make FreeBSD's kernel boot a RPi4B with modern RPi* firmware Message-ID: <EAD84A57-E8F0-4149-BCFC-8A06FF03B11B@yahoo.com> In-Reply-To: <9E9C739E-8308-472A-B797-05A37559DD00@googlemail.com> References: <9C037D3F-A440-4708-993D-117F313691BB@yahoo.com> <374EC3E5-4CB4-4336-A8B9-7A9CF6151691@yahoo.com> <BCCBE0D7-8BEB-4D6D-A017-9A59000F1E2B@yahoo.com> <9E9C739E-8308-472A-B797-05A37559DD00@googlemail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jan 7, 2023, at 00:37, Klaus K=C3=BCchemann = <maciphone2@googlemail.com> wrote: Hi Mark, > `ve tested your "early_dma=E2=80=9C patch now on the CM4(not tested on = the 4b)=E2=80=A6 > true, it can boot the latest firmware but after investigating in = what=E2=80=99s new=20 > in bcm2711-rpi-cm4.dtb I guess new features are mainly related to CPU = (L2-) caches features , > That=E2=80=99s why we get things like : "clk_fixed4: <Fixed clock> = disabled on ofwbus0 > clk_fixed4: Cannot FDT parameters.=E2=80=9C ... > so I think we currently won=E2=80=99t benefit much of the new = firmware. But being able to boot with the firmware should make it easier to deal with investigating other things related to more modern firmware. Also, it avoids running into the specific type of crash and having to figure out what is going on for it: one less problem to wade through. In fact, the modern firmware corrects mistakes in the .dtb's relative to the RPi4B PCIe description. For a long time different parts of the information were inconsistent with each other. But various OS's (including FreeBSD) ignore parts of the Device Tree information and do their own thing independent of parts of the Device Tree information present. This allowed various things to work even when ignoring things was not a deliberate way to avoid the inconsistency(s). FreeBSD may well have hardcoded RPi4B specifics that would be wrong for a CM4. But it is still better to target getting the CM4 going via using the corrected Device Tree information via targeting a more recent RPi* firmware vintage that has the fixed information --instead of targeting the old vintage with the known mistakes in the Device Tree. > I had a little hope that an earlier dma could shine a bit more light = on bugs like this :=20 > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D260131 The only BCM2711's that I've access to are RPi4B's so I do not see such problems. I've no clue about them. > But a tiny little hope can=E2=80=99t of course fix bugs :-) , so I = think we have to focus more on fixing existing bugs and=20 > Adding device drivers(of course the ToDo-list is not news but still = the truth;-) >=20 > Thanks for taking attention the current firmware things, always worth = to take a look into. Thanks for testing that the bcm_dma related patching applies to avoiding the specific crash type in more than my particular contexts. This weekend I'm trying to upgrade all the RPi4B boot media that I use to be using modern RPi* firmware (without boot crashes) as part of a general round of updating the FreeBSD version things are based on. (Also: RPi3B, RPi2B Rev 1.2, and [armv7] RPi2B Rev 1.1 boot media.) > Regards >=20 > K. >=20 >> Am 25.12.2022 um 17:37 schrieb Mark Millard <marklmi@yahoo.com>: >>=20 >> On Dec 24, 2022, at 20:14, Mark Millard <marklmi@yahoo.com> wrote: >>=20 >>> On Dec 24, 2022, at 19:15, Mark Millard <marklmi@yahoo.com> wrote: >>>=20 >>>> I finally ran into EARLY_DRIVER_MODULE, BUS_PASS_RESOURCE, >>>> BUS_PASS_ORDER_MIDDLE and the like and they allow being >>>> sure that the brcm,bcm2835-dma related setup has been done >>>> before any use of it is made, despite the order in the >>>> Device Tree: use an earlier pass for brcm,bcm2835-dma >>>> related attach. This avoids the kernel crashing during >>>> boot. >>>>=20 >>>> The example context used below has: serial console with >>>> USB3 SSD boot media (not requiring a usb_pgood_delay >>>> setting), booting a stable/13. The RPI4B is a C0T one (no >>>> 3 GiByte limitation, for example: the PCIe wrapper logic >>>> has been corrected). >>>>=20 >>>> stable/13's source code changes ( similarly for >>>> releng/13.1 ): >>>>=20 >>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c = b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>> index cab8639bb607..d8b49acfe332 100644 >>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver =3D { >>>>=20 >>>> static devclass_t bcm_dma_devclass; >>>>=20 >>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, = bcm_dma_devclass, 0, 0); >>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, = bcm_dma_devclass, >>>> + 0, 0, BUS_PASS_RESOURCE + BUS_PASS_ORDER_MIDDLE); >>>> MODULE_VERSION(bcm_dma, 1); >>>>=20 >>>>=20 >>>> For reference, a 13S snapshot with my kernel build replacing >>>> the snapshot's kernel, was booted with: >>>>=20 >>>> # strings /boot/msdos/start4.elf | grep VC_BUILD_ID_ >>>> VC_BUILD_ID_USER: dom >>>> VC_BUILD_ID_TIME: 11:09:05 >>>> VC_BUILD_ID_VARIANT: start >>>> VC_BUILD_ID_TIME: Oct 26 2022 >>>> VC_BUILD_ID_BRANCH: bcm2711_2 >>>> VC_BUILD_ID_HOSTNAME: buildbot >>>> VC_BUILD_ID_PLATFORM: raspberrypi_linux >>>> VC_BUILD_ID_VERSION: c72ad6b26ff40c91ef776b847436094ee63fabee = (clean) >>>>=20 >>>> There are new things present that FreeBSD reports >>>> but ignores, producing messages like: >>>>=20 >>>> clk_fixed4: <Fixed clock> disabled on ofwbus0 >>>> clk_fixed4: Cannot FDT parameters. >>>> device_attach: clk_fixed4 attach returned 6 >>>>=20 >>>> over and over during part of the boot. It seems to >>>> retry as it goes and thus produce so many messages. >>>>=20 >>>> There was also: >>>>=20 >>>> fb0: <BCM2835 VT framebuffer driver> on simplebus0 >>>> fb0: changing fb bpp from 0 to 24 >>>> mbox0: mbox response error >>>> fb0: bcm2835_mbox_fb_init failed, err=3D5 >>>> device_attach: fb0 attach returned 6 >>>>=20 >>>> genet0 is working. >>>>=20 >>>> I've not checked if the microsd card slot can be used. >>>>=20 >>>> I used the normal FreeBSD U-Boot since I was not booting >>>> the NVM3 media that requires extra time (usb_pgood_delay >>>> would be assigned in my own U-Boot builds). >>>>=20 >>>> For reference, I used my typical sort of config.txt : >>>>=20 >>>> # more /boot/msdos/config.txt >>>> [all] >>>> arm_64bit=3D1 >>>> dtparam=3Daudio=3Don,i2c_arm=3Don,spi=3Don >>>> dtoverlay=3Dmmc >>>> dtoverlay=3Ddisable-bt >>>> device_tree_address=3D0x4000 >>>> kernel=3Du-boot.bin >>>>=20 >>>> [pi4] >>>> hdmi_safe=3D1 >>>> armstub=3Darmstub8-gic.bin >>>>=20 >>>> # >>>> [all] >>>> # >>>> # Local addition that avoids USB3 SSD boot failures that look like: >>>> # uhub_reattach_port: port ? reset failed, error=3DUSB_ERR_TIMEOUT >>>> # uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling = port ? >>>> initial_turbo=3D60 >>>> # U-Boot that has, for example, a built-in usb_pgood_delay = assignment >>>> # for a media specific issue added: >>>> #kernel=3Du-boot.bin.2022.10.arm64 >>>> # >>>> # Local additions: >>>> enable_uart=3D1 >>>> uart_2ndstage=3D1 >>>> dtdebug=3D1 >>>> disable_commandline_tags=3D1 >>>> disable_overscan=3D1 >>>> #gpu_mem_1024=3D32 >>>> # >>>> #program_usb_boot_mode=3D1 >>>> #program_usb_boot_timeout=3D1 >>>>=20 >>>> # Old RPi3's/RPi2Bv1.2's may ignore [pi4] and the like. >>>> # That would make the below inappropriate for such contexts. >>>> [pi4] >>>> # Locally avoid hdmi_safe's dislay scaling: >>>> hdmi_safe=3D0 >>>> # >>>> armstub=3Darmstub8-gic.bin >>>> # >>>> # Local additions: >>>> over_voltage=3D6 >>>> arm_freq=3D2000 >>>> sdram_freq_min=3D3200 >>>> force_turbo=3D1 >>>> # >>>> #total_mem=3D1024 >>>> #total_mem=3D991 >>>> [all] >>>>=20 >>>> [pi3]=20 >>>> armstub=3Darmstub8.bin >>>> dtoverlay=3Dpwm >>>> audio_pwm_mode=3D0 >>>> [all] >>>>=20 >>>>=20 >>>> As for main [so: 14], the devclass_t use is gone, thus >>>> the source code changes are: >>>>=20 >>>> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c = b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>> index 5f9ecb0b7981..83c4c493a66b 100644 >>>> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >>>> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver =3D { >>>> sizeof(struct bcm_dma_softc), >>>> }; >>>>=20 >>>> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0); >>>> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0, >>>> + BUS_PASS_RESOURCE + BUS_PASS_ORDER_MIDDLE); >>>> MODULE_VERSION(bcm_dma, 1); >>>>=20 >>>>=20 >>>=20 >>> I should note that the above is not likely to be >>> the most appropriate in detail. The boot reports: >>>=20 >>> # dmesg -a | grep bcm_dma0 >>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq = 31,32,33,34,35,36,37,38,39,40,41 on simplebus0 >>> bcm_dma0: cannot allocate interrupt >>> device_attach: bcm_dma0 attach returned 6 >>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq = 31,32,33,34,35,36,37,38,39,40,41 on simplebus0 >>> bcm_dma0: cannot allocate interrupt >>> device_attach: bcm_dma0 attach returned 6 >>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq = 31,32,33,34,35,36,37,38,39,40,41 on simplebus0 >>> bcm_dma0: cannot allocate interrupt >>> device_attach: bcm_dma0 attach returned 6 >>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq = 31,32,33,34,35,36,37,38,39,40,41 on simplebus0 >>> bcm_dma0: cannot allocate interrupt >>> device_attach: bcm_dma0 attach returned 6 >>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq = 31,32,33,34,35,36,37,38,39,40,41 on simplebus0 >>>=20 >>> where that last (working) one has the message >>> context: >>>=20 >>> gic0: <ARM Generic Interrupt Controller> mem = 0x40041000-0x40041fff,0x40042000-0x40043fff,0x40044000-0x40045fff,0x400460= 00-0x40047fff irq 30 on simplebus0 >>> gic0: pn 0x2, arch 0x2, rev 0x1, implementer 0x43b irqs 256 >>> bcm_dma0: <BCM2835 DMA Controller> mem 0x7e007000-0x7e007aff irq = 31,32,33,34,35,36,37,38,39,40,41 on simplebus0 >>>=20 >>> So something involving BUS_PASS_INTERRUPT or later >>> (but before, say, BUS_PASS_SUPPORTDEV) may be more >>> appropriate. Possibly: >>>=20 >>> BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE >>>=20 >>> (so after gic0). >>>=20 >>>=20 >>=20 >> So, I'm now using . . . >> (leading whitespace possibly not accurately preserved) >>=20 >>=20 >> stable/13's source code changes are ( similarly for >> releng/13.1 ): >>=20 >> # git -C /usr/13S-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c >> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c = b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >> index cab8639bb607..6d521d6dcace 100644 >> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c >> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >> @@ -766,5 +766,6 @@ static driver_t bcm_dma_driver =3D { >>=20 >> static devclass_t bcm_dma_devclass; >>=20 >> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, bcm_dma_devclass, = 0, 0); >> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, = bcm_dma_devclass, >> + 0, 0, BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE); >> MODULE_VERSION(bcm_dma, 1); >>=20 >>=20 >> main's [so: 14's] source code changes are: >>=20 >> # git -C /usr/main-src/ diff sys/arm/broadcom/bcm2835/bcm2835_dma.c >> diff --git a/sys/arm/broadcom/bcm2835/bcm2835_dma.c = b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >> index 5f9ecb0b7981..d901447df1e9 100644 >> --- a/sys/arm/broadcom/bcm2835/bcm2835_dma.c >> +++ b/sys/arm/broadcom/bcm2835/bcm2835_dma.c >> @@ -764,5 +764,6 @@ static driver_t bcm_dma_driver =3D { >> sizeof(struct bcm_dma_softc), >> }; >>=20 >> -DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0); >> +EARLY_DRIVER_MODULE(bcm_dma, simplebus, bcm_dma_driver, 0, 0, >> + BUS_PASS_INTERRUPT + BUS_PASS_ORDER_LATE); >> MODULE_VERSION(bcm_dma, 1); >>=20 =3D=3D=3D Mark Millard marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?EAD84A57-E8F0-4149-BCFC-8A06FF03B11B>