Date: Sun, 21 Jul 2019 22:07:41 -1000 From: David Cornejo <dave@dogwood.com> To: Robert Crowston <crowston@protonmail.com> Cc: "freebsd-arm@freebsd.org" <freebsd-arm@freebsd.org> Subject: Re: Raspberry Pi 4 boot hangs in sched_idletd. Message-ID: <CAFnjQbt3TNdjgis2sZ7vrH0nsiP3R-i3zaQ0FtjL1hmZr6gMcQ@mail.gmail.com> In-Reply-To: <bZhr6z5_4M6PcYR8ZarvkmvnLEb9KLGiBgffRczi8BR8tGB4FzDhvddWuubPAJCuRfebzCJA2Uf5RSd9dUF4tNkkhRa9Pn27ruXia9s2C_M=@protonmail.com> References: <bZhr6z5_4M6PcYR8ZarvkmvnLEb9KLGiBgffRczi8BR8tGB4FzDhvddWuubPAJCuRfebzCJA2Uf5RSd9dUF4tNkkhRa9Pn27ruXia9s2C_M=@protonmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Is there some sort of guide or instructions for using a JTAG debugger on the FreeBSD kernel available somewhere? thanks, dave c On Sun, Jul 21, 2019 at 4:14 AM Robert Crowston via freebsd-arm <freebsd-arm@freebsd.org> wrote: > > I need a bit of a hand with this. I've been working on getting FreeBSD 13= .0-Current up and running on my pi4. I'm using the GENERIC configuration fo= r now. I have a JTAG hardware debugger so I've got a pretty good introspect= ion on what's going on at the detail level, but I'm missing some of the big= ger picture. > > The first problem is, this board has two interrupt controllers on the FDT= ; the bcm2836-l1-intc local interrupt controller (local_intc) that was pres= ent on the RPi3, and also a new gic400. Both the drivers call intr_pic_clai= m_root(), which causes a panic. If I remove the gic400 from the FDT, very l= ittle of the hardware enumeration succeeds and the kernel panics because th= ere is no event timer. If I remove the local_intc, a few devices, including= the BCM2835 DMA controller, the SD card controllers, and the GPIO drivers = fail to start, but the rest of the hardware enumeration finishes. I don't k= now enough about the hardware topology to figure out which one is the real = root and fix the drivers accordingly. > > So, without local_intc, we get another problem. Late in the boot sequence= , the idle thread gets swapped in, and there are still threads on the sleep= queue, including thread0, but no other threads get put on the run queue, a= nd the Pi loops through sched_idletd thread forever instead of finishing th= e boot. > > By poking things in the debugger I can progress as far as vfs_mountroot_w= ait(), and I think there is a race condition because the behaviour is diffe= rent if I step through manually without changing anything, but whatever hap= pens, eventually I end up stuck in the sched_idletd loop. > > I think perhaps event timer interrupts are not being delivered so the sle= ep queue never gets moved to the run queue?, but I'm guessing here. > > More detail: > > So far I'm in mi_startup(), and about ~1150 out of ~1200 SYSINIT tasks ar= e done. Eventually we get to initialize the usbus. One of its worker thread= s calls _sleep(): > > Breakpoint 6, _sleep (ident=3D0xffff000000c5eb28 <pause_wchan>, lock=3D0x= 0, priority=3D0, > wmesg=3D0xffff000000775d55 "USBWAIT", sbt=3D47244637, pr=3D0, flags= =3D256) > at /skeleton/root/sandbox/src/sys/kern/kern_synch.c:139 > 139 td =3D curthread; > (gdb) bt > #0 _sleep (ident=3D0xffff000000c5eb28 <pause_wchan>, lock=3D0x0, priorit= y=3D0, > wmesg=3D0xffff000000775d55 "USBWAIT", sbt=3D47244637, pr=3D0, flags= =3D256) > at /skeleton/root/sandbox/src/sys/kern/kern_synch.c:139 > #1 0xffff00000029db08 in usbd_req_set_address (udev=3D0xfffffd0000d23000= , mtx=3D0x0, > addr=3D<optimized out>) at /skeleton/root/sandbox/src/sys/dev/usb/usb= _request.c:1580 > #2 0xffff00000028d05c in usb_alloc_device (parent_dev=3D<optimized out>,= bus=3D0xffff0000405fe000, > parent_hub=3D0x0, depth=3D<optimized out>, port_index=3D<optimized ou= t>, port_no=3D<optimized out>, > speed=3DUSB_SPEED_HIGH, mode=3D<optimized out>) > at /skeleton/root/sandbox/src/sys/dev/usb/usb_device.c:1824 > #3 0xffff000000281524 in usb_bus_attach (pm=3D<optimized out>) > at /skeleton/root/sandbox/src/sys/dev/usb/controller/usb_controller.c= :767 > #4 0xffff00000029b750 in usb_process (arg=3D0xffff0000405fe130) > at /skeleton/root/sandbox/src/sys/dev/usb/usb_process.c:178 > #5 0xffff0000003b78c0 in fork_exit (callout=3D0xffff00000029b65c <usb_pr= ocess>, > arg=3D0xffff0000405fe130, frame=3D0xffff00004024cba0) > at /skeleton/root/sandbox/src/sys/kern/kern_fork.c:1056 > #6 0xffff00000071fe88 in fork_trampoline () > at /skeleton/root/sandbox/src/sys/arm64/arm64/swtch.S:214 > > _sleep() swaps the usb_process thread off the CPU. If I step through, ano= ther few threads are put on the CPU, including taskqueue_thread_loop, soaio= _kproc_loop, random_kthread. Sometimes I also see the interrupt event threa= d, but not always. Sometimes if I step, I'll find thread0 does get back on = the CPU, in which case we can progress as far as release_aps(), but then th= e other CPUs get stuck in sched_idletd and eventually CPU0 will hang in smp= _rendezvous() waiting for the other CPUs. > > Does anyone have any ideas what I should investigate next? > > Boot log attached. > > > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" --=20 Kailua, Hawai=CA=BBi US +1 (808) 728-3050 UK +44 (020) 3286 2808
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFnjQbt3TNdjgis2sZ7vrH0nsiP3R-i3zaQ0FtjL1hmZr6gMcQ>