From owner-freebsd-arm@freebsd.org Mon Jul 22 08:07:54 2019 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DBC88AB258 for ; Mon, 22 Jul 2019 08:07:54 +0000 (UTC) (envelope-from dave@dogwood.com) Received: from mail-oi1-x244.google.com (mail-oi1-x244.google.com [IPv6:2607:f8b0:4864:20::244]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4A6F89B6E for ; Mon, 22 Jul 2019 08:07:53 +0000 (UTC) (envelope-from dave@dogwood.com) Received: by mail-oi1-x244.google.com with SMTP id t76so28966142oih.4 for ; Mon, 22 Jul 2019 01:07:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dogwood.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=DHEFFhr4FL0tSFdJWCFmYqU+9liMRDZz357HscXh9lk=; b=X8l+dLvLpagSfLKd5iC6iWUyQLzxPjT/HZvSW2ioUA4WV6yLHO1HdSo4BWyZ88fjsk 6XD8BRYvnOeuEQNIHaHhQJ1VPpxEb+ZF7+z7pAmhcAwl1w9fpUk+iAvJcDuWOLCaV+Yd n/3wJN7sC16rjh/o8ZxOaXhypQEv3rhCro/nI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=DHEFFhr4FL0tSFdJWCFmYqU+9liMRDZz357HscXh9lk=; b=mFZ+TfW1krxQokLDLxq8qKro1vYNLIjelosfaLSsaYg87iIMrwjraL0LAAIzIeAN7B lwBpEddENcN+GuGuntuWIfcfj3h1hLR7294y/jEelsTCoR9FbKYlu1CW0AvL2nVZ/RvO +6X8sQMZabgsALj0wJDAVXPC5+c40NUPqui3s/kyBOssvkoHP7t0jc0xKLzy4UYCPQvX rvNCjDXsozuyk2GhRv57gvPARdTvbswP/uPOdJOMwzjb4HpsiEuzAAgtPaWapwhWNO6u LAu0lsSEbNu/p9r/Oz19SaDGN5Xeurc2eyHafneLKcaZ/ICjbXAYcfnfG1vu/aMe2DBl 3qPA== X-Gm-Message-State: APjAAAXHiRhJX6MRwCjNc6ZivgNaQqtzUaC7ylSLTd8yO6+G7qGE4jzK cFUYuZKbFrvLRPoRGZ2JIV4ksEgseCteHsqiDWOXEw== X-Google-Smtp-Source: APXvYqxLSTzUD0aGnVUAi6ACYF9QL7FtvHl/QH71R3HgSTWFshd4RNt9p0AUaj7alPcbOJclRrW4eZTZEfx0PiNe+kE= X-Received: by 2002:aca:5e88:: with SMTP id s130mr34714585oib.91.1563782872516; Mon, 22 Jul 2019 01:07:52 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: David Cornejo Date: Sun, 21 Jul 2019 22:07:41 -1000 Message-ID: Subject: Re: Raspberry Pi 4 boot hangs in sched_idletd. To: Robert Crowston Cc: "freebsd-arm@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: B4A6F89B6E X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=dogwood.com header.s=google header.b=X8l+dLvL; spf=pass (mx1.freebsd.org: domain of dave@dogwood.com designates 2607:f8b0:4864:20::244 as permitted sender) smtp.mailfrom=dave@dogwood.com X-Spamd-Result: default: False [-3.67 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[dogwood.com:s=google]; NEURAL_HAM_MEDIUM(-0.99)[-0.992,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; DMARC_NA(0.00)[dogwood.com]; NEURAL_HAM_SHORT(-0.49)[-0.495,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[dogwood.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com,aspmx.l.google.com,aspmx5.googlemail.com,aspmx4.googlemail.com,aspmx3.googlemail.com,alt2.aspmx.l.google.com,aspmx2.googlemail.com]; IP_SCORE(-0.67)[ip: (2.22), ipnet: 2607:f8b0::/32(-3.11), asn: 15169(-2.42), country: US(-0.05)]; FREEMAIL_TO(0.00)[protonmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jul 2019 08:07:54 -0000 Is there some sort of guide or instructions for using a JTAG debugger on the FreeBSD kernel available somewhere? thanks, dave c On Sun, Jul 21, 2019 at 4:14 AM Robert Crowston via freebsd-arm wrote: > > I need a bit of a hand with this. I've been working on getting FreeBSD 13= .0-Current up and running on my pi4. I'm using the GENERIC configuration fo= r now. I have a JTAG hardware debugger so I've got a pretty good introspect= ion on what's going on at the detail level, but I'm missing some of the big= ger picture. > > The first problem is, this board has two interrupt controllers on the FDT= ; the bcm2836-l1-intc local interrupt controller (local_intc) that was pres= ent on the RPi3, and also a new gic400. Both the drivers call intr_pic_clai= m_root(), which causes a panic. If I remove the gic400 from the FDT, very l= ittle of the hardware enumeration succeeds and the kernel panics because th= ere is no event timer. If I remove the local_intc, a few devices, including= the BCM2835 DMA controller, the SD card controllers, and the GPIO drivers = fail to start, but the rest of the hardware enumeration finishes. I don't k= now enough about the hardware topology to figure out which one is the real = root and fix the drivers accordingly. > > So, without local_intc, we get another problem. Late in the boot sequence= , the idle thread gets swapped in, and there are still threads on the sleep= queue, including thread0, but no other threads get put on the run queue, a= nd the Pi loops through sched_idletd thread forever instead of finishing th= e boot. > > By poking things in the debugger I can progress as far as vfs_mountroot_w= ait(), and I think there is a race condition because the behaviour is diffe= rent if I step through manually without changing anything, but whatever hap= pens, eventually I end up stuck in the sched_idletd loop. > > I think perhaps event timer interrupts are not being delivered so the sle= ep queue never gets moved to the run queue?, but I'm guessing here. > > More detail: > > So far I'm in mi_startup(), and about ~1150 out of ~1200 SYSINIT tasks ar= e done. Eventually we get to initialize the usbus. One of its worker thread= s calls _sleep(): > > Breakpoint 6, _sleep (ident=3D0xffff000000c5eb28 , lock=3D0x= 0, priority=3D0, > wmesg=3D0xffff000000775d55 "USBWAIT", sbt=3D47244637, pr=3D0, flags= =3D256) > at /skeleton/root/sandbox/src/sys/kern/kern_synch.c:139 > 139 td =3D curthread; > (gdb) bt > #0 _sleep (ident=3D0xffff000000c5eb28 , lock=3D0x0, priorit= y=3D0, > wmesg=3D0xffff000000775d55 "USBWAIT", sbt=3D47244637, pr=3D0, flags= =3D256) > at /skeleton/root/sandbox/src/sys/kern/kern_synch.c:139 > #1 0xffff00000029db08 in usbd_req_set_address (udev=3D0xfffffd0000d23000= , mtx=3D0x0, > addr=3D) at /skeleton/root/sandbox/src/sys/dev/usb/usb= _request.c:1580 > #2 0xffff00000028d05c in usb_alloc_device (parent_dev=3D,= bus=3D0xffff0000405fe000, > parent_hub=3D0x0, depth=3D, port_index=3D, port_no=3D, > speed=3DUSB_SPEED_HIGH, mode=3D) > at /skeleton/root/sandbox/src/sys/dev/usb/usb_device.c:1824 > #3 0xffff000000281524 in usb_bus_attach (pm=3D) > at /skeleton/root/sandbox/src/sys/dev/usb/controller/usb_controller.c= :767 > #4 0xffff00000029b750 in usb_process (arg=3D0xffff0000405fe130) > at /skeleton/root/sandbox/src/sys/dev/usb/usb_process.c:178 > #5 0xffff0000003b78c0 in fork_exit (callout=3D0xffff00000029b65c , > arg=3D0xffff0000405fe130, frame=3D0xffff00004024cba0) > at /skeleton/root/sandbox/src/sys/kern/kern_fork.c:1056 > #6 0xffff00000071fe88 in fork_trampoline () > at /skeleton/root/sandbox/src/sys/arm64/arm64/swtch.S:214 > > _sleep() swaps the usb_process thread off the CPU. If I step through, ano= ther few threads are put on the CPU, including taskqueue_thread_loop, soaio= _kproc_loop, random_kthread. Sometimes I also see the interrupt event threa= d, but not always. Sometimes if I step, I'll find thread0 does get back on = the CPU, in which case we can progress as far as release_aps(), but then th= e other CPUs get stuck in sched_idletd and eventually CPU0 will hang in smp= _rendezvous() waiting for the other CPUs. > > Does anyone have any ideas what I should investigate next? > > Boot log attached. > > > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org" --=20 Kailua, Hawai=CA=BBi US +1 (808) 728-3050 UK +44 (020) 3286 2808