Date: Sun, 18 Dec 2022 11:10:45 -0800 From: Mark Millard <marklmi@yahoo.com> To: freebsd-arm <freebsd-arm@freebsd.org> Subject: RPi4B boot -v "pci0: failed to allocate bus number" and "device_attach: pci0 attach returned 6"; more Message-ID: <7DDD8F49-B02C-484B-B92F-D894502B46DA@yahoo.com> References: <7DDD8F49-B02C-484B-B92F-D894502B46DA.ref@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
For reference, the example context used is
(long line output split for readability):
# uname -apKU
FreeBSD generic 13.1-STABLE FreeBSD 13.1-STABLE #0
stable/13-n253133-b51ee7ac252c: Wed Nov 23 03:36:16 UTC 2022
root@releng3.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC
arm64 aarch64 1301509 1301509
(So: not my build.)
=46rom the boot -v capture:
pcib0: <BCM2838-compatible PCI-express controller> mem =
0x7d500000-0x7d50930f irq 80,81 on simplebus2
pcib0: parsing FDT for ECAM0:
pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: 0x40000000
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: PCI addr: 0x0, CPU addr: 0x0, Size: 0x0
pcib0: Bus is not cache-coherent
pcib0: hardware identifies as revision 0x304.
pcib0: note: reported link speed is 5.0 GT/s.
pci1: <PCI bus> on pcib0
pci1: domain=3D0, physical bus=3D0
found-> vendor=3D0x14e4, dev=3D0x2711, revid=3D0x00
domain=3D0, bus=3D0, slot=3D0, func=3D0
class=3D06-04-00, hdrtype=3D0x01, mfdev=3D0
cmdreg=3D0x0000, statreg=3D0x0010, cachelnsz=3D0 (dwords)
lattimer=3D0x00 (0 ns), mingnt=3D0x00 (0 ns), maxlat=3D0x00 (0 =
ns)
intpin=3Da, irq=3D0
powerspec 3 supports D0 D3 current D0
secbus=3D1, subbus=3D1
pcib1: <PCI-PCI bridge> irq 91 at device 0.0 on pci1
pcib0: rman_reserve_resource: start=3D0xc0000000, end=3D0xc00fffff, =
count=3D0x100000
pcib1: domain 0
pcib1: secondary bus 1
pcib1: subordinate bus 1
pcib1: memory decode 0xc0000000-0xc00fffff
pci2: <PCI bus> on pcib1
pcib1: allocated bus range (1-1) for rid 0 of pci2
pci2: domain=3D0, physical bus=3D1
found-> vendor=3D0x1106, dev=3D0x3483, revid=3D0x01
domain=3D0, bus=3D1, slot=3D0, func=3D0
class=3D0c-03-30, hdrtype=3D0x00, mfdev=3D0
cmdreg=3D0x0000, statreg=3D0x0010, cachelnsz=3D0 (dwords)
lattimer=3D0x00 (0 ns), mingnt=3D0x00 (0 ns), maxlat=3D0x00 (0 =
ns)
intpin=3Da, irq=3D0
powerspec 3 supports D0 D3 current D0
MSI supports 4 messages, 64 bit
map[10]: type Memory, range 64, base 0, size 12, memory disabled
pcib1: slot 0 INTA is routed to irq 92
bcm_xhci0: <VL805 USB 3.0 controller (on the Raspberry Pi 4b)> irq 92 at =
device 0.0 on pci2
bcm_xhci0: note: xhci firmware not found.
bcm_xhci0: note: installing xhci firmware.
bcm_xhci0: note: xhci firmware detected; firmware is revision 138a1.
pcib1: allocated memory range (0xc0000000-0xc0000fff) for rid 10 of =
bcm_xhci0
bcm_xhci0: Lazy allocation of 0x1000 bytes rid 0x10 type 3 at 0xc0000000
bcm_xhci0: 32 bytes context size, 64-bit DMA
bcm_xhci0: attempting to allocate 1 MSI vectors (4 supported)
bcm_xhci0: using IRQ 93 for MSI
bcm_xhci0: MSI enabled
bcm_xhci0: (New XHCI DeviceId=3D0x34831106)
usbus0 on bcm_xhci0
bcm_xhci0: usbpf: Attached
bcm_xhci0: note: switched to 32-bit DMA.
pci0: <PCI bus> on pcib0
pci0: failed to allocate bus number
device_attach: pci0 attach returned 6
In order to get more directly comparable output, I tried
using lspci port and use of lspci on raspiOS64 (my
abbreviation) and fedora 37.
raspiOS64:
# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge =
(rev 10) (prog-if 00 [Normal decode])
Device tree node: =
/sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0
Flags: bus master, fast devsel, latency 0
Bus: primary=3D00, secondary=3D01, subordinate=3D01, =
sec-latency=3D0
I/O behind bridge: 00000000-00000fff [size=3D4K]
Memory behind bridge: c0000000-c00fffff [size=3D1M]
Prefetchable memory behind bridge: [disabled]
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=3D0000 Rev=3D0=
Len=3D028 <?>
Capabilities: [240] L1 PM Substates
01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host =
Controller (rev 01) (prog-if 30 [XHCI])
Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller
Device tree node: =
/sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
Flags: bus master, fast devsel, latency 0, IRQ 63
Memory at 600000000 (64-bit, non-prefetchable) [size=3D4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+
Capabilities: [c4] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: xhci_hcd
fedora:
# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge =
(rev 20) (prog-if 00 [Normal decode])
Device tree node: =
/sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0
Flags: bus master, fast devsel, latency 0, IRQ 36
Bus: primary=3D00, secondary=3D01, subordinate=3D01, =
sec-latency=3D0
Memory behind bridge: 00000000-000fffff [size=3D1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=3D0000 Rev=3D0=
Len=3D028 <?>
Capabilities: [240] L1 PM Substates
Kernel driver in use: pcieport
01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 =
Controller (rev 01) (prog-if 30 [XHCI])
Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 =
Controller
Device tree node: =
/sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
Flags: bus master, fast devsel, latency 0, IRQ 38
Memory at 600000000 (64-bit, non-prefetchable) [size=3D4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+
Capabilities: [c4] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: xhci_hcd
(00:00.0's "Memory behind bridge" displayed based on
different spaces, unfortunately. Not as uniform as I
was intending. Still. . .)
FreeBSD:
# lspci -v
00:00.0 PCI bridge: Broadcom Inc. and subsidiaries BCM2711 PCIe Bridge =
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 91
Bus: primary=3D00, secondary=3D01, subordinate=3D01, =
sec-latency=3D0
I/O behind bridge: 0000-0fff [size=3D4K] [16-bit]
Memory behind bridge: c0000000-c00fffff [size=3D1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [48] Power Management version 3
Capabilities: [ac] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [180] Vendor Specific Information: ID=3D0000 Rev=3D0=
Len=3D028 <?>
Capabilities: [240] L1 PM Substates
01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 =
Controller (rev 01) (prog-if 30 [XHCI])
Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 =
Controller
Flags: bus master, fast devsel, latency 0, IRQ 92
Memory at c0000000 (64-bit, non-prefetchable)
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+
Capabilities: [c4] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Note the 01:00.0's "Memory at": c0000000 vs. 600000000 for
the two linux exmaples. (Also: No size shown for FreeBSD.)
According to "PhilE Raspberry Pi Engineer & Forum Moderator" at:
=
https://forums.raspberrypi.com/viewtopic.php?t=3D288902&sid=3D792675eb2c65=
13e936b1cd9fe385a038&start=3D75
QUOTE
It's worth stating a few facts:
(1) The PCIe controller in BCM2711 has a single outbound window. This =
means that all addresses for PCIe devices have to be mapped in a single =
block, such that the spacing in PCIe space matches the spacing in host =
physical address space.
(2) The size of the outbound window must be a power of two, and the base =
address of the outbound window must be aligned to that size.
END QUOTE
According to bcm2711-peripherals.pdf figure 1 the standard
placement of PCIe space is 0x600000000..0x7ffffffff (avoiding
overlaps with anything else in any space).
For reference, the live Device Tree for the FreeBSD context
has for pcie@7d500000 :
ranges =3D <0x02000000 0x00000000 0xc0000000 =
0x00000006 0x00000000 0x00000000 0x40000000>;
The "0x00000006 0x00000000" part agrees with that "figure 1".
(More recent vintages agree for that part as well.)
I'm far from expert for such things but the 01:00.0 having
"Memory at c0000000 (64-bit, non-prefetchable)" looks
risky for PCI'e outbound memory mapping due to overlaps
having potential consequences.
An additional note of something that might be involved:
The RPi* firmware version that FreeBSD uses still has:
pcie@7d500000 {
. . .
pci@1,0 {
. . .
usb@1,0 {
. . .
but modern firmware (such as in the raspiOS64 and
fedora I used) has:
pcie@7d500000 {
. . .
pci@0,0 {
. . .
usb@0,0 {
. . .
I do not know if that is somehow related to
what I'm reporting or not.
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7DDD8F49-B02C-484B-B92F-D894502B46DA>