Date: Wed, 14 Feb 2024 10:16:02 -0800 From: Mark Millard <marklmi@yahoo.com> To: John Baldwin <jhb@FreeBSD.org> Cc: FreeBSD ARM List <freebsd-arm@freebsd.org>, Current FreeBSD <freebsd-current@freebsd.org>, Warner Losh <imp@bsdimp.com> Subject: Re: Recent commits reject RPi4B booting: pcib0 vs. pcib1 "rman_manage_region: <pcib1 memory window> request" leads to panic Message-ID: <CCA368C1-C8D7-4B54-999D-6F709C4E5039@yahoo.com> In-Reply-To: <36ECB040-7E09-47A9-AF71-DE546A78E9CA@yahoo.com> References: <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5.ref@yahoo.com> <76AB969F-5BC5-4116-8AF4-3ED2CABEBBA5@yahoo.com> <f4326cda-c602-439f-ab03-f66b3bca5ff2@FreeBSD.org> <1F704317-FDB8-4BDA-8A67-61CF48794DFE@yahoo.com> <9AFDF067-96E4-4E67-90D2-F40DAF3F138F@yahoo.com> <4C279710-5F88-4295-B1A4-7C395F3587E5@yahoo.com> <3A145420-399D-4EBD-9FF4-18924908AB1D@yahoo.com> <1298DF9C-0F82-4567-8E81-7332A608C7FC@yahoo.com> <d7b20565-6aa1-486f-a197-11fbc4d0c8dd@FreeBSD.org> <36ECB040-7E09-47A9-AF71-DE546A78E9CA@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Top posting a related but separate item:
I looked up some old (2022-Dec-17) lspci -v output from
a Linux boot. Note the "Memory at" value 600000000 (in
the 35 bit BCM2711 address space) and the "(64-bit,
non-prefetchable)" (and "[size=3D4K]").
01:00.0 USB controller: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 =
Controller (rev 01) (prog-if 30 [XHCI])
Subsystem: VIA Technologies, Inc. VL805/806 xHCI USB 3.0 =
Controller
Device tree node: =
/sys/firmware/devicetree/base/scb/pcie@7d500000/pci@0,0/usb@0,0
Flags: bus master, fast devsel, latency 0, IRQ 51
Memory at 600000000 (64-bit, non-prefetchable) [size=3D4K]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=3D1/4 Maskable- 64bit+
Capabilities: [c4] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: xhci_hcd
"Memory at 600000000 (64-bit, non-prefetchable)":
Violation of a PCIe standard?
On Feb 14, 2024, at 09:57, Mark Millard <marklmi@yahoo.com> wrote:
> On Feb 14, 2024, at 08:08, John Baldwin <jhb@FreeBSD.org> wrote:
>=20
>> On 2/12/24 5:57 PM, Mark Millard wrote:
>>> On Feb 12, 2024, at 16:36, Mark Millard <marklmi@yahoo.com> wrote:
>>>> On Feb 12, 2024, at 16:10, Mark Millard <marklmi@yahoo.com> wrote:
>>>>=20
>>>>> On Feb 12, 2024, at 12:00, Mark Millard <marklmi@yahoo.com> wrote:
>>>>>=20
>>>>>> [Gack: I was looking at the wrong vintage of source code, =
predating
>>>>>> your changes: wrong system used.]
>>>>>>=20
>>>>>>=20
>>>>>> On Feb 12, 2024, at 10:41, Mark Millard <marklmi@yahoo.com> =
wrote:
>>>>>>=20
>>>>>>> On Feb 12, 2024, at 09:32, John Baldwin <jhb@freebsd.org> wrote:
>>>>>>>=20
>>>>>>>> On 2/9/24 8:13 PM, Mark Millard wrote:
>>>>>>>>> Summary:
>>>>>>>>> pcib0: <BCM2838-compatible PCI-express controller> mem =
0x7d500000-0x7d50930f irq 80,81 on simplebus2
>>>>>>>>> pcib0: parsing FDT for ECAM0:
>>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: =
0x40000000
>>>>>>>>> . . .
>>>>>>>>> rman_manage_region: <pcib1 memory window> request: start =
0x600000000, end 0x6000fffff
>>>>>>>>> panic: Failed to add resource to rman
>>>>>>>>=20
>>>>>>>> Hmmm, I suspect this is due to the way that =
bus_translate_resource works which is
>>>>>>>> fundamentally broken. It rewrites the start address of a =
resource in-situ instead
>>>>>>>> of keeping downstream resources separate from the upstream =
resources. For example,
>>>>>>>> I don't see how you could ever release a resource in this =
design without completely
>>>>>>>> screwing up your rman. That is, I expect trying to detach a =
PCI device behind a
>>>>>>>> translating bridge that uses the current approach should =
corrupt the allocated
>>>>>>>> resource ranges in an rman long before my changes.
>>>>>>>>=20
>>>>>>>> That said, that doesn't really explain the panic. Hmm, the =
panic might be because
>>>>>>>> for PCI bridge windows the driver now passes RF_ACTIVE and the =
bus_translate_resource
>>>>>>>> hack only kicks in the activate_resource method of =
pci_host_generic.c.
>>>>>>>>=20
>>>>>>>>> Detail:
>>>>>>>>> . . .
>>>>>>>>> pcib0: <BCM2838-compatible PCI-express controller> mem =
0x7d500000-0x7d50930f irq 80,81 on simplebus2
>>>>>>>>> pcib0: parsing FDT for ECAM0:
>>>>>>>>> pcib0: PCI addr: 0xc0000000, CPU addr: 0x600000000, Size: =
0x40000000
>>>>>>>>=20
>>>>>>>> This indicates this is a translating bus.
>>>>>>>>=20
>>>>>>>>> pcib1: <PCI-PCI bridge> irq 91 at device 0.0 on pci0
>>>>>>>>> rman_manage_region: <pcib1 bus numbers> request: start 0x1, =
end 0x1
>>>>>>>>> pcib0: rman_reserve_resource: start=3D0xc0000000, =
end=3D0xc00fffff, count=3D0x100000
>>>>>>>>> rman_reserve_resource_bound: <PCIe Memory> request: =
[0xc0000000, 0xc00fffff], length 0x100000, flags 102, device pcib1
>>>>>>>>> rman_reserve_resource_bound: trying 0xffffffff =
<0xc0000000,0xfffff>
>>>>>>>>> considering [0xc0000000, 0xffffffff]
>>>>>>>>> truncated region: [0xc0000000, 0xc00fffff]; size 0x100000 =
(requested 0x100000)
>>>>>>>>> candidate region: [0xc0000000, 0xc00fffff], size 0x100000
>>>>>>>>> allocating from the beginning
>>>>>>>>> rman_manage_region: <pcib1 memory window> request: start =
0x600000000, end 0x6000fffff
>>>>>=20
>>>>> What you later typed does not match:
>>>>>=20
>>>>> 0x600000000
>>>>> 0x6000fffff
>>>>>=20
>>>>> You later typed:
>>>>>=20
>>>>> 0x60000000
>>>>> 0x600fffffff
>>>>>=20
>>>>> This seems to have lead to some confusion from using the
>>>>> wrong figure(s).
>>>>>=20
>>>>>>>> The fact that we are trying to reserve the CPU addresses in the =
rman is because
>>>>>>>> bus_translate_resource rewrote the start address in the =
resource after it was allocated.
>>>>>>>>=20
>>>>>>>> That said, I can't see why rman_manage_region would actually =
fail. At this point the
>>>>>>>> rman is empty (this is the first call to rman_manage_region for =
"pcib1 memory window"),
>>>>>>>> so only the check that should be failing are the checks against =
rm_start and
>>>>>>>> rm_end. For the memory window, rm_start is always 0, and =
rm_end is always
>>>>>>>> 0xffffffff, so both the old (0xc00000000 - 0xc00fffff) and new =
(0x60000000 - 0x600fffffff)
>>>>>>>> ranges are within those bounds.
>>>>>=20
>>>>> No:
>>>>>=20
>>>>> 0xffffffff
>>>>>=20
>>>>> .vs (actual):
>>>>>=20
>>>>> 0x600000000
>>>>> 0x6000fffff
>>=20
>> Ok, then this explains the failure if the "raw" addresses are above =
4G. I have
>> access to an emag I'm currently using to test fixes to =
pci_host_generic.c to
>> avoid corrupting struct resource objects. I'll post the diff once =
I've got
>> something verified to work.
>>=20
>>> It looks to me like in sys/dev/pci/pci_pci.c the:
>>> static void
>>> pcib_probe_windows(struct pcib_softc *sc)
>>> {
>>> . . .
>>> pcib_alloc_window(sc, &sc->mem, SYS_RES_MEMORY, 0, =
0xffffffff);
>>> . . .
>>> is just inappropriately restrictive about where in the system
>>> address space a PCIe can validly be mapped to on the high end.
>>> That, in turn, leads to the rejection on the RPi4B now that
>>> the range use is checked.
>>=20
>> No, the physical register in PCI-PCI bridges is only 32-bits. Only =
the
>> prefetchable BAR supports 64-bit addresses.
>=20
> Just for my edification . . .
>=20
> As I understand, SYS_RES_MEMORY for the BCM2711
> means the 35 bit addressing space in the BCM2711,
> not a PCIe device internal address range that
> corresponds. Am I wrong about that?
>=20
> If I'm wrong, what does identify the 35 bit
> addressing space in the BCM2711?
>=20
> If I'm correct, then the 0..0xffffffff
> seems to be from the wrong address space up
> front. Or, may be, the SYS_RES_MEMORY and the
> 0xffffffff argments are not related as I
> expected and the 0xffffffff is not a
> SYS_RES_MEMORY value?
>=20
>> This is why the host bridge
>> is doing a translation from the CPU side (0x600000000) to the PCI BAR
>> addresses (0xc0000000) so that the BAR addresses are down in the =
32-bit
>> address range. It's also true that many PCI devices only support =
32-bit
>> addresses in memory BARs. 64-bit BARs are an optional extension not
>> universally supported.
>>=20
>> The translation here is somewhat akin to a type of MMU where the CPU
>> addresses are mapped to PCI addresses. The problem here is that the
>> PCI BAR resources need to "stay" as PCI addresses since we depend on
>> being able to use rman_get_start/end to get the PCI addresses of
>> allocated resources, but pci_host_generic.c currently rewrites the
>> addresses.
>>=20
>> Probably I should remove rman_set_start/end entirely (Warner added =
them
>> back in 2004) as the methods don't do anything to deal with the =
fallout
>> that the rman.rm_list linked-list is no longer sorted by address once
>> some addresses get rewritten, etc.
>>=20
=3D=3D=3D
Mark Millard
marklmi at yahoo.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CCA368C1-C8D7-4B54-999D-6F709C4E5039>