Date: Wed, 13 Jan 2021 17:33:27 -0400 From: Mitchell Horne <mhorne@freebsd.org> To: Mark Millard <marklmi@yahoo.com> Cc: "mhorne@freebsd.org" <mhorne@freebsd.org>, "rwatson@freebsd.org" <rwatson@freebsd.org>, Ed Maste <emaste@freebsd.org>, freebsd-arm <freebsd-arm@freebsd.org>, bob prohaska <fbsd@www.zefox.net>, Gordon Bergling <gbe@freebsd.org>, =?UTF-8?Q?Klaus_K=C3=BCchemann?= <maciphone2@googlemail.com> Subject: Re: panic: Too many early devmap mappings Message-ID: <CADeAsy22X%2B87T9-sO3NkBDjQue=57EvaaL1=tJEz5y90pKe98g@mail.gmail.com> In-Reply-To: <3C95F8C0-73FA-4DEC-9F0A-6FFF9846E8A3@yahoo.com> References: <20210112233607.GA79348@www.zefox.net> <90C90797-A8A5-457C-AF07-800EA82F5F12@yahoo.com> <20210113002432.GA79600@www.zefox.net> <04FEAC11-5603-4D4E-8651-43AB37A10B46@yahoo.com> <3C95F8C0-73FA-4DEC-9F0A-6FFF9846E8A3@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 13, 2021 at 2:53 AM Mark Millard <marklmi@yahoo.com> wrote:
>
> On 2021-Jan-12, at 16:55, Mark Millard <marklmi at yahoo.com> wrote:
>
> > [I have a git bisect result for the failure: bbfa199cbc16.]
> >
> > On 2021-Jan-12, at 16:24, bob prohaska <fbsd at www.zefox.net> wrote:
> >
> >> On Tue, Jan 12, 2021 at 03:59:44PM -0800, Mark Millard wrote:
> >>>
> >>>
> >>> On 2021-Jan-12, at 15:49, bob prohaska <fbsd at www.zefox.net> wrote:
> >>>
> >>>> An RPi3 running -current updated on Jan. 10 installed a new world/kernel and
> >>>> when rebooted promptly crashed with
> >>>>
> >>>> ---<<BOOT>>---
> >>>> panic: Too many early devmap mappings
> >>>> cpuid = 0
> >>>> time = 1
> >>>> KDB: stack backtrace:
> >>>> (null)() at 0xffff00000011ad90
> >>>> pc = 0xffff000000760f70 lr = 0xffff00000011ad90
> >>>> sp = 0xffff0000011df330 fp = 0xffff0000011df530
> >>>>
> >>>> (null)() at 0xffff00000045c2d4
> >>>> pc = 0xffff00000011ad90 lr = 0xffff00000045c2d4
> >>>> sp = 0xffff0000011df540 fp = 0xffff0000011df5a0
> >>>>
> >>>> (null)() at 0xffff00000045c07c
> >>>> pc = 0xffff00000045c2d4 lr = 0xffff00000045c07c
> >>>> sp = 0xffff0000011df5b0 fp = 0xffff0000011df660
> >>>>
> >>>> (null)() at 0xffff0000007d8380
> >>>> pc = 0xffff00000045c07c lr = 0xffff0000007d8380
> >>>> sp = 0xffff0000011df670 fp = 0xffff0000011df670
> >>>>
> >>>> (null)() at 0xffff00000075dc98
> >>>> pc = 0xffff0000007d8380 lr = 0xffff00000075dc98
> >>>> sp = 0xffff0000011df680 fp = 0xffff0000011df6a0
> >>>>
> >>>> (null)() at 0xffff0000007710e4
> >>>> pc = 0xffff00000075dc98 lr = 0xffff0000007710e4
> >>>> sp = 0xffff0000011df6b0 fp = 0xffff0000011df6d0
> >>>>
> >>>> (null)() at 0xffff00000028850c
> >>>> pc = 0xffff0000007710e4 lr = 0xffff00000028850c
> >>>> sp = 0xffff0000011df6e0 fp = 0xffff0000011df7a0
> >>>>
> >>>> (null)() at 0xffff0000007c8788
> >>>> pc = 0xffff00000028850c lr = 0xffff0000007c8788
> >>>> sp = 0xffff0000011df7b0 fp = 0xffff0000011df830
> >>>>
> >>>> (null)() at 0xffff00000028a64c
> >>>> pc = 0xffff0000007c8788 lr = 0xffff00000028a64c
> >>>> sp = 0xffff0000011df840 fp = 0xffff0000011df850
> >>>>
> >>>> (null)() at 0xffff00000039b340
> >>>> pc = 0xffff00000028a64c lr = 0xffff00000039b340
> >>>> sp = 0xffff0000011df860 fp = 0xffff0000011df870
> >>>>
> >>>> (null)() at 0xffff0000004a6950
> >>>> pc = 0xffff00000039b340 lr = 0xffff0000004a6950
> >>>> sp = 0xffff0000011df880 fp = 0xffff0000011df8b0
> >>>>
> >>>> (null)() at 0xffff00000076d73c
> >>>> pc = 0xffff0000004a6950 lr = 0xffff00000076d73c
> >>>> sp = 0xffff0000011df8c0 fp = 0xffff0000011dfa00
> >>>>
> >>>> (null)() at 0xffff00000000089c
> >>>> pc = 0xffff00000076d73c lr = 0xffff00000000089c
> >>>> sp = 0xffff0000011dfa10 fp = 0x0000000000000000
> >>>>
> >>>> KDB: enter: panic
> >>>> [ thread pid 0 tid 0 ]
> >>>> Stopped at 0xffff0000004a6550
> >>>> db> reboot
> >>>> cpu_reset failed
> >>>>
> >>>> It had to be power-cycled to restart. It came back up readily with
> >>>> kernel.old, which reports main-c255664-g4d64c7243d26 compiled Jan 9.
> >>>>
> >>>> In particular, how does one recognize which revision fixes
> >>>> this problem, assuming it's a bug and not operator error?
> >>>> Presumably, it'll take at least several days to reach git.
> >>>
> >>> Discovered last night on 8GiByte RPi4B's relative to this:
> >>> Booting without a monitor changes the memory use and avoids
> >>> the panic. WIth the 1920x1080 monitor it fails. (Only kernels
> >>> with INVARIANTS make the check that panics, but need not
> >>> mean that others are operating well, even if it is not
> >>> obvious in a specific context.)
> >>>
> >>> Quoted from part of a message list item from last night:
> >>>
> >>> QUOTE
> >>> Going back to my 19cca0b9613d based debug kernel build that
> >>> has the printf's reporting the values used in the test, but
> >>> with no monitor attached, it boots fine and reports:
> >>>
> >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007ffffff000 size: 1000
> >>> pmap_mapdev early_boot: va: ffff007fffffe000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
> >>>
> >>> That compares to the previously reported failure figures from
> >>> having the monitor attached for that debug kernel:
> >>>
> >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: 1000
> >>> pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000
> >>> panic: Too many early devmap mappings
> >>>
> >>> where the code does:
> >>>
> >>> KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE,
> >>> ("Too many early devmap mappings"));
> >>>
> >>>
> >>> Looks like akva_devmap_vaddr gets smaller to make room above
> >>> for monitor related data and so va can end up being too small
> >>> by the criteria of this test.
> >>>
> >>> I've no clue who would be appropriate for dealing with this.
> >>> END QUOTE
> >>>
> >>> You may have provided a bound for a bisection
> >>>
> >>
> >> It looks as if unplugging the HDMI monitor (1920x1200) fixed the
> >> panic on the RPi3B+ as well.
> >>
> >> [the original subject line said "devmatch", which confused me hugely 8-)]
> >>
> >
> > A git bisect sequence on a 8 GiBYte RPi4B with a monitor plugged
> > in (to make it use more high kernel RAM such that the KASSERT
> > indicated above fails) resulted in:
> >
> > # git bisect good
> > bbfa199cbc1698631a0e932848e62dd76559d4d7 is the first bad commit
> > commit bbfa199cbc1698631a0e932848e62dd76559d4d7
> > Author: mhorne <mhorne@FreeBSD.org>
> > Date: Wed Dec 9 16:38:42 2020 -0400
> >
> > arm64: gdb(4) machine-dependent bits
> >
> > Everything required for remote kernel debugging over a serial
> > connection. For FDT-based systems, a debug port can be specified by
> > setting hw.fdt.dbgport to the desired device tree node in loader.conf.
> > For example, hw.fdt.dbgport="uart1", or
> > hw.fdt.dbgport="serial@ff1a0000".
> >
> > Looks good: emaste
> > Tested by: rwatson
> > MFC after: 2 weeks
> > Sponsored by: The FreeBSD Foundation
> > Differential Revision: https://reviews.freebsd.org/D27727
> >
> > sys/arm64/arm64/gdb_machdep.c | 112 ++++++++++++++++++++++++++++++++++++++++
> > sys/arm64/conf/GENERIC | 2 +-
> > sys/arm64/include/gdb_machdep.h | 81 +++++++++++++++++++++++++++++
> > sys/conf/files.arm64 | 1 +
> > 4 files changed, 195 insertions(+), 1 deletion(-)
> > create mode 100644 sys/arm64/arm64/gdb_machdep.c
> > create mode 100644 sys/arm64/include/gdb_machdep.h
> >
>
> I forgot to list the bugzilla for this:
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541
>
> I have added the notes, including:
>
> QUOTE
> Turns out that this "too much high kernel memory in use" issue happens for
> a combination of 2 things being true at the same time:
>
> A) Monitor attached (sufficiently large pixel count?)
> B) GDB enabled, per bbfa199cbc16 .
> END QUOTE
>
>
> ===
> Mark Millard
> marklmi at yahoo.com
> ( dsl-only.net went
> away in early 2018-Mar)
>
Hi all,
The problem should be fixed with commit 818390ce0ca5 to main. Please
let me know if you still encounter any issues after updating.
Cheers,
Mitchell
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CADeAsy22X%2B87T9-sO3NkBDjQue=57EvaaL1=tJEz5y90pKe98g>