From owner-freebsd-arm@freebsd.org Wed Jan 13 21:33:39 2021 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AB6614E96CB for ; Wed, 13 Jan 2021 21:33:39 +0000 (UTC) (envelope-from mhorne@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4DGLKb4NPHz3pKn; Wed, 13 Jan 2021 21:33:39 +0000 (UTC) (envelope-from mhorne@freebsd.org) Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) (Authenticated sender: mhorne) by smtp.freebsd.org (Postfix) with ESMTPSA id 838C52B173; Wed, 13 Jan 2021 21:33:39 +0000 (UTC) (envelope-from mhorne@freebsd.org) Received: by mail-qk1-f180.google.com with SMTP id b64so4191941qkc.12; Wed, 13 Jan 2021 13:33:39 -0800 (PST) X-Gm-Message-State: AOAM53388ZdjVCoiPf15uHkJIueBbR6pDN4oGvnAUtvH6SHSynTi+1n2 cJCXnZVZQU36JndPruG9pwQ7+sPVN4tqCcwPyiA= X-Google-Smtp-Source: ABdhPJy+oxa04fxGP14uHCsAIKK1tOQMmPlYxjM0BF0yMXUXaxvbg2ciIId1SfUGKP4GerOyzoSc3Gm1AYyzzdcQUkw= X-Received: by 2002:a25:2d61:: with SMTP id s33mr6345414ybe.36.1610573619105; Wed, 13 Jan 2021 13:33:39 -0800 (PST) MIME-Version: 1.0 References: <20210112233607.GA79348@www.zefox.net> <90C90797-A8A5-457C-AF07-800EA82F5F12@yahoo.com> <20210113002432.GA79600@www.zefox.net> <04FEAC11-5603-4D4E-8651-43AB37A10B46@yahoo.com> <3C95F8C0-73FA-4DEC-9F0A-6FFF9846E8A3@yahoo.com> In-Reply-To: <3C95F8C0-73FA-4DEC-9F0A-6FFF9846E8A3@yahoo.com> From: Mitchell Horne Date: Wed, 13 Jan 2021 17:33:27 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: panic: Too many early devmap mappings To: Mark Millard Cc: "mhorne@freebsd.org" , "rwatson@freebsd.org" , Ed Maste , freebsd-arm , bob prohaska , Gordon Bergling , =?UTF-8?Q?Klaus_K=C3=BCchemann?= Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2021 21:33:39 -0000 On Wed, Jan 13, 2021 at 2:53 AM Mark Millard wrote: > > On 2021-Jan-12, at 16:55, Mark Millard wrote: > > > [I have a git bisect result for the failure: bbfa199cbc16.] > > > > On 2021-Jan-12, at 16:24, bob prohaska wrote: > > > >> On Tue, Jan 12, 2021 at 03:59:44PM -0800, Mark Millard wrote: > >>> > >>> > >>> On 2021-Jan-12, at 15:49, bob prohaska wrote: > >>> > >>>> An RPi3 running -current updated on Jan. 10 installed a new world/kernel and > >>>> when rebooted promptly crashed with > >>>> > >>>> ---<>--- > >>>> panic: Too many early devmap mappings > >>>> cpuid = 0 > >>>> time = 1 > >>>> KDB: stack backtrace: > >>>> (null)() at 0xffff00000011ad90 > >>>> pc = 0xffff000000760f70 lr = 0xffff00000011ad90 > >>>> sp = 0xffff0000011df330 fp = 0xffff0000011df530 > >>>> > >>>> (null)() at 0xffff00000045c2d4 > >>>> pc = 0xffff00000011ad90 lr = 0xffff00000045c2d4 > >>>> sp = 0xffff0000011df540 fp = 0xffff0000011df5a0 > >>>> > >>>> (null)() at 0xffff00000045c07c > >>>> pc = 0xffff00000045c2d4 lr = 0xffff00000045c07c > >>>> sp = 0xffff0000011df5b0 fp = 0xffff0000011df660 > >>>> > >>>> (null)() at 0xffff0000007d8380 > >>>> pc = 0xffff00000045c07c lr = 0xffff0000007d8380 > >>>> sp = 0xffff0000011df670 fp = 0xffff0000011df670 > >>>> > >>>> (null)() at 0xffff00000075dc98 > >>>> pc = 0xffff0000007d8380 lr = 0xffff00000075dc98 > >>>> sp = 0xffff0000011df680 fp = 0xffff0000011df6a0 > >>>> > >>>> (null)() at 0xffff0000007710e4 > >>>> pc = 0xffff00000075dc98 lr = 0xffff0000007710e4 > >>>> sp = 0xffff0000011df6b0 fp = 0xffff0000011df6d0 > >>>> > >>>> (null)() at 0xffff00000028850c > >>>> pc = 0xffff0000007710e4 lr = 0xffff00000028850c > >>>> sp = 0xffff0000011df6e0 fp = 0xffff0000011df7a0 > >>>> > >>>> (null)() at 0xffff0000007c8788 > >>>> pc = 0xffff00000028850c lr = 0xffff0000007c8788 > >>>> sp = 0xffff0000011df7b0 fp = 0xffff0000011df830 > >>>> > >>>> (null)() at 0xffff00000028a64c > >>>> pc = 0xffff0000007c8788 lr = 0xffff00000028a64c > >>>> sp = 0xffff0000011df840 fp = 0xffff0000011df850 > >>>> > >>>> (null)() at 0xffff00000039b340 > >>>> pc = 0xffff00000028a64c lr = 0xffff00000039b340 > >>>> sp = 0xffff0000011df860 fp = 0xffff0000011df870 > >>>> > >>>> (null)() at 0xffff0000004a6950 > >>>> pc = 0xffff00000039b340 lr = 0xffff0000004a6950 > >>>> sp = 0xffff0000011df880 fp = 0xffff0000011df8b0 > >>>> > >>>> (null)() at 0xffff00000076d73c > >>>> pc = 0xffff0000004a6950 lr = 0xffff00000076d73c > >>>> sp = 0xffff0000011df8c0 fp = 0xffff0000011dfa00 > >>>> > >>>> (null)() at 0xffff00000000089c > >>>> pc = 0xffff00000076d73c lr = 0xffff00000000089c > >>>> sp = 0xffff0000011dfa10 fp = 0x0000000000000000 > >>>> > >>>> KDB: enter: panic > >>>> [ thread pid 0 tid 0 ] > >>>> Stopped at 0xffff0000004a6550 > >>>> db> reboot > >>>> cpu_reset failed > >>>> > >>>> It had to be power-cycled to restart. It came back up readily with > >>>> kernel.old, which reports main-c255664-g4d64c7243d26 compiled Jan 9. > >>>> > >>>> In particular, how does one recognize which revision fixes > >>>> this problem, assuming it's a bug and not operator error? > >>>> Presumably, it'll take at least several days to reach git. > >>> > >>> Discovered last night on 8GiByte RPi4B's relative to this: > >>> Booting without a monitor changes the memory use and avoids > >>> the panic. WIth the 1920x1080 monitor it fails. (Only kernels > >>> with INVARIANTS make the check that panics, but need not > >>> mean that others are operating well, even if it is not > >>> obvious in a specific context.) > >>> > >>> Quoted from part of a message list item from last night: > >>> > >>> QUOTE > >>> Going back to my 19cca0b9613d based debug kernel build that > >>> has the printf's reporting the values used in the test, but > >>> with no monitor attached, it boots fine and reports: > >>> > >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007ffffff000 size: 1000 > >>> pmap_mapdev early_boot: va: ffff007fffffe000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000 > >>> > >>> That compares to the previously reported failure figures from > >>> having the monitor attached for that debug kernel: > >>> > >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: 1000 > >>> pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: ffff008000000000 L2_SIZE: 200000 > >>> panic: Too many early devmap mappings > >>> > >>> where the code does: > >>> > >>> KASSERT(va >= VM_MAX_KERNEL_ADDRESS - L2_SIZE, > >>> ("Too many early devmap mappings")); > >>> > >>> > >>> Looks like akva_devmap_vaddr gets smaller to make room above > >>> for monitor related data and so va can end up being too small > >>> by the criteria of this test. > >>> > >>> I've no clue who would be appropriate for dealing with this. > >>> END QUOTE > >>> > >>> You may have provided a bound for a bisection > >>> > >> > >> It looks as if unplugging the HDMI monitor (1920x1200) fixed the > >> panic on the RPi3B+ as well. > >> > >> [the original subject line said "devmatch", which confused me hugely 8-)] > >> > > > > A git bisect sequence on a 8 GiBYte RPi4B with a monitor plugged > > in (to make it use more high kernel RAM such that the KASSERT > > indicated above fails) resulted in: > > > > # git bisect good > > bbfa199cbc1698631a0e932848e62dd76559d4d7 is the first bad commit > > commit bbfa199cbc1698631a0e932848e62dd76559d4d7 > > Author: mhorne > > Date: Wed Dec 9 16:38:42 2020 -0400 > > > > arm64: gdb(4) machine-dependent bits > > > > Everything required for remote kernel debugging over a serial > > connection. For FDT-based systems, a debug port can be specified by > > setting hw.fdt.dbgport to the desired device tree node in loader.conf. > > For example, hw.fdt.dbgport="uart1", or > > hw.fdt.dbgport="serial@ff1a0000". > > > > Looks good: emaste > > Tested by: rwatson > > MFC after: 2 weeks > > Sponsored by: The FreeBSD Foundation > > Differential Revision: https://reviews.freebsd.org/D27727 > > > > sys/arm64/arm64/gdb_machdep.c | 112 ++++++++++++++++++++++++++++++++++++++++ > > sys/arm64/conf/GENERIC | 2 +- > > sys/arm64/include/gdb_machdep.h | 81 +++++++++++++++++++++++++++++ > > sys/conf/files.arm64 | 1 + > > 4 files changed, 195 insertions(+), 1 deletion(-) > > create mode 100644 sys/arm64/arm64/gdb_machdep.c > > create mode 100644 sys/arm64/include/gdb_machdep.h > > > > I forgot to list the bugzilla for this: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252541 > > I have added the notes, including: > > QUOTE > Turns out that this "too much high kernel memory in use" issue happens for > a combination of 2 things being true at the same time: > > A) Monitor attached (sufficiently large pixel count?) > B) GDB enabled, per bbfa199cbc16 . > END QUOTE > > > === > Mark Millard > marklmi at yahoo.com > ( dsl-only.net went > away in early 2018-Mar) > Hi all, The problem should be fixed with commit 818390ce0ca5 to main. Please let me know if you still encounter any issues after updating. Cheers, Mitchell