From owner-freebsd-arm@freebsd.org Wed Jan 13 06:53:08 2021 Return-Path: Delivered-To: freebsd-arm@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4185F4D1EB9 for ; Wed, 13 Jan 2021 06:53:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic305-19.consmr.mail.gq1.yahoo.com (sonic305-19.consmr.mail.gq1.yahoo.com [98.137.64.82]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4DFynb1B9Jz3ldq for ; Wed, 13 Jan 2021 06:53:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1610520785; bh=QjS5zskQZuBwiYOQo2OdptKpU4hSDJhI5ZCLzh1BP4O=; h=Subject:From:Date:To:From:Subject:Reply-To; b=jVO/IGK0mA3lBcCEYrGzJktE3390zGLtkFhjazEUKUgWylUkqW2i7+9B8W2K58mWcDmt5OEuKrJOnnr4vb5aNwmO4XCa0SsKcH1zJcgqX4Ej5eUK6TY4C00+IY4DhxV8KpehqN82H+NUitSnIepCJLgn/UUfbyTqPElg9Hofwu+k5PrVmM5lwMOi9VHL7GKOJAHC7ZPGCKdRGhfj6p0yPK7XSGoIePq4kyGYUzW5BICXKfyQn5lBGQn0eSVAQcWWGDYTYGkDokq9V4c0+PNNvkihkpbEltl8euWiqfPHOEJdo5pmHoagvLAK4Ux/ElHvYcsyjR2VV89XdglZAzIuWQ== X-YMail-OSG: v98_SsIVM1m39rfXFvoyhABjff7luXxCOji38XSMKBiYIyhsTLVA.oqNOYFedKb HLLppFw_raDqMZjl4WG9YLZVvpnSaAWMgGVLdbDHr0no3UoJgA8Wv6f_77BbLSU6jUyEj2pVV6Ll BqOnVOCLiRdOfccUQTX976z9GFk0Zot2L2QtzlLkuy5hb3Pe9AzgrKnkDcpOMb7akwYtqTZoghU8 6WW3bVDQ.QXHLi_rIGl6mv.b3UW0iFiaN5_c4OLuq9Ct7iRk.eQjQfjyeRMZRYLxgNPdy5MWY5o1 qjLPznJ.cksDZrDJFe0bl0KY36bflBHjOGKZxJndrgEGg_cACADdV0rCb10YNnkrgwVJyPhJxAZH 7Bn0aiMBqw5HDwPjAlL8nDMNYo1OIubRKMbADKKqtwg8qmVxY8JAyNHuxE.Tj2eBh9iV2ZzcTYs9 SuxTA8Xf.JBSea6oRBU6zVGm8Y_Ayad0p5Grt6xP1Sn9dGS5bBAaxC6vQBXyTXJwa10mP9T.Y50L e9HGNiEOb5pxntnaJcHxJHNYfTNP11jHxo63vyqD5WTcE76bEICtCYi0L4Qt6kdh7B5y8P0bqjSb uH8c05BEPZhUuqCHW16Uqi3773_PoLEhE9ZH3CYEADri1EKRzh.T3Yef8WFxHN5dPOLXurfdG3YA UaJ1r8jj8d9cKHhEVuQ_2Q8NCH6SYZ30tUZCqJPAs3F4JXaNv1in_XPjHv5BVakoKZnOGH3V.xGf nEcsuWwssUBxZNFJrw9quzGfcqyC6tgx.JZtCEellaBPDmlITb19C2DIE4PHgU99fzM_PROInNq_ D8ivQGdw3QK_jHwhxDb5FkvRNM1UeW4MjdPrSpeqw519KXWNT2.Ezr1zC.na_ISx9DboLJBMyE03 0pGuhqy1JYpjn27nIc17ALd4NP3KhUAUmfr1fnP7Dv7j3Q38ccpJyEqPY7yl_sJVI1Csg_D9zZo3 Q44.EeMU5upB0pHzyREzxwg5DExtr3KwbahgTn2SHoGLz.jDaXq_wjkyE_eJk.dcvTb0RZvPOyeM akpfY_0VkMcEgRNeRgnjKCPGEh1Nz82NAJ7A4WbnzHxFXzDug.7lBKZH2z0QrUVDITSCkud1fuln wHDmhCExOzxw5I1Uve74Iev9wVbGz9JZQp1aSUZ1wRbDTcDSehQKSHej5nuHQxhJX6iy0hnI9H2V 6LJMxLWqIUx2C5qvM2JDBxLmorUdQ3spgrAcAsdGJEJbjETqkkpszuB8uCj64SQ131F5IG_FETqb r7Il.7N27.nEjUn1YrGcZzEuHaZhn_LeosFKqFgvOPvH6bDWotzFa3ai2bl4nhzYymIb8lFJomDR qBhCC.5nJqB3jjMlHvDupH4d7GBS8ekdZ60nfM6oXbr8D1_un.JPY8rXYtDgMzoTkd5slhkgKFUy 9JY_n1555fAT0acMkM_vz49wvvU4ofuyi52wIgXGkv80DrNHULjasB9A_SFWAkrY8yQfwdKGVe2J HKVsvxpuQLk3xGkSIkqGctPSLBuQcX3NMb.MlJoG6b.8zlM7MuKyzoszKOa1BcUgqO.qBPwWPu1U TzXJQxWlfpPVu66UIUQINv0se24vy6dqCEenm6eHYka8x5EPRSokFKwqU66AiFLfwXqkETBdoSKg JCNoujcq558VW1YC9pfs2ZZ0FkC.rBjxNFqWmXEH9Cvs9W1walGRfYUsJuSKfLdq2dckiv1_PtDu _h5cbaK3RYHDm4.greHprBH7r9OXpkRVGfWECmfdy6uVMFIAnKGtp.m.K8cqeTeHadzBfJKV9ogh __3ITJY.o_n9By8u8C8W3omiJMZKybFKNAKgPYiF6pG7Gy1VnDOS4soz4ihm0Re9uHqm8nJCKyKe 7fVkILpTaAMrhY3f4Wich_VtCKOjm5IWJo2rrNBwYo3IUHHIvkM7hnLqt232piU.GP8YQmWIiKp0 ILkjeRBWT5WlGfEB5PGKgi9zHzxc4c0KrGGo1uli1Tm.XqtVGUbmxzC3dPdbNhBJnnR4GFokB1OG bJxjJMC1myOFMZa46vdhS97gnAYMRQAGjhkhtpQQOq1TCw8Ey639J3zHjMD26ajEYWjw.uWVTVXr ue9vJlDZuC38K2EYDgRRr734OCtrY4sYY8ceRhz5z8EKS.JIcov0czeJ0zn3LWCD_ZGWcu5qp43P ssA6aDLB9HuVDPTaCwrKr_2ACRjOvTGSYGrSl7RX1sEsC2bC1eeJGfdVYaUu45B7QXXxAb6Q17Po qBNTeXLYFaoHtr2WNIWtvbYqKXGOYKbhXkYzj.vHapEW_OUdpnCNbqlo7YRl5LxJsaNY_s0hXfuz dtV_GiDDw4Ev7Jr6_QjAZopvnsrOhZXZIa5MABwjzVnQc.Wfq0brpFwpLj18RFPxIpdubW2YJP6x ypMlxeMxoOFE4oTH_K2F1HNPkRYIlfM0Z0ojErehm2IJBAx6CuNP6K3g2GKlljC3bcRYclYQ.zKS 1ibp5P64O4WP0RGcLbI1iXN0P9Zci2W4PB9MoXUQ7bf4Dm7LdN5TgdoMFi8QepDjHkt5q1rDjuHd HY2nzlPkUP9Bx.gLtxcjwLbDDP6vash7Xd167YKY5wswdNm9uGxXaZE2M3Nwv1NxpkJYAafkCqVY rKfm.n4WtpUPZGza.ce4ZbWpVJKMGxLiEjUTKHHiLCSV1T_Qr_3Co1vflM7B1Vjs4eu_Vc_0GdGC Ezl7GrXuMQkt3M5QpFalDvbr2kRgHaMOz5HcYzYhKPMRYGLb5pSD8oy3V4RRvnuy3p7j3hu00NeQ oOl_FNnyt1DvkbZMxUgSgaRfJjpMU8.vDihHFHn3zOKCbREZfS2M_IrBFM1LEoOVQwykTCtY9FH1 zqw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.gq1.yahoo.com with HTTP; Wed, 13 Jan 2021 06:53:05 +0000 Received: by smtp415.mail.gq1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 944e48b6a7e3d1d00e58d7d6467e880e; Wed, 13 Jan 2021 06:53:01 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.40.0.2.32\)) Subject: Re: panic: Too many early devmap mappings From: Mark Millard In-Reply-To: <04FEAC11-5603-4D4E-8651-43AB37A10B46@yahoo.com> Date: Tue, 12 Jan 2021 22:53:00 -0800 Cc: freebsd-arm , bob prohaska , Gordon Bergling , =?utf-8?Q?Klaus_K=C3=BCchemann?= Content-Transfer-Encoding: quoted-printable Message-Id: <3C95F8C0-73FA-4DEC-9F0A-6FFF9846E8A3@yahoo.com> References: <20210112233607.GA79348@www.zefox.net> <90C90797-A8A5-457C-AF07-800EA82F5F12@yahoo.com> <20210113002432.GA79600@www.zefox.net> <04FEAC11-5603-4D4E-8651-43AB37A10B46@yahoo.com> To: "mhorne@freebsd.org" , "rwatson@freebsd.org" , Ed Maste X-Mailer: Apple Mail (2.3654.40.0.2.32) X-Rspamd-Queue-Id: 4DFynb1B9Jz3ldq X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCPT_COUNT_SEVEN(0.00)[7]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[98.137.64.82:from]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; MIME_TRACE(0.00)[0:+]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; SPAMHAUS_ZRD(0.00)[98.137.64.82:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.82:from]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.82:from]; FREEMAIL_CC(0.00)[freebsd.org,www.zefox.net,googlemail.com]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-arm] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2021 06:53:08 -0000 On 2021-Jan-12, at 16:55, Mark Millard wrote: > [I have a git bisect result for the failure: bbfa199cbc16.] >=20 > On 2021-Jan-12, at 16:24, bob prohaska wrote: >=20 >> On Tue, Jan 12, 2021 at 03:59:44PM -0800, Mark Millard wrote: >>>=20 >>>=20 >>> On 2021-Jan-12, at 15:49, bob prohaska = wrote: >>>=20 >>>> An RPi3 running -current updated on Jan. 10 installed a new = world/kernel and=20 >>>> when rebooted promptly crashed with=20 >>>>=20 >>>> ---<>--- >>>> panic: Too many early devmap mappings >>>> cpuid =3D 0 >>>> time =3D 1 >>>> KDB: stack backtrace: >>>> (null)() at 0xffff00000011ad90 >>>> pc =3D 0xffff000000760f70 lr =3D 0xffff00000011ad90 >>>> sp =3D 0xffff0000011df330 fp =3D 0xffff0000011df530 >>>>=20 >>>> (null)() at 0xffff00000045c2d4 >>>> pc =3D 0xffff00000011ad90 lr =3D 0xffff00000045c2d4 >>>> sp =3D 0xffff0000011df540 fp =3D 0xffff0000011df5a0 >>>>=20 >>>> (null)() at 0xffff00000045c07c >>>> pc =3D 0xffff00000045c2d4 lr =3D 0xffff00000045c07c >>>> sp =3D 0xffff0000011df5b0 fp =3D 0xffff0000011df660 >>>>=20 >>>> (null)() at 0xffff0000007d8380 >>>> pc =3D 0xffff00000045c07c lr =3D 0xffff0000007d8380 >>>> sp =3D 0xffff0000011df670 fp =3D 0xffff0000011df670 >>>>=20 >>>> (null)() at 0xffff00000075dc98 >>>> pc =3D 0xffff0000007d8380 lr =3D 0xffff00000075dc98 >>>> sp =3D 0xffff0000011df680 fp =3D 0xffff0000011df6a0 >>>>=20 >>>> (null)() at 0xffff0000007710e4 >>>> pc =3D 0xffff00000075dc98 lr =3D 0xffff0000007710e4 >>>> sp =3D 0xffff0000011df6b0 fp =3D 0xffff0000011df6d0 >>>>=20 >>>> (null)() at 0xffff00000028850c >>>> pc =3D 0xffff0000007710e4 lr =3D 0xffff00000028850c >>>> sp =3D 0xffff0000011df6e0 fp =3D 0xffff0000011df7a0 >>>>=20 >>>> (null)() at 0xffff0000007c8788 >>>> pc =3D 0xffff00000028850c lr =3D 0xffff0000007c8788 >>>> sp =3D 0xffff0000011df7b0 fp =3D 0xffff0000011df830 >>>>=20 >>>> (null)() at 0xffff00000028a64c >>>> pc =3D 0xffff0000007c8788 lr =3D 0xffff00000028a64c >>>> sp =3D 0xffff0000011df840 fp =3D 0xffff0000011df850 >>>>=20 >>>> (null)() at 0xffff00000039b340 >>>> pc =3D 0xffff00000028a64c lr =3D 0xffff00000039b340 >>>> sp =3D 0xffff0000011df860 fp =3D 0xffff0000011df870 >>>>=20 >>>> (null)() at 0xffff0000004a6950 >>>> pc =3D 0xffff00000039b340 lr =3D 0xffff0000004a6950 >>>> sp =3D 0xffff0000011df880 fp =3D 0xffff0000011df8b0 >>>>=20 >>>> (null)() at 0xffff00000076d73c >>>> pc =3D 0xffff0000004a6950 lr =3D 0xffff00000076d73c >>>> sp =3D 0xffff0000011df8c0 fp =3D 0xffff0000011dfa00 >>>>=20 >>>> (null)() at 0xffff00000000089c >>>> pc =3D 0xffff00000076d73c lr =3D 0xffff00000000089c >>>> sp =3D 0xffff0000011dfa10 fp =3D 0x0000000000000000 >>>>=20 >>>> KDB: enter: panic >>>> [ thread pid 0 tid 0 ] >>>> Stopped at 0xffff0000004a6550 >>>> db> reboot >>>> cpu_reset failed >>>>=20 >>>> It had to be power-cycled to restart. It came back up readily with >>>> kernel.old, which reports main-c255664-g4d64c7243d26 compiled Jan = 9. >>>>=20 >>>> In particular, how does one recognize which revision fixes=20 >>>> this problem, assuming it's a bug and not operator error?=20 >>>> Presumably, it'll take at least several days to reach git. >>>=20 >>> Discovered last night on 8GiByte RPi4B's relative to this: >>> Booting without a monitor changes the memory use and avoids >>> the panic. WIth the 1920x1080 monitor it fails. (Only kernels >>> with INVARIANTS make the check that panics, but need not >>> mean that others are operating well, even if it is not >>> obvious in a specific context.) >>>=20 >>> Quoted from part of a message list item from last night: >>>=20 >>> QUOTE >>> Going back to my 19cca0b9613d based debug kernel build that >>> has the printf's reporting the values used in the test, but >>> with no monitor attached, it boots fine and reports: >>>=20 >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007ffffff000 size: = 1000 >>> pmap_mapdev early_boot: va: ffff007fffffe000 VM_MAX_KERNEL_ADDRESS: = ffff008000000000 L2_SIZE: 200000 >>>=20 >>> That compares to the previously reported failure figures from >>> having the monitor attached for that debug kernel: >>>=20 >>> pmap_mapdev early_boot: akva_devmap_vaddr: ffff007fff816000 size: = 1000 >>> pmap_mapdev early_boot: va: ffff007fff815000 VM_MAX_KERNEL_ADDRESS: = ffff008000000000 L2_SIZE: 200000 >>> panic: Too many early devmap mappings >>>=20 >>> where the code does: >>>=20 >>> KASSERT(va >=3D VM_MAX_KERNEL_ADDRESS - L2_SIZE, >>> ("Too many early devmap mappings")); >>>=20 >>>=20 >>> Looks like akva_devmap_vaddr gets smaller to make room above >>> for monitor related data and so va can end up being too small >>> by the criteria of this test. >>>=20 >>> I've no clue who would be appropriate for dealing with this. >>> END QUOTE >>>=20 >>> You may have provided a bound for a bisection >>>=20 >>=20 >> It looks as if unplugging the HDMI monitor (1920x1200) fixed the >> panic on the RPi3B+ as well.=20 >>=20 >> [the original subject line said "devmatch", which confused me hugely = 8-)]=20 >>=20 >=20 > A git bisect sequence on a 8 GiBYte RPi4B with a monitor plugged > in (to make it use more high kernel RAM such that the KASSERT > indicated above fails) resulted in: >=20 > # git bisect good > bbfa199cbc1698631a0e932848e62dd76559d4d7 is the first bad commit > commit bbfa199cbc1698631a0e932848e62dd76559d4d7 > Author: mhorne > Date: Wed Dec 9 16:38:42 2020 -0400 >=20 > arm64: gdb(4) machine-dependent bits >=20 > Everything required for remote kernel debugging over a serial > connection. For FDT-based systems, a debug port can be specified by > setting hw.fdt.dbgport to the desired device tree node in = loader.conf. > For example, hw.fdt.dbgport=3D"uart1", or > hw.fdt.dbgport=3D"serial@ff1a0000". >=20 > Looks good: emaste > Tested by: rwatson > MFC after: 2 weeks > Sponsored by: The FreeBSD Foundation > Differential Revision: https://reviews.freebsd.org/D27727 >=20 > sys/arm64/arm64/gdb_machdep.c | 112 = ++++++++++++++++++++++++++++++++++++++++ > sys/arm64/conf/GENERIC | 2 +- > sys/arm64/include/gdb_machdep.h | 81 +++++++++++++++++++++++++++++ > sys/conf/files.arm64 | 1 + > 4 files changed, 195 insertions(+), 1 deletion(-) > create mode 100644 sys/arm64/arm64/gdb_machdep.c > create mode 100644 sys/arm64/include/gdb_machdep.h >=20 I forgot to list the bugzilla for this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D252541 I have added the notes, including: QUOTE Turns out that this "too much high kernel memory in use" issue happens = for a combination of 2 things being true at the same time: A) Monitor attached (sufficiently large pixel count?) B) GDB enabled, per bbfa199cbc16 . END QUOTE =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)