From nobody Sun Jun 25 00:25:35 2023 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QpWwd0W26z4fwQp for ; Sun, 25 Jun 2023 00:25:53 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic309-22.consmr.mail.gq1.yahoo.com (sonic309-22.consmr.mail.gq1.yahoo.com [98.137.65.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4QpWwc50Cpz4YC9 for ; Sun, 25 Jun 2023 00:25:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1687652750; bh=TPFdGh8HRqUf+3b9o4Wkv5WDL6+QPc7vTdIVPu6SG0k=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=RlzuB6JtKa1S8yDMmirxEcu4wZ6XdUkq17sY4qNYAc+wmQt9azqbJ9JZnY7IemTFPG//+CNnYuKOsrqagjW4s1/Q1iMRP1xeYNeAxfmioDl6a9Kwbj0VasRqvKFIMeJYP6sZQwOCqSXAzhfgDm7cv92dRbnPKEHaRvUGkLGEUA+9cHaEkahZnP/2Dm+5sA3jnAEpXdpAOakhtUzyCBDQr3etTtVnpQnF4ibTiZOH6ATSTdafcBUoRtkjkT4m7bG29LpYCQQ1+hkeZpwcCniM5UxyMn7lBonBit25682pdyhj25QtFg4rOEBwRG/Kw6YRCZVUfmxM/wYeYkOaz/SeNw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1687652750; bh=j4oevzRlGsuLdJULt66EhMyXgcyHPvTxDqf/kT8cKd+=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=QubJj/iZn01ZYigKBziEA5am8h3P94MrHIIGD9cZH4ooCeDhYnMBDK2K/e9xcI8BovRBotbr8X9VSCiAQI5tNn1Wb+bWEB52x7+YwVYcvpFMV5RB8uoKrQVktJPsIKFNbJeSN+3VZAj+Xe8YHETCS/CQ7eCK7G80ZenO3CREFDuIPf3WGmy6UE9N91+uga6e/9Xw7pK/mrfs2MZyUXRVEfVXUdZFhF1Mwx7E+9m4jCNJ7B0XLRxlsXe20IVYhQH7LCQ/czqSqJkrNAkuy/anA8V37b+P0clzu7KceWiLQNsy9tgAQ5Iy+ypG2g0pqzONLmhKHpv0wg68Qn/JWARjdQ== X-YMail-OSG: FKwoVRkVM1l.QSauy4fOrWnokZ2yG9vOqU4vfLbJGNwMA98FxXqKuvRg8InhvkK cwdMsutNywPHCCR822EIb1J1lsy57SSc9Au84_Lsl5Op8oB4PAZz1ymzq3qpxR6a84hTgwSVj3td _fiNtoe0_taqe1h_NIsOrokRdg55fy6ZZqGfWaduWUaOx4O0yBR.HhZ23uVrYgBLaWg0GQSeQwwp tHO_cJ_wOdzD35KkWfAO5B4b2r1egwgoG5HoGTcB_J_QwKy_hPWwE4_BmDwkdxDTIYvV2kag5.GX 79XTpjT5tGurbO8AM7oetJlvV8cnlMf6_38yqjamCZhJiFOrxdonFE4f8rQM4fsURVUDkUYNqCH4 mB14vpcUXDu.BJKdIVropWoPSV9O8aaITi7ThZYLUNfWkHGZKsBsAQCVqk9SvtRR3BdXu2L.HSPC JtXZDjCqBzGV1U.TwTsoTMXiYN8_fqrV_1ZEky1mihjWzyz7lT8BLvQqXIX26qfWFxfOwSDlqKPN qUgObBXyhnTzxWTQbpYxwGsNTrrjcrqz4Hlx3Gc6Sw8fz3foQiKD0jH32VbYWHRnw3QAaXn.QAvh ae6mFJ9EHEk8r8HvKt4tadJ6EU4nZGMyEYRSagLltQ2pmtnA4esk_eBrXZ_m5YT89KxtBTZRAVKJ z0bj_om8QeT5_KfxNYjbdvPUV8YOHIEVt_3ltLOZurpSs_yvmoR_dCCYuJjowMLtjI7gYHwtWGQH 4PZhYF41bvYr22rVwHNRwqoJTM9f9pz2CskftMl82YTfIqktnJBeOEWvBCf.HvgFBln.toLgogy2 JqeKJ6dDHDU7vZ91f1b8Ua3UoWpZMzwD.fUEuKwbSJ59mZTyGYuu2XyTvYKFNSCsNkKjL9aKXSts YagIIRvJfVuzey3ax9oI_MZ3bcX4igVO_Ammyb9eeGHv9JAccBf8F7RqwPUxYB87XsJZ02XZCGLE N0MfCMN5ghASoUqYqOd26nnN7cjB240GgESFojax57S45RchXDTgTiw4AP9NsbTJ2PyEs35MrNgy 57PeFJjUNNS9dUeIRCVfW.sBVg5a7KNplQkMgw1Cj.QoCib8PKJQab7PqxM0vZse21vc.uV5cjtY 4WoKc1eUZt9PMD3yRzrfsns7Ht7CC7R6tPfZrpsKzbduWRKFKpVh55K8BGPkWPEMBobxaxLVS3W4 HcT1AYDIE.p7RFND8K2drfTkYYXFYrtNzkWKL7xH2VjrLekfHRd7F73Owt4FJXjxaUfPLDzVWcm6 sdhyX.N_fRlFgUQA3a7ieEM9tIJX5oiKIM_6u9lWe4atScKBgsnCLBTFbDpvPZ8TpxWv_dWOAVd4 ZwJN3rLCRmr5ifgdIW82Qpa.mTkhhtu7rrFaSis8nyd1dz5jDXixwbLh5Gzuv0KGz7Hfp_sx5eLi w5iTkqEj29oB6dUHjwusa_Eq7sGCTSalFpm.cbzeQ0DdCxRqEGQy4P5DOM8bhu9Y2oJla1SZv2.I sWEuxqWvumS.Ex_v4BngPofQJXjmxcoEsFZpKL5.JubTxSYG8wK7zHB65SZPVzC1h.bgClsUES0. yZAvT9rrcF4Hpjs7PUj8HZAcdMpWpE8oz6JIUU8fzym8k4JMH__bfORs_zRX.i8EleavZdz1u3QM EjNJybBaj2rHM.jMTNCl.Uayf_W4irmXo1pMFZG5j5pztjH49kh4dK7Qrje10i8GV1umgNshA0Y6 ul0JdaUgohD6uAK2vz8aWkvBpQyUEtHHhYp18jxTip5LLPmVLHZ2ZxSH344urVV54U5DyoW0IGZc Wt.ZptZkMQ9Bj5agfbcLbbIeC8M_6HuqAZo9JT.IfYAyYcyz2o.FXslEkQyL2Y.Bw6ku2ERK93u5 .FECOJSmgP_dX.AdWj41H6Qb9mkmzzFHcWlGgrhLBE2Uranv733dEGHGzoyPtZI.9Dkt8wcgr8Yq dzy6x97vTF.yHm0GIts1Tjto2rBouBgFdD2yg5OwleGQzbVG2As4.3JSpdxme8rhvMSfeaedtQtb Df2yHYsqOD3NFp1KMc1YumP.4oEAOh.9bqQcBwf963YCXZYaDaebISu_NMOeEAR187bITfgmYds7 o3UKvTya2O.ev6pGRUQb2aLNa.pBIdXSFz1M20nXaWR5qwBn_u0rhGptscNwGcjzoshkePw9avZe pqOcm9hgYZ8ZgmlkWC1LAjoN4mltVoWpBaV79TRh5qini4r_eyvNCAojD__Y8czyH4tc7cShSqZw ezdxsDrmWpkox_xe2eFu0.pceLiGAsFg.Q2BIFZroP9flyiFA3E1wwGM_.vfsAhpAArSLltUD6_3 OFpuRusg- X-Sonic-MF: X-Sonic-ID: dd9c2e7a-db3e-4472-a555-ba1cf66d471f Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.gq1.yahoo.com with HTTP; Sun, 25 Jun 2023 00:25:50 +0000 Received: by hermes--production-bf1-54475bbfff-lfx8r (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 075c9bb24c642fe578221f15a2e45d58; Sun, 25 Jun 2023 00:25:47 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.600.7\)) Subject: Re: aarch64 main-n263493-4e8d558c9d1c-dirty (so: 2023-Jun-10) Kyuafile run: "Fatal data abort" crash during vnet_register_sysinit From: Mark Millard In-Reply-To: Date: Sat, 24 Jun 2023 17:25:35 -0700 Cc: Current FreeBSD , freebsd-arm Content-Transfer-Encoding: quoted-printable Message-Id: <4A380699-7C9E-4E2E-8DCD-F9ECC2112667@yahoo.com> References: <3FD359F8-CFCC-400F-B6DE-B635B747DE7F.ref@yahoo.com> <3FD359F8-CFCC-400F-B6DE-B635B747DE7F@yahoo.com> To: John F Carr X-Mailer: Apple Mail (2.3731.600.7) X-Rspamd-Queue-Id: 4QpWwc50Cpz4YC9 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Jun 24, 2023, at 14:26, John F Carr wrote: >=20 >> On Jun 24, 2023, at 13:00, Mark Millard wrote: >>=20 >> The running system build is a non-debug build (but >> with symbols not stripped). >>=20 >> The HoneyComb's console log shows: >>=20 >> . . . >> GEOM_STRIPE: Device stripe.IMfBZr destroyed. >> GEOM_NOP: Device md0.nop created. >> g_vfs_done():md0.nop[READ(offset=3D5885952, length=3D8192)]error =3D = 5 >> GEOM_NOP: Device md0.nop removed. >> GEOM_NOP: Device md0.nop created. >> g_vfs_done():md0.nop[READ(offset=3D5935104, length=3D4096)]error =3D = 5 >> g_vfs_done():md0.nop[READ(offset=3D5935104, length=3D4096)]error =3D = 5 >> GEOM_NOP: Device md0.nop removed. >> GEOM_NOP: Device md0.nop created. >> GEOM_NOP: Device md0.nop removed. >> Fatal data abort: >> x0: ffffa02506e64400 >> x1: ffff0001ea401880 (g_raid3_post_sync + 3a145f8) >> x2: 4b >> x3: a343932b0b22fb30 >> x4: 0 >> x5: 3310b0d062d0e1d >> x6: 1d0e2d060d0b3103 >> x7: 0 >> x8: ea325df8 >> x9: ffff0001eec946d0 ($d.6 + 0) >> x10: ffff0001ea401880 (g_raid3_post_sync + 3a145f8) >> x11: 0 >> x12: 0 >> x13: ffff000000cd8960 (lock_class_mtx_sleep + 0) >> x14: 0 >> x15: ffffa02506e64405 >> x16: ffff0001eec94860 (_DYNAMIC + 160) >> x17: ffff00000063a450 (ifc_attach_cloner + 0) >> x18: ffff0001eb290400 (g_raid3_post_sync + 48a3178) >> x19: ffff0001eec94600 (vnet_epair_init_vnet_init + 0) >> x20: ffff000000fa5b68 (vnet_sysinit_sxlock + 18) >> x21: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0) >> x22: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0) >> x23: ffffa0000042e500 >> x24: ffffa0000042e500 >> x25: ffff000000ce0788 (linker_lookup_set_desc + 0) >> x26: ffffa0203cdef780 >> x27: ffff0001eec94698 (__set_sysinit_set_sym_if_epairmodule_sys_init = + 0) >> x28: ffff000000d8e000 (sdt_vfs_vop_vop_spare4_return + 0) >> x29: ffff0001eb290430 (g_raid3_post_sync + 48a31a8) >> sp: ffff0001eb290400 >> lr: ffff0001eec82a4c ($x.1 + 3c) >> elr: ffff0001eec82a60 ($x.1 + 50) >> spsr: 60000045 >> far: ffff0002d8fba4c8 >> esr: 96000046 >> panic: vm_fault failed: ffff0001eec82a60 error 1 >> cpuid =3D 14 >> time =3D 1687625470 >> KDB: stack backtrace: >> db_trace_self() at db_trace_self >> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >> vpanic() at vpanic+0x13c >> panic() at panic+0x44 >> data_abort() at data_abort+0x2fc >> handle_el1h_sync() at handle_el1h_sync+0x14 >> --- exception, esr 0x96000046 >> $x.1() at $x.1+0x50 >> vnet_register_sysinit() at vnet_register_sysinit+0x114 >> linker_load_module() at linker_load_module+0xae4 >> kern_kldload() at kern_kldload+0xfc >> sys_kldload() at sys_kldload+0x60 >> do_el0_sync() at do_el0_sync+0x608 >> handle_el0_sync() at handle_el0_sync+0x44 >> --- exception, esr 0x56000000 >> KDB: enter: panic >> [ thread pid 70419 tid 101003 ] >> Stopped at kdb_enter+0x44: str xzr, [x19, #3200] >> db>=20 >=20 > The failure appears to be initializing module if_epair. Yep: trying: # kldload if_epair.ko was enough to cause the crash. (Just a HoneyComb context at that point.) I tried media dd'd from the recent main snapshot, booting the same system. No crash. I moved my build boot media to some other systems and tested them: crashes. I tried my boot media built optimized for Cortex-A53 or Cortex-X1C/Cortex-A78C instead of Cortex-A72: no crashes. (But only one system can use the X1C/A78C code in that build.) So variation testing only gets the crashes for my builds that are code-optimized for Cortex-A72's. The same source tree vintage built for cortex-53 or Cortex-X1C/Cortex-A78C optimization does not get the crashes. But I also demonstrated an optmized for Cortex-A72 build from 2023-Mar that gets the crash. The last time I ran into one of these "crashes tied to cortex-a72 code optimization" examples it turned out to be some missing memory-model management code in FreeBSD's USB code. But being lucky enough to help identify a FreeBSD source code problem again seems not that likely. It could easily be a code generation error by clang for all I know. So, unless at some point I produce fairly solid evidence that the code actually running is messed up by FreeBSD source code, this should likely be treated as "blame the operator" and should likely be largely ignored as things are. (Just My Problem, as I want the Cortex-A72 optimized builds.) Sorry for the noise. > I see no recent changes in that module that would be likely to break = initialization. >=20 > a9bfd080d09a if_epair: do not transmit packets that exceed the = interface MTU > 4d846d260e2b spdx: The BSD-2-Clause-FreeBSD identifier is obsolete, = drop -FreeBSD > a6b55ee6be15 net: replace IFF_KNOWSEPOCH with IFF_NEEDSEPOCH > c69ae8419734 if_epair: also remove vlan metadata from mbufs > 29c9b1673305 epair: Remove unneeded includes and sort some of the rest My kyua run examples included a Cortex-A72 optimized system build from last 2023-Mar. It also crashes. It looks like my last kyua runs were back in 2022-Jan or so, associated with some ASAN and UBSAN experiments --and so would have been on amd64, not aarch64. Otherwise any aarch64 ones would be even older. I've no useful narrowing of the potential time frame for the problem starting. =3D=3D=3D Mark Millard marklmi at yahoo.com