From nobody Mon Nov 17 09:15:13 2025 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4d92CN6KdQz6GvcD for ; Mon, 17 Nov 2025 09:15:28 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-8.consmr.mail.gq1.yahoo.com (sonic315-8.consmr.mail.gq1.yahoo.com [98.137.65.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4d92CM1p8Tz3lrt for ; Mon, 17 Nov 2025 09:15:27 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=NKiXDjCr; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.32 as permitted sender) smtp.mailfrom=marklmi@yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1763370925; bh=fUXWt3VXnuDVo7J/NwtiWE3t5+EmNbuptEyICtk2t4s=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=NKiXDjCrK3F5Ea/OMALnpctB5O+51DKIOxZzNgJA/oer8oXzTnhFCLhB0utK2AjDvl+n1R6Mw3B8chfQalrFjHUbaoO38q3eFpluYQFYfaEE+u8HEH5X9ir6ceyLli1tke4/+RhaZi07BAsaHntbYyPiGkZIwslEHhhDsTGpAr2qsTmn0+ABeClQi2Y1oiaxtVkJNs+UZZB+Fg65heUerENZePQuARZN77sn5HwTNnDD6dyaB/NBVdRiWrwNLe2xUFBEP7PO/oggnjq8GjNT/78Wmqhv1rv97wBDH9v4u6Zxhox2PnJJjYEsnr2yX2AzMdae25cXpqP5dDcN3WdeLw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1763370925; bh=54C25OWGAkbKSknwyARZ7P/n3yCPtypmi5YsgV/R/4P=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=Kgw6fdNjBGOhMmNhE63AtrgOBT213Zmr7DNkUW8jzhoyAdHqv0VuEd/WD8wSEhNdywGHO7fH0L27EBed9LFiFMGpZ50kZzz9veTrwdA7UxEVTtjxf8nlm8JvDsqTd/4Dw7fWyTHLFP3nWRKpa2SnKZ2lCGwXflaEkPLSnsyOvuOoj/oPw5v2sDJyr/mZcOuObR6Opj6r0u/tqshj6FcQOhxXMxtKvEyW9ykFOakKd4DYP64L2Ci1Dmtr0FwHjyEJZ1pllx0K9Ln1Pr8zSLQl/eTz5Sn5Sj9ojBBQgB2G0lJlBYM4tPlNJ6oNKnV7vKSfMqvwweES+fWrisHMfSpjaw== X-YMail-OSG: cjNYDjYVM1nXKuHEOt99QUiPYq7v.dTsPDBzueJhSwlgbxO26xScvCp0igBeL0i zJV8HxgP81jO5BtnOpIQ2c2fiufM3Cy3g7UGqkPxWHBvRdeS9c7rIW3z4PF642kUu8RlK5qvV6UE c_998py.AIofbNmUE.0TivRdWWZIznewX07IJshfsaT3xZ2DiDcYwllQdW1WejHmZ.HWM8peZvnl OCMG0iegeeFelhrMhEeJHpDwuQo2KVm1gYHSYLV1jVuje2zEwGSfIteUw1l6Gk0XDtNUMbJ_7AQ5 1_5z8pyGDKA433b8b7SUFxbrFXehdg6qj59YVAGqr3wwRM7Z_sEjzVt.YxAAdeIdt5P5d3mHr4r9 YrDCs3zcauq43bdn0JdxcMW6.9U8B6TC9Gfy9e55wR_vWGsMY0x0QjFlcOlnbV3Hzubk69lVk2VS HMeFsMZiQWcPHbHzA6MASESwBC4Zmj.wYUuee1M1LpLKG5CwgTDZ9sNWHYK2ea8xd7YaVkmwa7x0 fdDu.6g5cD3ij1HXKaNdepb248MbkfAw2m0AMtew8NU0vp2xpYI_a0XDPp0ndFAUs5jgfGiYuIty Pi.VRBHy9QNSPSaUfkhOWIpJUz0SqzXB2oqeph0O25FfsAUoss5w6PltJci6C_5By0olsRQp4gEp od7G88NYaQOEIDDpAKaoAGd1sIQTynRtmAqbNdnQu5C2keprE1uowpRS7kdzScD7Kay8ofbbptva mdk9bSisbfAUEDoYRGV4hBadAG6gIVM5OJJv1dAwcunphNiou324DLqTAv.6zA3Tn0ozjSNM835e e8gqbdX1wNFr8PTDMpfwJoURKMiCqyL85C8rNrJ6jDftSsRiWQOwJb9n4TSNueiEgTV112AGSyAM nK5llCCl.7Zf5oIBzIt3D76aWqbBb83LdDL7zjQovzcBRBxM8VDdM5QX0lSxEWZcSfd2wxziR4ff AQ2W0UxKiTNLox2LZbSdon6IYVGFNZ4pbLLtaa3nx40buxq7QUXhNhQO.Aa2DfkwbwWMuWKxpIEe z3E.HUjNKmj7KhznM.pLdDnClhm78ZTiOyy_224dexCrM5lDekKHAKRKYOl3KEZ0IrDc74HisOqx QuqCyqHdPlZ.6WTVDL.Ol93txnNHH6fj6Cll0xvl_oiaLCLz0ekvfvXtMeUJ4Hd8LuJiNxSvXLVp Ty7MVKtQS8aWhzrauSW2vSUPoXpk.G9GGc0fX0bcYnw17zEqk3YFO2vQRvYBdiBkcikHBEYCWDWX FBWFl9_.D0l6KF4tz6GbXDwZtAnZGXhLo_1rmr_UB5uMtK8Y4VswICXzaKkPUKBXqoInVafhAFFB .gy3qU4vL5huYNdxsE5mRPABPlw0Z8Yd_u6rCs.ikHwtmoXD4X38zKDWPZQiCnfXI4LZJNqGJYHd fllgMpl2RzuJ0bOZvcS5g.O_QqIr_TfKFSqvQdCDbuclxQJe27TZzyiJOUcDVE6VWEUH3ibQBriz QWzGaqW7_V4KDlfzI1lZS9T4QdJfDKlqYUzSVYUViROYthDt9c3oisEPF6rWJAjhQjdV7oyqkmFv VrUcDoyHwDDmW26OiRikkWl2HHRuYa.8GFN8lX41gGFZQoFvQb_qAmg0kubKIpByWcKoqiQcVmsc vhwwVWAUz_M.52mNo7TRHXTx0fuPk9EOOde.MzEntg2p8hNGD9Szw2MIZUYiG7tAyxYgMDtkYTuQ jGGYBG1M9Nt2rkXxW4XRn2zMVEqdg_5LOjKZPQkoidfAXvM7ZI6mbdftLN66zyfFO1pg8OyxsJDS J5Zhfva6ZIMQOPO_eiwjMekJ9hLrZU2XJmHnaXBKXtcV7TOad2aelL9vkWxjhEScWsK.QdKm8fzh Slb8Y.CnYD_RPTeJ45fJCHEWeOpnjncm.01.7AXYZ7GXysMsTgZ_31eXb8aQeZPTgLs1DNrr.Ovy G6GzDqa8J481dc81ppqVClOqyCF5mtEJWAdVQvEYlQuJbnyQogfoCNFSyNM0Ej5cjNbGtenELmsu U41q8Dp_GlwMwRX7LcgtRd69KdvmcYXj3tVepz1Q1Yp6gjvzTBKNIJPYycTK82Bbb5lqTqrX7did QwrPfL7w5H35iYthszQ4FIZviXr7zFcSeSVfPV6ddybokHYeqDMIKYXkWxryEe_GZXBP4qU.MY1e QUCPWW_nqoctWYc5hStuL4I9nu7Etqs00kC5jF5lZKQ2VJN7UzVPXSsh9vxDu9pkt8c_ksu6xnF3 kzJI5WAewX0bJk0yqp9suR_YprwCmuoILBAnBi.qzhUKI9CiC8Tyf83bbcu4_VPmL6gH3_JdangF N9KYcY81Hi2ew7dloTw-- X-Sonic-MF: X-Sonic-ID: 0251c7dc-41c8-4f0a-b0fa-59220f13ad39 Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Mon, 17 Nov 2025 09:15:25 +0000 Received: by hermes--production-gq1-76c986f798-8dnv2 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 5290d0ea80a78a18d9330102fdb9be91; Mon, 17 Nov 2025 09:15:23 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3826.700.81\)) Subject: Re: Still seeing Failed assertion: "p[i] == 0" on armv7 buildworld From: Mark Millard In-Reply-To: <0157ECE7-759F-4C00-9656-CB2ECA65E19E@yahoo.com> Date: Mon, 17 Nov 2025 01:15:13 -0800 Cc: Warner Losh , bob prohaska , "Herbert J. Skuhra" , "freebsd-arm@freebsd.org" , FreeBSD Current Content-Transfer-Encoding: quoted-printable Message-Id: <6F882958-E901-489B-A758-BCCFE5D01FBA@yahoo.com> References: <4957be52-e57f-4f5f-9626-d0f706480fe1@FreeBSD.org> <87ldk9f4tt.wl-herbert@gojira.at> <0a93ab5f-3fdf-45a0-8b32-2df5ef4ad60a@FreeBSD.org> <0157ECE7-759F-4C00-9656-CB2ECA65E19E@yahoo.com> To: "mmel@freebsd.org" X-Mailer: Apple Mail (2.3826.700.81) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.53 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.53)[-0.527]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FREEMAIL_FROM(0.00)[yahoo.com]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; TO_DN_SOME(0.00)[]; MIME_TRACE(0.00)[0:+]; FROM_HAS_DN(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.32:from]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MLMMJ_DEST(0.00)[freebsd-current@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.32:from]; RCPT_COUNT_FIVE(0.00)[6] X-Rspamd-Queue-Id: 4d92CM1p8Tz3lrt On Nov 17, 2025, at 00:24, Mark Millard wrote: > On Nov 16, 2025, at 13:11, Michal Meloun wrote: >=20 >> On 16.11.2025 18:51, Warner Losh wrote: >>> Maybe try main with the following patch. Adrian noticed the TLS = mismatch. I don't think it will matter, but TLS thread model stuff = always gives me a big headache. If the following fails to apply, just = copy the JEMALLOC_TLS_MODEL line from i386 to arm. The default changed = elsewhere, but this wasn't updated here. >>> Warner >>=20 >> Unfortunately, that doesn't help. I'm out of ideas on how to debug = this, all of my attempts have failed. >>=20 >> The problem only occurs when Clang compiles a larger project and is = intermediate. Attempt to compile the clang generated reproducer is = always successful. >> It's clear that the parallelism introduced by make plays a = significant role. But the system never reached an OOM condition before = failure. >>=20 >> I would be grateful for any help and ideas on what to do next. >> Michal >=20 > [Note: The context is an official pkgbase distribution context > and so the /usr/src/ is not tied to git. /usr/src-investigation/ > is a copy of /usr/src/ that was then modified. Also, this is > via a armv7 chroot on the aarch64 Windows Dev Kit 2023, not > via armv7-only hardware.] >=20 > The crude hack reported later below has shown the first failure > indicated as happening during base_alloc_edata by reporting: >=20 > p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x11u >=20 > as the failure message. >=20 >=20 > # diff -u /usr/src/contrib/jemalloc/include/jemalloc/internal/ehooks.h = /usr/src-investigation/contrib/jemalloc/include/jemalloc/ > --- /usr/src/contrib/jemalloc/include/jemalloc/internal/ehooks.h = 2025-11-12 02:24:28.000000000 -0800 > +++ = /usr/src-investigation/contrib/jemalloc/include/jemalloc/internal/ehooks.h= 2025-11-16 23:47:10.965711000 -0800 > @@ -1,6 +1,7 @@ > #ifndef JEMALLOC_INTERNAL_EHOOKS_H > #define JEMALLOC_INTERNAL_EHOOKS_H >=20 > +#include > #include "jemalloc/internal/atomic.h" > #include "jemalloc/internal/extent_mmap.h" >=20 > @@ -158,6 +159,7 @@ > * This isn't really ehooks-specific (i.e. anyone can check for zeroed = memory). > * But incorrect zero information indicates an ehook bug. > */ > +__attribute__ ((visibility ("internal"))) extern volatile = sig_atomic_t which_base_extent_context; // HACK FOR DEBUGGING USE > static inline void > ehooks_debug_zero_check(void *addr, size_t size) { > assert(((uintptr_t)addr & PAGE_MASK) =3D=3D 0); > @@ -167,7 +169,45 @@ > /* Check the whole first page. */ > size_t *p =3D (size_t *)addr; > for (size_t i =3D 0; i < PAGE / sizeof(size_t); i++) { > - assert(p[i] =3D=3D 0); > +switch (which_base_extent_context) > +{ > +case 0x10u: // base_alloc > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x10u); > + which_base_extent_context=3D 0x0u; > + break; > +case 0x11u: // base_alloc_edata > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x11u); > + which_base_extent_context=3D 0x0u; > + break; > +case 0x12u: // base_new > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x12u); > + which_base_extent_context=3D 0x0u; > + break; > +case 0x13u: // base_boot > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x13u); > + which_base_extent_context=3D 0x0u; > + break; > +case 0x20u: // extent_commit_wrapper > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x20u); > + which_base_extent_context=3D 0x0u; > + break; > +case 0x21u: // extent_commit_zero > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x21u); > + which_base_extent_context=3D 0x0u; > + break; > +case 0x22u: // ecache_alloc_grow > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x22u); > + which_base_extent_context=3D 0x0u; > + break; > +case 0x00u: // None known > + assert(p[i] =3D=3D 0 && which_base_extent_context =3D=3D 0x00u); > + which_base_extent_context=3D 0x0u; > + break; > +default: // Some other context > + assert(p[i] =3D=3D 0 && which_base_extent_context !=3D 0x00u); > + which_base_extent_context=3D 0x0u; > +} > + //assert(p[i] =3D=3D 0); > } > /* > * And 4 spots within. There's a tradeoff here; the larger >=20 >=20 > # diff -u /usr/src/contrib/jemalloc/src/base.c = /usr/src-investigation/contrib/jemalloc/src/base.c > --- /usr/src/contrib/jemalloc/src/base.c 2025-11-12 02:24:28.000000000 = -0800 > +++ /usr/src-investigation/contrib/jemalloc/src/base.c 2025-11-16 = 23:50:14.396483000 -0800 > @@ -1,3 +1,4 @@ > +#include > #include "jemalloc/internal/jemalloc_preamble.h" > #include "jemalloc/internal/jemalloc_internal_includes.h" >=20 > @@ -340,12 +341,15 @@ > b0get(void) { > return b0; > } > + > +__attribute__ ((visibility ("internal"))) volatile sig_atomic_t = which_base_extent_context=3D0x0u; // HACK FOR DEBUGGING USE >=20 > base_t * > base_new(tsdn_t *tsdn, unsigned ind, const extent_hooks_t = *extent_hooks, > bool metadata_use_hooks) { > pszind_t pind_last =3D 0; > size_t extent_sn_next =3D 0; > +which_base_extent_context=3D 0x12u; >=20 > /* > * The base will contain the ehooks eventually, but it itself is > @@ -476,12 +480,14 @@ > */ > void * > base_alloc(tsdn_t *tsdn, base_t *base, size_t size, size_t alignment) = { > +which_base_extent_context=3D 0x10u; > return base_alloc_impl(tsdn, base, size, alignment, NULL); > } >=20 > edata_t * > base_alloc_edata(tsdn_t *tsdn, base_t *base) { > size_t esn; > +which_base_extent_context=3D 0x11u; > edata_t *edata =3D base_alloc_impl(tsdn, base, sizeof(edata_t), > EDATA_ALIGNMENT, &esn); > if (edata =3D=3D NULL) { > @@ -523,6 +529,7 @@ >=20 > bool > base_boot(tsdn_t *tsdn) { > +which_base_extent_context=3D 0x13u; > b0 =3D base_new(tsdn, 0, (extent_hooks_t = *)&ehooks_default_extent_hooks, > /* metadata_use_hooks */ true); > return (b0 =3D=3D NULL); >=20 >=20 > # diff -u /usr/src/contrib/jemalloc/src/extent.c = /usr/src-investigation/contrib/jemalloc/src/extent.c > --- /usr/src/contrib/jemalloc/src/extent.c 2025-11-12 = 02:24:28.000000000 -0800 > +++ /usr/src-investigation/contrib/jemalloc/src/extent.c 2025-11-16 = 23:49:55.820658000 -0800 > @@ -1,3 +1,4 @@ > +#include > #include "jemalloc/internal/jemalloc_preamble.h" > #include "jemalloc/internal/jemalloc_internal_includes.h" >=20 > @@ -90,11 +91,14 @@ > assert(edata =3D=3D NULL || edata_guarded_get(edata) =3D=3D guarded); > return edata; > } > + > +__attribute__ ((visibility ("internal"))) extern volatile = sig_atomic_t which_base_extent_context; // HACK FOR DEBUGGING USE >=20 > edata_t * > ecache_alloc_grow(tsdn_t *tsdn, pac_t *pac, ehooks_t *ehooks, ecache_t = *ecache, > edata_t *expand_edata, size_t size, size_t alignment, bool zero, > bool guarded) { > +which_base_extent_context=3D 0x22u; > assert(size !=3D 0); > assert(alignment !=3D 0); > witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn), > @@ -1114,6 +1118,7 @@ > bool > extent_commit_wrapper(tsdn_t *tsdn, ehooks_t *ehooks, edata_t *edata, > size_t offset, size_t length) { > +which_base_extent_context=3D 0x20u; > return extent_commit_impl(tsdn, ehooks, edata, offset, length, > /* growing_retained */ false); > } > @@ -1297,6 +1302,7 @@ > bool > extent_commit_zero(tsdn_t *tsdn, ehooks_t *ehooks, edata_t *edata, > bool commit, bool zero, bool growing_retained) { > +which_base_extent_context=3D 0x21u; > witness_assert_depth_to_rank(tsdn_witness_tsdp_get(tsdn), > WITNESS_RANK_CORE, growing_retained ? 1 : 0); Well, with this hack, the behavior looks to have changed to always fail leading-to an initial: *** [libzpool.so.2.full] Error code 1 The hack may disturb things too much and may not be sufficiently close to valid code for the context. Using -j1 got a first failure message: Failed assertion: p[i] =3D=3D 0 && which_base_extent_context =3D=3D = 0x22u That would be during ecache_alloc_grow. Still: *** [libzpool.so.2.full] Error code 1 =3D=3D=3D Mark Millard marklmi at yahoo.com