From nobody Mon Feb 7 23:16:31 2022 X-Original-To: freebsd-stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 3C99E19A7C11 for ; Mon, 7 Feb 2022 23:16:44 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic311-25.consmr.mail.gq1.yahoo.com (sonic311-25.consmr.mail.gq1.yahoo.com [98.137.65.206]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Jt28V4GNDz4ldF for ; Mon, 7 Feb 2022 23:16:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1644275795; bh=mhPQvh0pdrODV9FSWL/2GFN78vLg1i9eNw105odoD6M=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=qxZKYm1F0E3Loyf6udyulAh0aylph9eyAaRRaiC3sJVjaOQL8i0W3AWlX+EU1bXxySup04hV75kKueFpz0bIe2/fiEd195LAnQvjiDzLlCpPCb3EjLiCkkAD9yxjvXLXyjvmZCG+B/k1ISbI8tNQ1FAgRNF6OQPBedOBDejUqALOIbbjT3MweBSijWk1IyYYYxnAw/GQpZ+RVEXU2IFQJBxtSI/bQXvKxO0zWc008D7mYOocN1M3H5WFo9JT1YtAytcSCbAuLxj0EDMYypKKr5FqYMC26pwKHZ9TQkmS45msNh/dLjfWBGJ7vXvC5aBOsheu7F/aETiByn9DD2XJvg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1644275795; bh=VX4ETLsASk+HT0Re0FMqk6DLox2OgOg3Wfgk75s04tI=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=JSoKtfTQi5qgmrzWc3cszgEcwt85RY43ur2sLVuWRw3ch+mgb2q5GXnZpeSbsON48JAlKk2ilE974t2xwSAdmsNwTRX2k1krnjCuECo+aXEvJ7C8Yej3oZIY3cMnguKVy7YBXHkKCJYco90Bi3T5GW4ujTXvnrONtq+MtYcHahJbFNxoVYfxzRCyGNxycwExhIOQ6dCelnjFywidP5p9S2yy1AmNahWa1zy7Tgf8RLw1tahfFR4NkMivki8ZkUdRozfaIsBnd5gIE90lFwS/nZPZVHTGvYliNBtPP42/IaI5grN4KCxI/IIQ2sUQuV17yywkYPKqgvq0oCAozLb0yA== X-YMail-OSG: r4miil4VM1n7k22IT6AsmSiCShZUlCWQxMt9datKGE9kpfS49gCi99Ku6Vu3VY5 9Q_D07ke.XBdmIrVXNmzkTT3p8CAFsXhbBppOS74UZBCSsalkOuAwTdsO1L5GDAulNaREjiq4I7Y 5VW29I.PDuSWH_8gNfeDFQM2fho0qsOiu0puBQMds7EdVZsKf8CZ7CxsUGunhSR8fLcVvgLbRKNp sQyJLETPGj_HMtiFoSgo30DcmF1owzyHyJJY2EN3M5NtwntTdxOlCRZRIISRqTLayVwBH5Xl3W_7 p6IREWHcEfktp67nRylvyzcexbrEgzft_bnyf5uInTRdCryoTfGuSh8XSAin0ySpNoj4MGbsdgYP S.KQgeonx8WaMX4noWz_V65UdWXnlxyiKKkJ378CQ5npbf8oqlEV8m4KJNFhGqvFo3BwkOoFUVgD NoBRZt0l1w40XCg4e.7G.0H2eTjhmGzBbkBb.LUf15E9XiXl0gIItVIVdTDZ02TjnvrUS6EUbPTJ KUnusku5a5ak7A._3dNBvR0Q8wYmzXsASfF7ZwpyHx0a2_AfYxyBoqbnVCoeMItFQ.5ukyI._o7h 7wjXUsZjZ04.2n4xJvMxsF6Furgup_KN6IpjocXDR2R5VigCEEuWdZs6hVrPUuhTZehaBhmTDQ.K RyURzNqG2kYomsQU4e_cowcpyMAY5QJk9j0vgwh7XNobgQlhkRLQt9pEPd0smGuJLVvSke5290gs 3FnY5L_7FVW5C.hR.AaIHY5JrA3FZ8h19uORndFhtKjMVtN8wBiqvkog1UyF_xZZ58LDIKrEkapN IQLakf29j5GXStJ9aNO8vtAU32XRQcdfwGvmn09.WpCPS6.LGUC4XCSFYpTHAUfr_PL8BNRq0Kmg qd3pf_QUK51U.Wsm.tFwLmIzUdkmLeRGc.hM9plQJUAHj7FmFtTCnu.7GLrFgHyLksymIJxwCwMu xtbbLh1eIgzCBSLDx3n5yvYezENtmpkdMJwPEdBkAm2JP9kg6z83jpifD5AHBbRN0TtyF75pivmt yF3yCq4Ex5sLkXe8F_c9by8FwuPGGrdpkh.aOby9LX42_8qcifgkbFiota.eFSZVdv3jlJbI6jOM Jm3vKVUl0FEfX87fTeWT4ECmn.NsmcwNLLZSF8zpUwfTjTt6C3uHr7azorcTALjG1fNQ5NOgm3GR q.B56._synP9I_KJQCREDH.j3dFaR2z86adS0pQNlJJDRvYo_7IVpnif66iGmYWgItEvSfyGgENJ BSVLAGyg7_21fjMhmcPuyG9T1o8YoQ7k4a8tAplTikTYD5rRR5v8KO6YzdzbcfsDaTV4czF.NFI7 BpQPSZ8ORoCmCIyH_nVDIR7uFacQoy4E31KkJR7AdrJVv6c6OdAWzUffsKNBrafew72N2GQEsyHi k8qTzZWqBEMwAI9ziTaI_hK4KWE29bIXrH8oiW7aMxB0ox7KhVxV4swca265gftBqXjuqMO7Le2a N_Sl6IfnRmJHRWv4mKDoCZFjMt5nNuDz2aD9bPZK.X9yOsA1xZVZBQvVkH7sETWWLfK9mMO8GcMl cxR0_GDWrrs2YCp59DD0.6mTkylOh8JjOiU9jsys972gO6nYj5zMJxH1MzyP5PqgrSxB0sUTsb8e GMlKybcvvt50DFHu9VNnN5Ji6Ad5.fae5xXOedSARbGTE306ffGMNx91rToW8zhpwwUobOAiO7X9 Ht0mFjvje5Z4GEyfTmBJ7wfZAfHpHSosl9KvTy_jOVtTKV6nyDbIjEnYiu0g8j.Q4XNziRJ0h_Ii p2_SE3RhDVPps848ivI.fO3riOeG8nrhGRz0oCbh27j9T10djp5LNYNJqyYjT8S1Yw61x2FWfur_ N3BqZHXU3Hoord8Oa7n6xT8VisK2uPE8lHcKTBsJDp9jN_7AcQFNbBggTta5E4rtazZEmCBZobZo SQASu4R3gcZ2wdCHUP4GrYy5thfgL8jepE5QvGroJfBSgLtynNdpeWmTw.cSjXZ0ney7GrmXrFGK ZyqPvw4mJna8kmhEVhqcNeXXonSep7tWc6vd5wxRfrtY00meBHCoNj6c3oZJdq_513mNdeHmWD5g BTHCkna28wYAGIgAkl9WuN8x8WrAkIOwHOs10i.I6SwBCyCqyDdnsWra3QAeffGJ3WIvv6DlJ1IM oF964hTBrpy2EaDCYs9TVx9scSm_nC5x1Ac7Tmh06Qc4GOkNyfu.U.8crd21ti1Wn.Sjy3i6IqkQ mcyrvRJHlTfrfSzJaXrkWCLFZemgnPzX1dFUlfbW_UZfgfQfrYVUyPZFV4bMZWyKZLCQ_Grh7n.F u5GvCJMzF8aaBKQ-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic311.consmr.mail.gq1.yahoo.com with HTTP; Mon, 7 Feb 2022 23:16:35 +0000 Received: by kubenode524.mail-prod1.omega.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 98c090baffbeae86c5386226890028c9; Mon, 07 Feb 2022 23:16:33 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: Troubles building world on stable/13 [the little bit of evidence about the compiler failures: a jemalloc-tie/ASLR-tie?] From: Mark Millard In-Reply-To: <22832BFB-D1A2-4964-B7C0-3E8F97E9C5E0@yahoo.com> Date: Mon, 7 Feb 2022 15:16:31 -0800 Cc: Free BSD , FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: References: <20220202223208.GA78110@www.zefox.net> <70550346-BC53-458F-B01B-68559E5C9847@yahoo.com> <20220203015149.GA78722@www.zefox.net> <8A85F917-F4E8-4382-B777-15AF7401E616@yahoo.com> <20220204214403.GA85107@www.zefox.net> <20220205000800.GA85644@www.zefox.net> <51D494E4-6D8D-49C7-8F0C-FD53311264A5@yahoo.com> <20220205020612.GA85996@www.zefox.net> <22832BFB-D1A2-4964-B7C0-3E8F97E9C5E0@yahoo.com> To: bob prohaska X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Jt28V4GNDz4ldF X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=qxZKYm1F; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.50 / 15.00]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SUBJECT_HAS_QUESTION(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.206:from]; MLMMJ_DEST(0.00)[freebsd-stable]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.206:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N The primary evidence that I have gotten is dumps of part of the stack spanning the failure in the stable/13 c++ compiler. [Bob does his own buildworld buildkernel activities on the RPi3* in question, but normally absent the likes of make.conf and src.conf (or equivalents).] The following did lead to such a stack dump showing 8 bytes of 0xa5 that otherwise was 0x00. [The "junk:true" is a means of having jemalloc fill allocated memory with 0xa5 on allocation and 0x5a on deallocation (when jemalloc is built to allow such).] # ls -Tld /etc/malloc.conf=20 lrwxr-xr-x 1 root wheel 20 Feb 4 03:47:13 2022 /etc/malloc.conf -> = junk:true,abort:true It was around the beginning of the region that looked to have been stomped on: 0xffffffffa360: 00 00 00 00 00 00 00 00 a5 a5 a5 a5 a5 a5 a5 a5 = ................ Elsewhere in the dumped subregion of the stack (smaller addresses), there was also an example of "a5 a5 a5" : 0xffffffffae20: b8 3f 53 00 00 00 00 00 02 22 71 01 c1 a5 a5 a5 = .?S......"q..... There were no examples of "5a 5a" in the region and only the above examples of back to back a5's. (I did not check for back-to-back across lines.) What looks to be a valid fp/lr pair is: 0xffffffffa2d0: 70 a5 ff ff ff ff 00 00 6c 2b b7 02 00 00 00 00 = p.......l+...... But in the dump what the left part (fp part) refers to is: 0xffffffffa570: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... For reference for lr's "6c 2b b7 02": (lldb) disass -c 10 -a 0x2b72b6c c++`::ParseOptionalCXXScopeSpecifier(): 0x2b723fc <+0>: stp x29, x30, [sp, #-0x60]! 0x2b72400 <+4>: stp x28, x27, [sp, #0x10] 0x2b72404 <+8>: stp x26, x25, [sp, #0x20] 0x2b72408 <+12>: stp x24, x23, [sp, #0x30] 0x2b7240c <+16>: stp x22, x21, [sp, #0x40] 0x2b72410 <+20>: stp x20, x19, [sp, #0x50] 0x2b72414 <+24>: mov x29, sp 0x2b72418 <+28>: sub sp, sp, #0x250 ; =3D0x250=20 0x2b7241c <+32>: adrp x8, 9734 0x2b72420 <+36>: ldr x8, [x8, #0xe60] Most failures have the 0x01 after the ": ", but on occasion I've gotten one with 0x05 instead. The surrounding lines for the example at hand, that follow a simple, similarity-pattern, look like: 0xffffffffa450: 01 00 00 00 00 00 00 00 80 32 b2 55 00 00 00 00 = .........2.U.... 0xffffffffa460: 00 00 00 00 00 00 00 00 e2 34 b2 55 00 00 00 00 = .........4.U.... 0xffffffffa470: 01 00 00 00 00 00 00 00 00 33 b2 55 00 00 00 00 = .........3.U.... 0xffffffffa480: 00 00 00 00 00 00 00 00 f2 34 b2 55 00 00 00 00 = .........4.U.... 0xffffffffa490: 01 00 00 00 00 00 00 00 80 33 b2 55 00 00 00 00 = .........3.U.... 0xffffffffa4a0: 00 00 00 00 00 00 00 00 02 35 b2 55 00 00 00 00 = .........5.U.... 0xffffffffa4b0: 01 00 00 00 00 00 00 00 00 34 b2 55 00 00 00 00 = .........4.U.... 0xffffffffa4c0: 00 00 00 00 00 00 00 00 12 35 b2 55 00 00 00 00 = .........5.U.... 0xffffffffa4d0: 01 00 00 00 00 00 00 00 80 34 b2 55 00 00 00 00 = .........4.U.... 0xffffffffa4e0: 00 00 00 00 00 00 00 00 22 35 b2 55 00 00 00 00 = ........"5.U.... 0xffffffffa4f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa500: 00 00 00 00 00 00 00 00 5a 35 b2 55 00 00 00 00 = ........Z5.U.... 0xffffffffa510: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa520: 00 00 00 00 00 00 00 00 a2 35 b2 55 00 00 00 00 = .........5.U.... 0xffffffffa530: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa540: 00 00 00 00 00 00 00 00 ea 35 b2 55 00 00 00 00 = .........5.U.... 0xffffffffa550: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa560: 00 00 00 00 00 00 00 00 32 36 b2 55 00 00 00 00 = ........26.U.... 0xffffffffa570: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa580: 00 00 00 00 00 00 00 00 7a 36 b2 55 00 00 00 00 = ........z6.U.... 0xffffffffa590: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa5a0: 00 00 00 00 00 00 00 00 c2 36 b2 55 00 00 00 00 = .........6.U.... 0xffffffffa5b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa5c0: 00 00 00 00 00 00 00 00 0a 37 b2 55 00 00 00 00 = .........7.U.... 0xffffffffa5d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa5e0: 00 00 00 00 00 00 00 00 52 37 b2 55 00 00 00 00 = ........R7.U.... 0xffffffffa5f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa600: 00 00 00 00 00 00 00 00 9a 37 b2 55 00 00 00 00 = .........7.U.... 0xffffffffa610: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa620: 00 00 00 00 00 00 00 00 e2 37 b2 55 00 00 00 00 = .........7.U.... 0xffffffffa630: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa640: 00 00 00 00 00 00 00 00 2a 38 b2 55 00 00 00 00 = ........*8.U.... 0xffffffffa650: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa660: 00 00 00 00 00 00 00 00 72 38 b2 55 00 00 00 00 = ........r8.U.... 0xffffffffa670: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa680: 00 00 00 00 00 00 00 00 ba 38 b2 55 00 00 00 00 = .........8.U.... 0xffffffffa690: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa6a0: 00 00 00 00 00 00 00 00 02 39 b2 55 00 00 00 00 = .........9.U.... 0xffffffffa6b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa6c0: 00 00 00 00 00 00 00 00 4a 39 b2 55 00 00 00 00 = ........J9.U.... 0xffffffffa6d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa6e0: 00 00 00 00 00 00 00 00 92 39 b2 55 00 00 00 00 = .........9.U.... 0xffffffffa6f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa700: 00 00 00 00 00 00 00 00 da 39 b2 55 00 00 00 00 = .........9.U.... 0xffffffffa710: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa720: 00 00 00 00 00 00 00 00 22 3a b2 55 00 00 00 00 = ........":.U.... 0xffffffffa730: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa740: 00 00 00 00 00 00 00 00 6a 3a b2 55 00 00 00 00 = ........j:.U.... 0xffffffffa750: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa760: 00 00 00 00 00 00 00 00 b2 3a b2 55 00 00 00 00 = .........:.U.... 0xffffffffa770: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa780: 00 00 00 00 00 00 00 00 fa 3a b2 55 00 00 00 00 = .........:.U.... 0xffffffffa790: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa7a0: 00 00 00 00 00 00 00 00 42 3b b2 55 00 00 00 00 = ........B;.U.... 0xffffffffa7b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa7c0: 00 00 00 00 00 00 00 00 8a 3b b2 55 00 00 00 00 = .........;.U.... 0xffffffffa7d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa7e0: 00 00 00 00 00 00 00 00 d2 3b b2 55 00 00 00 00 = .........;.U.... 0xffffffffa7f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa800: 00 00 00 00 00 00 00 00 1a 3c b2 55 00 00 00 00 = .........<.U.... 0xffffffffa810: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa820: 00 00 00 00 00 00 00 00 62 3c b2 55 00 00 00 00 = ........b<.U.... 0xffffffffa830: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa840: 00 00 00 00 00 00 00 00 aa 3c b2 55 00 00 00 00 = .........<.U.... 0xffffffffa850: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa860: 00 00 00 00 00 00 00 00 f2 3c b2 55 00 00 00 00 = .........<.U.... 0xffffffffa870: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa880: 00 00 00 00 00 00 00 00 3a 3d b2 55 00 00 00 00 = ........:=3D.U.... 0xffffffffa890: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa8a0: 00 00 00 00 00 00 00 00 82 3d b2 55 00 00 00 00 = .........=3D.U.... 0xffffffffa8b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa8c0: 00 00 00 00 00 00 00 00 ca 3d b2 55 00 00 00 00 = .........=3D.U.... 0xffffffffa8d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa8e0: 00 00 00 00 00 00 00 00 12 3e b2 55 00 00 00 00 = .........>.U.... 0xffffffffa8f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa900: 00 00 00 00 00 00 00 00 5a 3e b2 55 00 00 00 00 = ........Z>.U.... 0xffffffffa910: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa920: 00 00 00 00 00 00 00 00 a2 3e b2 55 00 00 00 00 = .........>.U.... 0xffffffffa930: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa940: 00 00 00 00 00 00 00 00 ea 3e b2 55 00 00 00 00 = .........>.U.... 0xffffffffa950: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa960: 00 00 00 00 00 00 00 00 32 3f b2 55 00 00 00 00 = ........2?.U.... 0xffffffffa970: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa980: 00 00 00 00 00 00 00 00 7a 3f b2 55 00 00 00 00 = ........z?.U.... 0xffffffffa990: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa9a0: 00 00 00 00 00 00 00 00 c2 3f b2 55 00 00 00 00 = .........?.U.... 0xffffffffa9b0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa9c0: 00 00 00 00 00 00 00 00 0a 40 b2 55 00 00 00 00 = .........@.U.... 0xffffffffa9d0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffa9e0: 00 00 00 00 00 00 00 00 52 40 b2 55 00 00 00 00 = ........R@.U.... 0xffffffffa9f0: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffaa00: 00 00 00 00 00 00 00 00 9a 40 b2 55 00 00 00 00 = .........@.U.... 0xffffffffaa10: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffaa20: 00 00 00 00 00 00 00 00 e2 40 b2 55 00 00 00 00 = .........@.U.... 0xffffffffaa30: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffaa40: 00 00 00 00 00 00 00 00 2a 41 b2 55 00 00 00 00 = ........*A.U.... 0xffffffffaa50: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffaa60: 00 00 00 00 00 00 00 00 72 41 b2 55 00 00 00 00 = ........rA.U.... 0xffffffffaa70: 01 00 00 00 00 00 00 00 c0 0d b1 55 00 00 00 00 = ...........U.... 0xffffffffaa80: 00 00 00 00 00 00 00 00 ba 41 b2 55 00 00 00 00 = .........A.U.... When the 0x05's show up they are instead of the 0x01's, just after the ": ". After that the pattern is different. But quickly something looks like another fp/lr pair in memory, and tha, in turn, it references another: 0xffffffffaa90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = ................ 0xffffffffaaa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 = ................ 0xffffffffaab0: 00 00 00 00 00 00 00 00 44 c4 95 07 0e 02 46 57 = ........D.....FW 0xffffffffaac0: 10 ab ff ff ff ff 00 00 8c c6 aa 02 00 00 00 00 = ................ . . . 0xffffffffab10: 90 ac ff ff ff ff 00 00 e0 18 ab 02 00 00 00 00 = ................ . . . But after that the following does not seem to fit the pattern: 0xffffffffac90: 00 ac ff ff ff ff 00 00 44 c4 95 07 0e 02 46 57 = ........D.....FW and: 0xffffffffac00: 01 00 00 00 00 00 00 00 18 ae ff ff ff ff 00 00 = ................ The a5 sequences make me wonder if jemalloc assigned a memory allocation to stack space or was told to handle a stack address as if it was an assigned address for some aspects of an allocation (if that can even be requested). I wonder if there is any chance of ASLR being involved with the stack and memory allocation possibly overlapping. But I've really no clue. I've given up on trying to isolate what is going on for the compiler failures. I've only been able to see after the failure, not just before: debugger interactions with the compiler process in times close to the failure point in the code prevent the failure. I've not found any alternative that avoids such. This is on top of the issue that the plain-runs (no debugger) vary in behavior, sometimes running to completion, sometimes stopping at similar but varying places in the source code being processed. There is still no known way to get a full reproduction of failure details each time. (Which instance of the example type of source code being compiled at the point of failure does vary.) For reference: I've been using .sh/.cpp pairs that Bob published and a copy of the c++ from his system to investigate. The .cpp is large. Bob's RPi3* is a RAM+SWAP context of: 1 GiBYTe + 2 GiByte and I made such a context on a RPi3* as well. But I ran his stable/13 c++ on a system with a non-debug main [so: 14] kernel and either a main world or a stable/13 chroot. From the chroot: # uname -apKU FreeBSD Rock64_RPi_4_3_2v1p2 14.0-CURRENT FreeBSD 14.0-CURRENT #28 = main-n252475-e76c0108990b-dirty: Sat Jan 15 23:39:27 PST 2022 = root@CA72_16Gp_ZFS:/usr/obj/BUILDs/main-CA53-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA53 arm64 aarch64 1400047 1300524 # freebsd-version -ru 14.0-CURRENT 13.0-STABLE # ~/fbsd-based-on-what-commit.sh -C /usr/13S-src/ branch: stable/13 merge-base: a5f69859956049b5153b0e1b67f8f4a99622dc6f merge-base: CommitDate: 2022-01-15 12:55:32 +0000 a5f698599560 (HEAD -> stable/13, freebsd/stable/13) Ignore = debugger-injected signals left after detaching Bob's recent stable/13 context (kernel too) is more recent than mine. So the problems has been observed over a range of contexts. But, as I said, I've given up on finding a way to isolate whatever is going on. =3D=3D=3D Mark Millard marklmi at yahoo.com