From owner-freebsd-ppc@freebsd.org Fri May 8 16:14:34 2020 Return-Path: Delivered-To: freebsd-ppc@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 64B192DB642 for ; Fri, 8 May 2020 16:14:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-20.consmr.mail.gq1.yahoo.com (sonic306-20.consmr.mail.gq1.yahoo.com [98.137.68.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49Jb4m2wcTz40mW for ; Fri, 8 May 2020 16:14:32 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: TVENuvoVM1n_zA_0fs6Z719DKccdLvBZKI4IF2arNLX_N4maUdk63_v7VCedtYw kKmDxIpX7OF5E6wicfhOXIHMkD7lDSBRJFGS_kqn447wUaref25ZfPYh4YEESAzPpA2rIsETZB3z FYvkZLf5Po8MZyT3KX6Cain4buVLJUUhDIkLbXksSeWUo51VxQJMV3C6fCuJu6nWndqmA4Qh1mTt XMLaHVA8E9GNT_PfDLNvZKcnGbANy5qhCfkmZ4LOEF_GyAwAuZ1aHHwY4GOgyyoND1NHNxa04rat NulF6Fool9duKR9UmGva60q4Gl0HmRxyNNKQN253LMyOVhNlZ0B1iRkkEqig4JqhdM_aHxu0iktm AnoAodoa.Ws3BKJogkqMMnXCv91vkDRVyuxh7FDK3uf1SqN_2x9C_UpWLfrkwAET.ydd4aJEVT0N Jbz1FrB9uo8yq5KeD_uNskh1N2rHXtc30BDQOzrqsRDKcyZvcK3XcLvW6NMtMlM0UF5gHMNAA5uH 23SQcUX_sHjsarQTcf8.orhY4Xane0WX0kMO_QrZTnT00PQrlrGn0.GdH6_o8zwEAG.xcsFSLgTT _.yybkiZ.p3hwwxvTQEJWfzdQQDpj0FQcRE9gtcU8ONPqCXCZM14u73ArBF1F7dFJ8lEEru25dR5 0vErXFnRDQs.kxzebK4TWqJe24GHRSXhvLpALkZgqUp.BNqakGjSg97Y3OvQPdmLQAoPG_1S.Cje Hk9FfmYnXohSWqwBtfBgRzhd13NCi_uztl8yy2zdHs9e19MEuf8AvhwIX2AgBgSnRgde45zygkYH y.gPaQ8GewxsVdScud1nCUICCM1ACyu1IxRqZvu3QDrsoqhx5C3mTnYcvxZbAoC5QEoJ.1i3aIWy 4ooOqAK0xXFAp1VVpr3zp99dzwDc0PdtY3l6ar7sMa_uBpDCLaBry2ZCzQQmErOYj5_jeg0HH_sK 5mVHAUngzgcF_GJv29m7NC3viQH16l.e0mLx70pJv4goQuGhC_JoT9re8DZq3o9sbJIYTiTve8G9 xAqAgkZJUXQoPqkQD9bcr3cgUNMH3XCF55R0wJoH7TFpx5vYUZ3Qu4.DatFyku_ISXHer15cnlXN ThYqGhAVjHZ_cPf1N7EjWDuJxl6vtPrxbLgNPYG5eVrBVb0Y8meyYv9e9nPlCEYkPWDdGcVd8gvB Z77cDzOsF3Q8d0OJsBRBPyLjgMydpRbaffFyT0fAD22FU8rjdidlFmPmf9orIe6fRLZK4RKZh.gr rMRj5aMwaDpfT4G4KLlY9X.vCUNZ4eZasZkWtUnhoaq0ReXMJXFtCiWwEr3q_b9coYIqliudGjWR 7aUsa6GVnP53AU6aTKEurLaKhBF8tyndHLlpg8KVBobgq3YhFzOOsLv.rikJwACuuuc0eZlQh5rN TrcgPmggrtVjMJg_b2OXVrlBnJDEfhbGt0Tm_3IEz6t3qTdCQCG7XB7_ObBwezDw- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Fri, 8 May 2020 16:14:30 +0000 Received: by smtp421.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID f4875cfc9e4acaebf3d65b4cfeaaba16; Fri, 08 May 2020 16:14:26 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: svn commit: r360233 - in head: contrib/jemalloc . . . : This partially breaks a 2-socket 32-bit powerpc (old PowerMac G4) based on head -r360311 From: Mark Millard In-Reply-To: Date: Fri, 8 May 2020 09:14:24 -0700 Cc: Brandon Bergren , Justin Hibbits Content-Transfer-Encoding: quoted-printable Message-Id: <624CE71B-2C50-4E77-85A2-42D9FA140AD0@yahoo.com> References: <8479DD58-44F6-446A-9CA5-D01F0F7C1B38@yahoo.com> <17ACDA02-D7EF-4F26-874A-BB3E935CD072@yahoo.com> <695E6836-F860-4557-B7DE-CC1EDB347F18@yahoo.com> <121B9B09-141B-4DC3-918B-1E7CFB99E779@yahoo.com> <8AAB0462-3FA8-490C-8D8D-7C15B1C9E2DE@yahoo.com> <18E62746-80DB-4195-977D-4FF32D0129EE@yahoo.com> To: "vangyzen@freebsd.org" , svn-src-head@freebsd.org, FreeBSD Current , FreeBSD Hackers , FreeBSD PowerPC ML X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49Jb4m2wcTz40mW X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.40 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCPT_COUNT_SEVEN(0.00)[7]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.93)[-0.926,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.97)[-0.970,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (2.91), ipnet: 98.137.64.0/21(0.83), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[83.68.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[83.68.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-ppc@freebsd.org X-Mailman-Version: 2.1.32 Precedence: list List-Id: Porting FreeBSD to the PowerPC List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 May 2020 16:14:34 -0000 [More details for a sshd failure. The other examples are omitted. The sshd failure also shows a all-zeros-up-to-a-page-boundary issue, just for a different address range.] On 2020-May-7, at 12:06, Mark Millard wrote: >=20 > [mountd failure example: also at sz_size2index_lookup assert > for the same zero'd memory problem.] >=20 >> On 2020-May-7, at 00:46, Mark Millard wrote: >>=20 >> [__je_sz_size2index_tab seems messed up in 2 of the >> asserting contexts: first 384 are zero in both. More >> before that is also messed up (all zero). I show the >> details later below.] >>=20 >> On 2020-May-6, at 16:57, Mark Millard wrote: >>=20 >>> [This explores process crashes that happen during system >>> shutdown, in a context not having MALLOC_PRODUCTION=3D . >>> So assert failures are reported as the stopping points.] >>>=20 >>> It looks like shutdown -p now, shutdown -r now, and the >>> like can lead some processes to assert during their exit >>> attempt, including a sshd failure (that I've not seen >>> before), rpcbind, and nfsd. I show information about the >>> observed asserts for those below. >>>=20 >>>=20 >>> sshd hit an assert, failing slab =3D=3D extent_slab_get(extent) : >>>=20 >>> (gdb) bt=20 >>> #0 thr_kill () at thr_kill.S:4 >>> #1 0x50927170 in __raise (s=3D6) at = /usr/src/lib/libc/gen/raise.c:52 >>> #2 0x50886cc0 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 >>> #3 0x508834b0 in arena_dalloc (tsdn=3D, = ptr=3D, tcache=3D, alloc_ctx=3D, slow_path=3D) >>> at = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:315 >>> #4 idalloctm (tsdn=3D0x500dd040, ptr=3D0x5008a180, = tcache=3D0x500dd160, alloc_ctx=3D, is_internal=3D, slow_path=3D) >>> at = /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inli= nes_c.h:118 >>> #5 0x5087b0a4 in ifree (tsd=3D0x500dd040, ptr=3D0x5008a180, = tcache=3D0x500dd160, slow_path=3D) at = jemalloc_jemalloc.c:2590 >>> #6 0x5087acac in __je_free_default (ptr=3D0x5008a180) at = jemalloc_jemalloc.c:2784 >>> #7 0x5087b294 in __free (ptr=3D0x5008a180) at = jemalloc_jemalloc.c:2852 >>> #8 0x10029464 in server_accept_loop (config_s=3D, = sock_in=3D, sock_out=3D, = newsock=3D) at /usr/src/crypto/openssh/sshd.c:1185 >>> #9 main (ac=3D, av=3D0xffffde3c) at = /usr/src/crypto/openssh/sshd.c:2009 >>>=20 >>> . . . >>> (gdb) up >>> #2 0x50886cc0 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 >>> 67 (void)raise(SIGABRT); >>> (gdb) up >>> #3 0x508834b0 in arena_dalloc (tsdn=3D, = ptr=3D, tcache=3D, alloc_ctx=3D, slow_path=3D) >>> at = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:315 >>> 315 assert(slab =3D=3D extent_slab_get(extent)); >>>=20 >>> (gdb) list >>> 310 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); >>> 311 extent_t *extent =3D rtree_extent_read(tsdn, = &extents_rtree, >>> 312 rtree_ctx, (uintptr_t)ptr, true); >>> 313 assert(szind =3D=3D extent_szind_get(extent)); >>> 314 assert(szind < SC_NSIZES); >>> 315 assert(slab =3D=3D extent_slab_get(extent)); >>> 316 } >>> 317=09 >>> 318 if (likely(slab)) { >>> 319 /* Small allocation. */ >>>=20 >>> More fully: >>>=20 >>> 285 JEMALLOC_ALWAYS_INLINE void >>> 286 arena_dalloc(tsdn_t *tsdn, void *ptr, tcache_t *tcache, >>> 287 alloc_ctx_t *alloc_ctx, bool slow_path) { >>> 288 assert(!tsdn_null(tsdn) || tcache =3D=3D NULL); >>> 289 assert(ptr !=3D NULL); >>> 290=09 >>> 291 if (unlikely(tcache =3D=3D NULL)) { >>> 292 arena_dalloc_no_tcache(tsdn, ptr); >>> 293 return; >>> 294 } >>> 295=09 >>> 296 szind_t szind; >>> 297 bool slab; >>> 298 rtree_ctx_t *rtree_ctx; >>> 299 if (alloc_ctx !=3D NULL) { >>> 300 szind =3D alloc_ctx->szind; >>> 301 slab =3D alloc_ctx->slab; >>> 302 assert(szind !=3D SC_NSIZES); >>> 303 } else { >>> 304 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); >>> 305 rtree_szind_slab_read(tsdn, &extents_rtree, = rtree_ctx, >>> 306 (uintptr_t)ptr, true, &szind, &slab); >>> 307 } >>> 308=09 >>> 309 if (config_debug) { >>> 310 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); >>> 311 extent_t *extent =3D rtree_extent_read(tsdn, = &extents_rtree, >>> 312 rtree_ctx, (uintptr_t)ptr, true); >>> 313 assert(szind =3D=3D extent_szind_get(extent)); >>> 314 assert(szind < SC_NSIZES); >>> 315 assert(slab =3D=3D extent_slab_get(extent)); >>> 316 } >>> 317=09 >>> 318 if (likely(slab)) { >>> 319 /* Small allocation. */ >>> 320 tcache_dalloc_small(tsdn_tsd(tsdn), tcache, ptr, = szind, >>> 321 slow_path); >>> 322 } else { >>> 323 arena_dalloc_large(tsdn, ptr, tcache, szind, = slow_path); >>> 324 } >>> 325 } >>=20 >> . . . The machine code for: 309 if (config_debug) { 310 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); 311 extent_t *extent =3D rtree_extent_read(tsdn, = &extents_rtree, 312 rtree_ctx, (uintptr_t)ptr, true); 313 assert(szind =3D=3D extent_szind_get(extent)); 314 assert(szind < SC_NSIZES); 315 assert(slab =3D=3D extent_slab_get(extent)); 316 } was dropping the address in "extent" the next instruction after finding it: replacing with with a field's value. So by the time the status of the assert could be known, examining extent was a difficulty. So I touched the source code to force the address to be kept and to give a place to breakpoint on failure before calling another routine: # svnlite diff = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h Index: = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D --- = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h = (revision 360322) +++ = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h = (working copy) @@ -308,11 +308,11 @@ =20 if (config_debug) { rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); - extent_t *extent =3D rtree_extent_read(tsdn, = &extents_rtree, + extent_t * volatile extent =3D rtree_extent_read(tsdn, = &extents_rtree, rtree_ctx, (uintptr_t)ptr, true); assert(szind =3D=3D extent_szind_get(extent)); assert(szind < SC_NSIZES); - assert(slab =3D=3D extent_slab_get(extent)); + assert((slab =3D=3D extent_slab_get(extent)) ?true = :extent=3D=3DNULL); } =20 if (likely(slab)) { The ":extent=3D=3DNULL" should be guaranteed to produce :false as a = result but with more code involved to get there. It gave me a place to = breakpoint on failure. (gdb) bt -full #0 0x50883258 in arena_dalloc (tsdn=3D, ptr=3D, tcache=3D, alloc_ctx=3D, = slow_path=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:315 extent =3D 0x51007fc0 slab =3D szind =3D rtree_ctx =3D 0x500dd06c extent =3D #1 idalloctm (tsdn=3D0x500dd040, ptr=3D0x5008a180, tcache=3D0x500dd160, = alloc_ctx=3D, is_internal=3D, = slow_path=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inli= nes_c.h:118 No locals. #2 0x5087b0e4 in ifree (tsd=3D0x500dd040, ptr=3D0x5008a180, = tcache=3D0x500dd160, slow_path=3D) at = jemalloc_jemalloc.c:2590 alloc_ctx =3D {szind =3D 0, slab =3D true} rtree_ctx =3D usize =3D #3 0x5087acec in __je_free_default (ptr=3D0x5008a180) at = jemalloc_jemalloc.c:2784 tcache =3D 0x500dd160 tsd =3D #4 0x5087b2d4 in __free (ptr=3D0x5008a180) at jemalloc_jemalloc.c:2852 log_var =3D log_var =3D #5 0x10029464 in server_accept_loop (config_s=3D, = sock_in=3D, sock_out=3D, = newsock=3D) at /usr/src/crypto/openssh/sshd.c:1185 from =3D {ss_len =3D 16 '\020', ss_family =3D 2 '\002', = __ss_pad1 =3D "\317\200\300\250\001\031", __ss_align =3D 0,=20 __ss_pad2 =3D "\000\000\000\000\000\000\000\002Am", '\000' = , "\065\341I\000\000\000\000^\264\234\331", '\000' = , = "^\262\027\034-a\241H\000\000\000\000\000\000\000\000^\264\235Y,\024\247\0= 30\000\000\000\000\000\000\000\000V\312\331f6-N\370\000\000\000\000\000\00= 0\000\000\000\000\002\000\000\000\000\000\000\000\000\b"} rnd =3D '\000' startup_p =3D {6, 7} startups =3D 1 i =3D maxfd =3D 6 fdset =3D fromlen =3D ret =3D j =3D pid =3D laddr =3D raddr =3D #6 main (ac=3D, av=3D0xffffde3c) at = /usr/src/crypto/openssh/sshd.c:2009 config_s =3D {8, 9} on =3D 1 ssh =3D 0x0 logfile =3D newsock =3D -1 sock_out =3D -1 sock_in =3D -1 connection_info =3D i =3D opt =3D line =3D r =3D key =3D pubkey =3D keytype =3D fp =3D j =3D new_umask =3D already_daemon =3D remote_port =3D remote_ip =3D rdomain =3D --Type for more, q to quit, c to continue without paging-- laddr =3D authctxt =3D obytes =3D ibytes =3D (gdb) list 310 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); 311 extent_t * volatile extent =3D = rtree_extent_read(tsdn, &extents_rtree, 312 rtree_ctx, (uintptr_t)ptr, true); 313 assert(szind =3D=3D extent_szind_get(extent)); 314 assert(szind < SC_NSIZES); 315 assert((slab =3D=3D extent_slab_get(extent)) = ?true :extent=3D=3DNULL); 316 } 317=09 318 if (likely(slab)) { 319 /* Small allocation. */ (gdb) print/x extent $6 =3D 0x51007fc0 (gdb) print/x *extent $2 =3D {e_bits =3D 0x0, e_addr =3D 0x0, {e_size_esn =3D 0x0, e_bsize =3D = 0x0}, ql_link =3D {qre_next =3D 0x0, qre_prev =3D 0x0}, ph_link =3D = {phn_prev =3D 0x0, phn_next =3D 0x0, phn_lchild =3D 0x0}, {e_slab_data =3D= { bitmap =3D {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xffffffff, 0xffffffff, = 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, = 0xffffffff, 0xffffffff, 0xfff8}}, {e_alloc_time =3D { ns =3D 0x0}, e_prof_tctx =3D {repr =3D 0x0}}}} It looks like the prefix of the above has been stomped on to be zeros. Checking addresses as well: (gdb) x/64x extent 0x51007fc0: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51007fd0: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51007fe0: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51007ff0: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51008000: 0xffffffff 0xffffffff 0xffffffff = 0xffffffff 0x51008010: 0xffffffff 0xffffffff 0xffffffff = 0xffffffff 0x51008020: 0xffffffff 0xffffffff 0x0000fff8 = 0x00000000 0x51008030: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51008040: 0x00000800 0x0014f000 0x5008b000 = 0x00005000 0x51008050: 0x51008040 0x51008040 0x00000000 = 0x00000000 0x51008060: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51008070: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51008080: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x51008090: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x510080a0: 0x00000000 0x00000000 0x00000000 = 0x00000000 0x510080b0: 0x00000000 0x00000000 0x00000000 = 0x00000000 Note that the non-zero values start at: 0x51008000, again a page boundary, like the other examples in prior notes for other cases. This appears to be another example of memory having been stomped on/replaced that likely previously had the intended values. The first prior non-zero values are at: 0x51005a30: 0x00000000 0x51008740 0x0000000f = 0x01000000 0x51005a40: 0x51008740 0x0000000f 0x01000000 = 0x51008740 0x51005a50: 0x0000000f 0x01000000 0x51008740 = 0x0000000f 0x51005a60: 0x01000000 0x51008740 0x0000000f = 0x01000000 0x51005a70: 0x51008740 0x0000000f 0x01000000 = 0x51008740 0x51005a80: 0x0000000f 0x01000000 0x51008940 = 0x00000014 0x51005a90: 0x01000000 0x510089c0 0x00000014 = 0x01000000 0x51005aa0: 0x51008a40 0x00000018 0x01000000 = 0x51008ac0 0x51005ab0: 0x00000018 0x01000000 0x51008b40 = 0x00000018 0x51005ac0: 0x01000000 0x51008bc0 0x00000018 = 0x01000000 0x51005ad0: 0x51008c40 0x00000018 0x01000000 = 0x51008cc0 0x51005ae0: 0x00000001 0x01000000 0x00000000 = 0x00000000 So the pages for address ranges 0x51006yyy and 0x51007yyy seem to be all-zero, along with the tail of the page for the range 0x51005yyy. #2 (ifree) in the backtrace shows the alloc_ctx content: alloc_ctx =3D {szind =3D 0, slab =3D true} So slab=3D=3Dtrue but the bit in extent->e_bits=3D=3Dfalse, leading to the failed assert. Before going to sleep for the night, I could ssh into the old PowerMac without this detection. After getting up, trying the same got the failure detection. I did not have the machine doing anything else special between. The other examples in the other programs are similar: just waiting long enough with normal background processing going on eventually leads to the context for the next explicit use (or exit) to detect the problem. I'm still no where near identifying when the stomped-on memory range is trashed with zeros in code terms in any example program. But sshd, dhclient, rpcbind, nfsd, and sendmail all seem to have some common subject area(s) involved in their implementation. So I suspect something common across those is essentially involved. For reference: In #0: (gdb) print/x rtree_ctx $5 =3D 0x500dd06c (gdb) print/x *rtree_ctx $4 =3D {cache =3D {{leafkey =3D 0x50000000, leaf =3D 0x51004fc0}, = {leafkey =3D 0x1, leaf =3D 0x0}, {leafkey =3D 0x1, leaf =3D 0x0}, = {leafkey =3D 0x50c00000, leaf =3D 0x51008dc0}, {leafkey =3D 0x1,=20 leaf =3D 0x0} }, l2_cache =3D {{leafkey =3D 0x1, = leaf =3D 0x0}, {leafkey =3D 0x1, leaf =3D 0x0}, {leafkey =3D 0x1, leaf =3D= 0x0}, {leafkey =3D 0x1, leaf =3D 0x0}, {leafkey =3D 0x1, leaf =3D 0x0}, = { leafkey =3D 0x1, leaf =3D 0x0}, {leafkey =3D 0x1, leaf =3D 0x0}, = {leafkey =3D 0x1, leaf =3D 0x0}}} =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)