From owner-svn-src-head@freebsd.org Wed May 6 23:58:06 2020 Return-Path: Delivered-To: svn-src-head@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 009CE2E3C1F for ; Wed, 6 May 2020 23:58:06 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49HYSX446kz4PGv for ; Wed, 6 May 2020 23:58:04 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: .tMrsL8VM1lHXQP1QD1XvZPA.c2.3hOgHOxwWwSCQi09XDVpsXAd5ewGprIy6L1 bNqzZCaCCdY2_oNp9HMnBiRgfiyaGnEKtxgGWZdWmz1RJk4QhDnh23eirM4m3r.Cp94YkOaV0Uj8 O9awIIh7GssVjPFBgOzK3AQ6OyV5Wg009wwsVB8HZ.P7GFFvF4n1LnY7bJKCHIONZhrfVI7iGHZu F.iq.YkK.5wFY7EpOFv81jf30uZoxZqIsAV5jX.cE0iZvNDn8yh3zdcvPYlgOPz.IwEbFJq47R_7 PUY0Pk6QnDPKq9dROHiEr6hcaBs1cYNFzdE2tYQCGa29cOL_HtpRVCntJltJ9jrsQM7MP1I4a55c T0JLEdVJS9qVFJCyE35Kh9e0.9lqI7eaJyZ2f2L9OhRH.ec7k6aIgeuGbwiCl7guWS8wnRpRj1Bg RhurB93CwUj0tnHEpWh2GUP584a_4ds_HCr1fcWc.VHZFRXN2PtSB8s92qMIzthS9Yk4TSKrb_1s e8OQEyHk2zOlSJ_psMp0Gutl6_U2Z.jtnC.izImKdMyBTvCfEdX4xiIJ7U083wCIpxTEqRQ_jXN6 TiLh0Mxers3xJtQkAkAb32eII10xTvJEg_vcNQn5K6hfi_5Lp6URAFx1t6_SU62sXZZ4_uLjyjF2 xZd8KuAxpPmhFtFEBQ0rfy13B4P4e9NHfLz1SRsJpYpf.YPMU8kvglnsVD7UdUjfthU.fL88pXGD LXXFOoWUOAl06Vyr8cMGqWoH9aTSpr7BmGtvWTU7w4Z_U_Ai0CAObR51iPwT4slwjmbRHst.vwac yKC3IZoKuLFui2oYI1fdiBjAk9z.rtaBaLVjT7OR38SE7HJOCxCG_CMcB8Td.AARjZxj7l1S79Ot ciTVOg6xySj5cNq3fQvUXRggsU5AG8U5OmKz6KfS3qUQIzMuzwonRwUSB_dHtsKAyMWcSt5f1nBp 7sZ0T19os.GFqLkSmQbbDXKeKFLO13UtWQHEreoPW_ZDnDL9UnJ0Mba9uTu26E4wVGDqtagV52tQ s8VsNV0VLej2ISum1AMkBZSMdCYz_vBw6IltGJk0puzOyiblXVhn5I8WyWllgE1RileXEhUjbMuK dvP45LHE1ChrPbSXWyRpIBNkqmpKUZco83BwXUaTJmtksHC4DchQ7h4JFZqeWK9WoF5a7MiVYW0s LlJ_Ov54Q9nLSzZFgXTP5iAD4W4KtYIyolYDjkSBhqxotIY2i03RxPNDqJBHnI86Jepp9lyk0a5k b3xxYVEJ0fPg.o0ILqXANdpq2hMVUKAojQZxthATbCyUNEw_0jQY3S6DNBtrTR08nbynZ.gDhXc. XXZdgQwtOlr4EeOT1TxyJROSIIRELaKQSIuL4p4SMSpazgeSKBXhOkprKZi.UJJQD9BprLa7bqOP zxGyZxjAONZf6EQH51EvguF0u2wPsDIJeeRuyCFsz1.EPpmUO6BFN9p2Xw5OLIemj4iwU Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Wed, 6 May 2020 23:58:03 +0000 Received: by smtp408.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 275644a7f42f203160b12da5531da6aa; Wed, 06 May 2020 23:57:58 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: svn commit: r360233 - in head: contrib/jemalloc . . . : This partially breaks a 2-socket 32-bit powerpc (old PowerMac G4) based on head -r360311 From: Mark Millard In-Reply-To: <121B9B09-141B-4DC3-918B-1E7CFB99E779@yahoo.com> Date: Wed, 6 May 2020 16:57:55 -0700 Cc: Brandon Bergren Content-Transfer-Encoding: quoted-printable Message-Id: <8AAB0462-3FA8-490C-8D8D-7C15B1C9E2DE@yahoo.com> References: <8479DD58-44F6-446A-9CA5-D01F0F7C1B38@yahoo.com> <17ACDA02-D7EF-4F26-874A-BB3E935CD072@yahoo.com> <695E6836-F860-4557-B7DE-CC1EDB347F18@yahoo.com> <121B9B09-141B-4DC3-918B-1E7CFB99E779@yahoo.com> To: "vangyzen@freebsd.org" , svn-src-head@freebsd.org, FreeBSD Current , FreeBSD Hackers , FreeBSD PowerPC ML X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49HYSX446kz4PGv X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.05 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; RCPT_COUNT_FIVE(0.00)[6]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.65)[-0.648,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.90)[-0.905,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (5.24), ipnet: 98.137.64.0/21(0.83), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[84.68.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[84.68.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 May 2020 23:58:06 -0000 [This explores process crashes that happen during system shutdown, in a context not having MALLOC_PRODUCTION=3D . So assert failures are reported as the stopping points.] It looks like shutdown -p now, shutdown -r now, and the like can lead some processes to assert during their exit attempt, including a sshd failure (that I've not seen before), rpcbind, and nfsd. I show information about the observed asserts for those below. sshd hit an assert, failing slab =3D=3D extent_slab_get(extent) : (gdb) bt=20 #0 thr_kill () at thr_kill.S:4 #1 0x50927170 in __raise (s=3D6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x50886cc0 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 #3 0x508834b0 in arena_dalloc (tsdn=3D, ptr=3D, tcache=3D, alloc_ctx=3D, = slow_path=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:315 #4 idalloctm (tsdn=3D0x500dd040, ptr=3D0x5008a180, tcache=3D0x500dd160, = alloc_ctx=3D, is_internal=3D, = slow_path=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_inli= nes_c.h:118 #5 0x5087b0a4 in ifree (tsd=3D0x500dd040, ptr=3D0x5008a180, = tcache=3D0x500dd160, slow_path=3D) at = jemalloc_jemalloc.c:2590 #6 0x5087acac in __je_free_default (ptr=3D0x5008a180) at = jemalloc_jemalloc.c:2784 #7 0x5087b294 in __free (ptr=3D0x5008a180) at jemalloc_jemalloc.c:2852 #8 0x10029464 in server_accept_loop (config_s=3D, = sock_in=3D, sock_out=3D, = newsock=3D) at /usr/src/crypto/openssh/sshd.c:1185 #9 main (ac=3D, av=3D0xffffde3c) at = /usr/src/crypto/openssh/sshd.c:2009 . . . (gdb) up #2 0x50886cc0 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 67 (void)raise(SIGABRT); (gdb) up #3 0x508834b0 in arena_dalloc (tsdn=3D, ptr=3D, tcache=3D, alloc_ctx=3D, = slow_path=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_inlines_b.h:315 315 assert(slab =3D=3D extent_slab_get(extent)); (gdb) list 310 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); 311 extent_t *extent =3D rtree_extent_read(tsdn, = &extents_rtree, 312 rtree_ctx, (uintptr_t)ptr, true); 313 assert(szind =3D=3D extent_szind_get(extent)); 314 assert(szind < SC_NSIZES); 315 assert(slab =3D=3D extent_slab_get(extent)); 316 } 317=09 318 if (likely(slab)) { 319 /* Small allocation. */ More fully: 285 JEMALLOC_ALWAYS_INLINE void 286 arena_dalloc(tsdn_t *tsdn, void *ptr, tcache_t *tcache, 287 alloc_ctx_t *alloc_ctx, bool slow_path) { 288 assert(!tsdn_null(tsdn) || tcache =3D=3D NULL); 289 assert(ptr !=3D NULL); 290=09 291 if (unlikely(tcache =3D=3D NULL)) { 292 arena_dalloc_no_tcache(tsdn, ptr); 293 return; 294 } 295=09 296 szind_t szind; 297 bool slab; 298 rtree_ctx_t *rtree_ctx; 299 if (alloc_ctx !=3D NULL) { 300 szind =3D alloc_ctx->szind; 301 slab =3D alloc_ctx->slab; 302 assert(szind !=3D SC_NSIZES); 303 } else { 304 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); 305 rtree_szind_slab_read(tsdn, &extents_rtree, = rtree_ctx, 306 (uintptr_t)ptr, true, &szind, &slab); 307 } 308=09 309 if (config_debug) { 310 rtree_ctx =3D tsd_rtree_ctx(tsdn_tsd(tsdn)); 311 extent_t *extent =3D rtree_extent_read(tsdn, = &extents_rtree, 312 rtree_ctx, (uintptr_t)ptr, true); 313 assert(szind =3D=3D extent_szind_get(extent)); 314 assert(szind < SC_NSIZES); 315 assert(slab =3D=3D extent_slab_get(extent)); 316 } 317=09 318 if (likely(slab)) { 319 /* Small allocation. */ 320 tcache_dalloc_small(tsdn_tsd(tsdn), tcache, ptr, = szind, 321 slow_path); 322 } else { 323 arena_dalloc_large(tsdn, ptr, tcache, szind, = slow_path); 324 } 325 } rpcbind hit an assert, failing ret =3D=3D sz_size2index_compute(size) : (gdb) bt #0 thr_kill () at thr_kill.S:4 #1 0x502f2170 in __raise (s=3D6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x50251d04 in abort () at /usr/src/lib/libc/stdlib/abort.c:79 #3 0x5024f260 in sz_size2index_lookup (size=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/sz.h:159 #4 sz_size2index (size=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/sz.h:166 #5 imalloc_body (sopts=3D0xffffb360, dopts=3D0xffffb340, = tsd=3D0x5009a018) at jemalloc_jemalloc.c:2066 #6 0x50244874 in imalloc (sopts=3D0xffffb360, dopts=3D0xffffb340) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/tsd.h:331 #7 0x50244fe8 in __calloc (num=3D1, size=3D96) at = jemalloc_jemalloc.c:2498 #8 0x50265690 in svc_xprt_alloc () at /usr/src/lib/libc/rpc/svc.c:541 #9 0x502635f4 in makefd_xprt (fd=3D14, sendsize=3D9000, recvsize=3D9000) = at /usr/src/lib/libc/rpc/svc_vc.c:250 #10 0x502644b4 in rendezvous_request (xprt=3D0x5004c000, msg=3D) at /usr/src/lib/libc/rpc/svc_vc.c:315 #11 0x50265a98 in svc_getreq_common (fd=3D) at = /usr/src/lib/libc/rpc/svc.c:640 #12 0x50265d1c in svc_getreq_poll (pfdp=3D, pollretval=3D1)= at /usr/src/lib/libc/rpc/svc.c:739 #13 0x10018568 in my_svc_run () at = /usr/src/usr.sbin/rpcbind/rpcb_svc_com.c:1167 #14 0x10014ad8 in main (argc=3D, argv=3D) = at /usr/src/usr.sbin/rpcbind/rpcbind.c:250 (gdb) up 3 #3 0x5024f260 in sz_size2index_lookup (size=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/sz.h:159 159 assert(ret =3D=3D sz_size2index_compute(size)); (gdb) print ret $1 =3D 0 154 JEMALLOC_ALWAYS_INLINE szind_t 155 sz_size2index_lookup(size_t size) { 156 assert(size <=3D SC_LOOKUP_MAXCLASS); 157 szind_t ret =3D (sz_size2index_tab[(size + (ZU(1) << = SC_LG_TINY_MIN) - 1) 158 >> SC_LG_TINY_MIN]); 159 assert(ret =3D=3D sz_size2index_compute(size)); 160 return ret; 161 } nfsd hit an assert, failing ret =3D=3D sz_size2index_compute(size) (also, but a different caller of sz_size2index): (gdb) bt #0 thr_kill () at thr_kill.S:4 #1 0x502b2170 in __raise (s=3D6) at /usr/src/lib/libc/gen/raise.c:52 #2 0x50211cc0 in abort () at /usr/src/lib/libc/stdlib/abort.c:67 #3 0x50206104 in sz_index2size_lookup (index=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/sz.h:200 #4 sz_index2size (index=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/sz.h:207 #5 ifree (tsd=3D0x50094018, ptr=3D0x50041028, tcache=3D0x50094138, = slow_path=3D) at jemalloc_jemalloc.c:2583 #6 0x50205cac in __je_free_default (ptr=3D0x50041028) at = jemalloc_jemalloc.c:2784 #7 0x50206294 in __free (ptr=3D0x50041028) at jemalloc_jemalloc.c:2852 #8 0x50287ec8 in ns_src_free (src=3D0x50329004, srclistsize=3D) at /usr/src/lib/libc/net/nsdispatch.c:452 #9 ns_dbt_free (dbt=3D0x50329000) at = /usr/src/lib/libc/net/nsdispatch.c:436 #10 vector_free (vec=3D0x50329000, count=3D, esize=3D12, = free_elem=3D) at /usr/src/lib/libc/net/nsdispatch.c:253 #11 nss_atexit () at /usr/src/lib/libc/net/nsdispatch.c:578 #12 0x5028d958 in __cxa_finalize (dso=3D0x0) at = /usr/src/lib/libc/stdlib/atexit.c:240 #13 0x502117f8 in exit (status=3D0) at = /usr/src/lib/libc/stdlib/exit.c:74 #14 0x10013f9c in child_cleanup (signo=3D) at = /usr/src/usr.sbin/nfsd/nfsd.c:969 #15 #16 0x00000000 in ?? () (gdb) up 3 #3 0x50206104 in sz_index2size_lookup (index=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/sz.h:200 200 assert(ret =3D=3D sz_index2size_compute(index)); (ret is optimized out.) 197 JEMALLOC_ALWAYS_INLINE size_t 198 sz_index2size_lookup(szind_t index) { 199 size_t ret =3D (size_t)sz_index2size_tab[index]; 200 assert(ret =3D=3D sz_index2size_compute(index)); 201 return ret; 202 } Booting and immediately trying something like: service nfsd stop did not lead to a failure. But may be after a while it would and be less drastic than a reboot or power down. =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)