From owner-freebsd-hackers@freebsd.org Sun May 3 21:32:41 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DFA802BA1E0 for ; Sun, 3 May 2020 21:32:41 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49FfN849qmz4LRl for ; Sun, 3 May 2020 21:32:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: KnuII7sVM1nfAwvNbPtHtam40Uu8y6WCL4Z47wcV3hlZx0ZkXFkRJEK_ULFEmhU jrMOQc60Mr8XZKuajjcE0dqxRsQ.D0kEZy..Gmsxc5moHHHqV2s0C7L2AsTdRg7B0LQqK.511ONo IeMhZBvN1WbypwXPg4xQk8O2ZRbdMWqToHSWR9GRc3QPnLzih5xK7PR461Tf2yFrPMYC0QtwoN71 ZPP1HlMjSF7RUOSczLZhoEULqHuAxcYpF9RhwmWV4f4HDAPsCMH0NIJh370QSXAX6R6RieThPXLf ElFXInU86S3gyxVJVUyeNvv.PZQdpcJjQYO6FUwIg6DqdjEvxCNZpckGlmUOi9jbaOuCrIMknfk2 Z_sYORL8iSmhLPGpN_NHeTWknpmtPhWyqn40_9yGbrKrkSFMH5k53v0MGTrtcTIvN6S9EeMxvNPG e_MEPNfC1XAWDzG89UTh8l42xljp.wgoObTmMBespduGvoWGOVGDt8iu.epn74aCCLmq5Ld7zzm8 teytPIdS2V4cvhc0w2kUbrKYtbQdkW6GX0IzN.w3cI0LMunf9l8nZEdV6M8BJZn8mAtvm9QdoSmJ smV6YmEZxlVfPyKfiMx2pEERTXpKqOSeJCQjgDb1h_XnngBJ6G2ijO4AfxLUCJPJLhj52KTfDIxy HXGsFhZR57EaR0GC9DH6fUmfjs6Tdqw1iQKW6IbpOAV2pgq08EkxvxHOwOfkmmaajl8HNO31ldes Y2yPsWYPY08Hg5aFK3qeEsnF5MWWQhHqLr6WDxoziG_j7KR9T8_iVh2XgOYg2HteIT0XVtN.VpTW M6aTftNwBOGWsyDqhxOrYiEDanoO1D45fGEEUi5Ka9iGCtKAVDeYYA4phffIxmNBwpkFdBEJ9b4K QHp5Rb.C9HMnq7m7N1Ws__bPVDBHkPvHnVgxh67ZWYZr8a7HynVkZh6mm0kfl1HNH0WXFnlwERkx uZHDL2vVj3STIr36P5bEZ4cE.HKv9QSRS7i7RouW.UXehDJUBuoA9B7nVK1HU4.tWudtjKp49YNX bC_DyAqdbI1iZd9VZL8NmGP6V._Q2TSuzisY1CtTvnXby0VBhn1OEpHL_qIVKpBy.v3YM915ncA8 JhSKid05LjD.RZV_S_x.lJRqGT3hh7ULAhCJHZxa.mCDkEXjJK6RizZgoCyMUTuBhohnBa7mSZSD _PXv7_A6NiOprGs2pPx.Fet8XwH61i_vQqS3Iv0rXIhcTo3NFIdK4guEoNB6ZcSIeuzanv2XtIT8 48Gwhk5a9imkhIh4Q_EeHGSu6Ll1.kkAfZEjKYcxxAU4x.wZGgt2Wfq4IKNPE2ZzXQJY.R9V1_dM uTFKAWEsXj9zUjKD0Mjo5bwCl4MUJXZicVpR1AAKxqDwsTULRYCtyZAyx1A08Wxyja3YbOtPJCl. 2GKcTXrYvnvLcJSxBeCj15eEWNhZVRZQiCXW01CQIEOEsalzueEbqsgAhNKPY8W8meOUPJg-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Sun, 3 May 2020 21:32:37 +0000 Received: by smtp421.mail.bf1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 0baeaae5442914f50439d86f6563f26f; Sun, 03 May 2020 21:32:36 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: svn commit: r360233 - in head: contrib/jemalloc . . . : This partially breaks a 2-socket 32-bit powerpc (old PowerMac G4) based on head -r360311 From: Mark Millard In-Reply-To: <8479DD58-44F6-446A-9CA5-D01F0F7C1B38@yahoo.com> Date: Sun, 3 May 2020 14:32:34 -0700 Cc: Brandon Bergren Content-Transfer-Encoding: quoted-printable Message-Id: <17ACDA02-D7EF-4F26-874A-BB3E935CD072@yahoo.com> References: <8479DD58-44F6-446A-9CA5-D01F0F7C1B38@yahoo.com> To: "vangyzen@freebsd.org" , svn-src-head@freebsd.org, FreeBSD Current , FreeBSD Hackers , FreeBSD PowerPC ML X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49FfN849qmz4LRl X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.31 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; RCPT_COUNT_FIVE(0.00)[6]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.89)[-0.887,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.92)[-0.922,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (5.59), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[84.68.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[84.68.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 May 2020 21:32:41 -0000 [The bit argument ot bitmap_unset seems to be way too large.] On 2020-May-3, at 11:08, Mark Millard wrote: > [At around 4AM local time dhcient got a signal 11, > despite the jemalloc revert. The other exmaples > have not happened.] >=20 > On 2020-May-2, at 18:46, Mark Millard wrote: >=20 >> [I'm only claiming the new jemalloc is involved and that >> reverting avoids the problem.] >>=20 >> I've been reporting to some lists problems with: >>=20 >> dhclient >> sendmail >> rpcbind >> mountd >> nfsd >>=20 >> getting SIGSEGV (signal 11) crashes and some core >> dumps on the old 2-socket (1 core per socket) 32-bit >> PowerMac G4 running head -r360311. >>=20 >> Mika=C3=ABl Urankar sent a note suggesting that I try >> testing reverting head -r360233 for my head -r360311 >> context. He got it right . . . >>=20 >>=20 >> Context: >>=20 >> The problem was noticed by an inability to have >> other machines do a: >>=20 >> mount -onoatime,soft OLDPOWERMAC-LOCAL-IP:/... /mnt >>=20 >> sort of operation and to have succeed. By contrast, on >> the old PowerMac G4 I could initiate mounts against >> other machines just fine. >>=20 >> I do not see any such problems on any of (all based >> on head -r360311): >>=20 >> powerpc64 (old PowerMac G5 2-sockets with 2 cores each) >> armv7 (OrangePi+ 2ed) >> aarch64 (Rock64, RPi4, RPi3, >> OverDrive 1000, >> Macchiatobin Double Shot) >> amd64 (ThreadRipper 1950X) >>=20 >> So I expect something 32-bit powerpc specific >> is somehow involved, even if jemalloc is only >> using whatever it is. >>=20 >> (A kyua run with a debug kernel did not find other >> unexpected signal 11 sources on the 32-bit PowerMac >> compared to past kyua runs, at least that I noticed. >> There were a few lock order reversals that I do not >> know if they are expected or known-safe or not. >> I've reported those reversals to the lists as well.) >>=20 >>=20 >> Recent experiments based on the suggestion: >>=20 >> Doing the buildworld, buildkernel and installing just >> the new kernel and rebooting made no difference. >>=20 >> But then installing the new world and rebooting did >> make things work again: I no longer get core files >> for the likes of (old cores from before the update): >>=20 >> # find / -name "*.core" -print >> /var/spool/clientmqueue/sendmail.core >> /rpcbind.core >> /mountd.core >> /nfsd.core >>=20 >> Nor do I see the various notices for sendmail >> signal 11's that did not leave behind a core file >> --or for dhclient (no core file left behind). >> And I can mount the old PowerMac's drive from >> other machines just fine. >>=20 >>=20 >> Other notes: >>=20 >> I do not actively use sendmail but it was left >> to do its default things, partially to test if >> such default things are working. Unfortunately, >> PowerMacs have a problematical status under >> FreeBSD and my context has my historical >> experiments with avoiding various problems. >=20 > Looking, I see that I got a: >=20 > pid 572 (dhclient), jid 0, uid 0: exited on signal 11 (core dumped) >=20 > notice under the reverted build. No instances > of the other examples. This is the first that a > dhclient example has produced a .core file. >=20 > gdb indicates 0x5180936c for r7 in: >=20 > lwz r8,36(r7) >=20 > as leading to the failure. This was in > arena_dalloc_bin_locked_impl (where > arena_slab_reg_dalloc and bitmap_unset > were apparently inlined). >=20 > The chain for the example seems to be: > fork_privchld -> dispatch_imsg -> jemalloc >=20 > For reference . . . >=20 > # gdb dhclient /dhclient.core=20 > GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] > Copyright (C) 2020 Free Software Foundation, Inc. > License GPLv3+: GNU GPL version 3 or later = > . . . > Reading symbols from dhclient... > Reading symbols from /usr/lib/debug//sbin/dhclient.debug... > [New LWP 100089] > Core was generated by `dhclient: gem0 [priv]'. > Program terminated with signal SIGSEGV, Segmentation fault. > #0 bitmap_unset (bitmap=3D0x50407164, binfo=3D, = bit=3D167842154) at = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/bitmap.h:341= > 341 = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/bitmap.h: = No such file or directory. > (gdb) bt -full > #0 bitmap_unset (bitmap=3D0x50407164, binfo=3D, = bit=3D167842154) at = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/bitmap.h:341= > goff =3D > gp =3D 0x51809390 > propagate =3D > g =3D > i =3D > #1 arena_slab_reg_dalloc (slab=3D0x50407140, slab_data=3D0x50407164, = ptr=3D0x50088b50) at jemalloc_arena.c:273 > bin_info =3D > binind =3D 0 > regind =3D 167842154 > #2 arena_dalloc_bin_locked_impl (tsdn=3D0x5009f018, arena=3D, slab=3D, ptr=3D, junked=3D) at jemalloc_arena.c:1540 > slab_data =3D > binind =3D > bin_info =3D > bin =3D > nfree =3D > #3 0x502916a8 in __je_arena_dalloc_bin_junked_locked (tsdn=3D, arena=3D, extent=3D, ptr=3D) at jemalloc_arena.c:1559 > No locals. > #4 0x50250d2c in __je_tcache_bin_flush_small (tsd=3D0x5009f018, = tcache=3D, tbin=3D0x5009f1c0, binind=3D, = rem=3D24) at jemalloc_tcache.c:149 > ptr =3D > i =3D 0 > extent =3D 0x50407140 > bin_arena =3D 0x50400380 > bin =3D > ndeferred =3D 0 > merged_stats =3D > arena =3D 0x50400380 > nflush =3D 75 > __vla_expr0 =3D > item_extent =3D 0xffffd1f0 > #5 0x502508a0 in __je_tcache_event_hard (tsd=3D, = tcache=3D0x5009f108) at jemalloc_tcache.c:54 > tbin_info =3D > binind =3D 7 > tbin =3D 0x5009f1c0 > #6 0x5029a684 in __free (ptr=3D0x500530c0) at = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/rtree.h:374 > tcache =3D 0x5009f108 > tsd =3D > log_var =3D > log_var =3D > #7 0x10025994 in dispatch_imsg (ifix=3D, fd=3D10) at = /usr/powerpc32_src/sbin/dhclient/privsep.c:215 > hdr =3D {code =3D IMSG_SCRIPT_WRITE_PARAMS, len =3D 3225} > lease =3D {next =3D 0x0, expiry =3D 1588504529, renewal =3D = 1588504229, rebind =3D 1588504454, address =3D {len =3D 4, iabuf =3D = "\300\250\001i", '\000' }, nextserver =3D {len =3D 4,=20= > iabuf =3D '\000' }, server_name =3D 0x0, = filename =3D 0x0, medium =3D 0x0, is_static =3D 0, is_bootp =3D 0, = options =3D {{len =3D 0, data =3D 0x0}, {len =3D 4,=20 > data =3D 0x500530c8 "\377\377\377"}, {len =3D 0, data =3D = 0x0}, {len =3D 4, data =3D 0x500530d0 "\300\250\001\001"}, {len =3D 0, = data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D 4,=20 > data =3D 0x500530d8 "\300\250\001\001"}, {len =3D 0, data = =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D = 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, = { > len =3D 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len = =3D 20, data =3D 0x50055200 "hsd1.or.comcast.net."}, {len =3D 0, data =3D = 0x0} , {len =3D 4, data =3D 0x500530e0 ""}, {len =3D = 0,=20 > data =3D 0x0}, {len =3D 1, data =3D 0x500530e8 "\005"}, = {len =3D 4, data =3D 0x500530f0 "\300\250\001\001"}, {len =3D 0, data =3D = 0x0} }} > medium_len =3D > medium =3D > totlen =3D 3225 > filename_len =3D > filename =3D 0x0 > ret =3D > buf =3D > mtu =3D > servername_len =3D > servername =3D 0x0 > reason_len =3D > reason =3D > --Type for more, q to quit, c to continue without paging-- > prefix_len =3D > prefix =3D 0x500530c0 "new_" > i =3D 0 > optlen =3D 0 > #8 0x100189f4 in fork_privchld (fd=3D10, fd2=3D) at = /usr/powerpc32_src/sbin/dhclient/dhclient.c:2847 > pfd =3D {{fd =3D 10, events =3D 1, revents =3D 1}} > nfds =3D > #9 0x10017a80 in main (argc=3D, argv=3D) at /usr/powerpc32_src/sbin/dhclient/dhclient.c:505 > pipe_fd =3D {10, 11} > rights =3D {cr_rights =3D {1342801412, 18446706484155777024}} > immediate_daemon =3D 0 > i =3D 0 > ch =3D > otherpid =3D 8 > pw =3D 0x5039b9d8 > fd =3D > capmode =3D >=20 > (gdb) disass > Dump of assembler code for function arena_dalloc_bin_locked_impl: > 0x502916b8 <+0>: mflr r0 > 0x502916bc <+4>: stw r0,4(r1) > 0x502916c0 <+8>: stwu r1,-48(r1) > 0x502916c4 <+12>: stw r30,40(r1) > 0x502916c8 <+16>: stw r24,16(r1) > 0x502916cc <+20>: stw r25,20(r1) > 0x502916d0 <+24>: stw r26,24(r1) > 0x502916d4 <+28>: stw r27,28(r1) > 0x502916d8 <+32>: stw r28,32(r1) > 0x502916dc <+36>: stw r29,36(r1) > 0x502916e0 <+40>: bl 0x502916e4 = > 0x502916e4 <+44>: mr r27,r3 > 0x502916e8 <+48>: mflr r30 > 0x502916ec <+52>: addis r30,r30,14 > 0x502916f0 <+56>: addi r30,r30,7788 > 0x502916f4 <+60>: mr r28,r4 > 0x502916f8 <+64>: lwz r4,5856(r30) > 0x502916fc <+68>: lwz r3,4(r5) > 0x50291700 <+72>: mr r29,r5 > 0x50291704 <+76>: andi. r5,r7,1 > 0x50291708 <+80>: mr r26,r6 > 0x5029170c <+84>: lbz r4,0(r4) > 0x50291710 <+88>: rlwinm r5,r3,14,25,31 > 0x50291714 <+92>: mulli r24,r5,224 > 0x50291718 <+96>: mulli r25,r5,44 > 0x5029171c <+100>: cmpwi cr1,r4,0 > 0x50291720 <+104>: cror 4*cr5+lt,4*cr1+eq,gt > 0x50291724 <+108>: bge cr5,0x50291a2c = > 0x50291728 <+112>: lwz r4,0(r29) > 0x5029172c <+116>: lwz r6,6036(r30) > 0x50291730 <+120>: lwz r7,8(r29) > 0x50291734 <+124>: rlwinm r8,r5,2,0,29 > 0x50291738 <+128>: li r9,1 > 0x5029173c <+132>: add r24,r28,r24 > 0x50291740 <+136>: lwzx r6,r6,r8 > 0x50291744 <+140>: subf r7,r7,r26 > 0x50291748 <+144>: mulhwu r6,r6,r7 > 0x5029174c <+148>: rlwinm r7,r6,29,3,29 > 0x50291750 <+152>: add r7,r29,r7 > =3D> 0x50291754 <+156>: lwz r8,36(r7) > 0x50291758 <+160>: clrlwi r10,r6,27 > 0x5029175c <+164>: slw r9,r9,r10 > 0x50291760 <+168>: xor r9,r9,r8 > 0x50291764 <+172>: cmplwi r8,0 > 0x50291768 <+176>: stw r9,36(r7) > 0x5029176c <+180>: bne 0x502917e4 = > 0x50291770 <+184>: lwz r7,4408(r30) > 0x50291774 <+188>: mulli r8,r5,44 > 0x50291778 <+192>: add r5,r7,r8 > 0x5029177c <+196>: lwz r5,16(r5) > 0x50291780 <+200>: cmplwi r5,2 > 0x50291784 <+204>: blt 0x502917e4 = . . . >=20 > (gdb) info reg > r0 0x502916a8 1344870056 > r1 0xffffd1a0 4294955424 > r2 0x500a6018 1342857240 > r3 0x0 0 > r4 0x0 0 > r5 0x0 0 > r6 0xa01116a 167842154 > r7 0x5180936c 1367380844 > r8 0x0 0 > r9 0x1 1 > r10 0x1e 30 > r11 0x5005d114 1342558484 > r12 0x84000c00 2214595584 > r13 0x0 0 > r14 0xffffd1f0 4294955504 > r15 0xfffffffc 4294967292 > r16 0x4a 74 > r17 0x4b 75 > r18 0x0 0 > r19 0x504009a0 1346374048 > r20 0x0 0 > r21 0xffffd1f0 4294955504 > r22 0x620 1568 > r23 0x50400380 1346372480 > r24 0x50400380 1346372480 > r25 0x0 0 > r26 0x50088b50 1342737232 > r27 0x5009f018 1342828568 > r28 0x50400380 1346372480 > r29 0x50407140 1346400576 > r30 0x50373550 1345795408 > r31 0xffffd310 4294955792 > pc 0x50291754 0x50291754 = > msr > cr 0x42480c00 1112017920 > lr 0x502916e4 0x502916e4 = > ctr 0x5005d114 1342558484 > xer 0x0 0 > fpscr 0x0 0 > vscr > vrsave bitmap_unset (bitmap=3D0x50407164, binfo=3D, = bit=3D167842154) explains calculating: gp =3D 0x51809390 via bitmap+(bit/4/8): (gdb) print/x 0x50407164 +167842154/4/8=20 $16 =3D 0x51809390 The last potential bit/4/8 value to be able to access memory (without spanning a hole) is: (gdb) print *(bitmap+582566) $13 =3D 0 (gdb) print/x (bitmap+582566) $14 =3D 0x5063fffc So it looks like arena_slab_reg_dalloc produced an invalid bit value. Looking at that code shows that regind hold the parameter value that matches: static void arena_slab_reg_dalloc(extent_t *slab, arena_slab_data_t *slab_data, void = *ptr) { szind_t binind =3D extent_szind_get(slab); const bin_info_t *bin_info =3D &bin_infos[binind]; size_t regind =3D arena_slab_regind(slab, binind, ptr); =20 assert(extent_nfree_get(slab) < bin_info->nregs); /* Freeing an unallocated pointer can cause assertion failure. = */ assert(bitmap_get(slab_data->bitmap, &bin_info->bitmap_info, = regind)); bitmap_unset(slab_data->bitmap, &bin_info->bitmap_info, regind); extent_nfree_inc(slab); } The backtrace showed binind=3D=3D0 for arena_slab_reg_dalloc. That leaves: arena_slab_regind(slab, binind, ptr) as producing the odd value. size_t arena_slab_regind(extent_t *slab, szind_t binind, const void *ptr) { size_t diff, regind; /* Freeing a pointer outside the slab can cause assertion = failure. */ assert((uintptr_t)ptr >=3D (uintptr_t)extent_addr_get(slab)); assert((uintptr_t)ptr < (uintptr_t)extent_past_get(slab)); /* Freeing an interior pointer can cause assertion failure. */ assert(((uintptr_t)ptr - (uintptr_t)extent_addr_get(slab)) % (uintptr_t)bin_infos[binind].reg_size =3D=3D 0); diff =3D (size_t)((uintptr_t)ptr - = (uintptr_t)extent_addr_get(slab)); /* Avoid doing division with a variable divisor. */ regind =3D div_compute(&arena_binind_div_info[binind], diff); assert(regind < bin_infos[binind].nregs); return regind; } ptr =3D=3D 0x50088b50 slab =3D=3D 0x50407140 static inline void * extent_addr_get(const extent_t *extent) { assert(extent->e_addr =3D=3D PAGE_ADDR2BASE(extent->e_addr) || !extent_slab_get(extent)); return extent->e_addr; } (gdb) print *slab $17 =3D {e_bits =3D 0, e_addr =3D 0x0, {e_size_esn =3D 0, e_bsize =3D = 0}, ql_link =3D {qre_next =3D 0x0, qre_prev =3D 0x0}, ph_link =3D = {phn_prev =3D 0x0, phn_next =3D 0x0, phn_lchild =3D 0x0}, {e_slab_data =3D= {bitmap =3D { 0 }}, e_prof_tctx =3D {repr =3D 0x0}}} That looks wrong: all fields are zero, which is not likely to be the description of a slab. But I'll continue to be sure I get the reported value of bit. So extent_addr_get(slab)=3D=3Dslab->e_addr and slab->e_addr=3D=3D0x0 and diff=3D=3Dptr . (gdb) print/x arena_binind_div_info[binind] $19 =3D {magic =3D 0x20000000} static inline size_t div_compute(div_info_t *div_info, size_t n) { assert(n <=3D (uint32_t)-1); /* * This generates, e.g. mov; imul; shr on x86-64. On a 32-bit = machine, * the compilers I tried were all smart enough to turn this into = the * appropriate "get the high 32 bits of the result of a = multiply" (e.g. * mul; mov edx eax; on x86, umull on arm, etc.). */ size_t i =3D ((uint64_t)n * (uint64_t)div_info->magic) >> 32; #ifdef JEMALLOC_DEBUG assert(i * div_info->d =3D=3D n); #endif return i; } (gdb) print/x ((unsigned long long)0x50088b50 * (unsigned long = long)0x20000000) >> 32 $21 =3D 0xa01116a (gdb) print ((unsigned long long)0x50088b50 * (unsigned long = long)0x20000000) >> 32 $22 =3D 167842154 (As reported.) So returning to *slab being all zero . . . The slab value in the call chain seems to trace back to=3D __je_tcache_bin_flush_small code: bin_t *bin =3D &bin_arena->bins[binind]; . . . malloc_mutex_lock(tsd_tsdn(tsd), &bin->lock); . . . for (unsigned i =3D 0; i < nflush; i++) { void *ptr =3D *(tbin->avail - 1 - i); extent =3D item_extent[i]; assert(ptr !=3D NULL && extent !=3D NULL); if (extent_arena_get(extent) =3D=3D bin_arena) { = arena_dalloc_bin_junked_locked(tsd_tsdn(tsd), bin_arena, extent, ptr); . . . malloc_mutex_unlock(tsd_tsdn(tsd), &bin->lock); (So ptr's value here is later slab's value in the call chain.) The backtrace shows binind =3D 7 via __je_tcache_event_hard . (Not the same as the earlier binind.) #4 0x50250d2c in __je_tcache_bin_flush_small (tsd=3D0x5009f018, = tcache=3D, tbin=3D0x5009f1c0, binind=3D, = rem=3D24) at jemalloc_tcache.c:149 ptr =3D i =3D 0 extent =3D 0x50407140 bin_arena =3D 0x50400380 bin =3D ndeferred =3D 0 merged_stats =3D arena =3D 0x50400380 nflush =3D 75 __vla_expr0 =3D item_extent =3D 0xffffd1f0 (gdb) print/x bin_arena->bins[7] $44 =3D {lock =3D {{{prof_data =3D {tot_wait_time =3D {ns =3D 0x0}, = max_wait_time =3D {ns =3D 0x0}, n_wait_times =3D 0x0, n_spin_acquired =3D = 0x0, max_n_thds =3D 0x0, n_waiting_thds =3D {repr =3D 0x0},=20 n_owner_switches =3D 0x0, prev_owner =3D 0x0, n_lock_ops =3D = 0x0}, lock =3D 0x0, postponed_next =3D 0x504021d0}, witness =3D {name =3D = 0x0, rank =3D 0x0, comp =3D 0x0, opaque =3D 0x0, link =3D {qre_next =3D = 0x0,=20 qre_prev =3D 0x0}}, lock_order =3D 0x0}}, slabcur =3D = 0x50407140, slabs_nonfull =3D {ph_root =3D 0x0}, slabs_full =3D = {qlh_first =3D 0x0}, stats =3D {nmalloc =3D 0x64, ndalloc =3D 0x0, = nrequests =3D 0x1,=20 curregs =3D 0x64, nfills =3D 0x1, nflushes =3D 0x1, nslabs =3D 0x1, = reslabs =3D 0x0, curslabs =3D 0x1, mutex_data =3D {tot_wait_time =3D {ns = =3D 0x0}, max_wait_time =3D {ns =3D 0x0}, n_wait_times =3D 0x0,=20 n_spin_acquired =3D 0x0, max_n_thds =3D 0x0, n_waiting_thds =3D = {repr =3D 0x0}, n_owner_switches =3D 0x0, prev_owner =3D 0x0, n_lock_ops = =3D 0x0}}} That indicates: bin_arena->bins[7]->lock =3D 0x0 . Expected? Single threaded context? (gdb) print *item_extent[0] $27 =3D {e_bits =3D 0, e_addr =3D 0x0, {e_size_esn =3D 0, e_bsize =3D = 0}, ql_link =3D {qre_next =3D 0x0, qre_prev =3D 0x0}, ph_link =3D = {phn_prev =3D 0x0, phn_next =3D 0x0, phn_lchild =3D 0x0}, {e_slab_data =3D= {bitmap =3D { 0 }}, e_prof_tctx =3D {repr =3D 0x0}}} Other *item_extent[INDEX] that I tried got the same: all zeros. This is what contributed to the huge bit value. item_extent[] is based on the declaration: VARIABLE_ARRAY(extent_t *, item_extent, nflush); and: /* Declare a variable-length array. */ #if __STDC_VERSION__ < 199901L # ifdef _MSC_VER # include # define alloca _alloca # else # ifdef JEMALLOC_HAS_ALLOCA_H # include # else # include # endif # endif # define VARIABLE_ARRAY(type, name, count) \ type *name =3D alloca(sizeof(type) * (count)) #else # define VARIABLE_ARRAY(type, name, count) type name[(count)] #endif WARNING: C11 turned VLAs into a conditional feature (__STDC_NO_VLA__). Only C99 has it as required. Thus the above definition of VARIABLE_ARRAY is incomplete or limited to C99 and before relative the the language vintages. Looking around, the stack frames seem to span the space okay: (gdb) print/x &item_extent[75] $32 =3D 0xffffd31c (gdb) print/x &item_extent[0] $33 =3D 0xffffd1f0 r1 0xffffd1a0 4294955424 r14 0xffffd1f0 4294955504 r15 0xfffffffc 4294967292 r21 0xffffd1f0 4294955504 (gdb) print/x *(void**)0xffffd1a0 $36 =3D 0xffffd1d0 (gdb) print/x *(void**)0xffffd1d0 $37 =3D 0xffffd1e0 (gdb) print/x *(void**)0xffffd1e0 $38 =3D 0xffffd440 (gdb) print/x *(void**)0xffffd440 $39 =3D 0xffffd460 And I've run out of ideas for what else to look at (for now). (It is not like I understand jemalloc.) (Last I knew, 32-bit powerpc did not have red-zone stack-space criteria to leave room for signals to use.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)