From owner-freebsd-hackers@freebsd.org Fri May 1 03:10:30 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 128C82D0521 for ; Fri, 1 May 2020 03:10:30 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic308-8.consmr.mail.gq1.yahoo.com (sonic308-8.consmr.mail.gq1.yahoo.com [98.137.68.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49Cy1J56sVz4PXc for ; Fri, 1 May 2020 03:10:28 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: DI6KMC4VM1kxKVCSWhnCR67BZ28gPPvda9zxIxuIU_gDEdSGDbwVzm5AFVRdGGW .IsoFz3kUWRxhzHIBhg99b.491BhmrwH3Gw8Xe7CYbPTbQQsOZcyEymUkO_wqdb.ZIPE3yKaQnkQ pFlit9sd4gz8BtrLIdyth1WLkpY2F5RT_IvUs33Ir2HmdLJZLELZGzYeQ6eIFu3BEQ3lNQuVhhl2 0GBZBXEC82T_vxpOBUC4jwbZKvTDBMj9QQ3nrmSO.2cYawnQGnLzeA4_CK.OAQ2t2YWJxIIrCAne 9IZ_So3cYLiLHcUTPtrNzo8yN1tmQR.seb56IX6HZp9He2NL.RitMAGFXw4A8hVW3QCG_vZBo8Fc U8FUi9OLUJnQB7gHZ.XFneX_y_klfo4mYFZ34VJlHiH_LKMbLD6RrxHgynN6Vsj5rQhGt2WlBAkE .vXdnVBXaf9kf5ZsUrTT5IkIYazvvKYhFV9a4u4CvRm4Z0au10iZWxAYYEM.OnGq71rBwtQr3PRc DYI1fqsYWcZIroxp6cMlFgLRO9J9ks8z7iq9RZ6DHet8lTHf9i9f04rQWvafLAYEL.yC9gb4PX3z .9AGJZEX5abE2okQkJO5buoPPYYX5jnm5tn.LISimaa2uRuXahBfW3.xLvO748lhv6hqrL3a_OOg WELSgyCAuSTdLh8sL2vVZRY4YnYwKSKXkGP4PpkpcNWo65YXBHfJhb__d_rS8L.sGBl.exl5NuSO nYOCq_mbRG5gAvPYtVjY2.J0AcsgmktNmOypypVdPeZh7Ho30bXyyeFU6cAb_RCpCSzi73LjJaLH fQBPsUp.NeQn5FgCTLG_FAjpTci5.xdpNdBG1HwS1wcxTQGSbxLTlRlqsskF4QHNl.zWTvVzlglC ATASGHfz_Xq23tmRyA63AuQKPnBZtkt9u8gV.dSLXcy9Lj2OW.hyob.cTr7r9Dg1DmSy75WBrkS6 8I0oca1EI3yQtmgYMklRrLu3_yA9lPgKgQeDx6_qjIK.8PVkZy8g9BxKtIv5kTP79Nol23Q8CnAq bNgOoHZYr4.2KUh2CWboXvrUwozhf.Rgr519FX.9aKpANbh3OpAKtydi80gaHF1uRK8Uv5wNSbla Tov2hTZ9U5WupeR1P44gTUDxRyW1JRqX89uKKeGKQiSrsyT2mo9k5irAbrXyIj7dHJqy_BuZoxvN FnW_H2GiPUanqrNPAFobj_NhlSFsxqlBA4I.2_evgYQxHtxzvfbUZSL3mSN_9KuWyCmAcgHOuI0C N4Za.ADu54AG7Sflhd1AJfVJfE8iWIbXK8jKcGE8ur6EiEAnKNAGgj2co_wtEel5w73ftaUmZyvQ rHtNlyQuBSQ0acqgiHoAAeJRAdtTPIhylWU6CvA4mjvcGC0vwaOmcF4Ilvyyz5pA7sqWVm5ab_1r L6Pkoktxm32WuNJDQViTqbA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic308.consmr.mail.gq1.yahoo.com with HTTP; Fri, 1 May 2020 03:10:27 +0000 Received: by smtp431.mail.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 830c4bc3c4cb612231800e45fe4163bf; Fri, 01 May 2020 03:10:21 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: 32-bit powerpc head -r360311 has signal 11 process crashes (old PowerMac dual-socket, 1 core each) Date: Thu, 30 Apr 2020 20:10:20 -0700 References: <76D3A482-98BC-44C3-B84D-504A012CA8D8@yahoo.com> To: FreeBSD PowerPC ML , FreeBSD Hackers In-Reply-To: Message-Id: <7ACAC4BB-AFAB-4048-9F27-26B2115F183E@yahoo.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49Cy1J56sVz4PXc X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.46 / 15.00]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; SH_EMAIL_ZRD(0.00)[0.0.0.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.970,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; SH_EMAIL_DBL_DONT_QUERY_IPS(0.00)[0.0.0.0]; NEURAL_HAM_LONG(-0.99)[-0.985,0]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[32.68.137.98.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (0.42), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; RWL_MAILSPIKE_POSSIBLE(0.00)[32.68.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 May 2020 03:10:30 -0000 [Turns out one sendmail left a sendmail.core behind. rendezvous_request is not on the call stack.] > On 2020-Apr-30, at 19:41, Mark Millard wrote: >=20 > [After some time dhclient and sendmail did have the problem > despite the debug-kernel being in use.] >=20 > On 2020-Apr-29, at 21:29, Mark Millard wrote: >=20 >> [Notes on using the artifact.ci head -3260311 debug-kernel added.] >>=20 >> On 2020-Apr-29, at 02:27, Mark Millard wrote: >>=20 >>> Since updating from head -r359427 based to head -r360311 base >>> (my own non-debug builds), various things report segmentation >>> faults. It appears that rendezvous_request may always be >>> involved. I've not had time (yet) to deal with substituting a >>> debug kernel from: >>>=20 >>> = https://artifact.ci.freebsd.org/snapshot/head/r360311/powerpc/powerpc/ >>>=20 >>> I expect to do so at some point. >>>=20 >>>=20 >>> I give 2 examples below (mountd and rpcbind). Both involve >>> rendezvous_request and svc_getreq_common . >>>=20 >>> First mountd : >>>=20 >>> # gdb mountd /mountd.core=20 >>> GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] >>> Copyright (C) 2020 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later = >>> . . . >>> Reading symbols from mountd... >>> Reading symbols from /usr/lib/debug//usr/sbin/mountd.debug... >>> [New LWP 100105] >>> Core was generated by `/usr/sbin/mountd -r'. >>> Program terminated with signal SIGSEGV, Segmentation fault. >>> #0 atomic_load_u (a=3D, mo=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/atomic.h:70 >>>=20 >>> warning: Source file is more recent than executable. >>> 70 JEMALLOC_GENERATE_INT_ATOMICS(unsigned, u, LG_SIZEOF_INT) >>> (gdb) bt >>> #0 atomic_load_u (a=3D, mo=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/atomic.h:70 >>> #1 rtree_leaf_elm_szind_read (tsdn=3D, = rtree=3D, elm=3D, dependent=3D) at /usr/src/contrib/jemalloc/include/jemalloc/internal/rtree.h:230 >>> #2 rtree_szind_slab_read (tsdn=3D0x50094018, rtree=3D, rtree_ctx=3D0x50094044, key=3D1869636193, dependent=3Dtrue, = r_szind=3D, r_slab=3D) >>> at /usr/src/contrib/jemalloc/include/jemalloc/internal/rtree.h:504 >>> #3 ifree (tsd=3D, ptr=3D0x6f706261, = tcache=3D, slow_path=3Dfalse) at jemalloc_jemalloc.c:2574 >>> #4 __je_free_default (ptr=3D0x6f706261) at jemalloc_jemalloc.c:2775 >>> #5 0x50235db0 in __free (ptr=3D0x6f706261) at = jemalloc_jemalloc.c:2852 >>> #6 0x5026525c in freenetconfigent (netconfigp=3D0x50049170) at = /usr/src/lib/libc/rpc/getnetconfig.c:540 >>> #7 0x50260d8c in __rpc_sockinfo2netid (sip=3D, = netid=3D) at /usr/src/lib/libc/rpc/rpc_generic.c:573 >>> #8 0x502521f0 in makefd_xprt (fd=3D10, sendsize=3D9000, = recvsize=3D9000) at /usr/src/lib/libc/rpc/svc_vc.c:270 >>> #9 0x50252fa4 in rendezvous_request (xprt=3D0x5007b120, = msg=3D) at /usr/src/lib/libc/rpc/svc_vc.c:315 >>> #10 0x50254588 in svc_getreq_common (fd=3D) at = /usr/src/lib/libc/rpc/svc.c:640 >>> #11 0x502543d0 in svc_getreqset (readfds=3D) at = /usr/src/lib/libc/rpc/svc.c:611 >>> #12 0x1001434c in main (argc=3D, argv=3D0xffffde3c) = at /usr/src/usr.sbin/mountd/mountd.c:683 >>>=20 >>> (gdb) disass >>> Dump of assembler code for function __je_free_default: >>> 0x50235244 <+0>: mflr r0 >>> 0x50235248 <+4>: stw r0,4(r1) >>> 0x5023524c <+8>: stwu r1,-80(r1) >>> 0x50235250 <+12>: stw r30,72(r1) >>> 0x50235254 <+16>: stw r21,36(r1) >>> 0x50235258 <+20>: stw r22,40(r1) >>> 0x5023525c <+24>: stw r23,44(r1) >>> 0x50235260 <+28>: stw r24,48(r1) >>> 0x50235264 <+32>: stw r25,52(r1) >>> 0x50235268 <+36>: stw r26,56(r1) >>> 0x5023526c <+40>: stw r27,60(r1) >>> 0x50235270 <+44>: stw r28,64(r1) >>> 0x50235274 <+48>: stw r29,68(r1) >>> 0x50235278 <+52>: bl 0x5023527c <__je_free_default+56> >>> 0x5023527c <+56>: mr r28,r3 >>> 0x50235280 <+60>: mflr r30 >>> 0x50235284 <+64>: addis r30,r30,14 >>> 0x50235288 <+68>: addi r30,r30,-20816 >>> 0x5023528c <+72>: lwz r4,64(r30) >>> 0x50235290 <+76>: lwz r5,6188(r30) >>> 0x50235294 <+80>: lwz r4,0(r4) >>> 0x50235298 <+84>: stw r4,32(r1) >>> 0x5023529c <+88>: lbz r4,0(r5) >>> 0x502352a0 <+92>: cmplwi r4,0 >>> 0x502352a4 <+96>: bne 0x502353c4 <__je_free_default+384> >>> 0x502352a8 <+100>: cmplwi r28,0 >>> 0x502352ac <+104>: beq 0x50235378 <__je_free_default+308> >>> 0x502352b0 <+108>: addi r3,r30,4332 >>> 0x502352b4 <+112>: rlwinm r24,r28,0,0,9 >>> 0x502352b8 <+116>: bl 0x502f06b4 = <00000000.plt_pic32.__tls_get_addr> >>> 0x502352bc <+120>: rlwinm r26,r28,13,25,28 >>> 0x502352c0 <+124>: mr r29,r3 >>> 0x502352c4 <+128>: lbz r3,0(r3) >>> 0x502352c8 <+132>: cmplwi r3,0 >>> 0x502352cc <+136>: bne 0x502353fc <__je_free_default+440> >>> 0x502352d0 <+140>: add r25,r29,r26 >>> 0x502352d4 <+144>: lwz r3,44(r25) >>> 0x502352d8 <+148>: addi r27,r29,44 >>> 0x502352dc <+152>: cmplw r3,r24 >>> 0x502352e0 <+156>: bne 0x50235578 <__je_free_default+820> >>> 0x502352e4 <+160>: lwz r3,48(r25) >>> 0x502352e8 <+164>: rlwinm r4,r28,20,22,31 >>> 0x502352ec <+168>: mulli r4,r4,12 >>> 0x502352f0 <+172>: add r3,r3,r4 >>> =3D> 0x502352f4 <+176>: lwz r6,4(r3) >>> 0x502352f8 <+180>: lwz r5,4368(r30) >>> 0x502352fc <+184>: addi r26,r29,288 >>> 0x50235300 <+188>: lbz r4,8(r3) >>> 0x50235304 <+192>: rlwinm r3,r6,2,0,29 >>> 0x50235308 <+196>: lwz r7,28(r29) >>> 0x5023530c <+200>: lwzx r5,r5,r3 >>> 0x50235310 <+204>: lwz r8,24(r29) >>> 0x50235314 <+208>: andi. r4,r4,1 >>> 0x50235318 <+212>: addc r4,r7,r5 >>> 0x5023531c <+216>: addze r5,r8 >>> 0x50235320 <+220>: stw r5,24(r29) >>> 0x50235324 <+224>: stw r4,28(r29) >>> 0x50235328 <+228>: ble 0x50235534 <__je_free_default+752> >>> . . . >>>=20 >>> (gdb) info reg >>> r0 0x50235c04 1344494596 >>> r1 0xffffcfb0 4294954928 >>> r2 0x5009b018 1342812184 >>> r3 0x2448 9288 >>> r4 0x500940ac 1342783660 >>> r5 0x2448 9288 >>> r6 0x500940cc 1342783692 >>> r7 0x1 1 >>> r8 0x0 0 >>> r9 0x80808080 2155905152 >>> r10 0xc 12 >>> r11 0x502e3b50 1345207120 >>> r12 0x500491a0 1342476704 >>> r13 0x0 0 >>> r14 0x1 1 >>> r15 0x10040000 268697600 >>> r16 0x0 0 >>> r17 0x10040000 268697600 >>> r18 0x2 2 >>> r19 0x0 0 >>> r20 0x1 1 >>> r21 0x5007b164 1342681444 >>> r22 0xffffd2dc 4294955740 >>> r23 0x80 128 >>> r24 0x6f400000 1866465280 >>> r25 0x50094080 1342783616 >>> r26 0x68 104 >>> r27 0x50094044 1342783556 >>> r28 0x6f706261 1869636193 >>> r29 0x50094018 1342783512 >>> r30 0x5031012c 1345388844 >>> r31 0x10040000 268697600 >>> pc 0x502352f4 0x502352f4 = <__je_free_default+176> >>> msr >>> cr 0x242008a4 606079140 >>> lr 0x50235c04 0x50235c04 = <__je_free_default+2496> >>> ctr 0x0 0 >>> xer 0x0 0 >>> fpscr 0x0 0 >>> vscr >>> vrsave >>>=20 >>>=20 >>> Then rpcbind : >>>=20 >>> # gdb rpcbind /rpcbind.core=20 >>> GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] >>> Copyright (C) 2020 Free Software Foundation, Inc. >>> License GPLv3+: GNU GPL version 3 or later = >>> . . . >>> Reading symbols from rpcbind... >>> Reading symbols from /usr/lib/debug//usr/sbin/rpcbind.debug... >>> [New LWP 100098] >>> Core was generated by `/usr/sbin/rpcbind'. >>> Program terminated with signal SIGSEGV, Segmentation fault. >>> #0 __xdrrec_setnonblock (xdrs=3D0x500472d8, maxrec=3D9000) at = /usr/src/lib/libc/xdr/xdr_rec.c:607 >>> 607 rstrm->nonblock =3D TRUE; >>> (gdb) bt >>> #0 __xdrrec_setnonblock (xdrs=3D0x500472d8, maxrec=3D9000) at = /usr/src/lib/libc/xdr/xdr_rec.c:607 >>> #1 0x502440d4 in rendezvous_request (xprt=3D, = msg=3D) at /usr/src/lib/libc/rpc/svc_vc.c:348 >>> #2 0x50245588 in svc_getreq_common (fd=3D) at = /usr/src/lib/libc/rpc/svc.c:640 >>> #3 0x5024580c in svc_getreq_poll (pfdp=3D, = pollretval=3D1) at /usr/src/lib/libc/rpc/svc.c:739 >>> #4 0x10018568 in my_svc_run () at = /usr/src/usr.sbin/rpcbind/rpcb_svc_com.c:1167 >>> #5 0x10014ad8 in main (argc=3D, argv=3D) at /usr/src/usr.sbin/rpcbind/rpcbind.c:250 >>>=20 >>> (gdb) disass >>> Dump of assembler code for function __xdrrec_setnonblock: >>> 0x50250468 <+0>: lwz r5,12(r3) >>> 0x5025046c <+4>: li r6,1 >>> 0x50250470 <+8>: cmplwi r4,0 >>> =3D> 0x50250474 <+12>: stw r6,64(r5) >>> 0x50250478 <+16>: bne 0x50250480 <__xdrrec_setnonblock+24> >>> 0x5025047c <+20>: lwz r4,60(r5) >>> 0x50250480 <+24>: li r3,1 >>> 0x50250484 <+28>: stw r4,92(r5) >>> 0x50250488 <+32>: blr >>> End of assembler dump. >>>=20 >>> (gdb) info reg >>> r0 0x5c 92 >>> r1 0xffffb400 4294947840 >>> r2 0x500a1018 1342836760 >>> r3 0x500472d8 1342468824 >>> r4 0x2328 9000 >>> r5 0x2020 8224 >>> r6 0x1 1 >>> r7 0xffffb364 4294947684 >>> r8 0x500472f4 1342468852 >>> r9 0x0 0 >>> r10 0x20 32 >>> r11 0x502d8ea0 1345162912 >>> r12 0x24200880 606079104 >>> r13 0x0 0 >>> r14 0x0 0 >>> r15 0xffffbc28 4294949928 >>> r16 0x10002848 268445768 >>> r17 0x10040000 268697600 >>> r18 0x2 2 >>> r19 0x0 0 >>> r20 0x1 1 >>> r21 0x5004c044 1342488644 >>> r22 0xffffb63c 4294948412 >>> r23 0x80 128 >>> r24 0x50048010 1342472208 >>> r25 0x14 20 >>> r26 0xffffb630 4294948400 >>> r27 0x500472d0 1342468816 >>> r28 0xe 14 >>> r29 0x50047220 1342468640 >>> r30 0x5030112c 1345327404 >>> r31 0x10040000 268697600 >>> pc 0x50250474 0x50250474 = <__xdrrec_setnonblock+12> >>> msr >>> cr 0x44200080 1142947968 >>> lr 0x502440d4 0x502440d4 = >>> ctr 0x502d8ea0 1345162912 >>> xer 0x0 0 >>> fpscr 0x0 0 >>> vscr >>> vrsave >>>=20 >>>=20 >>> dhclient and sendmail have notices of signal 11's >>> but I do not find any .core files around for them. >>>=20 >>> Prior to this upgrade I was having no such problems >>> with the 32-bit powerpc PowerMac. >>=20 >> I substituted the debug-kernel from: >>=20 >> = https://artifact.ci.freebsd.org/snapshot/head/r360311/powerpc/powerpc/kern= el.txz >>=20 >> and with it there is no evidence so far of the >> problem(s) occurring. Since the same world build is >> in use in both contexts, it looks like the kernel >> is what makes the difference for the problem(s). >>=20 >> With the debug-kernel avoiding the problem, I've yet >> to figure out how to gather evidence. >>=20 >=20 > About 8.5 hours later than the boot completing > it got a: >=20 > kernel: pid 659 (dhclient), jid 0, uid 65: exited on signal 11 >=20 > About 7 hours later it started getting sendmail messaged > similar to: >=20 > kernel: pid 3722 (sendmail), jid 0, uid 25: exited on signal 11 >=20 > These seem to be 0.5 hours part. Between the 3rd and 4th > is a lock order reversal notice that I've separately > submitted to the lists (but that do not yet appear there, > last I checked). I found that there was a sendmail.core around: # gdb /usr/libexec/sendmail/sendmail = /var/spool/clientmqueue/sendmail.core=20 GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later = . . . Reading symbols from /usr/libexec/sendmail/sendmail... Reading symbols from = /usr/lib/debug//usr/libexec/sendmail/sendmail.debug... [New LWP 100163] Core was generated by `sendmail: running queue: = /var/spool/clientmqueue'. Program terminated with signal SIGSEGV, Segmentation fault. #0 bitmap_unset (bitmap=3D0x50a0af68, binfo=3D, = bit=3D167949476) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/bitmap.h:341 warning: Source file is more recent than executable. 341 g =3D *gp; (gdb) bt -full #0 bitmap_unset (bitmap=3D0x50a0af68, binfo=3D, = bit=3D167949476) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/bitmap.h:341 goff =3D gp =3D propagate =3D g =3D i =3D #1 arena_slab_reg_dalloc (slab=3D0x50a0af40, slab_data=3D0x50a0af68, = ptr=3D0x5015a520) at jemalloc_arena.c:368 bin_info =3D binind =3D regind =3D 167949476 #2 arena_dalloc_bin_locked_impl (tsdn=3D0x50144018, arena=3D, bin=3D, binind=3D, slab=3D, ptr=3D, junked=3D) at jemalloc_arena.c:1695 bin_info =3D slab_data =3D 0x50a0af68 nfree =3D #3 0x506a1148 in __je_arena_dalloc_bin_junked_locked (tsdn=3D, arena=3D, bin=3D, binind=3D, extent=3D, ptr=3D) at jemalloc_arena.c:1714 No locals. #4 0x50656080 in __je_tcache_bin_flush_small (tsd=3D0x50144018, = tcache=3D, tbin=3D0x501441d8, binind=3D6, rem=3D25) at = jemalloc_tcache.c:189 ptr =3D i =3D 0 extent =3D bin_arena_ind =3D 0 bin_arena =3D 0x50a00380 bin =3D 0x50a02e08 ndeferred =3D 0 binshard =3D 0 merged_stats =3D arena =3D 0x50a00380 nflush =3D 71 __vla_expr0 =3D item_extent =3D 0xffffb940 #5 0x50655bb8 in __je_tcache_event_hard (tsd=3D, = tcache=3D0x50144138) at jemalloc_tcache.c:55 tbin_info =3D binind =3D 6 tbin =3D 0x501441d8 #6 0x506ab5bc in __je_free_default (ptr=3D0x5012b480) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/rtree.h:374 tcache =3D tsd =3D #7 0x506abdb0 in __free (ptr=3D0x5012b480) at jemalloc_jemalloc.c:2852 log_var =3D log_var =3D #8 0x5074f170 in __clean_env (freeVars=3D255) at = /usr/src/lib/libc/stdlib/getenv.c:243 envNdx =3D 0 #9 __clean_env_destructor () at /usr/src/lib/libc/stdlib/getenv.c:409 No locals. #10 0x5010d45c in objlist_call_fini (list=3D, root=3D0x0, = lockstate=3D0xffffc0a8) at /usr/src/libexec/rtld-elf/rtld.c:2694 saved_msg =3D 0x5013e144 "Undefined symbol = \"_nss_cache_cycle_prevention_function\"" elm =3D fini_addr =3D index =3D #11 0x50105b24 in rtld_exit () at /usr/src/libexec/rtld-elf/rtld.c:3081 lockstate =3D {lockstate =3D 2, env =3D {{_sjb =3D {269512264, = 0, 269484032, 269514775, 269615104, 269510192, 0, 0, -16280, 1343500292, = 1343400236, 0 , 2, 1348094285, 904808974,=20 1143081540, 1, 269512264, 1343411584, 269463008, 0, = 269514775, 269512264, 1, 1350263456, 269407312, 2, 269407852, 269407296, = -15868, -15888, -15808, 1343400236, 0, -15872,=20 1343238092, 1350066476, 0, -15840, 1349172772, = 1350897796, 1343434832, 4, 1343434752, 1350066476, 269235216, -15792, = 1349700248, 269615104, 269514775, 0, 1350263400, 0, 1349699420}}}} #12 0x50732448 in __cxa_finalize (dso=3D0x0) at = /usr/src/lib/libc/stdlib/atexit.c:240 phdr_info =3D {dlpi_addr =3D 1343533080, dlpi_name =3D = 0xffffc2e0 "\377\377\310\260\020\006@\f", dlpi_phdr =3D 0x0, dlpi_phnum = =3D 20500, dlpi_adds =3D 1, dlpi_subs =3D 1157546361094329728,=20 dlpi_tls_modid =3D 269463008, dlpi_tls_data =3D 0x0} has_phdr =3D p =3D 0x507b6268 n =3D fn =3D {fn_type =3D 1, fn_ptr =3D {std_func =3D 0x50105ad4 = , cxa_func =3D 0x50105ad4 }, fn_arg =3D 0x0, = fn_dso =3D } #13 0x506b62e8 in exit (status=3D0) at = /usr/src/lib/libc/stdlib/exit.c:74 No locals. #14 0x1006400c in finis (drop=3D, cleanup=3D1, = exitstat=3D0) at /usr/src/contrib/sendmail/src/main.c:3105 _h =3D {eh_value =3D 0x0, eh_context =3D {{_sjb =3D {0, 0, 0, 0, = 0, 1343533080, -15648, 268844484, 610404932, 0, 1, 269512264, = 1343411584, 269463008, 0, 269514775, 269512264, 0, 269484032,=20 269514775, 269615104, 269510192, 0, -15632, -15628, 1, = 1, 0 , 184, 268523004, 0, -13888, 1343356928, = 1350589132, 904808974, 0, -8184, 3, 268500992, 268566528,=20 268566528, 23, 1343489540, 0, 0, 0, 0, 1, 1, 0, 1224, 0, = 0, 0, 0, 771751936, 0, 0, 0, 0, 0, 0, 0, 0, 0}}}, eh_parent =3D 0x0, = eh_state =3D 2} pidpath =3D '\000' ... pid =3D #15 0x10085d6c in run_work_group (wgrp=3D, = flags=3D) at /usr/src/contrib/sendmail/src/queue.c:2276 sequenceno =3D 1 now =3D rpool =3D 0x5012d580 e =3D 0x10106e48 endgrp =3D 0 qdir =3D qgrp =3D i =3D h =3D full =3D more =3D njobs =3D #16 0x10084a18 in runqueue (forkflag=3D, = verbose=3D, persistent=3D269484032, runall=3D0) at = /usr/src/contrib/sendmail/src/queue.c:1545 rwgflags =3D 1 wasblocked =3D curnum =3D 0 oldgroup =3D 0 ret =3D 1 cursh =3D i =3D 0 #17 0x100628e4 in main (argc=3D, argv=3D, = envp=3D) at /usr/src/contrib/sendmail/src/main.c:2555 qtype =3D "Queue runner@00:30:00 for /var/spool/clientmqueue", = '\000' , "\377\370", '\000' , = "\377\377\314`", '\000' , = "\377\377\327\060\000\000\000\000\000\000\000\000g\247\220+\000\350\067\36= 4\377\377\314\210P\024\060\004PW\337\330\377\377\315\000P\022\251,\000\000= \000\000\377\377\314\360P\020\206\344" dtype =3D "+queueing@00:30:00\000\060D = \f\240\000\000\000\000Pcl\240Pcl\250\000\000\360\062\000\000\f\000P\017\36= 0\000B", '\000' starttime =3D 1588220070 rnamebuf =3D "root", '\000' traf_st =3D {st_dev =3D 0, st_ino =3D 1, st_nlink =3D 0, st_mode = =3D 0, st_padding0 =3D 0, st_uid =3D 0, st_gid =3D 0, st_padding1 =3D 0, = st_rdev =3D 0, st_atim =3D {tv_sec =3D 0, tv_nsec =3D 0}, st_mtim =3D { tv_sec =3D 0, tv_nsec =3D 0}, st_ctim =3D {tv_sec =3D 0, = tv_nsec =3D 0}, st_birthtim =3D {tv_sec =3D 0, tv_nsec =3D 0}, st_size =3D= 1343500292, st_blocks =3D 0, st_blksize =3D 0, st_flags =3D 0, st_gen =3D= 0,=20 st_spare =3D {25769795072, 18446706793393422336, = 18446706485499010232, 1073743008, 1153877804845105152, = 5769342402355519307, 18446708107653414912, 5769342402355519472, = 34359738372,=20 1343489540}} buf =3D "[IPv6:0:0:0:0:0:0:0:1]", '\000' ... jbuf =3D "localhost.my.domain", '\000' emptyenviron =3D {0x0} foregroundqueue =3D queuepersistent =3D quarantining =3D queuegroup =3D conffile =3D 0x0 sysloglabel =3D authinfo =3D nullserver =3D 0x0 debug =3D queuerun =3D run_in_foreground =3D warn_f_flag =3D warn_C_flag =3D 0 p_flags =3D 0x0 safecf =3D 1 qgrp =3D extraprivs =3D 606210628 cftype =3D 2 fill_errno =3D 0 smdebug =3D i =3D dp =3D av =3D p =3D j =3D pw =3D from =3D hp =3D negate =3D ep =3D new =3D forged =3D 0 st =3D tls_ok =3D save_val =3D e =3D (gdb) disass Dump of assembler code for function arena_dalloc_bin_locked_impl: 0x506a1158 <+0>: mflr r0 0x506a115c <+4>: stw r0,4(r1) 0x506a1160 <+8>: stwu r1,-48(r1) 0x506a1164 <+12>: stw r30,40(r1) 0x506a1168 <+16>: stw r23,12(r1) 0x506a116c <+20>: stw r24,16(r1) 0x506a1170 <+24>: stw r25,20(r1) 0x506a1174 <+28>: stw r26,24(r1) 0x506a1178 <+32>: stw r27,28(r1) 0x506a117c <+36>: stw r28,32(r1) 0x506a1180 <+40>: stw r29,36(r1) 0x506a1184 <+44>: bl 0x506a1188 = 0x506a1188 <+48>: mr r26,r3 0x506a118c <+52>: mflr r30 0x506a1190 <+56>: addis r30,r30,14 0x506a1194 <+60>: addi r30,r30,20388 0x506a1198 <+64>: mr r27,r4 0x506a119c <+68>: lwz r3,6040(r30) 0x506a11a0 <+72>: andi. r4,r9,1 0x506a11a4 <+76>: mr r25,r8 0x506a11a8 <+80>: mr r28,r7 0x506a11ac <+84>: mr r29,r5 0x506a11b0 <+88>: lbz r3,0(r3) 0x506a11b4 <+92>: addi r23,r7,40 0x506a11b8 <+96>: mulli r24,r6,48 0x506a11bc <+100>: cmpwi cr1,r3,0 0x506a11c0 <+104>: cror 4*cr5+lt,4*cr1+eq,gt 0x506a11c4 <+108>: bge cr5,0x506a14e4 = 0x506a11c8 <+112>: lwz r3,4(r28) 0x506a11cc <+116>: lwz r4,6264(r30) 0x506a11d0 <+120>: lwz r5,8(r28) 0x506a11d4 <+124>: li r8,1 0x506a11d8 <+128>: rlwinm r6,r3,16,23,29 0x506a11dc <+132>: lwzx r6,r4,r6 0x506a11e0 <+136>: subf r5,r5,r25 0x506a11e4 <+140>: lwz r4,0(r28) 0x506a11e8 <+144>: mulhwu r6,r6,r5 0x506a11ec <+148>: rlwinm r5,r6,29,3,29 =3D> 0x506a11f0 <+152>: lwzx r7,r23,r5 0x506a11f4 <+156>: clrlwi r9,r6,27 0x506a11f8 <+160>: slw r8,r8,r9 0x506a11fc <+164>: xor r8,r8,r7 0x506a1200 <+168>: cmplwi r7,0 0x506a1204 <+172>: stwx r8,r23,r5 0x506a1208 <+176>: bne 0x506a1280 = 0x506a120c <+180>: lwz r5,4440(r30) 0x506a1210 <+184>: rlwinm r7,r3,14,25,31 0x506a1214 <+188>: mulli r7,r7,48 0x506a1218 <+192>: add r5,r5,r7 0x506a121c <+196>: lwz r5,20(r5) 0x506a1220 <+200>: cmplwi r5,2 0x506a1224 <+204>: blt 0x506a1280 = 0x506a1228 <+208>: lwz r8,4440(r30) 0x506a122c <+212>: rlwinm r6,r6,27,5,31 0x506a1230 <+216>: li r9,2 0x506a1234 <+220>: add r7,r8,r7 0x506a1238 <+224>: addi r7,r7,24 0x506a123c <+228>: li r8,1 . . . (gdb) info reg r0 0x506a1148 1349128520 r1 0xffffb8f0 4294949104 r2 0x5014b018 1343533080 r3 0x0 0 r4 0x0 0 r5 0x1405694 20993684 r6 0xa02b4a4 167949476 r7 0x50a0af40 1352707904 r8 0x1 1 r9 0x1 1 r10 0xffffc2e0 4294951648 r11 0x50102f94 1343238036 r12 0x84220a44 2216823364 r13 0x0 0 r14 0x0 0 r15 0x0 0 r16 0x47 71 r17 0x47 71 r18 0x0 0 r19 0xffffb940 4294949184 r20 0x18 24 r21 0x0 0 r22 0xffffb940 4294949184 r23 0x50a0af68 1352707944 r24 0x120 288 r25 0x5015a520 1343595808 r26 0x50144018 1343504408 r27 0x50a00380 1352663936 r28 0x50a0af40 1352707904 r29 0x50a02e08 1352674824 r30 0x5078612c 1350066476 r31 0xffffba50 4294949456 pc 0x506a11f0 0x506a11f0 = msr cr 0x424a0a44 1112148548 lr 0x506a1188 0x506a1188 = ctr 0x50102f94 1343238036 xer 0x0 0 fpscr 0x82024000 -2113781760 vscr vrsave =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)