From owner-freebsd-hackers@freebsd.org Fri May 1 02:41:10 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DEDC52CF8D4 for ; Fri, 1 May 2020 02:41:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic317-21.consmr.mail.gq1.yahoo.com (sonic317-21.consmr.mail.gq1.yahoo.com [98.137.66.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49CxMT0vkrz4Mgq for ; Fri, 1 May 2020 02:41:08 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: IZXnkPQVM1n2FtUz7GeerFhNx7kDkrN3qZilfhCHoFoOHXF720WM8uTC9FaOdoh 610z1GkAZ_ep8VwE.LbZI7cJfmw1FraMbIf.oOAobKpL_YexRl1yb3Lh8TgjOMKRonScSR9jcKW6 D1uy7u_YxYGYUMi2ujeCgLf5dO5KoNKvzONvoOtZ1uuRClmalnI5v6kd53YvOzM0LcnNm.9EsQwb HxE6RpVjypGicKo54s99Fa1qlO0b9L4V1yXZMbevnYd37OuVpYu3xodZyRlt9mXbmfMv8ezk7zT_ 0MfHd_RvCLKbF86fupEsVopzNiS.aifBsywUniNXvfjaSG_YDoXjEsBKHJaSGb3Tmi.f3RRugJtn 2HYCqLEDvFE22WZykayFD1IuLUasIB_.ZkxsdyrZL4ZE1Bpjtkt1OAknnD2WHwqZBFsiMzTud0m4 2nqJr.YLI_3Hp.M267BIfOL73ksRyMvjnds5ZZoGeZXye6Utz_6OBy2EgmRc4.BfB64OTbNtrmUJ ViknHGR8te5HfxMBmMHK3w4hXXqzCt740NeLt9.VqMuw4JkW_yN1SGy51WqhbUPuR1qqmGp3aDXn 3fJBCIHMu9myodNK7kX7hIs9hJhY_hqxdXPia42ihYoTWc2SfH1saxJEY1v8ib3obuDB3EUxu_8B yeZfBVfmQKd5IlaiwcoequHD4e2qYXD0NEW9ZpcPKxhmov4uh8AfLJel9jzznHhTQ0Gu1hOKdUH5 823nUaU0fLsZR4x11i1Maitctq1o3JFjt.mEj5UKjNPmkhLZH71.a3NI_JooW9hHODSN8dpXkFGV q2GF7NEjK0lGyhtXJkOBL4HpZK2C5WH885UzoY2izGw_KOeIvIzGzdUL7EdFiSJZX3oAgIdmIFp9 rxJ6emuuqDCi_NxpC2.EJ_bFY6fxGAJyhL2.n3YqYB_hyPewF2R2RxG9dVWM16esJQG7UQFGxLUi rECWtqCuHyL8MDVi8hqPAI2qoI9YG1sPcaTAfHGkdFr1D81lVfGtKzyfMZ1sxRbt2z3mEuAv3UNZ 4tO9ogn9qqxIEFQMhc.yhj9hJ5FERhhdT5r4bLFqjFNih.xuJw6DLI1mAl2Hf.CX7GTffvqVK3tE cmy8lcaac8.CdkUpsnNMyoY9fiSziIVyv.EEGeT05N6r9J1FK9gCT1m0j8Nglp1v9fHmaxXA65F3 3g7QqJEN3XHG3Jr7AcZJvTobve1L5TOe345MNGzfWCPyX2joHlmZFlxZlr7JQExs8FwB0bksZjbN v4aQ5EwNlFJCtCPxuG3GbYDeOt5aRxJ54hqSBL4w.EFtt51pDAPwg6fGmrschuBWQ9PQgy0FF5Sh 1ofP9LpNQftztLjI8Llp5kWZVrCMPg6jU.XEdjgO2AXEkVGuo6y51p0G55gXarrFaOfw6nyf.h.l 8NsQnQWDQkIJQqRn5ea2eHN0- Received: from sonic.gate.mail.ne1.yahoo.com by sonic317.consmr.mail.gq1.yahoo.com with HTTP; Fri, 1 May 2020 02:41:06 +0000 Received: by smtp413.mail.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 2a53e3b74aaf11553f38d5ce95aad27d; Fri, 01 May 2020 02:41:01 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: 32-bit powerpc head -r360311 has signal 11 process crashes with rendezvous_request involved (old PowerMac dual-socket, 1 core each) Date: Thu, 30 Apr 2020 19:41:00 -0700 References: <76D3A482-98BC-44C3-B84D-504A012CA8D8@yahoo.com> To: FreeBSD PowerPC ML , FreeBSD Hackers In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49CxMT0vkrz4Mgq X-Spamd-Bar: - X-Spamd-Result: default: False [-1.62 / 15.00]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.59)[-0.594,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-0.53)[-0.527,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (3.98), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[147.66.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[147.66.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 May 2020 02:41:10 -0000 [After some time dhclient and sendmail did have the problem despite the debug-kernel being in use.] On 2020-Apr-29, at 21:29, Mark Millard wrote: > [Notes on using the artifact.ci head -3260311 debug-kernel added.] >=20 > On 2020-Apr-29, at 02:27, Mark Millard wrote: >=20 >> Since updating from head -r359427 based to head -r360311 base >> (my own non-debug builds), various things report segmentation >> faults. It appears that rendezvous_request may always be >> involved. I've not had time (yet) to deal with substituting a >> debug kernel from: >>=20 >> = https://artifact.ci.freebsd.org/snapshot/head/r360311/powerpc/powerpc/ >>=20 >> I expect to do so at some point. >>=20 >>=20 >> I give 2 examples below (mountd and rpcbind). Both involve >> rendezvous_request and svc_getreq_common . >>=20 >> First mountd : >>=20 >> # gdb mountd /mountd.core=20 >> GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] >> Copyright (C) 2020 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later = >> . . . >> Reading symbols from mountd... >> Reading symbols from /usr/lib/debug//usr/sbin/mountd.debug... >> [New LWP 100105] >> Core was generated by `/usr/sbin/mountd -r'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 atomic_load_u (a=3D, mo=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/atomic.h:70 >>=20 >> warning: Source file is more recent than executable. >> 70 JEMALLOC_GENERATE_INT_ATOMICS(unsigned, u, LG_SIZEOF_INT) >> (gdb) bt >> #0 atomic_load_u (a=3D, mo=3D) at = /usr/src/contrib/jemalloc/include/jemalloc/internal/atomic.h:70 >> #1 rtree_leaf_elm_szind_read (tsdn=3D, = rtree=3D, elm=3D, dependent=3D) at /usr/src/contrib/jemalloc/include/jemalloc/internal/rtree.h:230 >> #2 rtree_szind_slab_read (tsdn=3D0x50094018, rtree=3D, rtree_ctx=3D0x50094044, key=3D1869636193, dependent=3Dtrue, = r_szind=3D, r_slab=3D) >> at /usr/src/contrib/jemalloc/include/jemalloc/internal/rtree.h:504 >> #3 ifree (tsd=3D, ptr=3D0x6f706261, tcache=3D, slow_path=3Dfalse) at jemalloc_jemalloc.c:2574 >> #4 __je_free_default (ptr=3D0x6f706261) at jemalloc_jemalloc.c:2775 >> #5 0x50235db0 in __free (ptr=3D0x6f706261) at = jemalloc_jemalloc.c:2852 >> #6 0x5026525c in freenetconfigent (netconfigp=3D0x50049170) at = /usr/src/lib/libc/rpc/getnetconfig.c:540 >> #7 0x50260d8c in __rpc_sockinfo2netid (sip=3D, = netid=3D) at /usr/src/lib/libc/rpc/rpc_generic.c:573 >> #8 0x502521f0 in makefd_xprt (fd=3D10, sendsize=3D9000, = recvsize=3D9000) at /usr/src/lib/libc/rpc/svc_vc.c:270 >> #9 0x50252fa4 in rendezvous_request (xprt=3D0x5007b120, = msg=3D) at /usr/src/lib/libc/rpc/svc_vc.c:315 >> #10 0x50254588 in svc_getreq_common (fd=3D) at = /usr/src/lib/libc/rpc/svc.c:640 >> #11 0x502543d0 in svc_getreqset (readfds=3D) at = /usr/src/lib/libc/rpc/svc.c:611 >> #12 0x1001434c in main (argc=3D, argv=3D0xffffde3c) at = /usr/src/usr.sbin/mountd/mountd.c:683 >>=20 >> (gdb) disass >> Dump of assembler code for function __je_free_default: >> 0x50235244 <+0>: mflr r0 >> 0x50235248 <+4>: stw r0,4(r1) >> 0x5023524c <+8>: stwu r1,-80(r1) >> 0x50235250 <+12>: stw r30,72(r1) >> 0x50235254 <+16>: stw r21,36(r1) >> 0x50235258 <+20>: stw r22,40(r1) >> 0x5023525c <+24>: stw r23,44(r1) >> 0x50235260 <+28>: stw r24,48(r1) >> 0x50235264 <+32>: stw r25,52(r1) >> 0x50235268 <+36>: stw r26,56(r1) >> 0x5023526c <+40>: stw r27,60(r1) >> 0x50235270 <+44>: stw r28,64(r1) >> 0x50235274 <+48>: stw r29,68(r1) >> 0x50235278 <+52>: bl 0x5023527c <__je_free_default+56> >> 0x5023527c <+56>: mr r28,r3 >> 0x50235280 <+60>: mflr r30 >> 0x50235284 <+64>: addis r30,r30,14 >> 0x50235288 <+68>: addi r30,r30,-20816 >> 0x5023528c <+72>: lwz r4,64(r30) >> 0x50235290 <+76>: lwz r5,6188(r30) >> 0x50235294 <+80>: lwz r4,0(r4) >> 0x50235298 <+84>: stw r4,32(r1) >> 0x5023529c <+88>: lbz r4,0(r5) >> 0x502352a0 <+92>: cmplwi r4,0 >> 0x502352a4 <+96>: bne 0x502353c4 <__je_free_default+384> >> 0x502352a8 <+100>: cmplwi r28,0 >> 0x502352ac <+104>: beq 0x50235378 <__je_free_default+308> >> 0x502352b0 <+108>: addi r3,r30,4332 >> 0x502352b4 <+112>: rlwinm r24,r28,0,0,9 >> 0x502352b8 <+116>: bl 0x502f06b4 = <00000000.plt_pic32.__tls_get_addr> >> 0x502352bc <+120>: rlwinm r26,r28,13,25,28 >> 0x502352c0 <+124>: mr r29,r3 >> 0x502352c4 <+128>: lbz r3,0(r3) >> 0x502352c8 <+132>: cmplwi r3,0 >> 0x502352cc <+136>: bne 0x502353fc <__je_free_default+440> >> 0x502352d0 <+140>: add r25,r29,r26 >> 0x502352d4 <+144>: lwz r3,44(r25) >> 0x502352d8 <+148>: addi r27,r29,44 >> 0x502352dc <+152>: cmplw r3,r24 >> 0x502352e0 <+156>: bne 0x50235578 <__je_free_default+820> >> 0x502352e4 <+160>: lwz r3,48(r25) >> 0x502352e8 <+164>: rlwinm r4,r28,20,22,31 >> 0x502352ec <+168>: mulli r4,r4,12 >> 0x502352f0 <+172>: add r3,r3,r4 >> =3D> 0x502352f4 <+176>: lwz r6,4(r3) >> 0x502352f8 <+180>: lwz r5,4368(r30) >> 0x502352fc <+184>: addi r26,r29,288 >> 0x50235300 <+188>: lbz r4,8(r3) >> 0x50235304 <+192>: rlwinm r3,r6,2,0,29 >> 0x50235308 <+196>: lwz r7,28(r29) >> 0x5023530c <+200>: lwzx r5,r5,r3 >> 0x50235310 <+204>: lwz r8,24(r29) >> 0x50235314 <+208>: andi. r4,r4,1 >> 0x50235318 <+212>: addc r4,r7,r5 >> 0x5023531c <+216>: addze r5,r8 >> 0x50235320 <+220>: stw r5,24(r29) >> 0x50235324 <+224>: stw r4,28(r29) >> 0x50235328 <+228>: ble 0x50235534 <__je_free_default+752> >> . . . >>=20 >> (gdb) info reg >> r0 0x50235c04 1344494596 >> r1 0xffffcfb0 4294954928 >> r2 0x5009b018 1342812184 >> r3 0x2448 9288 >> r4 0x500940ac 1342783660 >> r5 0x2448 9288 >> r6 0x500940cc 1342783692 >> r7 0x1 1 >> r8 0x0 0 >> r9 0x80808080 2155905152 >> r10 0xc 12 >> r11 0x502e3b50 1345207120 >> r12 0x500491a0 1342476704 >> r13 0x0 0 >> r14 0x1 1 >> r15 0x10040000 268697600 >> r16 0x0 0 >> r17 0x10040000 268697600 >> r18 0x2 2 >> r19 0x0 0 >> r20 0x1 1 >> r21 0x5007b164 1342681444 >> r22 0xffffd2dc 4294955740 >> r23 0x80 128 >> r24 0x6f400000 1866465280 >> r25 0x50094080 1342783616 >> r26 0x68 104 >> r27 0x50094044 1342783556 >> r28 0x6f706261 1869636193 >> r29 0x50094018 1342783512 >> r30 0x5031012c 1345388844 >> r31 0x10040000 268697600 >> pc 0x502352f4 0x502352f4 <__je_free_default+176> >> msr >> cr 0x242008a4 606079140 >> lr 0x50235c04 0x50235c04 = <__je_free_default+2496> >> ctr 0x0 0 >> xer 0x0 0 >> fpscr 0x0 0 >> vscr >> vrsave >>=20 >>=20 >> Then rpcbind : >>=20 >> # gdb rpcbind /rpcbind.core=20 >> GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] >> Copyright (C) 2020 Free Software Foundation, Inc. >> License GPLv3+: GNU GPL version 3 or later = >> . . . >> Reading symbols from rpcbind... >> Reading symbols from /usr/lib/debug//usr/sbin/rpcbind.debug... >> [New LWP 100098] >> Core was generated by `/usr/sbin/rpcbind'. >> Program terminated with signal SIGSEGV, Segmentation fault. >> #0 __xdrrec_setnonblock (xdrs=3D0x500472d8, maxrec=3D9000) at = /usr/src/lib/libc/xdr/xdr_rec.c:607 >> 607 rstrm->nonblock =3D TRUE; >> (gdb) bt >> #0 __xdrrec_setnonblock (xdrs=3D0x500472d8, maxrec=3D9000) at = /usr/src/lib/libc/xdr/xdr_rec.c:607 >> #1 0x502440d4 in rendezvous_request (xprt=3D, = msg=3D) at /usr/src/lib/libc/rpc/svc_vc.c:348 >> #2 0x50245588 in svc_getreq_common (fd=3D) at = /usr/src/lib/libc/rpc/svc.c:640 >> #3 0x5024580c in svc_getreq_poll (pfdp=3D, = pollretval=3D1) at /usr/src/lib/libc/rpc/svc.c:739 >> #4 0x10018568 in my_svc_run () at = /usr/src/usr.sbin/rpcbind/rpcb_svc_com.c:1167 >> #5 0x10014ad8 in main (argc=3D, argv=3D) at /usr/src/usr.sbin/rpcbind/rpcbind.c:250 >>=20 >> (gdb) disass >> Dump of assembler code for function __xdrrec_setnonblock: >> 0x50250468 <+0>: lwz r5,12(r3) >> 0x5025046c <+4>: li r6,1 >> 0x50250470 <+8>: cmplwi r4,0 >> =3D> 0x50250474 <+12>: stw r6,64(r5) >> 0x50250478 <+16>: bne 0x50250480 <__xdrrec_setnonblock+24> >> 0x5025047c <+20>: lwz r4,60(r5) >> 0x50250480 <+24>: li r3,1 >> 0x50250484 <+28>: stw r4,92(r5) >> 0x50250488 <+32>: blr >> End of assembler dump. >>=20 >> (gdb) info reg >> r0 0x5c 92 >> r1 0xffffb400 4294947840 >> r2 0x500a1018 1342836760 >> r3 0x500472d8 1342468824 >> r4 0x2328 9000 >> r5 0x2020 8224 >> r6 0x1 1 >> r7 0xffffb364 4294947684 >> r8 0x500472f4 1342468852 >> r9 0x0 0 >> r10 0x20 32 >> r11 0x502d8ea0 1345162912 >> r12 0x24200880 606079104 >> r13 0x0 0 >> r14 0x0 0 >> r15 0xffffbc28 4294949928 >> r16 0x10002848 268445768 >> r17 0x10040000 268697600 >> r18 0x2 2 >> r19 0x0 0 >> r20 0x1 1 >> r21 0x5004c044 1342488644 >> r22 0xffffb63c 4294948412 >> r23 0x80 128 >> r24 0x50048010 1342472208 >> r25 0x14 20 >> r26 0xffffb630 4294948400 >> r27 0x500472d0 1342468816 >> r28 0xe 14 >> r29 0x50047220 1342468640 >> r30 0x5030112c 1345327404 >> r31 0x10040000 268697600 >> pc 0x50250474 0x50250474 = <__xdrrec_setnonblock+12> >> msr >> cr 0x44200080 1142947968 >> lr 0x502440d4 0x502440d4 = >> ctr 0x502d8ea0 1345162912 >> xer 0x0 0 >> fpscr 0x0 0 >> vscr >> vrsave >>=20 >>=20 >> dhclient and sendmail have notices of signal 11's >> but I do not find any .core files around for them. >>=20 >> Prior to this upgrade I was having no such problems >> with the 32-bit powerpc PowerMac. >=20 > I substituted the debug-kernel from: >=20 > = https://artifact.ci.freebsd.org/snapshot/head/r360311/powerpc/powerpc/kern= el.txz >=20 > and with it there is no evidence so far of the > problem(s) occurring. Since the same world build is > in use in both contexts, it looks like the kernel > is what makes the difference for the problem(s). >=20 > With the debug-kernel avoiding the problem, I've yet > to figure out how to gather evidence. >=20 About 8.5 hours later than the boot completing it got a: kernel: pid 659 (dhclient), jid 0, uid 65: exited on signal 11 About 7 hours later it started getting sendmail messaged similar to: kernel: pid 3722 (sendmail), jid 0, uid 25: exited on signal 11 These seem to be 0.5 hours part. Between the 3rd and 4th is a lock order reversal notice that I've separately submitted to the lists (but that do not yet appear there, last I checked). =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)