From owner-freebsd-current@freebsd.org Sun May 3 18:08:36 2020 Return-Path: Delivered-To: freebsd-current@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 894DA2E4476 for ; Sun, 3 May 2020 18:08:36 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-20.consmr.mail.gq1.yahoo.com (sonic313-20.consmr.mail.gq1.yahoo.com [98.137.65.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49FYrf4GwXz41sZ for ; Sun, 3 May 2020 18:08:34 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: 1oX6N4AVM1kMEt_vPxuFXH9vEhh1wYomaWReYDYjULLXvDflLYHJiezPy_PUB3r JgJ0FHTSFm9MJZ9RL7pKFvscGotD95w45Lba1uwxAPX0Wz4YlbPdRn76Vrzg2lrQF51vIBEM7Iia NtbtlW_Brm6UT1HX7GAtvzWR80EFMGGGz2IxTYxbUKsbNO2ZRCU.f7yLKWTl_GZx2MjtpW4oU77B UiaQwBSwN7TgKBWI0FcwPinkrOZioD.XUQS24i_ACCIdPVccgJUNR8vCHdhGTVTHM.9.DUcXfGX8 U1EBvDfBOBt8hVIgi7KdHHjVU8qK53uLEnnQeqx0Wc7aZ7Bu3ZQZjJUKo253ddI0xW08i6kp0MVt rXUeUXRHukehLn.2svajv90uPVWzeBV4.FRx1_ym85CCzZGUJ3QOLJUH_NNylGBkZfa5eP2VV9EE APhM0cZ8o8GNrHPvulT6lqSZLjWRLoKAIEAVt6fhyiKTQhN6lDc57o6AOVSbvrQdNbXKdqfA8smm OzCws63s9R_LMSjL7q6Yn8WYHxI5pWPfxFSR6TKRNJkocJ6cEttlSIJn.haWkabQCA5pBCI3_Ss7 oeipxHzpQJnl6cSKFNyxWUSbw8TQXnp1B8oqfofzH3MaISfml7ksJqzW.zPuOOrtgYESFCO2kw2V giR6umdL5MEF4fVQWZvBBsZRxDDh5RPFWK.fOKr51txJw1oHcK8j.1.KmcsQ.bxdHtIPV6RQK5vV 0bv9z_3qG_CEdWLIIswO34W.T22RRVUEvT0NCKEX5nFvLXJtHBh7Ij9055ZqSnK71IUTgL5MdgBv 7LLCwL598HOfs27rntQDl5lRfmSFULqFuKYrOie4.iAehxD0l9Gb76eM4v2rkPUt6wQNxn_HvY0v vEoIFxL9sJK.bU5Cop21fIF48lhB0c9eeIe6T2rTGPAY_Rya22MA6FxI7zMURQe7iH73fWgog9Od dWh1E8H3HvlCjHqf_QmDXGaznD2j1n6H5zpMarpY_eIk97qicL.UfrSYXGksHBsHxEL4NOCJXPMZ 4i.qz.tdb0MG1zZf38.LFrGUmHJlDFnr_O2Mt6abokrw9_v.jIuqIyoFGUJK8wxmr_kAh9yqbja2 fH3uW9lcyP_JkGP3STEX0NSBUiQKuyGxVk_kt7Msz7307g2Pn2yA3fG0XROxErkhR2X8HIMrIPM0 u9x9LaIXJz6.EHoNO.QTcl4Zf52KxOklCmQO3cNV1zJTmH5E3ldU_1QdQ8wKM7PSY4kSoYOQGpWV M4NVExzZ6bpbUXH1bLcgyl2aRFzL._Bcp1A9nJ5R_H.HuZPrxw82zeVlbpm6jGsJjpjgu158aKYY hgVzf3kpsPO1ZCcnQDyt7PAX.R1Y6WzJA6ThHJYtKSA3r5ISa_p..VDrk6zTkQM.GWvpzHCbjlU1 eNJ3xXdJdnvZlnifZo2xn.vGythTSa3G1APVLxceufCV1EwU56ZaikDdf6WDJj9RHmsc- Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sun, 3 May 2020 18:08:32 +0000 Received: by smtp422.mail.ne1.yahoo.com (VZM Hermes SMTP Server) with ESMTPA ID 233fcb4833509f60a9288f2901074f87; Sun, 03 May 2020 18:08:28 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: svn commit: r360233 - in head: contrib/jemalloc . . . : This partially breaks a 2-socket 32-bit powerpc (old PowerMac G4) based on head -r360311 From: Mark Millard In-Reply-To: Date: Sun, 3 May 2020 11:08:27 -0700 Cc: Brandon Bergren Content-Transfer-Encoding: quoted-printable Message-Id: <8479DD58-44F6-446A-9CA5-D01F0F7C1B38@yahoo.com> References: To: "vangyzen@freebsd.org" , svn-src-head@freebsd.org, FreeBSD Current , FreeBSD Hackers , FreeBSD PowerPC ML X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49FYrf4GwXz41sZ X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.49 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; RCPT_COUNT_FIVE(0.00)[6]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.989,0]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(0.00)[ip: (1.71), ipnet: 98.137.64.0/21(0.82), asn: 36647(0.66), country: US(-0.05)]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[83.65.137.98.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[83.65.137.98.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 May 2020 18:08:36 -0000 [At around 4AM local time dhcient got a signal 11, despite the jemalloc revert. The other exmaples have not happened.] On 2020-May-2, at 18:46, Mark Millard wrote: > [I'm only claiming the new jemalloc is involved and that > reverting avoids the problem.] >=20 > I've been reporting to some lists problems with: >=20 > dhclient > sendmail > rpcbind > mountd > nfsd >=20 > getting SIGSEGV (signal 11) crashes and some core > dumps on the old 2-socket (1 core per socket) 32-bit > PowerMac G4 running head -r360311. >=20 > Mika=C3=ABl Urankar sent a note suggesting that I try > testing reverting head -r360233 for my head -r360311 > context. He got it right . . . >=20 >=20 > Context: >=20 > The problem was noticed by an inability to have > other machines do a: >=20 > mount -onoatime,soft OLDPOWERMAC-LOCAL-IP:/... /mnt >=20 > sort of operation and to have succeed. By contrast, on > the old PowerMac G4 I could initiate mounts against > other machines just fine. >=20 > I do not see any such problems on any of (all based > on head -r360311): >=20 > powerpc64 (old PowerMac G5 2-sockets with 2 cores each) > armv7 (OrangePi+ 2ed) > aarch64 (Rock64, RPi4, RPi3, > OverDrive 1000, > Macchiatobin Double Shot) > amd64 (ThreadRipper 1950X) >=20 > So I expect something 32-bit powerpc specific > is somehow involved, even if jemalloc is only > using whatever it is. >=20 > (A kyua run with a debug kernel did not find other > unexpected signal 11 sources on the 32-bit PowerMac > compared to past kyua runs, at least that I noticed. > There were a few lock order reversals that I do not > know if they are expected or known-safe or not. > I've reported those reversals to the lists as well.) >=20 >=20 > Recent experiments based on the suggestion: >=20 > Doing the buildworld, buildkernel and installing just > the new kernel and rebooting made no difference. >=20 > But then installing the new world and rebooting did > make things work again: I no longer get core files > for the likes of (old cores from before the update): >=20 > # find / -name "*.core" -print > /var/spool/clientmqueue/sendmail.core > /rpcbind.core > /mountd.core > /nfsd.core >=20 > Nor do I see the various notices for sendmail > signal 11's that did not leave behind a core file > --or for dhclient (no core file left behind). > And I can mount the old PowerMac's drive from > other machines just fine. >=20 >=20 > Other notes: >=20 > I do not actively use sendmail but it was left > to do its default things, partially to test if > such default things are working. Unfortunately, > PowerMacs have a problematical status under > FreeBSD and my context has my historical > experiments with avoiding various problems. Looking, I see that I got a: pid 572 (dhclient), jid 0, uid 0: exited on signal 11 (core dumped) notice under the reverted build. No instances of the other examples. This is the first that a dhclient example has produced a .core file. gdb indicates 0x5180936c for r7 in: lwz r8,36(r7) as leading to the failure. This was in arena_dalloc_bin_locked_impl (where arena_slab_reg_dalloc and bitmap_unset were apparently inlined). The chain for the example seems to be: fork_privchld -> dispatch_imsg -> jemalloc For reference . . . # gdb dhclient /dhclient.core=20 GNU gdb (GDB) 9.1 [GDB v9.1 for FreeBSD] Copyright (C) 2020 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later = . . . Reading symbols from dhclient... Reading symbols from /usr/lib/debug//sbin/dhclient.debug... [New LWP 100089] Core was generated by `dhclient: gem0 [priv]'. Program terminated with signal SIGSEGV, Segmentation fault. #0 bitmap_unset (bitmap=3D0x50407164, binfo=3D, = bit=3D167842154) at = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/bitmap.h:341= 341 = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/bitmap.h: = No such file or directory. (gdb) bt -full #0 bitmap_unset (bitmap=3D0x50407164, binfo=3D, = bit=3D167842154) at = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/bitmap.h:341= goff =3D gp =3D 0x51809390 propagate =3D g =3D i =3D #1 arena_slab_reg_dalloc (slab=3D0x50407140, slab_data=3D0x50407164, = ptr=3D0x50088b50) at jemalloc_arena.c:273 bin_info =3D binind =3D 0 regind =3D 167842154 #2 arena_dalloc_bin_locked_impl (tsdn=3D0x5009f018, arena=3D, slab=3D, ptr=3D, junked=3D) at jemalloc_arena.c:1540 slab_data =3D binind =3D bin_info =3D bin =3D nfree =3D #3 0x502916a8 in __je_arena_dalloc_bin_junked_locked (tsdn=3D, arena=3D, extent=3D, ptr=3D) at jemalloc_arena.c:1559 No locals. #4 0x50250d2c in __je_tcache_bin_flush_small (tsd=3D0x5009f018, = tcache=3D, tbin=3D0x5009f1c0, binind=3D, = rem=3D24) at jemalloc_tcache.c:149 ptr =3D i =3D 0 extent =3D 0x50407140 bin_arena =3D 0x50400380 bin =3D ndeferred =3D 0 merged_stats =3D arena =3D 0x50400380 nflush =3D 75 __vla_expr0 =3D item_extent =3D 0xffffd1f0 #5 0x502508a0 in __je_tcache_event_hard (tsd=3D, = tcache=3D0x5009f108) at jemalloc_tcache.c:54 tbin_info =3D binind =3D 7 tbin =3D 0x5009f1c0 #6 0x5029a684 in __free (ptr=3D0x500530c0) at = /usr/powerpc32_src/contrib/jemalloc/include/jemalloc/internal/rtree.h:374 tcache =3D 0x5009f108 tsd =3D log_var =3D log_var =3D #7 0x10025994 in dispatch_imsg (ifix=3D, fd=3D10) at = /usr/powerpc32_src/sbin/dhclient/privsep.c:215 hdr =3D {code =3D IMSG_SCRIPT_WRITE_PARAMS, len =3D 3225} lease =3D {next =3D 0x0, expiry =3D 1588504529, renewal =3D = 1588504229, rebind =3D 1588504454, address =3D {len =3D 4, iabuf =3D = "\300\250\001i", '\000' }, nextserver =3D {len =3D 4,=20= iabuf =3D '\000' }, server_name =3D 0x0, = filename =3D 0x0, medium =3D 0x0, is_static =3D 0, is_bootp =3D 0, = options =3D {{len =3D 0, data =3D 0x0}, {len =3D 4,=20 data =3D 0x500530c8 "\377\377\377"}, {len =3D 0, data =3D = 0x0}, {len =3D 4, data =3D 0x500530d0 "\300\250\001\001"}, {len =3D 0, = data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D 4,=20 data =3D 0x500530d8 "\300\250\001\001"}, {len =3D 0, data = =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D = 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, = { len =3D 0, data =3D 0x0}, {len =3D 0, data =3D 0x0}, {len = =3D 20, data =3D 0x50055200 "hsd1.or.comcast.net."}, {len =3D 0, data =3D = 0x0} , {len =3D 4, data =3D 0x500530e0 ""}, {len =3D = 0,=20 data =3D 0x0}, {len =3D 1, data =3D 0x500530e8 "\005"}, = {len =3D 4, data =3D 0x500530f0 "\300\250\001\001"}, {len =3D 0, data =3D = 0x0} }} medium_len =3D medium =3D totlen =3D 3225 filename_len =3D filename =3D 0x0 ret =3D buf =3D mtu =3D servername_len =3D servername =3D 0x0 reason_len =3D reason =3D --Type for more, q to quit, c to continue without paging-- prefix_len =3D prefix =3D 0x500530c0 "new_" i =3D 0 optlen =3D 0 #8 0x100189f4 in fork_privchld (fd=3D10, fd2=3D) at = /usr/powerpc32_src/sbin/dhclient/dhclient.c:2847 pfd =3D {{fd =3D 10, events =3D 1, revents =3D 1}} nfds =3D #9 0x10017a80 in main (argc=3D, argv=3D) = at /usr/powerpc32_src/sbin/dhclient/dhclient.c:505 pipe_fd =3D {10, 11} rights =3D {cr_rights =3D {1342801412, 18446706484155777024}} immediate_daemon =3D 0 i =3D 0 ch =3D otherpid =3D 8 pw =3D 0x5039b9d8 fd =3D capmode =3D (gdb) disass Dump of assembler code for function arena_dalloc_bin_locked_impl: 0x502916b8 <+0>: mflr r0 0x502916bc <+4>: stw r0,4(r1) 0x502916c0 <+8>: stwu r1,-48(r1) 0x502916c4 <+12>: stw r30,40(r1) 0x502916c8 <+16>: stw r24,16(r1) 0x502916cc <+20>: stw r25,20(r1) 0x502916d0 <+24>: stw r26,24(r1) 0x502916d4 <+28>: stw r27,28(r1) 0x502916d8 <+32>: stw r28,32(r1) 0x502916dc <+36>: stw r29,36(r1) 0x502916e0 <+40>: bl 0x502916e4 = 0x502916e4 <+44>: mr r27,r3 0x502916e8 <+48>: mflr r30 0x502916ec <+52>: addis r30,r30,14 0x502916f0 <+56>: addi r30,r30,7788 0x502916f4 <+60>: mr r28,r4 0x502916f8 <+64>: lwz r4,5856(r30) 0x502916fc <+68>: lwz r3,4(r5) 0x50291700 <+72>: mr r29,r5 0x50291704 <+76>: andi. r5,r7,1 0x50291708 <+80>: mr r26,r6 0x5029170c <+84>: lbz r4,0(r4) 0x50291710 <+88>: rlwinm r5,r3,14,25,31 0x50291714 <+92>: mulli r24,r5,224 0x50291718 <+96>: mulli r25,r5,44 0x5029171c <+100>: cmpwi cr1,r4,0 0x50291720 <+104>: cror 4*cr5+lt,4*cr1+eq,gt 0x50291724 <+108>: bge cr5,0x50291a2c = 0x50291728 <+112>: lwz r4,0(r29) 0x5029172c <+116>: lwz r6,6036(r30) 0x50291730 <+120>: lwz r7,8(r29) 0x50291734 <+124>: rlwinm r8,r5,2,0,29 0x50291738 <+128>: li r9,1 0x5029173c <+132>: add r24,r28,r24 0x50291740 <+136>: lwzx r6,r6,r8 0x50291744 <+140>: subf r7,r7,r26 0x50291748 <+144>: mulhwu r6,r6,r7 0x5029174c <+148>: rlwinm r7,r6,29,3,29 0x50291750 <+152>: add r7,r29,r7 =3D> 0x50291754 <+156>: lwz r8,36(r7) 0x50291758 <+160>: clrlwi r10,r6,27 0x5029175c <+164>: slw r9,r9,r10 0x50291760 <+168>: xor r9,r9,r8 0x50291764 <+172>: cmplwi r8,0 0x50291768 <+176>: stw r9,36(r7) 0x5029176c <+180>: bne 0x502917e4 = 0x50291770 <+184>: lwz r7,4408(r30) 0x50291774 <+188>: mulli r8,r5,44 0x50291778 <+192>: add r5,r7,r8 0x5029177c <+196>: lwz r5,16(r5) 0x50291780 <+200>: cmplwi r5,2 0x50291784 <+204>: blt 0x502917e4 = msr cr 0x42480c00 1112017920 lr 0x502916e4 0x502916e4 = ctr 0x5005d114 1342558484 xer 0x0 0 fpscr 0x0 0 vscr vrsave =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)