From owner-freebsd-hackers@freebsd.org Tue Oct 16 04:00:25 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62C4C10DFE76 for ; Tue, 16 Oct 2018 04:00:25 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-21.consmr.mail.gq1.yahoo.com (sonic306-21.consmr.mail.gq1.yahoo.com [98.137.68.84]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B814978E47 for ; Tue, 16 Oct 2018 04:00:24 +0000 (UTC) (envelope-from marklmi@yahoo.com) X-YMail-OSG: z17AQoIVM1mbFSZ8AMx5keokcT1PbUjyxr1ezsLrKpbUDw5qAmuAM.VMo5Gik4t N4sE2VL7jB8Ygf0FccRtQ1TUoLmCWb_5hIVIfvJPbxR16N7VSVKW03uZ9QHA1.kz7nQE8wGVI_GP JY__t3tiCQSCIXa7WdwG5.RJxL8QgO.fR4WJ2QdrT4sokem.XdoMSVpu_UUBCUJ1hGgMBWOkyWsU eTRS9VvmxrEpYqYSOpkfe11kL_ILSVqrSVT2PmHYMGt7TT3aya0ksou.7jFUXl.Hc2ZO1sGYjms6 Wf56SBolvP0hXq34x0jKKu43HjWtibzwXjjuGr28IXEvxt.anLH1_0CDbzLUZVYWhXcO9RuqKAXR bMCJhTBRh1ru2iSiZPGWzkM.MWcdnjOF.VcPr4y.qAQNFdqDiG6LeppiIMta6wueo1SdAufDW8PV .dkSXVwdCc_lVQTeXRGyIY21V1J_KmyiuigFMZ6mdWmrP99dpsFXp52BylCfibqIdW.VQ55AcKQv CXBkHd9fXPbcwP.njaV7SZywy2UqrxB9AUWjJppbVVHO0BGvB9yK.pHZvsD8ZG1w8nhc7roWL0WF ArMFytAZut2mWOHTcNZ6jqKXcDnE41fh.If0YtpslBt0HFqfyv.vnIlsZOuM.WbRjGO0hzziRyIq 8VTwaDICKgwR8v4Imp4nEyQxYvFHd6o12SAT4nLpCrH.6sV73sPHYo6nQrTDX8_F_.9xDn.ojwk8 uKufXRQexAnhyQMv7f_DOtRDrF3kwjzX1Aoaq7vBU9XXcKMnX4BSYNfgr9vhJJqYtYiFtAUPUWMO ho3NherbyWZlGpbo6OkKmh0w_hvkod2BIdi5y5jJsQF2Qi990FmZP3rlWTEVFGJF4zw.PF5LNVfv ROek40xAQgUI3LV7ka7iEl48dTcPvxarkbzSma.FtL3tDP2DenTktDUUHdJHMjvwtkse6_iPFqJa 1D2HQxyURmkF6CXsXMHgba0wgF7wEHQ_00FMYTJVgPRE_ein_dxqoBmCD4Hiedjoo6GdzPZ62ook skq.TEmJN93Mehsc0Q_1.zMLK9efPp8TUnm55OmqyJj4H_pQuiXRgvdSoOWi_oUZmVCNUefNRgzl NoJszC.0dXmfNYA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Tue, 16 Oct 2018 04:00:18 +0000 Received: from c-76-115-7-162.hsd1.or.comcast.net (EHLO [192.168.1.25]) ([76.115.7.162]) by smtp431.mail.gq1.yahoo.com (Oath Hermes SMTP Server) with ESMTPA ID dd93d55728f71f0de1169e4749adc2d5; Tue, 16 Oct 2018 04:00:17 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: FYI: powerpc64 headbuilt via devel/powerpc64-xtoolchain-gcc and C++ exceptions for user code built by system-clang or devel/powerpc64-gcc (as of head -r339076 and ports -r480180) From: Mark Millard In-Reply-To: <0539C16B-1603-4639-914A-0308578C7262@yahoo.com> Date: Mon, 15 Oct 2018 21:00:17 -0700 Cc: Justin Hibbits Content-Transfer-Encoding: quoted-printable Message-Id: <3B69D483-AEBA-47CA-B140-7445089EB064@yahoo.com> References: <0539C16B-1603-4639-914A-0308578C7262@yahoo.com> To: FreeBSD PowerPC ML , FreeBSD Toolchain , FreeBSD Mailing List X-Mailer: Apple Mail (2.3445.9.1) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Oct 2018 04:00:25 -0000 [I've found the problem at the low level for my context of using WITH_LIBCPLUSPLUS=3D with the likes of devel/powerpc64-gcc but I do not have a solution for WITH_LIBCPLUSPLUS=3D so far. I give details of what I found.] On 2018-Oct-14, at 12:40 AM, Mark Millard wrote: > On 2018-Oct-12, at 1:59 PM, Mark Millard wrote: >=20 >> I built a powerpc64 head -r339076 via devel/powerpc64-gcc >> and the like that built clang as cc as well (and >> WITHOUT_LIB32). This included use of base/binutils to >> the the powerpc64 set up. The system and kernel are >> non-debug builds (but with symbols). [system-clang is not >> used for buildworld buildkernel because of known >> issues (last I tried).] >>=20 >> booting, buildworld, buildkernel, poudriere building >> what totaled to be somewhat under 400 ports all seem >> to work. But . . . >>=20 >> It been a long time since I've done something analogous >> and a significant item in the result is different than in >> the past once I started testing the throwing of C++ >> exceptions in code produced by system-clang or by >> devel/powerpc64-gcc : >>=20 >> Such code ends up stuck using around 100% of a CPU. >> An example is the program: >>=20 >> # more exception_test.cpp >> #include >>=20 >> int main(void) >> { >> try { throw std::exception(); } >> catch (std::exception& e) {} >> return 0; >> } >>=20 >> For system-clang it ended up with: >>=20 >> # ldd a.out >> a.out: >> libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x81006d000) >> libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x810184000) >> libm.so.5 =3D> /lib/libm.so.5 (0x8101ab000) >> libc.so.7 =3D> /lib/libc.so.7 (0x8101eb000) >> libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x810554000) >>=20 >> That program goes into an possibly unbounded execution. >> (Historically when this program had problems it would >> stop and produce a core file.) >>=20 >> When compiled by devel/powerpc64-gcc the a.out that results >> does the same thing. ( = /usr/local/bin/powerpc64-unknown-freebsd12.0-c++=20 >> as the compiler path ) So this is not really clang specific >> in any way. This ended up with: >>=20 >> # ldd a.out >> a.out: >> libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x81006d000) >> libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x810184000) >> libm.so.5 =3D> /lib/libm.so.5 (0x8101ab000) >> libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x8101eb000) >> libc.so.7 =3D> /lib/libc.so.7 (0x810211000) >>=20 >> (That should not have involved clang or llvm at all.) >>=20 >> But compiled by lang/gcc8's g++8 the a.out that results works >> fine. This ends up with: >>=20 >> # ldd a.out >> a.out: >> libstdc++.so.6 =3D> /usr/local/lib/gcc8/libstdc++.so.6 = (0x81006e000) >> libm.so.5 =3D> /lib/libm.so.5 (0x8102c7000) >> libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x810307000) >> libc.so.7 =3D> /lib/libc.so.7 (0x81032d000) >>=20 >> It is not clear if using base/gcc as system cc >> would do any better than using system-clang does >> or than devel/powerpc64-gcc does: it is sort of >> a variant of devel/powerpc64-gcc . >>=20 >> It will probably be some time before I figure out >> much about what is going on. >>=20 >> Two things common to the problem cases are: >>=20 >> libc++.so.1 =3D> /usr/lib/libc++.so.1 (0x81006d000) >> libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x810184000) >>=20 >> lang/gcc8 avoids those being involved. >>=20 >>=20 >> Notes: >>=20 >> . . . >>=20 >> WITHOUT_LIB32=3D is because, for every post-gcc 4.2.1 >> that I've tried, the lib32 produced misuses R30 in >> crtbeginS code (vs. the ABI for FreeBSD) and 32-bit >> code just produces core files from the bad so-called >> address dereference that results. >>=20 >> I'd rather have throwing C++ exceptions working and >> lack of lib32 than have lib32 but not have throwing >> C++ exceptions working. But at the moment how to have >> such is not obvious when fairly modern compilers >> and toolchains are involved.=20 >=20 > Here is what I've found so far. >=20 > The code is looping in the following routine. > (I've inserted 2 NOTE: lines for what the > sustained looping is like.) >=20 . . . (the routine was _Unwind_RaiseException) . . . So far I've found that the following in _Unwind_RaiseException stays invariant once initialized --despite the uw_frame_state_for and uw_update_context calls in _Unwind_RaiseException 's loop that normally update fs: (gdb) print fs $15 =3D {regs =3D {reg =3D {{loc =3D {reg =3D 0, offset =3D 0, exp =3D = 0x0}, how =3D REG_UNSAVED} , {loc =3D {reg =3D = 18446744073709551608, offset =3D -8,=20 exp =3D 0xfffffffffffffff8 }, how =3D REG_SAVED_OFFSET}, {loc =3D = {reg =3D 0, offset =3D 0, exp =3D 0x0}, how =3D REG_UNSAVED} , { loc =3D {reg =3D 16, offset =3D 16, exp =3D 0x10 }, how =3D REG_SAVED_OFFSET}, {loc =3D {reg =3D = 0, offset =3D 0, exp =3D 0x0}, how =3D REG_UNSAVED} },=20= prev =3D 0x0}, cfa_offset =3D 0, cfa_reg =3D 1, cfa_exp =3D 0x0, = cfa_how =3D CFA_REG_OFFSET, pc =3D 0x8101999f8, personality =3D 0, = data_align =3D -8, code_align =3D 4, retaddr_column =3D 65,=20 fde_encoding =3D 27 '\033', lsda_encoding =3D 255 '?', saw_z =3D 1 = '\001', signal_frame =3D 0 '\0', eh_ptr =3D 0x0} It turns out that pc value 0x8101999f8 is a little after the libcxxrt.so call to _Unwind_RaiseException that is in throw_exception. But _Unwind_RaiseException returning would be a failure and would end up in a non-returning, error-reporting code path. In other words: this is not an appropriate context for following the return path to unwind out of _Unwind_RaiseException and its internal caller (throw_exception). It got to that address from lr containing the address of the instruction after the one that does bl to the _Unwind_RaiseException routine. Overall it needs to unwind past this in the normal case but is stuck handling the error/no-return case as "the" case. Supporting details follow. What lead up to 0x8101999f8 for initialization was the lr value related to calling _Unwind_RaiseException (see the address in lr below, also copied to r5): . . . (gdb) c Continuing. Breakpoint 9, 1: x/i $pc 0x8101f35d8 <_Unwind_RaiseException+216>: bl = 0x8101f2bc0 Current language: auto; currently minimal where the register values being supplied are (see r5 and lr): (gdb) info reg r0 0x8101999f0 34629851632 r1 0x3fffffffffffc320 4611686018427372320 r2 0x810217900 34630367488 r3 0x3fffffffffffd280 4611686018427376256 r4 0x3fffffffffffd930 4611686018427377968 r5 0x8101999f0 34629851632 r6 0xa0 160 r7 0x0 0 r8 0x1 1 r9 0x8101aac10 34629921808 r10 0x1 1 r11 0x28 40 r12 0x28000282 671089282 r13 0x81005d020 34628554784 r14 0x0 0 r15 0x0 0 r16 0x0 0 r17 0x0 0 r18 0x0 0 r19 0x0 0 r20 0x0 0 r21 0x0 0 r22 0x0 0 r23 0x0 0 r24 0x0 0 r25 0x0 0 r26 0x0 0 r27 0x3fffffffffffd280 4611686018427376256 r28 0x810041060 34628440160 r29 0x3fffffffffffc390 4611686018427372432 r30 0x3fffffffffffcd10 4611686018427374864 r31 0x810041008 34628440072 pc 0x8101f35d8 34630219224 ps 0x0 0 cr 0x0 0 lr 0x8101999f0 34629851632 ctr 0x8101f3500 34630219008 xer 0x0 0 fpscr 0xfff80000 -524288 vscr 0x0 0 vrsave 0x0 0 The pc listed in print fs (0x8101999f8) is in the following from libcxxrt, as is the value in r5 and lr (0x8101999f0): (some blank lines inserted to help identify the area and some related material) (gdb) disass throw_exception Dump of assembler code for function throw_exception: 0x0000000810199960 : mflr r0 0x0000000810199964 : std r31,-8(r1) 0x0000000810199968 : mr r31,r3 0x000000081019996c : std r0,16(r1) 0x0000000810199970 : stdu r1,-128(r1) 0x0000000810199974 : bl 0x810197ab0 = 0x0000000810199978 : ld r10,8(r3) 0x000000081019997c : mr r9,r3 0x0000000810199980 : cmpdi cr7,r10,0 0x0000000810199984 : std r10,24(r31) 0x0000000810199988 : beq- cr7,0x810199a10 = 0x000000081019998c : ld r10,0(r9) 0x0000000810199990 : cmpdi cr7,r10,0 0x0000000810199994 : std r10,32(r31) 0x0000000810199998 : beq- cr7,0x8101999d0 = 0x000000081019999c : lwz r10,48(r9) 0x00000008101999a0 : addi r3,r31,88 0x00000008101999a4 : addi r10,r10,1 0x00000008101999a8 : stw r10,48(r9) 0x00000008101999ac : bl 0x81018e500 = <00000018.plt_call._Unwind_RaiseException@@GCC_3.0> 0x00000008101999b0 : ld r2,40(r1) 0x00000008101999b4 : addi r1,r1,128 0x00000008101999b8 : mr r4,r31 0x00000008101999bc : ld r0,16(r1) 0x00000008101999c0 : ld r31,-8(r1) 0x00000008101999c4 : mtlr r0 0x00000008101999c8 : b 0x8101996b0 = 0x00000008101999cc : nop 0x00000008101999d0 : nop 0x00000008101999d4 : addi r3,r31,88 0x00000008101999d8 : ld r10,-30008(r2) 0x00000008101999dc : std r10,32(r31) 0x00000008101999e0 : lwz r10,48(r9) 0x00000008101999e4 : addi r10,r10,1 0x00000008101999e8 : stw r10,48(r9) 0x00000008101999ec : bl 0x81018e500 = <00000018.plt_call._Unwind_RaiseException@@GCC_3.0> 0x00000008101999f0 : ld r2,40(r1) 0x00000008101999f4 : addi r1,r1,128 0x00000008101999f8 : mr r4,r31 0x00000008101999fc : ld r0,16(r1) 0x0000000810199a00 : ld r31,-8(r1) 0x0000000810199a04 : mtlr r0 0x0000000810199a08 : b 0x8101996b0 = 0x0000000810199a0c : nop 0x0000000810199a10 : nop 0x0000000810199a14 : ld r10,-30000(r2) 0x0000000810199a18 : std r10,24(r31) 0x0000000810199a1c : b 0x81019998c = 0x0000000810199a20 : .long 0x0 0x0000000810199a24 : .long 0x90001 0x0000000810199a28 : lwz r0,0(r1) End of assembler dump. For: 0x00000008101999f8 (above) objdump shows: 00000000000159f8 (below): 0000000000015960 <.__cxa_end_catch+0x460> mflr r0 0000000000015964 <.__cxa_end_catch+0x464> std r31,-8(r1) 0000000000015968 <.__cxa_end_catch+0x468> mr r31,r3 000000000001596c <.__cxa_end_catch+0x46c> std r0,16(r1) 0000000000015970 <.__cxa_end_catch+0x470> stdu r1,-128(r1) 0000000000015974 <.__cxa_end_catch+0x474> bl 0000000000013ab0 = <._ZdaPv+0x590> 0000000000015978 <.__cxa_end_catch+0x478> ld r10,8(r3) 000000000001597c <.__cxa_end_catch+0x47c> mr r9,r3 0000000000015980 <.__cxa_end_catch+0x480> cmpdi cr7,r10,0 0000000000015984 <.__cxa_end_catch+0x484> std r10,24(r31) 0000000000015988 <.__cxa_end_catch+0x488> beq cr7,0000000000015a10 = <.__cxa_end_catch+0x510> 000000000001598c <.__cxa_end_catch+0x48c> ld r10,0(r9) 0000000000015990 <.__cxa_end_catch+0x490> cmpdi cr7,r10,0 0000000000015994 <.__cxa_end_catch+0x494> std r10,32(r31) 0000000000015998 <.__cxa_end_catch+0x498> beq cr7,00000000000159d0 = <.__cxa_end_catch+0x4d0> 000000000001599c <.__cxa_end_catch+0x49c> lwz r10,48(r9) 00000000000159a0 <.__cxa_end_catch+0x4a0> addi r3,r31,88 00000000000159a4 <.__cxa_end_catch+0x4a4> addi r10,r10,1 00000000000159a8 <.__cxa_end_catch+0x4a8> stw r10,48(r9) 00000000000159ac <.__cxa_end_catch+0x4ac> bl 000000000000a500 = 00000000000159b0 <.__cxa_end_catch+0x4b0> ld r2,40(r1) 00000000000159b4 <.__cxa_end_catch+0x4b4> addi r1,r1,128 00000000000159b8 <.__cxa_end_catch+0x4b8> mr r4,r31 00000000000159bc <.__cxa_end_catch+0x4bc> ld r0,16(r1) 00000000000159c0 <.__cxa_end_catch+0x4c0> ld r31,-8(r1) 00000000000159c4 <.__cxa_end_catch+0x4c4> mtlr r0 00000000000159c8 <.__cxa_end_catch+0x4c8> b 00000000000156b0 = <.__cxa_end_catch+0x1b0> 00000000000159cc <.__cxa_end_catch+0x4cc> nop 00000000000159d0 <.__cxa_end_catch+0x4d0> nop 00000000000159d4 <.__cxa_end_catch+0x4d4> addi r3,r31,88 00000000000159d8 <.__cxa_end_catch+0x4d8> ld r10,-30008(r2) 00000000000159dc <.__cxa_end_catch+0x4dc> std r10,32(r31) 00000000000159e0 <.__cxa_end_catch+0x4e0> lwz r10,48(r9) 00000000000159e4 <.__cxa_end_catch+0x4e4> addi r10,r10,1 00000000000159e8 <.__cxa_end_catch+0x4e8> stw r10,48(r9) 00000000000159ec <.__cxa_end_catch+0x4ec> bl 000000000000a500 = 00000000000159f0 <.__cxa_end_catch+0x4f0> ld r2,40(r1) 00000000000159f4 <.__cxa_end_catch+0x4f4> addi r1,r1,128 00000000000159f8 <.__cxa_end_catch+0x4f8> mr r4,r31 00000000000159fc <.__cxa_end_catch+0x4fc> ld r0,16(r1) 0000000000015a00 <.__cxa_end_catch+0x500> ld r31,-8(r1) 0000000000015a04 <.__cxa_end_catch+0x504> mtlr r0 0000000000015a08 <.__cxa_end_catch+0x508> b 00000000000156b0 = <.__cxa_end_catch+0x1b0> 0000000000015a0c <.__cxa_end_catch+0x50c> nop 0000000000015a10 <.__cxa_end_catch+0x510> nop 0000000000015a14 <.__cxa_end_catch+0x514> ld r10,-30000(r2) 0000000000015a18 <.__cxa_end_catch+0x518> std r10,24(r31) 0000000000015a1c <.__cxa_end_catch+0x51c> b 000000000001598c = <.__cxa_end_catch+0x48c> 0000000000015a20 <.__cxa_end_catch+0x520> .long 0x0 0000000000015a24 <.__cxa_end_catch+0x524> .long 0x90001 0000000000015a28 <.__cxa_end_catch+0x528> lwz r0,0(r1) And dwarfdump shows starting at 0x00015960 as: < 117><0x00015960:0x00015a2c><> 0x00015960: =20 0x00015968: =20 0x00015974: =20 0x000159b8: =20 0x000159c8: =20 0x000159d0: =20 0x000159f8: =20 0x00015a08: =20 0x00015a10: =20 fde section offset 4120 0x00001018 cie offset for fde: 4124 0x0000101c 0 DW_CFA_advance_loc 8 (2 * 4) 1 DW_CFA_register r65 =3D r0 4 DW_CFA_offset r31 -8 (1 * -8) 6 DW_CFA_advance_loc 12 (3 * 4) 7 DW_CFA_def_cfa_offset 128 10 DW_CFA_offset_extended_sf r65 16 (-2 * -8) 13 DW_CFA_advance_loc 68 (17 * 4) 14 DW_CFA_remember_state 15 DW_CFA_def_cfa_offset 0 17 DW_CFA_advance_loc 16 (4 * 4) 18 DW_CFA_restore_extended r65 20 DW_CFA_restore r31 21 DW_CFA_advance_loc 8 (2 * 4) 22 DW_CFA_restore_state 23 DW_CFA_advance_loc 40 (10 * 4) 24 DW_CFA_remember_state 25 DW_CFA_def_cfa_offset 0 27 DW_CFA_advance_loc 16 (4 * 4) 28 DW_CFA_restore_extended r65 30 DW_CFA_restore r31 31 DW_CFA_advance_loc 8 (2 * 4) 32 DW_CFA_restore_state 33 DW_CFA_nop 34 DW_CFA_nop /usr/src/contrib/libstdc++/libsupc++/eh_throw.cc has something that /usr/src/contrib/libcxxrt/exception.cc does not have in its error handling code path: a use of __cxa_begin_catch=20 in __cxxabiv1::__cxa_throw : #ifdef _GLIBCXX_SJLJ_EXCEPTIONS _Unwind_SjLj_RaiseException (&header->unwindHeader); #else _Unwind_RaiseException (&header->unwindHeader); #endif // Some sort of unwinding error. Note that terminate is a handler. __cxa_begin_catch (&header->unwindHeader); std::terminate (); It looks to me like __cxa_begin_catch might do either of terminate or _Unwind_Resume and that the (conditional) _Unwind_Resume case is possibly needed here for the normal execution. There are also no other calls (bl's) before the terminate: more directly indicated to not return. I do not know if one or both of those points helped the code unwind correctly or not. But it seems suggestive. Other notes: I've demonstrated the problem on my prior head -r333594 environment that had been build via a 6.3 vintage of devel/powerpc64-gcc (but that was then updated to ports -r469844 and so had 6.4 of devel/powerpc64-gcc installed). Also: base/binutils was of a 2.30 vintage and base/gcc was of a 6.3 vintage. (system-clang was not cc, base/gcc provided system-cc.) Compiling to produce the a.out via: /usr/bin/powerpc64-unknown-freebsd12.0-g++ (and so via a 6.3 vintage g++) made no difference. It still had the problem. I have taken to having buildworld buildkernel use -gdwarf-2 so that /usr/libexec/gdb has access to the information in a format it is (mostly) designed for. (/usr/local/bin/gdb is broken by the thrown-C++-exception problem that I'm investigating: gdb internally throws exceptions in normal operation.) =3D=3D=3D Mark Millard marklmi at yahoo.com ( dsl-only.net went away in early 2018-Mar)