From owner-freebsd-toolchain@FreeBSD.ORG Wed Aug 24 08:18:23 2011 Return-Path: Delivered-To: freebsd-toolchain@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 655A1106566B; Wed, 24 Aug 2011 08:18:23 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id F3F508FC0C; Wed, 24 Aug 2011 08:18:22 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p7O8IJ7A030121 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 24 Aug 2011 11:18:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p7O8IJq7054726; Wed, 24 Aug 2011 11:18:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p7O8IJ7Y054725; Wed, 24 Aug 2011 11:18:19 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 24 Aug 2011 11:18:19 +0300 From: Kostik Belousov To: Dimitry Andric Message-ID: <20110824081819.GI17489@deviant.kiev.zoral.com.ua> References: <86fwkvt9me.fsf@gmail.com> <4E541D02.1040506@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="g2QKYTO1/ZKqu+7N" Content-Disposition: inline In-Reply-To: <4E541D02.1040506@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Test Rat , freebsd-toolchain@freebsd.org Subject: Re: [clang] rtld-elf/rtld.c and stack traces in gdb(1) X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Aug 2011 08:18:23 -0000 --g2QKYTO1/ZKqu+7N Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 23, 2011 at 11:34:58PM +0200, Dimitry Andric wrote: > On 2011-08-21 11:08, Test Rat wrote: > >I often get corrupted traces with clang world, the cause seems to be in= =20 > >rtld. > ... > > (gdb) bt > > #0 0x00000008009455ac in ?? () > > #1 0x0000000800944fa7 in ?? () >=20 > After some digging, this turned out to be caused by the empty function > r_debug_state() in libexec/rtld-elf/rtld.c. This function is just a > necessary hook for gdb, but since it is completely empty, calls to it in > the same compilation unit simply don't generate any code, even if the > function is marked as __noinline. >=20 > The attached patch fixes this, by marking the function __noinline, and > inserting an empty asm statement, that pretends to clobber memory. It > generates no extra code, and forces clang to emit calls to r_debug_state > throughout rtld.c. It looks rather hackish, though. >=20 > An alternative solution would be to move the r_debug_state() function to > another .c file, which should work OK, until we eventually start using > link time optimization... :) >=20 >=20 > >And compiling rtld with clang + -O0 makes it crash. >=20 > This is caused by yet another interesting problem, which is in the > _rtld() function in rtld.c. It is run at the very beginning of rtld, > when relocations have not yet been processed. This initial code must be > very careful to *not* use any relocated symbols, or problems will arise. >=20 > The early initialization goes like: >=20 > ... >=20 > /* Initialize and relocate ourselves. */ > assert(aux_info[AT_BASE] !=3D NULL); > init_rtld((caddr_t) aux_info[AT_BASE]->a_un.a_ptr, aux_info); >=20 > __progname =3D obj_rtld.path; > argv0 =3D argv[0] !=3D NULL ? argv[0] : "(null)"; > environ =3D env; >=20 > The init_rtld() function takes care of the initial relocations, after > which 'global' symbols like __progname and environ can be used. >=20 > However, at -O0, clang still reorders the retrieval of the __progname > offset to just *before* the init_rtld() call, and assigns it afterwards: >=20 > ... > .LBB0_16: # %cond.end > movq __progname@GOTPCREL(%rip), %rax <-- gets the offs= et=20 > here > leaq -224(%rbp), %rsi > .loc 1 329 5 #=20 > /usr/src/libexec/rtld-elf/rtld.c:329:5 > movq -168(%rbp), %rcx > movq 8(%rcx), %rdi > movq %rax, -1504(%rbp) # 8-byte Spill <-- saves offset = on=20 > stack > callq init_rtld > .loc 1 331 5 #=20 > /usr/src/libexec/rtld-elf/rtld.c:331:5 > movq obj_rtld+24(%rip), %rax > movq -1504(%rbp), %rcx # 8-byte Reload <-- loads offset= =20 > from stack > movq %rax, (%rcx) <-- stores value = in=20 > __progname >=20 > It's not clear to me why clang does this reordering even when > optimization is off, but it is normally legal, and quite usual. > However, in case of this early initialization, it is fatal, as > __progname@GOTPCREL(%rip) will still be junk, or zero... >=20 > With optimization, such reorderings are even more likely, but for some > reason, we have always been lucky that it turned out OK. A possible > solution would be to move the code after the init_rtld() call to another > function, and call that, but this could also be defeated again by > inlining. :( I think you can try to insert another compiler memory barrier after the init_rtld. > Index: libexec/rtld-elf/rtld.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- libexec/rtld-elf/rtld.c (revision 225105) > +++ libexec/rtld-elf/rtld.c (working copy) > @@ -143,7 +143,7 @@ static void ld_utrace_log(int, void *, void *, siz > static void rtld_fill_dl_phdr_info(const Obj_Entry *obj, > struct dl_phdr_info *phdr_info); > =20 > -void r_debug_state(struct r_debug *, struct link_map *); > +void r_debug_state(struct r_debug *, struct link_map *) __noinline; > =20 > /* > * Data declarations. > @@ -2780,6 +2780,14 @@ linkmap_delete(Obj_Entry *obj) > void > r_debug_state(struct r_debug* rd, struct link_map *m) > { > + /* > + * The following is a hack to force the compiler to emit calls to > + * this function, even when optimizing. If the function is empty, > + * the compiler is not obliged to emit any code for calls to it, > + * even when marked __noinline. However, gdb depends on those > + * calls being made. > + */ > + __asm __volatile("" : : : "memory"); > } This is a reasonable change, IMO. Also, we still compile rtld and csu in the hosted environment, which is the lie to the compiler. --g2QKYTO1/ZKqu+7N Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk5Us8sACgkQC3+MBN1Mb4iohgCg9IrpApQtadl18L7riRr8wcOs DI0AoODMEBK9GHN6S0J6DF2Y9a95QeTQ =MpWV -----END PGP SIGNATURE----- --g2QKYTO1/ZKqu+7N--