Date: Wed, 24 Aug 2011 11:18:19 +0300 From: Kostik Belousov <kostikbel@gmail.com> To: Dimitry Andric <dim@freebsd.org> Cc: Test Rat <ttsestt@gmail.com>, freebsd-toolchain@freebsd.org Subject: Re: [clang] rtld-elf/rtld.c and stack traces in gdb(1) Message-ID: <20110824081819.GI17489@deviant.kiev.zoral.com.ua> In-Reply-To: <4E541D02.1040506@FreeBSD.org> References: <86fwkvt9me.fsf@gmail.com> <4E541D02.1040506@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--g2QKYTO1/ZKqu+7N
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Tue, Aug 23, 2011 at 11:34:58PM +0200, Dimitry Andric wrote:
> On 2011-08-21 11:08, Test Rat wrote:
> >I often get corrupted traces with clang world, the cause seems to be in=
=20
> >rtld.
> ...
> > (gdb) bt
> > #0 0x00000008009455ac in ?? ()
> > #1 0x0000000800944fa7 in ?? ()
>=20
> After some digging, this turned out to be caused by the empty function
> r_debug_state() in libexec/rtld-elf/rtld.c. This function is just a
> necessary hook for gdb, but since it is completely empty, calls to it in
> the same compilation unit simply don't generate any code, even if the
> function is marked as __noinline.
>=20
> The attached patch fixes this, by marking the function __noinline, and
> inserting an empty asm statement, that pretends to clobber memory. It
> generates no extra code, and forces clang to emit calls to r_debug_state
> throughout rtld.c. It looks rather hackish, though.
>=20
> An alternative solution would be to move the r_debug_state() function to
> another .c file, which should work OK, until we eventually start using
> link time optimization... :)
>=20
>=20
> >And compiling rtld with clang + -O0 makes it crash.
>=20
> This is caused by yet another interesting problem, which is in the
> _rtld() function in rtld.c. It is run at the very beginning of rtld,
> when relocations have not yet been processed. This initial code must be
> very careful to *not* use any relocated symbols, or problems will arise.
>=20
> The early initialization goes like:
>=20
> ...
>=20
> /* Initialize and relocate ourselves. */
> assert(aux_info[AT_BASE] !=3D NULL);
> init_rtld((caddr_t) aux_info[AT_BASE]->a_un.a_ptr, aux_info);
>=20
> __progname =3D obj_rtld.path;
> argv0 =3D argv[0] !=3D NULL ? argv[0] : "(null)";
> environ =3D env;
>=20
> The init_rtld() function takes care of the initial relocations, after
> which 'global' symbols like __progname and environ can be used.
>=20
> However, at -O0, clang still reorders the retrieval of the __progname
> offset to just *before* the init_rtld() call, and assigns it afterwards:
>=20
> ...
> .LBB0_16: # %cond.end
> movq __progname@GOTPCREL(%rip), %rax <-- gets the offs=
et=20
> here
> leaq -224(%rbp), %rsi
> .loc 1 329 5 #=20
> /usr/src/libexec/rtld-elf/rtld.c:329:5
> movq -168(%rbp), %rcx
> movq 8(%rcx), %rdi
> movq %rax, -1504(%rbp) # 8-byte Spill <-- saves offset =
on=20
> stack
> callq init_rtld
> .loc 1 331 5 #=20
> /usr/src/libexec/rtld-elf/rtld.c:331:5
> movq obj_rtld+24(%rip), %rax
> movq -1504(%rbp), %rcx # 8-byte Reload <-- loads offset=
=20
> from stack
> movq %rax, (%rcx) <-- stores value =
in=20
> __progname
>=20
> It's not clear to me why clang does this reordering even when
> optimization is off, but it is normally legal, and quite usual.
> However, in case of this early initialization, it is fatal, as
> __progname@GOTPCREL(%rip) will still be junk, or zero...
>=20
> With optimization, such reorderings are even more likely, but for some
> reason, we have always been lucky that it turned out OK. A possible
> solution would be to move the code after the init_rtld() call to another
> function, and call that, but this could also be defeated again by
> inlining. :(
I think you can try to insert another compiler memory barrier after
the init_rtld.
> Index: libexec/rtld-elf/rtld.c
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> --- libexec/rtld-elf/rtld.c (revision 225105)
> +++ libexec/rtld-elf/rtld.c (working copy)
> @@ -143,7 +143,7 @@ static void ld_utrace_log(int, void *, void *, siz
> static void rtld_fill_dl_phdr_info(const Obj_Entry *obj,
> struct dl_phdr_info *phdr_info);
> =20
> -void r_debug_state(struct r_debug *, struct link_map *);
> +void r_debug_state(struct r_debug *, struct link_map *) __noinline;
> =20
> /*
> * Data declarations.
> @@ -2780,6 +2780,14 @@ linkmap_delete(Obj_Entry *obj)
> void
> r_debug_state(struct r_debug* rd, struct link_map *m)
> {
> + /*
> + * The following is a hack to force the compiler to emit calls to
> + * this function, even when optimizing. If the function is empty,
> + * the compiler is not obliged to emit any code for calls to it,
> + * even when marked __noinline. However, gdb depends on those
> + * calls being made.
> + */
> + __asm __volatile("" : : : "memory");
> }
This is a reasonable change, IMO.
Also, we still compile rtld and csu in the hosted environment, which is the
lie to the compiler.
--g2QKYTO1/ZKqu+7N
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (FreeBSD)
iEYEARECAAYFAk5Us8sACgkQC3+MBN1Mb4iohgCg9IrpApQtadl18L7riRr8wcOs
DI0AoODMEBK9GHN6S0J6DF2Y9a95QeTQ
=MpWV
-----END PGP SIGNATURE-----
--g2QKYTO1/ZKqu+7N--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110824081819.GI17489>