Date: Sat, 30 Aug 2014 22:36:57 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Bryan Drewery <bdrewery@FreeBSD.org> Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r270803 - head/libexec/rtld-elf Message-ID: <20140830193657.GR2737@kib.kiev.ua> In-Reply-To: <5400D873.70707@FreeBSD.org> References: <201408291044.s7TAiwmI077897@svn.freebsd.org> <540094DB.5050307@FreeBSD.org> <20140829172311.GP2737@kib.kiev.ua> <5400D873.70707@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--JpBuuFAFSbNvDzdI
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, Aug 29, 2014 at 02:45:55PM -0500, Bryan Drewery wrote:
> My only uses so far have been observing which file is loaded for a
> library with dlopen(3) and where symbols are resolved from. Seeing where
> symbols are resolved from is useful for both dlopen(3)/dlsym(3) usage
> and LD_PRELOAD.
Ok.
Note that there are no traces of why the symbols where resolved from
the specified object, and, why that symbol was selected (one object
may contain several symbols with the same name, for several reasons).
>=20
> I've written an application that "optionally" allows using some
> libraries such as libtcl to be used without requiring the library at
> build time or startup. At build time the library's headers are used to
> generate wrapper functions for every symbol needed. The library is not
> linked in. When the binary is ran it starts up fine if libtcl is
> missing. At runtime if TCL is enabled then it will attempt to
> dlopen(libtcl.so.VER) using the VER its symbols was compiled against. If
> it cannot find it then the feature is disabled. It uses dlsym(3) to load
> all symbols into a table and the wrapper functions use that table via
> hash table lookups.
Note that ELF PLT symbols (i.e. function call sites) resolution is lazy
by default, i.e. the only tricky part to do for this with the standard
tools is to avoid writing the DT_NEEDED tag into the binary you construct.
Newer GNU ld has interesting options like --unresolved-symbols=3Dignore-all,
or --warn-unresolved-symbols, which seemingly allow to link the binary
even with unresolved symbols.
If you use then dlopen(libname, RTLD_LAZY | RTLD_GLOBAL) before calling
the functions from the library, you should get what you want without
needing to laborously provide stubs and use dlsym().
And, this cannot work for non-PLT relocations, e.g. references to global
variables.
>=20
> This all pretty convoluted and only done to avoid requiring a library at
> startup. The application is distributed pre-compiled for specific OS
> releases and not intended to be compiled manually.
>=20
> Anyway some examples from GNU's rtld:
>=20
> > # env LD_DEBUG=3Dhelp ./app
> > Valid options for the LD_DEBUG environment variable are:
> >=20
> > libs display library search paths
> > reloc display relocation processing
> > files display progress for input file
> > symbols display symbol table processing
> > bindings display information about symbol binding
> > versions display version dependencies
> > all all previous options combined
> > statistics display relocation statistics
> > unused determined unused DSOs
> > help display this help message and exit
> >=20
> > To direct the debugging output into a file instead of standard output
> > a filename can be specified using the LD_DEBUG_OUTPUT environment varia=
ble.
>=20
> With libs:
>=20
> > # env LD_DEBUG=3Dlibs ./app
> > ...
> > 9328: find library=3Dlibtcl.so [0]; searching
> > 9328: search cache=3D/etc/ld.so.cache
> > 9328: search path=3D/lib64/tls/x86_64:/lib64/tls:/lib64/x86_=
64:/lib64:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/x86_64:/usr/lib64=
(system search path)
> > 9328: trying file=3D/lib64/tls/x86_64/libtcl.so
> > 9328: trying file=3D/lib64/tls/libtcl.so
> > 9328: trying file=3D/lib64/x86_64/libtcl.so
> > 9328: trying file=3D/lib64/libtcl.so
> > 9328: trying file=3D/usr/lib64/tls/x86_64/libtcl.so
> > 9328: trying file=3D/usr/lib64/tls/libtcl.so
> > 9328: trying file=3D/usr/lib64/x86_64/libtcl.so
> > 9328: trying file=3D/usr/lib64/libtcl.so
> > 9328:
> > 9328: find library=3Dlibpthread.so.0 [0]; searching
> > 9328: search path=3D/usr/lib64 (system search path)
> > 9328: trying file=3D/usr/lib64/libpthread.so.0
> > 9328: search cache=3D/etc/ld.so.cache
> > 9328: trying file=3D/lib64/libpthread.so.0
> > 9328:
> > 9328:
> > 9328: calling init: /lib64/libpthread.so.0
> > 9328:
> > 9328:
> > 9328: calling init: /usr/lib64/libtcl.so
> > 9328:
> > 9328: find library=3Dlibnss_compat.so.2 [0]; searching
> > 9328: search cache=3D/etc/ld.so.cache
> > 9328: trying file=3D/lib64/libnss_compat.so.2
> > 9328:
> > 9328: find library=3Dlibnsl.so.1 [0]; searching
> > 9328: search cache=3D/etc/ld.so.cache
> > 9328: trying file=3D/lib64/libnsl.so.1
> > 9328:
> > 9328:
> > 9328: calling init: /lib64/libnsl.so.1
> > 9328:
> > 9328:
> > 9328: calling init: /lib64/libnss_compat.so.2
> > 9328:
> > 9328: find library=3Dlibnss_nis.so.2 [0]; searching
> > 9328: search cache=3D/etc/ld.so.cache
> > 9328: trying file=3D/lib64/libnss_nis.so.2
> > 9328:
> > 9328: find library=3Dlibnss_files.so.2 [0]; searching
> > 9328: search cache=3D/etc/ld.so.cache
> > 9328: trying file=3D/lib64/libnss_files.so.2
> > 9328:
> > 9328:
> > 9328: calling init: /lib64/libnss_files.so.2
> > 9328:
> > 9328:
> > 9328: calling init: /lib64/libnss_nis.so.2
> > ^C
> > 9328: calling fini: /usr/lib64/libtcl.so [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libm.so.6 [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libdl.so.2 [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libpthread.so.0 [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libnss_compat.so.2 [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libnss_nis.so.2 [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libnsl.so.1 [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libnss_files.so.2 [0]
> > 9328:
> > 9328:
> > 9328: calling fini: /lib64/libc.so.6 [0]
>=20
> With symbols:
>=20
> > # LD_DEBUG=3Dsymbols ./app
> > ...
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib=
dl.so.2 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib/g=
cc/x86_64-pc-linux-gnu/4.8.2/libstdc++.so.6 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib=
m.so.6 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib/g=
cc/x86_64-pc-linux-gnu/4.8.2/libgcc_s.so.1 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib=
c.so.6 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/ld-=
linux-x86-64.so.2 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib64=
/libcrypto.so.1.0.0 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/lib64/lib=
z.so.1 [0]
> > 9355: symbol=3DTcl_SetObjResult; lookup in file=3D/usr/lib64=
/libtcl.so [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D./netplay =
[0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib=
dl.so.2 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib/g=
cc/x86_64-pc-linux-gnu/4.8.2/libstdc++.so.6 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib=
m.so.6 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib/g=
cc/x86_64-pc-linux-gnu/4.8.2/libgcc_s.so.1 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib=
c.so.6 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/ld-=
linux-x86-64.so.2 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib64=
/libcrypto.so.1.0.0 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/lib64/lib=
z.so.1 [0]
> > 9355: symbol=3DTcl_GetObjResult; lookup in file=3D/usr/lib64=
/libtcl.so [0]
>=20
>=20
> I would love to work on adding such a feature to ours, especially if it
> can assist with debugging rtld itself. I would think with some
> predict_false() checks we could avoid much of the performance penalties.
As I said, for debugging rtld, currently existing or, probably, any
predefined set of debugging statements is mostly useless. A startup
problems usually results in coredumps, which is compfortably debugged
by loading core with debugging symbols (-g) in gdb.
If you can determine useful set of the debugging output for application
programmers to introspect the rtld activity, I am all for adding it
and enabling LD_DEBUG unconditionally.
>=20
> As a side note I think I may have found an rtld issue (on 8.4) with -32
> handling. Being a production machine I don't want to attempt installing
> a DEBUG rtld to figure out the problem. Having LD_DEBUG would have been
> useful on this as well.
>=20
> The issue I ran into with -32 was from converting a i386 system to
> amd64. I have hundreds of out-of-ports binaries that need to be manually
> recompiled. To allow delaying this task for a bit I copied all of the
> libraries from /usr/local/lib from the old i386 system to
> /usr/local/lib32/compat/system and added all of the paths in there to
> ldconfig:
>=20
> > [~] # ldconfig -32 -r|head -n3
> > /var/run/ld-elf32.so.hints:
> > search directories: /usr/lib32:/usr/local/lib32/compat/system:/=
usr/local/lib32/compat/system/mysql:/usr/local/lib32/compat/system/gnutls3:=
/usr/local/lib32/compat/system/pth:/usr/local/lib32/compat/system/qt4:/usr/=
local/lib32/compat/system/tdom0.8.3:/usr/local/lib32/compat/system/gcc46:/u=
sr/local/lib32/compat/system/gcc47:/usr/local/lib32/compat/system/gcc48:/us=
r/local/lib32/compat/system/gcc49:/usr/local/lib32/compat/system/event2:/us=
r/local/lib32/compat/pkg:/usr/local/lib32/compat
>=20
> It worked for 90% of the system but then I ran into some libraries that
> refused to find their own dependencies. The issue seemingly is that
> dependencies for libraries that are also in the -32 path are not looked u=
p.
>=20
> For example:
>=20
> > [~] # ldd -a /usr/local/lib32/compat/system/libssl.so
> > /usr/local/lib32/compat/system/libssl.so:
> > libcrypto.so.8 =3D> not found (0x0)
> > libthr.so.3 =3D> /usr/lib32/libthr.so.3 (0x310fb000)
> > libc.so.7 =3D> /usr/lib32/libc.so.7 (0x29165000)
> > /usr/lib32/libthr.so.3:
> > libc.so.7 =3D> /usr/lib32/libc.so.7 (0x29165000)
> > [~] # ldconfig -32 -r|grep libcrypto.so.8
> > 105:-lcrypto.8 =3D> /usr/local/lib32/compat/system/libcrypto.so=
=2E8
> > [~] # readelf -a /usr/local/lib32/compat/system/libssl.so|egrep "(path|=
NEEDED)"
> > 0x00000001 (NEEDED) Shared library: [libcrypto.so.=
8]
> > 0x00000001 (NEEDED) Shared library: [libthr.so.3]
> > 0x00000001 (NEEDED) Shared library: [libc.so.7]
> > 0x0000000f (RPATH) Library rpath: [/usr/local/lib]
>=20
> My guess is it is the RPATH. I am not sure without a simple debug ability.
>=20
> --=20
> Regards,
> Bryan Drewery
>=20
--JpBuuFAFSbNvDzdI
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBAgAGBQJUAifZAAoJEJDCuSvBvK1BnEIP/jK7QVFtc8K3qj3q016N3ZNd
ZWYGWbEOFi9MUKzvySqUwkAQaSOBsqjc9ezUGR+RyQJn1oO5pXBQ2LQDObZq2R5+
1iyNFx3WzqaTf25OGh8bUQzhunWjwkcaKYtOR+ZzAVAH97298qNhDlxK9buA7FnI
AxjQFKkVYd8wZUt5XP4GUzruXQSPU6C077L9MrppJpXi5HRoVPjHoM5Wqt8jdeEc
KpVhyFp5uNp2naAJNsBULU7tRGa7Sso+wQHnMIAPZkdCMI9XNNb9Y7E2k3EkNIdI
I2XMbDD+ZLC/DFmhZBIvygIX9jgpi1tnvulLjLqlL/Tzy5QChxsZ+vsiTFgCQwGU
bwmeWuf6PcW27FN/ncT74GZ8mc197UI9vSXrqNfbLbanU0pvgWKIL2tyePWofItb
bHSwj2Wv75IASeogIbfECqwi5MtzomhtebHVtaBH5Kdsrl1S8DivS8U7l7sJsd+1
vLG0gg95aLS3IvTY5F8zVpcWlZvRnjrCGUhjsR9VR67Sb8L2O4AS2AvBBDj8KqYH
x/c+9v2LtSS5WTRuMMV54a86KWw9OhbWJwcWaJV67u0PlUvsMiBBBAW6cPIwKfTr
dQrKnGsHryKZfcHMrPsxR8/Jl1++6JxvJz5TlHYqox3naF5hs7iOWkGZq628nEyl
EkxVzbgfWn55NiBzRqba
=Emte
-----END PGP SIGNATURE-----
--JpBuuFAFSbNvDzdI--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140830193657.GR2737>