Date: Wed, 21 Mar 2012 10:11:37 -0700 (PDT) From: Doug Ambrisko <ambrisko@ambrisko.com> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-arch@freebsd.org Subject: Re: rtld enhancement to add osversion, version 2 Message-ID: <201203211711.q2LHBbO2026327@ambrisko.com> In-Reply-To: <201203191659.34568.jhb@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
John Baldwin writes: | On Tuesday, March 13, 2012 1:29:17 pm Doug Ambrisko wrote: | > This is round 2 of my rtld enhancements. The primary goal it to have | > rtld look in different places for libraries depending on the legacy | > binary that we want to run. This is especially a problem with binaries | > linked to libraries from ports since the version of a library in | > /usr/local/lib is the same whether it was FreeBSD 6, 7, 8 etc. until the | > library itself changes version due to an interface change. At work we | > need to run 3rd party binaries on our box of which we don't have source. | > Having a FreeBSD 6 binaries load /usr/local/lib/libiconv.so.3 that was | > built for FreeBSD 9 is not good. It is worse when libiconv.so.3 is | > linked to libc.so.7 when the FreeBSD 6 binary needs libc.so.6. | > | > I solved this by having rtld extract the OSVERSION from the binary | > and then use that to determine what to do. In my prior version, | > I inserted that into the library directory. That meant to pull | > in libc it would look at: | > /lib/603000/libc.so.6 | > /lib/6/libc.so.6 | > /lib/libc.so.6 | > to find libc.so.6. This meant a lot more look ups. Also I found it | > had a problem in that if we had an ambiguous name say | > /usr/local/lib/libcrypto.so.5 on the FreeBSD 6 machine and | /lib/libcrypto.so.5 | > on the new FreeBSD 8 system, just doing the search logic would get | > a hit on /lib/libcrypto.so.5 before it got to /usr/local/lib/libcrypto.so.5. | > So then it would mean we'd have to put all FreeBSD lib's in /lib/6 to | > beat the search path. This wasn't a good solution. To solve the | > performance and path issue, I now follow the 32 bit hints file name | > model. Now it does a lookup of the hints file based on the osversion | > and major. So now it looks for the hints file as: | > /var/run/ld-elf-603000.so.hints | > /var/run/ld-elf-6.so.hints | > /var/run/ld-elf.so.hints | > This is faster and has more unique paths for FreeBSD 6 libraries since | > the FreeBSD 6 search paths would be in the hints file. I modified | > ldconfig to accept an "-os=<version>" option to create this hints file. | > I tweaked /etc/rc* to make this easy to setup like this: | > ldconfig_os_versions="6" | > ldconfig_6_path="/usr/local/lib/compat/6" | | I think this is a definite improvement from before, thanks! | | > This doesn't solve the LD_LIBRARY_PATH or LD_PRELOAD. Solving that | > I still insert and iterate over the osverion, major and none into the | > path to find the library. The reason for this is that a FreeBSD 8 | > binary could exec a FreeBSD 6 binary that execs a FreeBSD 7 binary. | > If for the FreeBSD 6 binary we needed to set a custom LD_LIBRARY_PATH | > and the FreeBSD 7 binary find a library via the FreeBSD 6 search path | > then the FreeBSD 7 binary would die. By adding in the osversion search | > path I can put the FreeBSD 6 library into the search path + the directory | > 6. Then only FreeBSD 6 binaries can find it. An example: | > LD_LIBRARY_PATH=/usr/custom_software/lib | > /usr/custom_software/lib/6/libfoo.so.6 | > then only the FreeBSD 6 binary could load it. Since the searches | > would be for the FreeBSD 6 binary: | > /usr/custom_software/lib/603000/libfoo.so.6 | > /usr/custom_software/lib/6/libfoo.so.6 | > /usr/custom_software/lib/libfoo.so.6 | > and for FreeBSD 7 binary: | > /usr/custom_software/lib/702000/libfoo.so.6 | > /usr/custom_software/lib/7/libfoo.so.6 | > /usr/custom_software/lib/libfoo.so.6 | > Only the FreeBSD 6 binary would load /usr/custom_software/lib/6/libfoo.so.6. | > I do the same search with LD_PRELOAD. | | Hmm, I'm still not quite fan of this. Perhaps you could add an extension to | ldconfig and the hints file to handle this case? That is, a way to store | path mappings so you could do something like: | | ldconfig -os=6 -p /usr/local/lib /usr/local/lib/6 I'll have to look to see how the hints file could update rtld. It is an interesting idea. Maybe libmap.conf would be better place for this. I haven't looked at how libmap works. Maybe introduce: /etc/libmap-<OSVERSION>.conf that maps paths as well. So with the above example. /etc/libmap-6.conf would contain /usr/custom_software/lib /usr/custom_software/lib/6:/usr/custom_software/lib for example. | Or maybe you could make it an extension of how 'm' worked, so you could make | directories accept an optional set of colon-separated paths that they serve | as aliases for: | | ldconfig -os=6 -m /usr/local/lib/6:/usr/local/lib:/usr/lib I don't really get how this is solving the LD_LIBRARY_PATH/LD_PRELOAD since "-m" is solving the general case with the hints file which I did first. | (That would even fit into your existing rc.d script changes I believe). Then | rtld would keep this internal directory mapping and be able to map the | '/usr/local/lib' and '/usr/lib' directories in LD_PRELOAD and LD_LIBRARY_PATH | to /usr/local/lib/6. The advantage of this is the same as with your previous | change that all the mappings are configurable and not hard-coded into rtld | itself. | > Final, is that prior binaries built on FreeBSD i386 but run on FreeBSD amd64 | > that set LD_* environemnt variables would fail on FreeBSD amd64 as is | > since it didn't set the equivalent LD32_*. To address this for COMPAT_32BIT | > I have rtld look for LD32_* first and then check for the LD_* second. | > This way legacy applications work fine. | | Hmm, so Yahoo! had some patches to handle this as well. I think Yahoo's | patches were different though. They actually patched the 32-bit libc to | capture attempts to get or set LD_* and convert them to actually get/set | LD32_* instead. I'm not sure which approach is best, but it might be worth | asking Peter why Yahoo! did it that way and if there were reasons for that | approach vs. doing it in rtld. I think the primary reason was so that you | could set LD_LIBRARY_PATH or LD_PRELOAD to reference 64-bit libraries and | not have it break 32-bit apps, but if a 32-bit app tried to set a variable | before launching another app it would still DTRT. This means you would have to have a modified libc in these environment and it wouldn't help in the static binary case. I think we are safe in the case LD_ for a 32 bit binary effecting a 64 bit since rtld should not allow loading a the wrong type of lib. into a binary. I can do some more testing around that area. I was trying to keep all changes in the host environment so we can run unchanged binaries from other vendors. | I do think this is definitely a problem worth solving. Thanks, Doug A.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201203211711.q2LHBbO2026327>