From owner-freebsd-arch@FreeBSD.ORG Wed Mar 21 17:11:43 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C9C2F106564A; Wed, 21 Mar 2012 17:11:43 +0000 (UTC) (envelope-from ambrisko@ambrisko.com) Received: from mail.ambrisko.com (mail.ambrisko.com [70.91.206.90]) by mx1.freebsd.org (Postfix) with ESMTP id 8EB2B8FC16; Wed, 21 Mar 2012 17:11:43 +0000 (UTC) X-Ambrisko-Me: Yes Received: from server2.ambrisko.com (HELO internal.ambrisko.com) ([192.168.1.2]) by ironport.ambrisko.com with ESMTP; 21 Mar 2012 10:11:43 -0700 Received: from ambrisko.com (localhost [127.0.0.1]) by internal.ambrisko.com (8.14.4/8.14.4) with ESMTP id q2LHBbVf026328; Wed, 21 Mar 2012 10:11:37 -0700 (PDT) (envelope-from ambrisko@ambrisko.com) Received: (from ambrisko@localhost) by ambrisko.com (8.14.4/8.14.4/Submit) id q2LHBbO2026327; Wed, 21 Mar 2012 10:11:37 -0700 (PDT) (envelope-from ambrisko) From: Doug Ambrisko Message-Id: <201203211711.q2LHBbO2026327@ambrisko.com> In-Reply-To: <201203191659.34568.jhb@freebsd.org> To: John Baldwin Date: Wed, 21 Mar 2012 10:11:37 -0700 (PDT) X-Mailer: ELM [version 2.4ME+ PL124d (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="US-ASCII" Cc: freebsd-arch@freebsd.org Subject: Re: rtld enhancement to add osversion, version 2 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Mar 2012 17:11:43 -0000 John Baldwin writes: | On Tuesday, March 13, 2012 1:29:17 pm Doug Ambrisko wrote: | > This is round 2 of my rtld enhancements. The primary goal it to have | > rtld look in different places for libraries depending on the legacy | > binary that we want to run. This is especially a problem with binaries | > linked to libraries from ports since the version of a library in | > /usr/local/lib is the same whether it was FreeBSD 6, 7, 8 etc. until the | > library itself changes version due to an interface change. At work we | > need to run 3rd party binaries on our box of which we don't have source. | > Having a FreeBSD 6 binaries load /usr/local/lib/libiconv.so.3 that was | > built for FreeBSD 9 is not good. It is worse when libiconv.so.3 is | > linked to libc.so.7 when the FreeBSD 6 binary needs libc.so.6. | > | > I solved this by having rtld extract the OSVERSION from the binary | > and then use that to determine what to do. In my prior version, | > I inserted that into the library directory. That meant to pull | > in libc it would look at: | > /lib/603000/libc.so.6 | > /lib/6/libc.so.6 | > /lib/libc.so.6 | > to find libc.so.6. This meant a lot more look ups. Also I found it | > had a problem in that if we had an ambiguous name say | > /usr/local/lib/libcrypto.so.5 on the FreeBSD 6 machine and | /lib/libcrypto.so.5 | > on the new FreeBSD 8 system, just doing the search logic would get | > a hit on /lib/libcrypto.so.5 before it got to /usr/local/lib/libcrypto.so.5. | > So then it would mean we'd have to put all FreeBSD lib's in /lib/6 to | > beat the search path. This wasn't a good solution. To solve the | > performance and path issue, I now follow the 32 bit hints file name | > model. Now it does a lookup of the hints file based on the osversion | > and major. So now it looks for the hints file as: | > /var/run/ld-elf-603000.so.hints | > /var/run/ld-elf-6.so.hints | > /var/run/ld-elf.so.hints | > This is faster and has more unique paths for FreeBSD 6 libraries since | > the FreeBSD 6 search paths would be in the hints file. I modified | > ldconfig to accept an "-os=" option to create this hints file. | > I tweaked /etc/rc* to make this easy to setup like this: | > ldconfig_os_versions="6" | > ldconfig_6_path="/usr/local/lib/compat/6" | | I think this is a definite improvement from before, thanks! | | > This doesn't solve the LD_LIBRARY_PATH or LD_PRELOAD. Solving that | > I still insert and iterate over the osverion, major and none into the | > path to find the library. The reason for this is that a FreeBSD 8 | > binary could exec a FreeBSD 6 binary that execs a FreeBSD 7 binary. | > If for the FreeBSD 6 binary we needed to set a custom LD_LIBRARY_PATH | > and the FreeBSD 7 binary find a library via the FreeBSD 6 search path | > then the FreeBSD 7 binary would die. By adding in the osversion search | > path I can put the FreeBSD 6 library into the search path + the directory | > 6. Then only FreeBSD 6 binaries can find it. An example: | > LD_LIBRARY_PATH=/usr/custom_software/lib | > /usr/custom_software/lib/6/libfoo.so.6 | > then only the FreeBSD 6 binary could load it. Since the searches | > would be for the FreeBSD 6 binary: | > /usr/custom_software/lib/603000/libfoo.so.6 | > /usr/custom_software/lib/6/libfoo.so.6 | > /usr/custom_software/lib/libfoo.so.6 | > and for FreeBSD 7 binary: | > /usr/custom_software/lib/702000/libfoo.so.6 | > /usr/custom_software/lib/7/libfoo.so.6 | > /usr/custom_software/lib/libfoo.so.6 | > Only the FreeBSD 6 binary would load /usr/custom_software/lib/6/libfoo.so.6. | > I do the same search with LD_PRELOAD. | | Hmm, I'm still not quite fan of this. Perhaps you could add an extension to | ldconfig and the hints file to handle this case? That is, a way to store | path mappings so you could do something like: | | ldconfig -os=6 -p /usr/local/lib /usr/local/lib/6 I'll have to look to see how the hints file could update rtld. It is an interesting idea. Maybe libmap.conf would be better place for this. I haven't looked at how libmap works. Maybe introduce: /etc/libmap-.conf that maps paths as well. So with the above example. /etc/libmap-6.conf would contain /usr/custom_software/lib /usr/custom_software/lib/6:/usr/custom_software/lib for example. | Or maybe you could make it an extension of how 'm' worked, so you could make | directories accept an optional set of colon-separated paths that they serve | as aliases for: | | ldconfig -os=6 -m /usr/local/lib/6:/usr/local/lib:/usr/lib I don't really get how this is solving the LD_LIBRARY_PATH/LD_PRELOAD since "-m" is solving the general case with the hints file which I did first. | (That would even fit into your existing rc.d script changes I believe). Then | rtld would keep this internal directory mapping and be able to map the | '/usr/local/lib' and '/usr/lib' directories in LD_PRELOAD and LD_LIBRARY_PATH | to /usr/local/lib/6. The advantage of this is the same as with your previous | change that all the mappings are configurable and not hard-coded into rtld | itself. | > Final, is that prior binaries built on FreeBSD i386 but run on FreeBSD amd64 | > that set LD_* environemnt variables would fail on FreeBSD amd64 as is | > since it didn't set the equivalent LD32_*. To address this for COMPAT_32BIT | > I have rtld look for LD32_* first and then check for the LD_* second. | > This way legacy applications work fine. | | Hmm, so Yahoo! had some patches to handle this as well. I think Yahoo's | patches were different though. They actually patched the 32-bit libc to | capture attempts to get or set LD_* and convert them to actually get/set | LD32_* instead. I'm not sure which approach is best, but it might be worth | asking Peter why Yahoo! did it that way and if there were reasons for that | approach vs. doing it in rtld. I think the primary reason was so that you | could set LD_LIBRARY_PATH or LD_PRELOAD to reference 64-bit libraries and | not have it break 32-bit apps, but if a 32-bit app tried to set a variable | before launching another app it would still DTRT. This means you would have to have a modified libc in these environment and it wouldn't help in the static binary case. I think we are safe in the case LD_ for a 32 bit binary effecting a 64 bit since rtld should not allow loading a the wrong type of lib. into a binary. I can do some more testing around that area. I was trying to keep all changes in the host environment so we can run unchanged binaries from other vendors. | I do think this is definitely a problem worth solving. Thanks, Doug A.