Date: Mon, 22 Jun 2015 22:54:16 -0400 From: "John W. O'Brien" <john@saltant.com> To: Johannes Jost Meixner <xmj@FreeBSD.org> Cc: FreeBSD Python List <freebsd-python@FreeBSD.org> Subject: Re: ipython / numpy / scipy weirdness Message-ID: <5588CA58.2030402@saltant.com> In-Reply-To: <5565A72C.50801@saltant.com> References: <555FDE72.4010303@FreeBSD.org> <55646E74.4080206@saltant.com> <556558ED.8030506@FreeBSD.org> <5565A72C.50801@saltant.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --4orMr0mNU2kiqGorlNJALpEhEAI0k8mLf Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 5/27/15 7:14 AM, John W. O'Brien wrote: > [adding freebsd-python@ back] >=20 > On 5/27/15 1:41 AM, Johannes Jost Meixner wrote: >> On 05/26/2015 16:00, John W. O'Brien wrote: >>> On 5/22/15 9:57 PM, Johannes Jost Meixner wrote: >>>> Trying my hand at Data SCIENCE (tm) with a combination of >>>> ipython notebooks and numpy/scipy, I seem to have stumbled upon >>>> weirdness we already considered fixed: >>>> >>>> ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by=20 >>>> /usr/local/lib/gcc48/libgfortran.so.3 not found >>>> >>>> Supposedly, we have a commit to show for fixing it in=20 >>>> https://svnweb.freebsd.org/ports?view=3Drevision&revision=3D369447, >>>> >>>> and yet... I can't get numpy nor scipy to work *without* the >>>> following: >>>> >>> [patch] >>>> % ipython notebook (new site opens in a browser) >>>> >>>> create a notebook write into it, 'import numpy' write into it, >>>> 'import scipy >>>> >>>> run the notebook (hit the 'play' button) >> >>> I am afraid to say that I fail to reproduce this problem. >> >>> % ipython --no-banner >> >> I can't produce it with the console. Only with `ipython notebook` and >> entering those then within a browser. Could you try that? >=20 > Yes. In fact, I did it that way first, and then on the console. For wha= t > it's worth, I only have headless boxen at the ready, so the other run > looked like this: >=20 > fbsd% ipython notebook --no-browser --port=3D8080 >=20 > osx% ssh -L 8080:localhost:8080 fbsd >=20 > [open http://localhost:8080/ in browser on osx] >=20 > It's curious that you see different results between the console and > browser front-ends. I don't know enough about the ipython architecture > to infer useful next steps, but that seems significant. While doing QA on math/py-pandas 0.16.2, I began to investigate a new test failure in the io.pytables module and promptly stumbled upon the wil= y ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by /usr/local/lib/gcc48/libgfortran.so.3 not found error. This does not appear to have anything to do with the test failure I have yet to track down, but since it's relevant to your inquiry from last month, I wanted to share a new insight I've gained. On my system, this works fine. % python Python 2.7.9 (default, May 13 2015, 01:30:13) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information= =2E >>> from numpy.linalg import _umath_linal, lapack_lite >>> But this barfs. % python Python 2.7.9 (default, May 13 2015, 01:30:13) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information= =2E >>> import tables Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/site-packages/tables/__init__.py", line 82, in <module> from tables.utilsextension import ( File "numpy.pxd", line 155, in init tables.utilsextension (tables/utilsextension.c:15443) File "/usr/local/lib/python2.7/site-packages/numpy/__init__.py", line 170, in <module> from . import add_newdocs File "/usr/local/lib/python2.7/site-packages/numpy/add_newdocs.py", line 13, in <module> from numpy.lib import add_newdoc File "/usr/local/lib/python2.7/site-packages/numpy/lib/__init__.py", line 18, in <module> from .polynomial import * File "/usr/local/lib/python2.7/site-packages/numpy/lib/polynomial.py", line 19, in <module> from numpy.linalg import eigvals, lstsq, inv File "/usr/local/lib/python2.7/site-packages/numpy/linalg/__init__.py", line 51, in <module> from .linalg import * File "/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 29, in <module> from numpy.linalg import lapack_lite, _umath_linalg ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by /usr/local/lib/gcc48/libgfortran.so.3 not found >>> But this works fine. % python Python 2.7.9 (default, May 13 2015, 01:30:13) [GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10 Type "help", "copyright", "credits" or "license" for more information= =2E >>> import numpy >>> import tables >>> Here's what the major players look like behind the scenes. $ readelf -d /usr/local/lib/python2.7/site-packages/{numpy/linalg/{_umath_linalg,lapac= k_lite},tables/utilsextension}.so \ | egrep "^File|libgcc|libgfortran|rpath" File: /usr/local/lib/python2.7/site-packages/numpy/linalg/_umath_linalg.so 0x0000000000000001 (NEEDED) Shared library: [libgfortran.so.3] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so= =2E1] 0x000000000000000f (RPATH) Library rpath: [/usr/local/lib/gcc48] File: /usr/local/lib/python2.7/site-packages/numpy/linalg/lapack_lite= =2Eso 0x0000000000000001 (NEEDED) Shared library: [libgfortran.so.3] 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so= =2E1] 0x000000000000000f (RPATH) Library rpath: [/usr/local/lib/gcc48] File: /usr/local/lib/python2.7/site-packages/tables/utilsextension.so= 0x0000000000000001 (NEEDED) Shared library: [libgcc_s.so.1] And here are some examples of the innards from the libraries in question.= % objdump -T /usr/local/lib/gcc48/libgfortran.so.3.0.0 \ | grep GCC_4.6.0 \ | head -3 0000000000000000 DF *UND* 0000000000000000 GCC_4.6.0 __divtf3= 0000000000000000 DF *UND* 0000000000000000 GCC_4.6.0 __gttf2 0000000000000000 DF *UND* 0000000000000000 GCC_4.6.0 __eqtf2 % objdump -T /usr/local/lib/gcc48/libgcc_s.so.1 \ | egrep "__(divtf3|gttf2|eqtf2)" 00000000000084e0 g DF .text 0000000000000126 GCC_4.6.0 __eqtf2 0000000000008610 g DF .text 00000000000001b5 GCC_4.6.0 __gttf2 0000000000007a60 g DF .text 0000000000000a7a GCC_4.6.0 __divtf3= % objdump -T /lib/libgcc_s.so.1 \ | egrep "__(divtf3|gttf2|eqtf2)" What I think I understand now about what this means and how these things are related is as follows. When I import tables first, there comes a point when the runtime linker loads utilsextension.so, sees that it needs libgcc_s.so.1 but doesn't specify where to find it, loads it from the default /lib, and carries on... Right up to the point when utilsextensions.so tries to load numpy, and the runtime linker is presented with the first of _umath_linalg.so and lapack_lite.so. If I were a runtime linker, I would look at the list of dependencies, encounter libgcc_s.so.1, see that I've already provided that to this process, get to the libgfortran.so.3 dependency, and barf on the missing/inadequate symbols. When I import numpy first, the runtime linker obeys the rpath and loads libgcc_s.so.1 from the gcc48 port location, which thereby satisfies libgfortran.so.3 from the same. At that point, importing tables works because utilsextensions.so isn't picky about which libgcc_s.so.1 is offered, so it doesn't matter that the runtime linker hands it the one from gcc48 instead of the one from base. The insight this gives me about the general problem is that *any* python package containing an .so with a requirement on libgcc_s.so.1 that can be satisfied from base is probably sufficient to make numpy and company fail with this pesky import error if it happens to be imported first. I've looked on my system for examples of these with % find /usr/local/lib/python2.7/site-packages -name "*.so" \ | xargs readelf -d \ | egrep "^File|libgcc|libgfortran|rpath" \ | less and see a handful that might qualify in scipy, numexpr, matplotlib, and some others. I spot checked, and none seem to trigger this problem. Given their common flavor, I wonder whether they all happen to import numpy internally before loading the potentially-problematic .so module. However, this could (and does [0]) affect a port that depends indirectly on numpy (as x11-toolkits/py-wxPython28 does via graphics/py-opengl). I hope this helps you, and perhaps others who stumble over this. It's doesn't directly address the scenario you described in your original message, but perhaps it will give you a new way to think about it. [0] [PATCH] x11-toolkits/py-wxPython28: Fail embedding_in_wx5.py - gcc48/libgfortran.so.3 not found https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D196862 Regards, John --4orMr0mNU2kiqGorlNJALpEhEAI0k8mLf Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2 iQIcBAEBCgAGBQJViMpbAAoJEGEAJpqmdExHYYIP/3M22WjNdVZGT5mMPTzZqIY/ qCTVPhNSqM2qACq4jOz6FudNelG2DCiI07/HFCWZanbJRuDLks2UAC5R948JqUrG rbkHlfETtrj+RaCpsN96Cy610/Hvtf3tKHBkeU3M9ydnw+RlE1M6ee3RK9iW/7+6 wMegajgbIxI9x1gnCJVdllaA1AfXePWvBqX24lRRaYLsEu/QWm4MCTOWKNaMVbf0 mAhZI5iq0fYWncGiwYcYj1UQb/uveDvkUn+gxq4Id3ZvsodMWjgRZ8aSb5TtEH03 xP+pyPKio91+fmrJr3kqMhjOBNUslz901kaX5PwaIuZizHS+FIB5TRSpqGKyYkbp 7A7rv6SkeLGq7my2FlFYWJy4sInEwNVp+D0HEtpIv721tWG5mM5Dn1h0Tdl2Pgyo iSXqaeUZ3LCAHxyS1LbiCFatHq9ofA3cKoiMCUNib14czG6ZuuDUZE+qjG/eh7hk rqljCtJBBm/ypiSqkrWRSPEfcbDjtu4nkpWnymuw0dZON7UquGq2FUmhNRQo6urw d0M+4A3HUhSb3p1mQQvdjbQCOu3FuNn2rwSvYG2+7LOYyby9FEnCn56xZEbhtWw4 noWZYaTxkXuMFLCmGI+omoSQhMDUkHnQZv8qpMcMklOOEn1fZn5/Lz2kziLzSwLa 2FC1HleQLuGXYQpBhe4y =J8K0 -----END PGP SIGNATURE----- --4orMr0mNU2kiqGorlNJALpEhEAI0k8mLf--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5588CA58.2030402>