From owner-freebsd-python@FreeBSD.ORG  Tue May 13 03:06:08 2014
Return-Path: <owner-freebsd-python@FreeBSD.ORG>
Delivered-To: python@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 9F16D485;
 Tue, 13 May 2014 03:06:08 +0000 (UTC)
Received: from homiemail-a97.g.dreamhost.com (sub4.mail.dreamhost.com
 [69.163.253.135])
 by mx1.freebsd.org (Postfix) with ESMTP id 791E92292;
 Tue, 13 May 2014 03:06:07 +0000 (UTC)
Received: from homiemail-a97.g.dreamhost.com (localhost [127.0.0.1])
 by homiemail-a97.g.dreamhost.com (Postfix) with ESMTP id 497D228606F;
 Mon, 12 May 2014 20:06:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=saltant.com; h=message-id
 :date:from:mime-version:to:cc:subject:references:in-reply-to:
 content-type; s=saltant.com; bh=6tws3VWgEXutW3HGKLXKn4Kz4Rg=; b=
 GeMRqZTS7JJ7ixnuRcGHfFPI1i1+okXD1bZTu++CiT+HzscZIlmPNPTaz6XzFGcz
 wl1rZ2gE3Kearu8kB+rT9AlS1sMeatCWk/zi3W11XkWHs5VHRq5mkkB66rbXwHsH
 5Uvz/75jDWZms6QcsJpDCKRYjLBSTk8HE9q04qe0ot0=
Received: from dreck.saltant.net (dreck.saltant.net [72.78.188.150])
 (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits))
 (No client certificate requested)
 (Authenticated sender: john@saltant.com)
 by homiemail-a97.g.dreamhost.com (Postfix) with ESMTPSA id DB661286059;
 Mon, 12 May 2014 20:06:00 -0700 (PDT)
Message-ID: <53718C14.1040807@saltant.com>
Date: Mon, 12 May 2014 23:05:56 -0400
From: "John W. O'Brien" <john@saltant.com>
Organization: Saltant Solutions
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8;
 rv:24.0) Gecko/20100101 Thunderbird/24.5.0
MIME-Version: 1.0
To: Irjohn Junus <i.junus@gmail.com>
Subject: math/py-numpy cannot satisfy libgfortran dependency on
 libgcc_s/GCC_4.6.0
 [WAS: Re: py27-pandas-0.13.1_1 conflict with py27-MySQLdb-1.2.3_4]
References: <CALXkR+8ivP3m2aJN+m88Sz+4BodAxnYioBvgUotFZf_-TVPLgg@mail.gmail.com>	<536C40B0.2060705@saltant.com>
 <CALXkR+_r4kG8CHO8R2MOsGHK4JDeEMa28MN4rzZ1hAtd+zAu4g@mail.gmail.com>
In-Reply-To: <CALXkR+_r4kG8CHO8R2MOsGHK4JDeEMa28MN4rzZ1hAtd+zAu4g@mail.gmail.com>
X-Enigmail-Version: 1.6
Content-Type: multipart/signed; micalg=pgp-sha512;
 protocol="application/pgp-signature";
 boundary="OJHnpXUV1METTqspoUOTie7LGatUvageN"
Cc: python@freebsd.org, Jimmy Olgeni <olgeni@freebsd.org>
X-BeenThere: freebsd-python@freebsd.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: FreeBSD-specific Python issues <freebsd-python.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-python>,
 <mailto:freebsd-python-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-python/>
List-Post: <mailto:freebsd-python@freebsd.org>
List-Help: <mailto:freebsd-python-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-python>,
 <mailto:freebsd-python-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 13 May 2014 03:06:08 -0000

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--OJHnpXUV1METTqspoUOTie7LGatUvageN
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 5/8/14 11:33 PM, Irjohn Junus wrote:
> Hi John,
>=20
> No worries and yes I'm still having the problem.
>=20
> I remember seeing this message:
> =3D=3D=3D>>> pkg-message for gcc-4.7.3_1
> To ensure binaries built with this toolchain find appropriate versions
> of the necessary run-time libraries, you may want to link using
>=20
>   -Wl,-rpath=3D/usr/local/lib/gcc47
>=20
> after upgrading gcc port a few days ago, which is described in your lin=
k
> [0]. Do I issue this command as root? Need advice here, thanks.
>=20
> Regards,
> Irjohn
>=20
>=20
> On Fri, May 9, 2014 at 10:42 AM, John W. O'Brien <john@saltant.com
> <mailto:john@saltant.com>> wrote:
>=20
>     On 4/18/14 8:45 AM, Irjohn Junus wrote:
>     > Dear All,
>     >
>     > I'm running 9.1-RELEASE and today just after freebsd-update to p1=
1 and
>     > updating ports to the latest found out that pandas won't import a=
fter
>     > MySQLdb is imported first. Strangely, it will import no problem i=
f
>     > MySQLdb is imported after:
>=20
>     Hi Irjohn,
>=20
>     Sorry for the long delay. Are you still having this problem?
>=20
>     >>>> import MySQLdb
>     >>>> import pandas as pd
>     > /lib/libgcc_s.so.1: version GCC_4.6.0 required by
>     > /usr/local/lib/gcc47/libgfortran.so.3 not found
>     [...]
>     > Traceback (most recent call last):
>     [...]
>     >   File
>     "/usr/local/lib/python2.7/site-packages/numpy/linalg/linalg.py",
>     > line 29, in <module>
>     >     from numpy.linalg import lapack_lite, _umath_linalg
>     > ImportError: /lib/libgcc_s.so.1: version GCC_4.6.0 required by
>     > /usr/local/lib/gcc47/libgfortran.so.3 not found
>=20
>     It looks like this thread [0], these commits [1] [2], and this PR [=
3]
>     are related.
>=20
>     [0]
>     https://lists.freebsd.org/pipermail/freebsd-toolchain/2014-April/00=
1149.html
>=20
>     [1] https://svnweb.freebsd.org/ports?view=3Drevision&revision=3D351=
167
>=20
>     [2] https://svnweb.freebsd.org/ports?view=3Drevision&revision=3D351=
159
>=20
>     [3] https://www.freebsd.org/cgi/query-pr.cgi?pr=3D185902

Irjohn,

This is just a little out of my depth, so I'm going to talk through it
out loud while I investigate. Presumably somebody more knowledgeable
than I will step in to correct me if I get something wrong.

First off, there are a few open PRs ([0], [1]) on math/py-numpy, both of
which seem to have been triggered when ports started using GCC 4.7 by
default [2].

What I know from reading gcc(1) is that "-Wl,<option>" passes <option>
to the linker, and from ld(1) is that "-rpath=3D<path>" embeds <path> in
the output file so that the runtime linker knows where to look for
shared objects. It makes sense why this would help resolve the traceback
you reported because one thing (linalg.py + ??? + libgfortran.so.3) is
looking for another thing (libgcc_s.so.1), finds it, yet remains
unsatisfied. Setting the rpath in the right place in the former will
help it find the right one of the latter.

I dug a little more to learn how to find out what rpath is embedded in a
given file. It turns out that readelf(1) with the "-d" option to show an
ELF file's dynamic section is just the ticket. Here is some output from
my system.

    $ cd /usr/local/lib/gcc47
    $ readelf -d libgfortran.so.3 | grep RPATH
     0x000000000000000f (RPATH)              Library rpath:
[/usr/local/lib/gcc47]
    $ cd /usr/local/lib/python2.7/site-packages/numpy/linalg
    $ readelf -d {_umath_linalg.so,lapack_lite.so} | grep RPATH
     0x000000000000000f (RPATH)              Library rpath:
[/usr/local/lib/gcc47]
     0x000000000000000f (RPATH)              Library rpath:
[/usr/local/lib/gcc47]

In fact, this tool will also tell you which shared objects a file needs.

    $ readelf -d {_umath_linalg.so,lapack_lite.so} | grep libgfortran
     0x0000000000000001 (NEEDED)             Shared library:
[libgfortran.so.3]
     0x0000000000000001 (NEEDED)             Shared library:
[libgfortran.so.3]

So this line of python

    >>> from numpy.linalg import lapack_lite, _umath_linalg

tries to load two binary shared objects, both of which require libgfortra=
n.

Moving right along. If everything goes as it should, how is numpy
supposed to learn the correct rpath? Time to dive into
math/py-numpy/Makefile and the ports machinery.

In math/py-numpy, I find "USES=3Dfortran" and some fishy business in the
pre-configure target involving GCCLIBDIR. The former is pretty
interesting because it pulls in /usr/ports/Mk/Uses/fortran.mk to set all
kinds of useful things, including

    FFLAGS+=3D        -Wl,-rpath=3D${LOCALBASE}/lib/gcc${_GCC_VER}
    FCFLAGS+=3D       -Wl,-rpath=3D${LOCALBASE}/lib/gcc${_GCC_VER}
    LDFLAGS+=3D       -Wl,-rpath=3D${LOCALBASE}/lib/gcc${_GCC_VER} \
                    -L${LOCALBASE}/lib/gcc${_GCC_VER} -B${LOCALBASE}/bin

It also sets FC=3Dgfortran${_GCC_VER}; a fact I can use to further probe
the GCCLIBDIR funny business in the numpy port. During the pre-configure
target, ports determines GCCLIBDIR like this:

    $ gfortran47 -print-file-name=3Dlibgfortran.so \
        | sed -e s/libgfortran.so//
    /usr/local/lib/gcc47/gcc/x86_64-portbld-freebsd9.2/4.7.3/../../../

Which simplifies, of course, to /usr/local/lib/gcc47. So far, so good.

I've built math/py-numpy on my machine, and will begin to examine the
build logs [3] for clues about what to dig into next (e.g. differences
between my setting CC=3Dclang et al, and not). In the mean time...

Questions to you:

1.  What is the rpath stored in your copy of those numpy files? That
    is, what is the output from

        cd /usr/local/lib/python2.7/site-packages/numpy/linalg
        readelf -d _umath_linalg.so | grep RPATH
        readelf -d lapack_lite.so | grep RPATH

    ?

2.  What is the contents of
/usr/local/lib/python2.7/site-packages/numpy/distutils/site.cfg on your
machine?

2.  What is the contents of /etc/make.conf on your machine?

3.  How to you install and upgrade your ports? Build from source by
    hand? Build from source with the help of portmaster, poudriere, etc?
    Install binary packages with pkg_* or ports-mgmt/pkg (pkgng)?

One thing that still puzzles me is what databases/py-MySQLdb would have
to do with it.

[0] ports/188114: math/py-numpy is broken on 9.2-STABLE
    http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dports/188114
[1] ports/188327: math/py-numpy: Numpy is broken
    http://www.freebsd.org/cgi/query-pr.cgi?pr=3Dports/188327
[2] https://svnweb.freebsd.org/ports?view=3Drevision&revision=3D347809
[3]
https://pkg.saltant.net/poudriere/bulk/92amd64-current/2014-05-12_21h56m0=
1s/logs/py27-numpy-1.8.0_1,1.log

Regards,
John


--OJHnpXUV1METTqspoUOTie7LGatUvageN
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJTcYwYAAoJEORay8JGGICYo00IANjAEmXtv/dMIJrZTwjBV2QZ
f8cWuUpII18DIFgo94cjSi6Ak3sxqwiVvuBdXnL0CeoNwbHmTQdG5oIHN97TziZx
cf3qEN58UKdg2JyZ0anuJ+wMR2/EMhdlw4XVKrsSScamgtXtWHa5Xn8BgutWWOGP
2zHaH22kMp1pLGQgvVROO1cSZkBmmIK7JEfwTktSC9w8LxfPSxjLiG2nt2MHlVsS
xfiSX2tu/bIlxA2QorAjz5rXX7bWj5TJYANCEkJjLryy/1aFnmMkSpKrgcg9wP8E
TjSRY3wdnUAqJh6XIMI9qsZIJWrPmBYEpyg3gc5GWcQe1PoIwQOE60KiWr1fUxw=
=VSnp
-----END PGP SIGNATURE-----

--OJHnpXUV1METTqspoUOTie7LGatUvageN--