Skip site navigation (1)Skip section navigation (2) 
Date:      Mon, 9 Nov 2009 04:37:52 -0600 (CST)
From:      Scott Bennett <bennett@cs.niu.edu>
To:        "b. f." <bf1783@googlemail.com>, dougb@FreeBSD.org
Cc:        maho@freebsd.org, freebsd-questions@FreeBSD.org
Subject:   Re: math/py-numpy vs. math/atlas-devel
Message-ID:  <200911091037.nA9Abq9b021278@mp.cs.niu.edu>
next in thread | raw e-mail | index | archive | help 
     On Sun, 08 Nov 2009 23:59:29 -0800 Doug Barton <dougb@FreeBSD.org>
wrote:
>First, sorry I missed the original problem report, I'm not subscribed
>to -questions. Usually ports questions (including those about tools)
>are handled on freebsd-ports@freebsd.org, but OTOH if you're not sure
>where to send a question it's always better to start on the -questions
>list. :)

     The responses I got the last time I posted something to -ports
convinced me not to post there anymore.  I'm still subscribed, however,
so I still see what transpires there.
>
>Second, thanks to b.f. for the very thorough analysis of the problem.

     Yes, indeed, thanks.  It was fascinating.

>I will respond and trim a bit as I go.
>
>Third, I've cc'ed maho@ since the genesis of the problem is that
>math/atlas and math/atlas-devel aren't setting CONFLICTS when
>apparently they should be. maho, if you need any help with this let me
>know.

     I've left the Cc in as well because IIRC, Maho was heavily involved
in the upgrade of immense numbers of ports to gfortran42.  The next obstacle
in math/py-numpy outlined further below may well be a missed relic of that
effort.
>
>b. f. wrote:
>> Scott Bennett wrote:
>>> I would like to install science/gnudatalanguage but have been running
>>> into various obstacles.  Lars Engels very kindly just fixed one of them
>>> (devel/lasi) (Thanks, Lars!), so now I'm on to the next one, which may not
>>> be a showstopper, but it's at least a nuisance.  One of the gnudatalanguage
>>> dependencies is math/py-numpy, which, in turn, depends upon math/atlas.
>>> However, I have math/atlas-devel installed and do not wish to revert to an
>>> older, slower version of the ATLAS library.  Adding a "-x atlas-\*" onto
>>> the portmaster command to build math/py-numpy fails to prevent portmaster
>>>from trying to build math/atlas. 
>
>As was pointed out, that definitely won't work. With the glob patterns
>for exclusions, or specifying ports on the command line you want to be
>as conservative as possible. Also, it's not necessary to include a *
>at the end, portmaster will handle that for you.
>
>>> Creating a
>>> /var/db/pkg/atlas-3.8.3_1,1/+IGNOREME file in addition doesn't help.
>
>+IGNOREME files are only relevant to installed ports.
>
>>> How
>>> can I force math/py-numpy to accept the already installed math/atlas-devel
>>> libraries?
>>> Thanks in advance for any help!
>
>The easiest way for you to accomplish this would have been to use '-x
>atlas', the second-easiest would have been to use the -i command line
>option. See the man page for more information on that.

     Yes, thanks much to both of you.  Cutting the option back to just
"-x atlas" did the trick, allowing math/atlas to be skipped.  Here's
what it looks like giving the desired action:

===>>> Port directory: /usr/ports/math/py-numpy

===>>> Gathering distinfo list for installed ports

===>>> Launching 'make checksum' for math/py-numpy in background
===>>> Gathering dependency list for math/py-numpy from ports
===>>> Starting recursive 'make config' check
===>>> Checking dependency: devel/py-nose
===>>> Launching child to update devel/py-nose
	math/py-numpy >> devel/py-nose

===>>> Port directory: /usr/ports/devel/py-nose
===>>> Launching 'make checksum' for devel/py-nose in background
===>>> Gathering dependency list for devel/py-nose from ports
===>>> Starting recursive 'make config' check
===>>> Checking dependency: devel/py-setuptools
===>>> Checking dependency: lang/python26
===>>> Recursive 'make config' check complete for devel/py-nose
	math/py-numpy >> devel/py-nose

===>>> Continuing 'make config' dependency check for math/py-numpy
===>>> Checking dependency: lang/gcc44
===>>> Checking dependency: lang/python26
===>>> Checking dependency: math/atlas
===>>> Skipping math/atlas
       because it matches the pattern: *atlas*

===>>> Checking dependency: math/suitesparse
===>>> Recursive 'make config' check complete for math/py-numpy

===>>> Starting build for math/py-numpy <<<===

===>>> Starting check for build dependencies
===>>> Gathering dependency list for math/py-numpy from ports
===>>> Starting dependency check
===>>> Checking dependency: lang/gcc44
===>>> Checking dependency: lang/python26
===>>> Checking dependency: math/atlas
===>>> Skipping math/atlas
       because it matches the pattern: *atlas*

===>>> Checking dependency: math/suitesparse
===>>> Dependency check complete for math/py-numpy

===>  Cleaning for py26-numpy-1.3.0_2,1

===>  Found saved configuration for py26-numpy-1.3.0_2,1
===>  Extracting for py26-numpy-1.3.0_2,1
>
>> Congratulations, you've managed to defeat three sets of safeguards in
>> portmaster.

     Sigh.  It has *always* been that way for me.  By age 15 I was already
stumbling over compiler bugs, library bugs, operating system bugs, and even
the occasional, very peculiar sort of hardware bug.  It's enough to make
one believe in gremlins developing interests in areas other than aviation. :-}
In the great majority of those cases, I wasn't doing anything out of the
ordinary, but somehow I managed to trigger the bugs anyway.
>> 
>> Problem #1
>> 
>> math/atlas and math/atlas-devel don't have proper CONFLICTS entries in
>> their Makefiles (this should be fixed),
>
>Yes, this analysis was essentially correct.

     Okay.
>
>>  Problem #2:
>> 
>> You're using the -x flag incorrectly.  Or portmaster isn't
>> implementing it properly when the port to be excluded is not actually
>> installed, take your pick. 
>
>The way that -x is implemented currently really depends on the user's
>definition of the exclude pattern, and is designed to exclude things
>as early as possible so as to avoid what would ultimately become
>wasted effort. It's hard to justify modifying the code to guess that
>something we've already started working on might match what we think
>the user MEANT instead of what they SAID.
>
     Understood.  As noted above, it does work correctly with the shorter
string.

>> Problem #3:
>> 
>> The +IGNOREME checks in portmaster aren't properly implemented for
>> this case, so they fail to prevent math/atlas from being built.
>
>s/properly implemented for/designed to handle/
>
>If you think about this for a second, the entries in /var/db/pkg are
>exclusively related to installed ports, so trying to use an +IGNOREME
>file for this purpose doesn't make sense, although I can sympathize
>with the OPs sense of desperation here. :)

     Actually, it does make sense, or at least I thought so.  If a
directory for atlas exists in /var/db/pkg, presumably portmaster will
think that the port *is* installed, even if it is not, in fact, installed.
Then the +IGNOREME file in that directory ought to prevent further action
on that port.
>
>> So what can you do while these problems are being fixed? 
>
>Just to be clear, I didn't see anything that needs to be fixed in your
>message, and I hope my explanation makes it clear why. If you think
>that there are actual bugs that need to be fixed please feel free to
>create a thread on -ports to discuss them.
>
     Anyway, the math/py-numpy port now proceeds to build without bothering
with math/atlas.  It quickly goes astray when it doesn't recognize that any
of gfortran4[345] has been installed--I guess there no longer is a FORTRAN
compiler included in the base system--and instead tries to use lang/g95,
which has also been installed.  Of course, this won't work because the ATLAS
library needs to have been compiled with the same compiler as the programs
that use it.

===>  Configuring for py26-numpy-1.3.0_2,1
Running from numpy source directory.
F2PY Version 2
blas_opt_info:
blas_mkl_info:
  libraries mkl,vml,guide not found in /usr/lib
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
  FOUND:
    libraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r']
    library_dirs = ['/usr/local/lib']
    language = c
    include_dirs = ['/usr/local/include']

/usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/command/config.py:361: DeprecationWarning: 
+++++++++++++++++++++++++++++++++++++++++++++++++
Usage of get_output is deprecated: please do not 
use it anymore, and avoid configuration checks 
involving running executable on the target machine.
+++++++++++++++++++++++++++++++++++++++++++++++++

  DeprecationWarning)
customize GnuFCompiler
Found executable /usr/local/bin/gfortran44
gnu: no Fortran 90 compiler found
gnu: no Fortran 90 compiler found
customize Gnu95FCompiler
customize Gnu95FCompiler
customize Gnu95FCompiler using config
compiling '_configtest.c':

/* This file is generated from numpy/distutils/system_info.py */
void ATL_buildinfo(void);
int main(void) {
  ATL_buildinfo();
  return 0;
}
C compiler: gcc44 -DNDEBUG -O2 -fno-strict-aliasing -pipe -march=prescott -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x100000 -O2 -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44 -fPIC

compile options: '-c'
gcc44: _configtest.c
gcc44 _configtest.o -L/usr/local/lib -lalapack_r -lf77blas_r -lcblas_r -latlas_r -o _configtest
/usr/bin/ld: _configtest: hidden symbol `__powisf2' in /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o) is referenced by DSO
collect2: ld returned 1 exit status
/usr/bin/ld: _configtest: hidden symbol `__powisf2' in /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/libgcc.a(_powisf2.o) is referenced by DSO
collect2: ld returned 1 exit status
failure.
removing: _configtest.c _configtest.o
Status: 255
Output: 
  FOUND:
    libraries = ['alapack_r', 'f77blas_r', 'cblas_r', 'atlas_r']
    library_dirs = ['/usr/local/lib']
    language = c
    define_macros = [('NO_ATLAS_INFO', 2)]
    include_dirs = ['/usr/local/include']

lapack_opt_info:
lapack_mkl_info:
mkl_info:
  libraries mkl,vml,guide not found in /usr/lib
  libraries mkl,vml,guide not found in /usr/local/lib
  libraries mkl,vml,guide not found in /usr/local/lib/gcc44/gcc/i386-portbld-freebsd7.2/4.4.3/../../../
  NOT AVAILABLE

  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /usr/local/lib
numpy.distutils.system_info.atlas_threads_info
Setting PTATLAS=ATLAS
/usr/ports/math/py-numpy/work/numpy-1.3.0/numpy/distutils/system_info.py:999: UserWarning: 
*********************************************************************
    Lapack library (from ATLAS) is probably incomplete:
      size of /usr/local/lib/libalapack_r.so is 3832k (expected >4000k)

    Follow the instructions in the KNOWN PROBLEMS section of the file
    numpy/INSTALL.txt.
*********************************************************************

  warnings.warn(message)


The above sequence gets more or less repeated several more times, and
after much compiling, running of python26, and other activities, the
process runs aground as follows.


creating build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/fft
compile options: '-Inumpy/core/include -Ibuild/src.freebsd-7.2-STABLE-i386-2.6/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.6 -c'
gcc44: numpy/fft/fftpack_litemodule.c
gcc44: numpy/fft/fftpack.c
cc -shared -pthread -O2 -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44 build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/fft/fftpack_litemodule.o build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/fft/fftpack.o -Lbuild/temp.freebsd-7.2-STABLE-i386-2.6 -o build/lib.freebsd-7.2-STABLE-i386-2.6/numpy/fft/fftpack_lite.so
building 'numpy.linalg.lapack_lite' extension
compiling C sources
C compiler: gcc44 -DNDEBUG -O2 -fno-strict-aliasing -pipe -march=prescott -D__wchar_t=wchar_t -DTHREAD_STACK_SIZE=0x100000 -O2 -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44 -fPIC

creating build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg
compile options: '-DATLAS_INFO="\"3.9.11\"" -I/usr/local/include -Inumpy/core/include -Ibuild/src.freebsd-7.2-STABLE-i386-2.6/numpy/core/include/numpy -Inumpy/core/src -Inumpy/core/include -I/usr/local/include/python2.6 -c'
gcc44: numpy/linalg/lapack_litemodule.c
gcc44: numpy/linalg/python_xerbla.c
cc -shared -pthread -O2 -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44 build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_litemodule.o build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/python_xerbla.o -L/usr/local/lib -Lbuild/temp.freebsd-7.2-STABLE-i386-2.6 -lalapack_r -lalapack_r -lf77blas_r -lcblas_r -latlas_r -lgfortran -lm -lpthread -o build/lib.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_lite.so
/usr/bin/ld: cannot find -lgfortran
/usr/bin/ld: cannot find -lgfortran
error: Command "cc -shared -pthread -O2 -fno-strict-aliasing -pipe -march=prescott -Wl,-rpath=/usr/local/lib/gcc44 build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_litemodule.o build/temp.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/python_xerbla.o -L/usr/local/lib -Lbuild/temp.freebsd-7.2-STABLE-i386-2.6 -lalapack_r -lalapack_r -lf77blas_r -lcblas_r -latlas_r -lgfortran -lm -lpthread -o build/lib.freebsd-7.2-STABLE-i386-2.6/numpy/linalg/lapack_lite.so" failed with exit status 1
*** Error code 1

Stop in /usr/ports/math/py-numpy.

===>>> make failed for math/py-numpy
===>>> Aborting update

     I looked in /usr/local/lib and didn't spot libraries by name for
gfortran[345], so I assume they have either been renamed or included in
with the corresponding gcc libraries.  What is the best way to proceed?
Will simply deleting the instances of "-lgfortran" do the job?  Or do I
need to specify the library explicitly?


                                  Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet:       bennett at cs.niu.edu                              *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good  *
* objection to the introduction of that bane of all free governments *
* -- a standing army."                                               *
*    -- Gov. John Hancock, New York Journal, 28 January 1790         *
**********************************************************************

Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200911091037.nA9Abq9b021278>