compiling scipy with pgcc 9.0-3 on cray linux 2.2

I am trying to compile scipy 0.7.0 on the cray linux 2.2 nodes on the NICS kraken system. I’ve successfully built lapack, ATLAS, python 2.6.4, and numpy 1.3.0 but am having problems with scipy. Running “python setup.py install” in the scipy source directory yields the following error:


compiling C sources
C compiler: cc -DNDEBUG -fastsse -fPIC

creating build/temp.linux-x86_64-2.6/build
creating build/temp.linux-x86_64-2.6/build/src.linux-x86_64-2.6
creating build/temp.linux-x86_64-2.6/build/src.linux-x86_64-2.6/scipy
creating build/temp.linux-x86_64-2.6/build/src.linux-x86_64-2.6/scipy/fftpack
compile options: ‘-Ibuild/src.linux-x86_64-2.6 -I/lustre/scratch/jvanclev/opt/lib/python2.6/site-packages/numpy/core/include -I/lustre/scratch/jvanclev/opt/include/python2.6 -c’
cc: build/src.linux-x86_64-2.6/fortranobject.c
/opt/cray/xt-asyncpe/3.3/bin/cc: INFO: linux target is being used
cc: scipy/fftpack/src/zfftnd.c
/opt/cray/xt-asyncpe/3.3/bin/cc: INFO: linux target is being used
PGC-W-0156-Type not specified, ‘int’ assumed (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0039-Use of undeclared variable caches_zfftnd_fftpack (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0095-Type cast required for this conversion (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0155-Pointer value created from a nonlong integral type (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0095-Type cast required for this conversion (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0155-Pointer value created from a nonlong integral type (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0095-Type cast required for this conversion (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0095-Type cast required for this conversion (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0095-Type cast required for this conversion (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0155-Pointer value created from a nonlong integral type (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0095-Type cast required for this conversion (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-W-0155-Pointer value created from a nonlong integral type (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0054-Subscript operator () applied to non-array (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0059-Struct or union required on left of . or → (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-F-0008-Error limit exceeded (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC/x86-64 Linux 9.0-3: compilation aborted

I was unable to find any information on a problem in scipy with this source, so I was wondering whether there might be something specific to compiling with pgcc. Any thoughts?

Thanks!

best,

Jeremy

Hi Jeremy,

Without having the source, I’m guessing that there is missing a header file definition.

This is the key error:

PGC-W-0156-Type not specified, ‘int’ assumed (scipy/fftpack/src/zfftnd_fftpack.c: 21)
PGC-S-0039-Use of undeclared variable caches_zfftnd_fftpack (scipy/fftpack/src/zfftnd_fftpack.c: 21)

How is “caches_zfftnd_fftpack” defined? Is is something like “FOO caches_zfftnd_fftpack” and if so, what is “FOO” and how is it defined?

  • Mat

Hi Matt,

That’s something I looked for, but I couldn’t find a problem with the code. Here is the relevant code from
zfftnd_fftpack.c (line 21 is the last line quoted below):

GEN_CACHE(zfftnd_fftpack, (int n, int rank)
, complex_double * ptr; int *iptr; int rank;
, ((caches_zfftnd_fftpack_.n == n)
&& (caches_zfftnd_fftpack.rank == rank))
, caches_zfftnd_fftpack[id].n = n;
caches_zfftnd_fftpack[id].ptr =
(complex_double *) malloc(2 * sizeof(double) * n);
caches_zfftnd_fftpack[id].iptr =
(int *) malloc(4 * rank * sizeof(int));
,
free(caches_zfftnd_fftpack[id].ptr);
free(caches_zfftnd_fftpack[id].iptr);
, 10)


And here is the relevant code from the header (fftpack.h):

define GEN_CACHE(name,CACHEARG,CACHETYPE,CHECK,MALLOC,FREE,CACHESIZE)
typedef struct {
int n;
CACHETYPE
} cache_type_##name;
static cache_type_##name caches_##name[CACHESIZE];
static int nof_in_cache_##name = 0;
static int last_cache_id_##name = 0;
static int get_cache_id_##name CACHEARG {
int i,id = -1;
for (i=0;i<nof_in_cache_##name;i++)
if (CHECK) {
id=i;
break;
}
if (id>=0) goto exit;
if (nof_in_cache_##name<CACHESIZE) {
id = nof_in_cache_##name++;
} else {
id = (last_cache_id_##name<CACHESIZE-1)?last_cache_id_##name+1:0;
/fprintf(stderr,“Removing cache item n=%d\n”,caches_##name[id].n);/
FREE
caches_##name[id].n = 0;
}
/fprintf(stderr,“New cache item n=%d\n”,n);/
caches_##name[id].n = n;
MALLOC
exit:
last_cache_id_##name = id;
return id;
}
static void destroy_##name##caches(void) {
int id;
for (id=0;id<nof_in_cache
##name;++id) {
FREE
caches_##name[id].n = 0;
}
nof_in_cache_##name = last_cache_id_##name = 0;
}
…_


Hi Jeremy,

I’ve sent this error on to our engineers for further investigation (TPR#16377). It appears to me to be a preprocessing bug where the line "static cache_type_##name caches_##name[CACHESIZE];" is getting processed to "static cache_type_zffnd_fftpackcaches_zffnd_fftpack[CACHESIZE];" (no space between fftpack and caches).

For reasons I don’t fully understand, the error only occurs when "static void destroy_##name##caches(void) {" is also used in the GEN_CACHE macro. If you change this line to "static void destroy_caches##name(void) {", then zfftnd.c will compile.

Thanks,
Mat

Hi Mat,

Thanks for looking into this! That does sound weird. Any suggestions for a short term fix or do you think I should just wait until the engineers have a chance to take a look?

Jeremy

Hi Jeremy,

While not ideal, I think the easiest thing to do would be to first preprocess the file (“pgcc -P zfftnd.c”). Next, edit the resulting “.i” file and add the missing space. Finally, compile the edited “.i” file (“pgcc -fast -c zfftnd.i”).

Hope this helps,
Mat

I’ve now been able to compile zfftnd.c, but the scipy compilation fails at another step:

building ‘scipy.spatial.ckdtree’ extension
compiling C sources
C compiler: cc -DNDEBUG -fastsse -fPIC

compile options: ‘-I/lustre/scratch/jvanclev/opt/lib/python2.6/site-packages/numpy/core/include -I/lustre/scratch/jvanclev/opt/include/python2.6 -c’
cc: scipy/spatial/ckdtree.c
/opt/cray/xt-asyncpe/3.3/bin/cc: INFO: linux target is being used
PGC-W-0106-Shift count out of range (scipy/spatial/ckdtree.c: 220)
PGC/x86-64 Linux 9.0-3: compilation completed with warnings
cc -shared build/temp.linux-x86_64-2.6/scipy/spatial/ckdtree.o -Lbuild/temp.linux-x86_64-2.6 -o build/lib.linux-x86_64-2.6/scipy/spatial/ckdtree.so
/opt/cray/xt-asyncpe/3.3/bin/cc: INFO: linux target is being used
/usr/bin/ld: build/temp.linux-x86_64-2.6/scipy/spatial/ckdtree.o: relocation R_X86_64_PC32 against __Pyx_ConsumeWhitespace' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: final link failed: Bad value /opt/cray/xt-asyncpe/3.3/bin/cc: INFO: linux target is being used /usr/bin/ld: build/temp.linux-x86_64-2.6/scipy/spatial/ckdtree.o: relocation R_X86_64_PC32 against __Pyx_ConsumeWhitespace’ can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
error: Command “cc -shared build/temp.linux-x86_64-2.6/scipy/spatial/ckdtree.o -Lbuild/temp.linux-x86_64-2.6 -o build/lib.linux-x86_64-2.6/scipy/spatial/ckdtree.so” failed with exit status 2

I know the source is compiled with -fPIC, but it still gives an error when linking. Here is a link to the ckdtree.c source:
http://projects.scipy.org/scipy/browser/branches/0.7.x/scipy/spatial/ckdtree.c

Any thoughts?

Hi Jeremy,

I’m trying to recreate the problem here but need a bit of help. I’m able to get the build to use pgcc instead of gcc by setting CC in my environment. However, I’m not able to override the C flags it’s using. How did you work around this issue?

Thanks,
Mat

Hi Mat,

So far as I understand it, running “python setup.py install” defaults to the compiler flags that were used to compile python. When I compiled python, I used the following configure command:
./configure --prefix=/lustre/scratch/jvanclev/opt CC=cc CXX=CC OPT=-fastsse LINKFORSHARED=-Wl,–export-dynamic

These options get saved in a master Makefile in /usr/lib/python2.6/config/Makefile (or where ever your python installation is); presumably, you can edit this file to change the compile options, but I haven’t tried this.

In case it helps, here is the section of this Makefile from my python installation on the cray system that deals with the C compiler

=== Variables set by configure

VERSION= 2.6
srcdir= .


CC= cc
CXX= CC
MAINCC= $(CC)
LINKCC= $(PURIFY) $(MAINCC)
AR= ar
RANLIB= ranlib
SVNVERSION= echo Unversioned directory

Shell used by make (some versions default to the login shell, which is bad)

SHELL= /bin/sh

Use this to make a link between python$(VERSION) and python in $(BINDIR)

LN= ln

Portable install script (configure doesn’t always guess right)

INSTALL= /usr/bin/install -c
INSTALL_PROGRAM=${INSTALL}
INSTALL_SCRIPT= ${INSTALL}
INSTALL_DATA= ${INSTALL} -m 644

Shared libraries must be installed with executable mode on some systems;

rather than figuring out exactly which, we always give them executable mode.

Also, making them read-only seems to be a good idea…

INSTALL_SHARED= ${INSTALL} -m 555

MAKESETUP= $(srcdir)/Modules/makesetup

Compiler options

OPT= -DNDEBUG -fastsse
BASECFLAGS=
CFLAGS= $(BASECFLAGS) $(OPT) $(EXTRA_CFLAGS)

Both CPPFLAGS and LDFLAGS need to contain the shell’s value for setup.py to

be able to build extension modules using the directories specified in the

environment variables

CPPFLAGS= -I. -IInclude -I$(srcdir)/Include
LDFLAGS=
LDLAST=
SGI_ABI=
CCSHARED= -fPIC
LINKFORSHARED= -Wl,–export-dynamic

Extra C flags added for building the interpreter object files.

CFLAGSFORSHARED=

C flags used for building the interpreter object files

PY_CFLAGS= $(CFLAGS) $(CPPFLAGS) $(CFLAGSFORSHARED) -DPy_BUILD_CORE

\

Machine-dependent subdirectories

MACHDEP= linux2

Install prefix for architecture-independent files

prefix= /lustre/scratch/jvanclev/opt

Install prefix for architecture-dependent files

exec_prefix= ${prefix}

Install prefix for data files

datarootdir= ${prefix}/share

Hi Jeremy,

Now that I’m back in the office, I was able to recreate the error. Turns out to be another compiler problem where pgcc is removing what it thinks is a vestigial function. I have sent a report to our engineers (TPR#16393) and hopefully we can have this fixed soon.

In the mean time, you can add the internal compiler flag “-Mx,2,0x1000” to the compilation of ckdtree.c as well as vonmises_cython.c. " -Mx,2,0x1000" tells the compiler to not perform vestigial elimination. Note that internal flags can and do change from release to release.

Hope this helps,
Mat

Hi Mat,

Your advice worked perfectly; I was able to successfully compile scipy. Within python, I ran some the numpy and scipy tests, many of which failed. I’m going to look into it and see if the problem isn’t my compilation of lapack/ATLAS. I’ll post again if something comes up related to PGI.

Thanks for all your help!

Jeremy

Jeremy,

TPR 16377 was corrected in the 10.1 release.

regards,
dave