MPI with pgf90 and MPICH on an opteron cluster

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi,

I am trying to compile COSMOMC (a cosmological parameter fitting package) on a cluster of operton machines, running v6 of the Portland compilers and MPICH.

Everything seems to go smoothly until the final link stage (I have used hardcoded paths to be sure that everything is using the compiler and version I expect), when the mpi related functions are not found.

The output from th make file is as follows (camb is a library that is built during the install - I know that this is working, and it does not seem to be related to this problem, in any case).

I would be very grateful for any hints or tips. There is apparently an MPICH library in the appropriate directory.

/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c WMAP_likelihood.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c utils.F90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c settings.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c cmbtypes.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c cmbdata.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c WeakLen.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c twodf.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c mpk.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c supernovae.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c CMB_Cls_simple.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c paramdef.F90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c propose.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c params_CMB.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c calclike.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c conjgrad_wrapper.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c EstCovmat.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c postprocess.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c MCMC.f90
/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -fast -DMPI -I…/
camb -I/usr/local/cluster/mpi/include -c driver.F90


/usr/local/cluster/pgi/linux86-64/6.0/bin/pgf90 -o …/cosmomc
WMAP_likelihood.o utils.o
settings.o cmbtypes.o cmbdata.o WeakLen.o twodf.o mpk.o supernovae.o
CMB_Cls_simple.o paramdef.o propose.o params_CMB.o calclike.o
conjgrad_wrapper.o
EstCovmat.o postprocess.o MCMC.o driver.o -L…/camb -lcamb -L/usr/
local/cluster/pgi/linux86-64/6.0/lib
-llapack -lblas -L/usr/local/cluster/mpi/lib -lmpich -fast -
DMPI -I…/camb -I/usr/local/cluster/mpi/include

A previously compiled program (“camb”) is made available locally as a
library. That seems to be workig. However, the mpi functions are
still not being seen by the linker;

paramdef.o: In function paramdef_dostop_': paramdef.o(.text+0xf5): undefined reference to mpi_wtime_’
paramdef.o(.text+0x295): undefined reference to mpi_finalize_' paramdef.o: In function paramdef_doabort_‘:
paramdef.o(.text+0x3b2): undefined reference to mpi_abort_' paramdef.o: In function paramdef_addmpiparams_’:
paramdef.o(.text+0x42ba): undefined reference to mpi_wtime_' paramdef.o(.text+0x4496): undefined reference to mpi_isend_’
paramdef.o(.text+0x44ea): undefined reference to mpi_irecv_' paramdef.o(.text+0x4742): undefined reference to mpi_testall_’
paramdef.o(.text+0x47d0): undefined reference to mpi_testall_' paramdef.o(.text+0x48a7): undefined reference to mpi_issend_’
paramdef.o(.text+0x48e6): undefined reference to mpi_iprobe_' paramdef.o(.text+0x4930): undefined reference to mpi_recv_’
paramdef.o(.text+0x552e): undefined reference to mpi_bcast_' paramdef.o(.text+0x558f): undefined reference to mpi_bcast_’
paramdef.o: In function paramdef_checklimitsconverge_': paramdef.o(.text+0x8808): undefined reference to mpi_bcast_’
driver.o: In function MAIN_': driver.o(.text+0x4c): undefined reference to mpi_init_’
driver.o(.text+0x258): undefined reference to mpi_comm_rank_' driver.o(.text+0x2fe): undefined reference to mpi_comm_size_’
driver.o(.text+0x303): undefined reference to `mpi_wtime_’

2

Hi Parallel,

Looks like your missing the MPICH Fortran library (libfmpich.a) which acts as the interface from Fortran to C. First make sure it was built and installed in “/usr/local/cluster/mpi/lib”. (If it’s not there, you’ll need to get it built before proceeding. See the PGI MPICH Guide if youe need help. ) Next add “-lfmpich” before “-lmpich” on the link line.

Hope this helps,
Mat

3

I added the -lfmpich and got the same response. Do the MPI libraries have to be built with the portland compilers in order to work? I understand that the MPI libraries in the -L/usr/local/cluster/mpi/lib directory on this cluster are built with some other set of tools.

There is definitely a libfmpich.a in this directory, however.

4

If you built the MPI libraries with a F90 compiler, then there may be runtime dependencies. But given that the undefined object names are from MPICH, I don’t think this is the case.

With a F77 compiler you shouldn’t have compatability problems. However, if you’ve used g77 then the symbol names may be different. g77 appends two underscores while most other compilers append one. Try performing the following command:

nm /usr/local/cluster/mpi/lib/libfmpich.a | grep mpi_recv

Are there one or two underscores at the end of “mpi_recv”? If there are two, try recompiling with the “-Msecond_underscore” flag to have pgf90 append a second underscore to symbol names.

  • Mat
5

I forgot to mention that if the library was built with g77, you’ll may need to add “-g77libs” to the link line to get the g77 libraries linked in.

  • Mat