undefined reference to `rdma_get_src_port@RDMACM_1.0'

jand · October 20, 2016, 4:24pm

Hi,

after installing 16.9 on ubuntu 14.04, I get the following linking error:

mpif90 -I./include -Mcuda -o ./bin/smc_cuda.x ./obj/psrjmh_pt_auto.o ./obj/rjmcmc_com.o ./obj/read_input.o ./obj/nrtype.o ./obj/nr.o ./obj/nrutil.o ./obj/median.o ./obj/loglhood.o ./obj/cuda_refl_levin.o -lblas -llapack -I./include
/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so: undefined reference to `rdma_get_src_port@RDMACM_1.0’

I installed the compiler with the MPI option. Any idea what may cause this issue?

Thanks, Jan

tull · October 24, 2016, 11:37pm

Add the switches

-v -Wl,-t

and then add the output dialog associated with the failed link,
or send it to trs@pgroup.com (if you can send source that demonstrates,
that would be good).

Nit: lapack calls routines from blas, so -llapack should come before -lblas

dave

tull · November 8, 2016, 10:23pm

We have since learned that the
rdma_get_src_port@RDMACM_1.0
symbol comes from the Infiniband Libraries.

A properly configured OFED (install in the default locations),
should make this work, since when you indicate you want
GPU support in OpenMPI, the expectation is that you have an
Infiniband network with OFED.

To correct, reinstall, and say ‘no’ to having GPU mpi support,
or have the OFED running with infiniband.

dave

jand · January 30, 2017, 8:17pm

Hi Dave,

finally getting back to this…

I re-installed the compiler (this time using 16.10) and chose ‘no’ when queried for GPU MPI support.

However, the problem persists (see below). Are there any other possible causes or solutions?

Thanks, Jan

mpif90 -C -L/usr/lib -llapack -lblas -o ./bin/prjmh_temper_rf ray3d/ray3d.o ray3d/ray3d_com.o ray3d/ray3d_par.o ray3d/theo.o ray3d/qlayer.o ./obj/data_type.o ./obj/rjmcmc_com.o ./obj/alloc_obj.o ./obj/read_input.o ./obj/loglhood.o ./obj/nrtype.o ./obj/nr.o ./obj/nrutil.o ./obj/svdcmp.o ./obj/mod_sac_io.o ./obj/four1.o ./obj/fourrow.o ./obj/realft.o ./obj/convlv.o ./obj/quicksort.o ./obj/pythag.o ./obj/prjmh_temper_rf.o ./obj/libraysum.a ./obj/libswd.a
/usr/bin/ld: warning: libmpi.so.12, needed by /opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempif08.so, may conflict with libmpi.so.1
/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12: undefined reference to rdma_get_src_port@RDMACM_1.0' /opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12: undefined reference to rdma_get_dst_port@RDMACM_1.0’
make: *** [bin/prjmh_temper_rf] Error 2

tull · January 30, 2017, 8:55pm

Try
mpif90 -C -L/usr/lib -L./obj -o ./bin/prjmh_temper_rf ray3d/ray3d.o ray3d/ray3d_com.o ray3d/ray3d_par.o ray3d/theo.o ray3d/qlayer.o ./obj/data_type.o ./obj/rjmcmc_com.o ./obj/alloc_obj.o ./obj/read_input.o ./obj/loglhood.o ./obj/nrtype.o ./obj/nr.o ./obj/nrutil.o ./obj/svdcmp.o ./obj/mod_sac_io.o ./obj/four1.o ./obj/fourrow.o ./obj/realft.o ./obj/convlv.o ./obj/quicksort.o ./obj/pythag.o ./obj/prjmh_temper_rf.o -llapack -lblas -lraysum -lswd -Wl,-t -v

the libs should always come after the objects, and in the order where any references made in an object or library, show up later in the link line.

dave

jand · January 30, 2017, 9:46pm

Hi Dave,

thanks for the suggestion. The issue persists with the command you recommend (see below for screen output).

I can also say that I have used this source code and makefile on several machines (linux and OSX) and with several compilers (GNU, intel and PGI) and have not had this issue before.

Thanks, Jan

mpif90 -C -L/usr/lib -L./obj -o ./bin/prjmh_temper_rf ray3d/ray3d.o ray3d/ray3d_com.o ray3d/ray3d_par.o ray3d/theo.o ray3d/qlayer.o ./obj/data_type.o ./obj/rjmcmc_com.o ./obj/alloc_obj.o ./obj/read_input.o ./obj/loglhood.o ./obj/nrtype.o ./obj/nr.o ./obj/nrutil.o ./obj/svdcmp.o ./obj/mod_sac_io.o ./obj/four1.o ./obj/fourrow.o ./obj/realft.o ./obj/convlv.o ./obj/quicksort.o ./obj/pythag.o ./obj/prjmh_temper_rf.o -llapack -lblas -lraysum -lswd -Wl,-t -v
Export PGI=/opt/pgi

/usr/bin/ld /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /opt/pgi/linux86-64/16.9/lib/trace_init.o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o /opt/pgi/linux86-64/16.9/lib/initmp.o /opt/pgi/linux86-64/16.9/lib/f90main.o --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /opt/pgi/linux86-64/16.9/lib/pgi.ld -L/usr/lib -L./obj -L/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -L/opt/pgi/linux86-64/16.9/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 ray3d/ray3d.o ray3d/ray3d_com.o ray3d/ray3d_par.o ray3d/theo.o ray3d/qlayer.o ./obj/data_type.o ./obj/rjmcmc_com.o ./obj/alloc_obj.o ./obj/read_input.o ./obj/loglhood.o ./obj/nrtype.o ./obj/nr.o ./obj/nrutil.o ./obj/svdcmp.o ./obj/mod_sac_io.o ./obj/four1.o ./obj/fourrow.o ./obj/realft.o ./obj/convlv.o ./obj/quicksort.o ./obj/pythag.o ./obj/prjmh_temper_rf.o -llapack -lblas -lraysum -lswd -t -rpath ‘$ORIGIN/…/…/lib:$ORIGIN/…/…/…/…/lib’ -rpath /opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib --enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -rpath /opt/pgi/linux86-64/16.9/lib -o ./bin/prjmh_temper_rf -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lpgmp -lnuma -lpthread -lnspgc -lpgc -lrt -lpthread -lm -lgcc -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/x86_64-linux-gnu/crtn.o
/usr/bin/ld: mode elf_x86_64
/usr/lib/x86_64-linux-gnu/crt1.o
/usr/lib/x86_64-linux-gnu/crti.o
/opt/pgi/linux86-64/16.9/lib/trace_init.o
/usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o
/opt/pgi/linux86-64/16.9/lib/initmp.o
/opt/pgi/linux86-64/16.9/lib/f90main.o
ray3d/ray3d.o
ray3d/ray3d_com.o
ray3d/ray3d_par.o
ray3d/theo.o
ray3d/qlayer.o
./obj/data_type.o
./obj/rjmcmc_com.o
./obj/alloc_obj.o
./obj/read_input.o
./obj/loglhood.o
./obj/nrtype.o
./obj/nr.o
./obj/nrutil.o
./obj/svdcmp.o
./obj/mod_sac_io.o
./obj/four1.o
./obj/fourrow.o
./obj/realft.o
./obj/convlv.o
./obj/quicksort.o
./obj/pythag.o
./obj/prjmh_temper_rf.o
-llapack (/usr/lib/liblapack.so)
-lblas (/usr/lib/libblas.so)
(./obj/libraysum.a)raysum.o
(./obj/libraysum.a)readwrite.o
(./obj/libraysum.a)phaselist.o
(./obj/libraysum.a)buildmodel.o
(./obj/libraysum.a)trace.o
(./obj/libraysum.a)recipes00.o
(./obj/libraysum.a)eigenvec.o
(./obj/libraysum.a)matrixops.o
(./obj/libraysum.a)eispack-cg.o
(./obj/libswd.a)dispersion.o
(./obj/libswd.a)lovmrx.o
(./obj/libswd.a)raymrx.o
(./obj/libswd.a)raydsp.o
-lmpi_usempif08 (/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempif08.so)
-lmpi_usempi_ignore_tkr (/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempi_ignore_tkr.so)
-lmpi_mpifh (/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_mpifh.so)
-lmpi (/usr/lib/libmpi.so)
-lpgf90rtl (/opt/pgi/linux86-64/16.9/lib/libpgf90rtl.so)
-lpgf90 (/opt/pgi/linux86-64/16.9/lib/libpgf90.so)
-lpgf90_rpm1 (/opt/pgi/linux86-64/16.9/lib/libpgf90_rpm1.so)
-lpgf902 (/opt/pgi/linux86-64/16.9/lib/libpgf902.so)
-lpgf90rtl (/opt/pgi/linux86-64/16.9/lib/libpgf90rtl.so)
-lpgftnrtl (/opt/pgi/linux86-64/16.9/lib/libpgftnrtl.so)
-lpgmp (/opt/pgi/linux86-64/16.9/lib/libpgmp.so)
-lnuma (/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libnuma.so)
/lib/x86_64-linux-gnu/libpthread.so.0
-lpgc (/opt/pgi/linux86-64/16.9/lib/libpgc.so)
-lrt (//usr/lib/x86_64-linux-gnu/librt.so)
/lib/x86_64-linux-gnu/libpthread.so.0
-lm (//usr/lib/x86_64-linux-gnu/libm.so)
/lib/x86_64-linux-gnu/libc.so.6
(/usr/lib/x86_64-linux-gnu/libc_nonshared.a)elf-init.oS
/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
-lgcc_s (/usr/lib/gcc/x86_64-linux-gnu/4.8/libgcc_s.so)
/usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o
/usr/lib/x86_64-linux-gnu/crtn.o
/usr/bin/ld: warning: libmpi.so.12, needed by /opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempif08.so, may conflict with libmpi.so.1
/opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12: undefined reference to rdma_get_src_port@RDMACM_1.0' /opt/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12: undefined reference to rdma_get_dst_port@RDMACM_1.0’
/usr/bin/ld: link errors found, deleting executable `./bin/prjmh_temper_rf’
pgfortran-Fatal-linker completed with exit code 1

jand · April 7, 2017, 6:42pm

This problem persists in the community edition.
Under the 17.1 release (on a different linux computer where I have a full license), I do not have the issue.

-Jan

tull · April 10, 2017, 9:16pm

The issue is the result of missing infiniband libraries and probably OFED.

The Open MPI libraries could be installed with or without GPU support.

If you choose “with GPU support” you get a version of the MPI libraries that
require you have Infiniband hardware with OFED software installed.

I would avoid the GPU support in MPI in 16.10.

Reinstall the 16.10 community edition, and say ‘yes’ to Openmpi installation,
but say ‘NO’ to 'Do you want to enable NVIDIA GPU support in Open MPI.

dave