I compiled an example mpihello with PGI on a Linux system. However, when I copy the executable file to another computer without PGI, it seems that the it does not work. And the error message is as follows:
[lji565@raijin1 mpihello]$ mpiexec -np 1 mpihello
mpihello: error while loading shared libraries: libmpi_usempif08.so.11: cannot open shared object file: No such file or directory
What could I do to solve this problem?
Many thanks
Moving executables to other target systems is essentially
creating an executable, determining what external references come
from shared libraries (libx.so instead of libx.a), and moving those libs as
well as the executable.
Once you know the shared libs and where the executable expects to find them,
you then move the libraries not on the target to the system to either
the same path on the target system, or you put them in a directory
designated by $LD_LIBRARY_PATH.
However, your example use mpi. The same or compatible MPI must be
installed and working on the target system before running the executable.
This can be done by installing the PGI compilers and MPI libs on your
target system.
It is not always enough to just copy the MPI libs over, if MPI
programs don’t run on the target system. But it might work.
You use the MPI version that you put in your $PATH.
export PATH=$PGI/linux86-64/16.10/bin:$PGI/linux86-64/2016/mpi/openmpi/bin:$PATH
Then build the executable
mpif90 -o mpihello mpihello.f
Then determine what shared libraries it needs.
% ldd mpihello
linux-vdso.so.1 => (0x00007ffce2bdf000)
libmpi_usempif08.so.11 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempif08.so.11 (0x00007fb6f7ea2000)
libmpi_usempi_ignore_tkr.so.6 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempi_ignore_tkr.so.6 (0x00007fb6f7c99000)
libmpi_mpifh.so.12 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_mpifh.so.12 (0x00007fb6f7a3c000)
libmpi.so.12 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12 (0x00007fb6f7457000)
libpgf90rtl.so => /home/tull/Downloads/pgi1610/linux86-64/16.10/lib/libpgf90rtl.so (0x00007fb6f7230000)
libpgf90.so => /home/tull/Downloads/pgi1610/linux86-64/16.10/lib/libpgf90.so (0x00007fb6f6c7a000)
libpgf90_rpm1.so => /home/tull/Downloads/pgi1610/linux86-64/16.10/lib/libpgf90_rpm1.so (0x00007fb6f6a78000)
libpgf902.so => /home/tull/Downloads/pgi1610/linux86-64/16.10/lib/libpgf902.so (0x00007fb6f6864000)
libpgftnrtl.so => /home/tull/Downloads/pgi1610/linux86-64/16.10/lib/libpgftnrtl.so (0x00007fb6f662f000)
libpgmp.so => /home/tull/Downloads/pgi1610/linux86-64/16.10/lib/libpgmp.so (0x00007fb6f63af000)
libnuma.so => /local/pgi/16.10/share_objects/lib64/libnuma.so (0x00007fb6f61ad000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb6f5f80000)
libpgc.so => /home/tull/Downloads/pgi1610/linux86-64/16.10/lib/libpgc.so (0x00007fb6f5cf8000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fb6f5aef000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fb6f57e9000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb6f5425000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fb6f520d000)
librdmacm.so.1 => /usr/lib/librdmacm.so.1 (0x00007fb6f4ff7000)
libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00007fb6f4de7000)
libopen-rte.so.12 => /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libopen-rte.so.12 (0x00007fb6f4ae4000)
libopen-pal.so.13 => /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libopen-pal.so.13 (0x00007fb6f47a1000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007fb6f4595000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb6f4391000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fb6f418e000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb6f8107000)
So, for example, you would put libpgf902.so on the target machine
in a directory called
/home/tull/Downloads/pgi1610/linux86-64/16.10/lib
OR put it in a LD_LIBRARY_PATH directory.
cp libpgf902.so /home/xxx/.
export LD_LIBRARY_PATH=/home/xxx:$LD_LIBRARY_PATH
and when you
ldd mpihello
on the target machine, you will now see it finds libpgf902.so in /home/xxx.
Only the PGI libs found in $PGI/linux86-64/16.10/REDIST
can be copied to other systems for use with executables.
You can reduce most of the work by adding -Bstatid_pgi
to your link line
mpif90 -o mpihello mpihello.f -Bstatic_pgi
will result in an executable with fewer external refs.
% ldd mpihello
linux-vdso.so.1 => (0x00007ffe251ea000)
libmpi_usempif08.so.11 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempif08.so.11 (0x00007f0da6d1c000)
libmpi_usempi_ignore_tkr.so.6 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_usempi_ignore_tkr.so.6 (0x00007f0da6b13000)
libmpi_mpifh.so.12 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_mpifh.so.12 (0x00007f0da68b6000)
libmpi.so.12 => /home/tull/Downloads/pgi1610/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12 (0x00007f0da62d1000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f0da60a3000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f0da5e9b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f0da5b95000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f0da57d0000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0da6f81000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f0da55b9000)
librdmacm.so.1 => /usr/lib/librdmacm.so.1 (0x00007f0da53a2000)
libibverbs.so.1 => /usr/lib/libibverbs.so.1 (0x00007f0da5193000)
libopen-rte.so.12 => /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libopen-rte.so.12 (0x00007f0da4e90000)
libopen-pal.so.13 => /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libopen-pal.so.13 (0x00007f0da4b4c000)
libnuma.so.1 => /usr/lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f0da4941000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f0da473d000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f0da4539000)
libpgf90.so => not found
libpgf90_rpm1.so => not found
libpgf902.so => not found
libpgf90rtl.so => not found
libpgftnrtl.so => not found
libpgmp.so => not found
libpgc.so => not found
libpgf90.so => not found
libpgf90_rpm1.so => not found
libpgf902.so => not found
libpgf90rtl.so => not found
libpgftnrtl.so => not found
libpgmp.so => not found
libpgc.so => not found
Notice only Linux systems routines and MPI libs are shared references.
The Linux system refs are satisfied by the target system’s version, so they
do not need to be copied. The MPI shared libs should be copied from the
installed area in $PGI/linux86-64/2016/mpi/openmpi/lib, and should be copied to the target system. But it is better if they are installed.
Hope this helps.
Hi jtull,
Thank you very much for your detailed explanation. Another issue I met is that the command ‘idd’ could not be found on my system. Do you know how to install it?
Best wishes,
Lulu Jiao
the ‘i’ in ldd is a lower case ‘L’.
ldd is a Linux command that should be there like any other
system util.
dave