MPI programs give the same error

I installed the community edition compiler on a machine and now any code that I try to run after compiling with mpif90, I get the same error. I tested a simple code

program main
   include 'mpif.h'
   integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)

   call MPI_INIT(ierror)
   call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
   call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
   print*, 'node', rank, ': Hello world'
   call MPI_FINALIZE(ierror)
   end

and it exits with the error

--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    ini file:file not found
But I couldn't open the help file:
    /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/share/openmpi/help-mpi-btl-openib.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no device params found
But I couldn't open the help file:
    /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/share/openmpi/help-mpi-btl-openib.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no device params found
But I couldn't open the help file:
    /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/share/openmpi/help-mpi-btl-openib.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    default subnet prefix
But I couldn't open the help file:
    /proj/pgi/linux86-64/2016/mpi/openmpi-1.10.2/share/openmpi/help-mpi-btl-openib.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
*** Process received signal ***
Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
Failing at address: 0x8 
[ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fac31c0d330]
[ 1] /usr/lib64/libmlx5-rdmav2.so(+0x27e10)[0x7fac2d3bbe10]
[ 2] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(+0x147b40)[0x7fac33209b40]
[ 3] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(+0x14700a)[0x7fac3320900a]
[ 4] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(+0x1462b2)[0x7fac332082b2]
[ 5] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(+0x145e15)[0x7fac33207e15]
[ 6] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(ompi_btl_openib_connect_base_select_for_loc]
[ 7] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(+0x12fbc9)[0x7fac331f1bc9]
[ 8] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(mca_btl_base_select+0x18f)[0x7fac331e5f8f]
[ 9] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(mca_bml_r2_component_init+0x23)[0x7fac331e5]
[10] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(mca_bml_base_init+0xa9)[0x7fac331e3b69]
[11] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(+0x26454b)[0x7fac3332654b]
[12] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(mca_pml_base_select+0x1f0)[0x7fac33323700]
[13] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(ompi_mpi_init+0x765)[0x7fac33165365]
[14] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi.so.12(MPI_Init+0x152)[0x7fac33191622]
[15] /home/usr/local/pgi/linux86-64/16.10/mpi/openmpi/lib/libmpi_mpifh.so.12(ompi_init_f+0x2c)[0x7fac336ef6ac]
[16] ./a.out[0x4014b3]
*** End of error message ***

Here is what I did with 16.10, the same release you have.

% setenv PGI /opt/pgi
% set path=($PGI/linux86-64/16.10/bin $PGI/linux86-64/2016/mpi/openmpi/bin $path)


% mpif90 -o mpihello mpihello.f
% mpirun -np 4 mpihello
thread 0 : Hello world
thread 2 : Hello world
thread 1 : Hello world
thread 3 : Hello world
% more mpihello.f
program main
include ‘mpif.h’
integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)

call MPI_INIT(ierror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
print*, ‘thread’, rank, ‘: Hello world’
call MPI_FINALIZE(ierror)
end


If it continues, send the output of

mpif90 -v -o mpihello mpihello.f

This is what I get when I do

mpif90 -v -o hello hello.f90



Export PGI=/home/vikram/usr/local/pgi

/home/vikram/usr/local/pgi/linux86-64/16.10/bin/pgf901 hello.f90 -opt 1 -nohpf -nostatic -x 19 0x400000 -quad -x 59 4 -x 15 2 -x 49 0x400
004 -x 51 0x20 -x 57 0x4c -x 58 0x10000 -x 124 0x1000 -tp haswell -x 57 0xfb0000 -x 58 0x78031040 -x 47 0x08 -x 48 4608 -x 49 0x100 -x 12
0 0x200 -stdinc /home/vikram/usr/local/pgi/linux86-64/16.10/include-gcc48:/home/vikram/usr/local/pgi/linux86-64/16.10/include:/usr/lib/gc
c/x86_64-linux-gnu/4.8/include:/usr/local/include:/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed:/usr/include/x86_64-linux-gnu:/usr/incl
ude -cmdline '+pgfortran hello.f90 -v -o hello -I/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include -I/home/vikram/usr
/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -Wl,-rpath,$ORIGIN/../../lib:$ORIGIN/../../../../lib -Wl,-rpath -Wl,/home/vikram/usr/lo
cal/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -Wl,--enable-new-dtags -L/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib
 -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi' -def unix -def __unix -def __unix__ -def linux -def __linux -def __linux__ -d
ef __NO_MATH_INLINES -def __LP64__ -def __x86_64 -def __x86_64__ -def __LONG_MAX__=9223372036854775807L -def '__SIZE_TYPE__=unsigned long
 int' -def '__PTRDIFF_TYPE__=long int' -def __THROW= -def __extension__= -def __amd_64__amd64__ -def __k8 -def __k8__ -def __SSE__ -def _
_MMX__ -def __SSE2__ -def __SSE3__ -def __SSSE3__ -def __STDC_HOSTED__ -idir /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.
2/include -idir /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -freeform -vect 48 -y 54 1 -x 70 0x40000000 -y 163 0xc0
000000 -x 189 0x10 -modexport /tmp/pgfortranCsKcuRBPpQpX.cmod -modindex /tmp/pgfortran8sKc0f5YQ631.cmdx -output /tmp/pgfortranCsKcuYCzRYk
X.ilm
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 17)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 24)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 31)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 38)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 45)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 52)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 59)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 66)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 73)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 80)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 87)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 94)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 101)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 108)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 115)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 122)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 129)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 136)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 143)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 150)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 157)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 164)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 171)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 178)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 185)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 192)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 199)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 206)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 213)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 220)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 227)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 234)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 241)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 248)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 255)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 262)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 269)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 276)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 283)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 290)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 297)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 304)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 311)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 318)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 325)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 332)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 339)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 346)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 353)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 360)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 367)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 374)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 381)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 388)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 395)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 402)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 409)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 416)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 423)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 430)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 437)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 444)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 451)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 458)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 469)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 476)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 483)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 497)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 504)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 511)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 518)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 525)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 532)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 539)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 546)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 553)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 560)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 567)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 574)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 581)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 588)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 595)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 602)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 609)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 616)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 623)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 630)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 637)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 644)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 651)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 658)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 665)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 672)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 679)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 686)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 693)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 700)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 707)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 714)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 721)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 728)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 735)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 742)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 749)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 756)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 763)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 770)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 777)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 784)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 791)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 798)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 805)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 812)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 819)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 826)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 833)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 840)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 847)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 854)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 861)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 868)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 875)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 889)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 896)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 903)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/mp
if-sizeof.h: 910)
PGF90-I-0035-Predefined intrinsic size loses intrinsic property (hello.f90: 7)
129 inform,   0 warnings,   0 severes, 0 fatal for main
PGF90/x86-64 Linux 16.10-0: compilation completed with informational messages

/home/vikram/usr/local/pgi/linux86-64/16.10/bin/pgf902 /tmp/pgfortranCsKcuYCzRYkX.ilm -fn hello.f90 -opt 1 -x 51 0x20 -x 119 0xa10000 -x 
122 0x40 -x 123 0x1000 -x 127 4 -x 127 17 -x 19 0x400000 -x 28 0x40000 -x 120 0x10000000 -x 70 0x8000 -x 122 1 -x 125 0x20000 -quad -x 59
 4 -tp haswell -x 120 0x1000 -x 124 0x1400 -y 15 2 -x 57 0x3b0000 -x 58 0x48000000 -x 49 0x100 -x 120 0x200 -astype 0 -x 70 0x40000000 -x
 124 1 -y 163 0xc0000000 -x 189 0x10 -y 189 0x4000000 -cmdline '+pgfortran hello.f90 -v -o hello -I/home/vikram/usr/local/pgi/linux86-64/
2016/mpi/openmpi-1.10.2/include -I/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -Wl,-rpath,$ORIGIN/../../lib:$ORIGIN/
../../../../lib -Wl,-rpath -Wl,/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -Wl,--enable-new-dtags -L/home/vikram/us
r/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi' -asm /tmp/pgfortranCsKcu0b4
gmAT.s
  0 inform,   0 warnings,   0 severes, 0 fatal for main
PGF90/x86-64 Linux 16.10-0: compilation successful

/usr/bin/as /tmp/pgfortranCsKcu0b4gmAT.s -I/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/include/ -I/home/vikram/usr/loca
l/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/ -o /tmp/pgfortranCsKcuntwwYMx.o

/home/vikram/usr/local/pgi/linux86-64/16.10/bin/pgappend -noerror /tmp/pgfortranCsKcuntwwYMx.o -name .IPDINFO /tmp/pgfortranCsKcuRBPpQpX.
cmod -name .IPEINFO /tmp/pgfortran8sKc0f5YQ631.cmdx
/usr/bin/ld /usr/lib/x86_64-linux-gnu/crt1.o /usr/lib/x86_64-linux-gnu/crti.o /home/vikram/usr/local/pgi/linux86-64/16.10/lib/trace_init.
o /usr/lib/gcc/x86_64-linux-gnu/4.8/crtbegin.o /home/vikram/usr/local/pgi/linux86-64/16.10/lib/initmp.o /home/vikram/usr/local/pgi/linux8
6-64/16.10/lib/f90main.o --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /home/vikram/usr/local/pgi/linux86-64/1
6.10/lib/pgi.ld -L/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib -L/home/vikram/usr/local/pgi/linux86-64/16.10/lib -L/
usr/lib64 -L/usr/lib/gcc/x86_64-linux-gnu/4.8 /tmp/pgfortranCsKcuntwwYMx.o -rpath '$ORIGIN/../../lib:$ORIGIN/../../../../lib' -rpath /hom
e/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib --enable-new-dtags -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmp
i -rpath /home/vikram/usr/local/pgi/linux86-64/16.10/lib -o hello -lpgf90rtl -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lpgmp -
lnuma -lpthread -lnspgc -lpgc -lrt -lpthread -lm -lgcc -lc -lgcc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/4.8/crtend.o /usr/lib/x86_64-linux
-gnu/crtn.o
Unlinking /tmp/pgfortranCsKcuYCzRYkX.ilm
Unlinking /tmp/pgfortran8sKc0edd3Lvw.stb
Unlinking /tmp/pgfortranCsKcuRBPpQpX.cmod
Unlinking /tmp/pgfortran8sKc0f5YQ631.cmdx
Unlinking /tmp/pgfortranCsKcu0b4gmAT.s
Unlinking /tmp/pgfortran8sKc0wqwSuUV.ll
Unlinking /tmp/pgfortranCsKcuntwwYMx.o

Did you try running the program? I think it built. I wonder if the problem
is the machines running the program.

I saw the same ‘Predefined intrinsic size’ informational messages, but it
ran.

Try this version, which lists the hostnames for each process started.

% more mpihello_2.f
program hello
include ‘mpif.h’
integer ierr, myproc,hostnm, nprocs
character*64 hostname
call mpi_init(ierr)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD, nprocs, ierr)
call mpi_comm_rank(MPI_COMM_WORLD, myproc, ierr)
ierr=setvbuf3f(6,2,0)
ierr=hostnm(hostname)
write(6,100) myproc,nprocs,hostname
100 format(1x,“hello - I am process”,i3," of", i3, " on host ",A64)
call mpi_finalize(ierr)
end


mpif90 -o mpihello_2 mpihello_2.f
mpirun -np 4 mpihello_2

and see what happens.

Hi.

Thanks for the reply.

Compiling was never the problem. It was compiling the code fine. But running it caused the error messages.

Here’s the error I get.

mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
mlx5: Warning: Timestamp available,
but failed to mmap() hca core clock page.
--------------------------------------------------------------------------
WARNING: There are more than one active ports on host 'gpu-plx', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical IB
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate IB subnet that is
used between connected MPI processes must have different subnet ID
values.

Please see this FAQ entry for more details:

  http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid

NOTE: You can turn off this warning by setting the MCA parameter
      btl_openib_warn_default_gid_prefix to 0.
--------------------------------------------------------------------------
[gpu-plx:12023] *** Process received signal ***
[gpu-plx:12023] Signal: Segmentation fault (11)
[gpu-plx:12023] Signal code: Address not mapped (1)
[gpu-plx:12023] Failing at address: 0x8
[gpu-plx:12024] *** Process received signal ***
[gpu-plx:12024] Signal: Segmentation fault (11)
[gpu-plx:12024] Signal code: Address not mapped (1)
[gpu-plx:12024] Failing at address: 0x8
[gpu-plx:12023] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fb49d6a1330]
[gpu-plx:12024] [ 1] /usr/lib64/libmlx5-rdmav2.so(+0x27e10)[0x7f907acf2e10]
[gpu-plx:12024] [ 2] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x147b40)[0x7fb49ec9db40]
[gpu-plx:12023] [ 3] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x147b40)[0x7f9080b40b40]
[gpu-plx:12024] [ 3] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x14700a)[0x7fb49ec9d00a]
[gpu-plx:12023] [ 4] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x14700a)[0x7f9080b4000a]
[gpu-plx:12024] [ 4] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x1462b2)[0x7fb49ec9c2b2]
[gpu-plx:12023] [ 5] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x1462b2)[0x7f9080b3f2b2]
[gpu-plx:12024] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x145e15)[0x7fb49ec9be15]
[gpu-plx:12023] [ 6] [ 5] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x145e15)[0x7f9080b3ee15]
[gpu-plx:12024] [ 6] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_btl_openib_connect_base_select_f
or_local_port+0xfa)[0x7fb49ec9817a]
[gpu-plx:12023] [ 7] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_btl_openib_connect_base_select_f
or_local_port+0xfa)[0x7f9080b3b17a]
[gpu-plx:12024] [ 7] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x12fbc9)[0x7fb49ec85bc9]
[gpu-plx:12023] [ 8] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x12fbc9)[0x7f9080b28bc9]
[gpu-plx:12024] [ 8] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_btl_base_select+0x18f)[0x7fb49ec7
9f8f]
[gpu-plx:12023] [ 9] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_r2_component_init+0x23)[0x7fb
49ec79833]
[gpu-plx:12023] [10] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_btl_base_select+0x18f)[0x7f9080b1
cf8f]
[gpu-plx:12024] [ 9] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_base_init+0xa9)[0x7fb49ec77b6
9]
[gpu-plx:12023] [11] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_r2_component_init+0x23)[0x7f9
080b1c833]
[gpu-plx:12024] [10] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_base_init+0xa9)[0x7f9080b1ab6
9]
[gpu-plx:12024] [11] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x26454b)[0x7fb49edba54b]
[gpu-plx:12023] [12] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x26454b)[0x7f9080c5d54b]
[gpu-plx:12024] [12] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_pml_base_select+0x1f0)[0x7fb49edb
7700]
[gpu-plx:12023] [13] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_mpi_init+0x765)[0x7fb49ebf9365]
[gpu-plx:12023] [14] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_pml_base_select+0x1f0)[0x7f9080c5
a700]
[gpu-plx:12024] [13] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(MPI_Init+0x152)[0x7fb49ec25622]
[gpu-plx:12023] [15] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_mpi_init+0x765)[0x7f9080a9c365]
[gpu-plx:12024] [14] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(MPI_Init+0x152)[0x7f9080ac8622]
[gpu-plx:12024] [15] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_mpifh.so.12(ompi_init_f+0x2c)[0x7fb49f1836a
c][gpu-plx:12023] [16] ./a.out[0x401533]
[gpu-plx:12023] *** End of error message ***
/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_mpifh.so.12(ompi_init_f+0x2c)[0x7f90810266ac]
[gpu-plx:12024] [16] ./a.out[0x401533]
[gpu-plx:12024] *** End of error message ***
[gpu-plx:12026] *** Process received signal ***
[gpu-plx:12026] Signal: Segmentation fault (11)
[gpu-plx:12026] Signal code: Address not mapped (1)
[gpu-plx:12026] Failing at address: 0x8
[gpu-plx:12025] *** Process received signal ***
[gpu-plx:12025] Signal: Segmentation fault (11)
[gpu-plx:12025] Signal code: Address not mapped (1)
[gpu-plx:12025] Failing at address: 0x8
[gpu-plx:12026] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f50b7ec6330]
[gpu-plx:12026] [ 1] /usr/lib64/libmlx5-rdmav2.so(+0x27e10)[0x7f50b3537e10]
[gpu-plx:12026] [ 2] [gpu-plx:12025] [ 0] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x147b40)[0x7f5
0b94c2b40]
[gpu-plx:12026] [ 3] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7fa10a15c330]
[gpu-plx:12025] [ 1] /usr/lib64/libmlx5-rdmav2.so(+0x27e10)[0x7fa1057fce10]
[gpu-plx:12025] [ 2] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x14700a)[0x7f50b94c200a]
[gpu-plx:12026] [ 4] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x147b40)[0x7fa10b758b40]
[gpu-plx:12025] [ 3] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x1462b2)[0x7f50b94c12b2]
[gpu-plx:12026] [ 5] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x14700a)[0x7fa10b75800a]
[gpu-plx:12025] [ 4] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x145e15)[0x7f50b94c0e15]
[gpu-plx:12026] [ 6] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x1462b2)[0x7fa10b7572b2]
[gpu-plx:12025] [ 5] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_btl_openib_connect_base_select_f
or_local_port+0xfa)[0x7f50b94bd17a]
[gpu-plx:12026] [ 7] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x145e15)[0x7fa10b756e15]
[gpu-plx:12025] [ 6] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x12fbc9)[0x7f50b94aabc9]
[gpu-plx:12026] [ 8] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_btl_openib_connect_base_select_f
or_local_port+0xfa)[0x7fa10b75317a]
[gpu-plx:12025] [ 7] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_btl_base_select+0x18f)[0x7f50b949
ef8f]
[gpu-plx:12026] [ 9] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x12fbc9)[0x7fa10b740bc9]
[gpu-plx:12025] [ 8] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_r2_component_init+0x23)[0x7f5
0b949e833]
[gpu-plx:12026] [10] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_btl_base_select+0x18f)[0x7fa10b73
4f8f]
[gpu-plx:12025] [ 9] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_base_init+0xa9)[0x7f50b949cb6
9]
[gpu-plx:12026] [11] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_r2_component_init+0x23)[0x7fa
10b734833]
[gpu-plx:12025] [10] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_bml_base_init+0xa9)[0x7fa10b732b6
9]
[gpu-plx:12025] [11] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x26454b)[0x7f50b95df54b]
[gpu-plx:12026] [12] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(+0x26454b)[0x7fa10b87554b]
[gpu-plx:12025] [12] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_pml_base_select+0x1f0)[0x7f50b95d
c700]
[gpu-plx:12026] [13] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_mpi_init+0x765)[0x7f50b941e365]
[gpu-plx:12026] [14] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(mca_pml_base_select+0x1f0)[0x7fa10b87
2700]
[gpu-plx:12025] [13] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(MPI_Init+0x152)[0x7f50b944a622]
[gpu-plx:12026] [15] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(ompi_mpi_init+0x765)[0x7fa10b6b4365]
[gpu-plx:12025] [14] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_mpifh.so.12(ompi_init_f+0x2c)[0x7f50b99a86a
c]
[gpu-plx:12026] [16] ./a.out[0x401533]
[gpu-plx:12026] *** End of error message ***
/home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi.so.12(MPI_Init+0x152)[0x7fa10b6e0622]
[gpu-plx:12025] [15] /home/vikram/usr/local/pgi/linux86-64/2016/mpi/openmpi-1.10.2/lib/libmpi_mpifh.so.12(ompi_init_f+0x2c)[0x7fa10bc3e6a
c]
[gpu-plx:12025] [16] ./a.out[0x401533]
[gpu-plx:12025] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 12024 on node gpu-plx exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
[gpu-plx:12021] 3 more processes have sent help message help-mpi-btl-openib.txt / default subnet prefix
[gpu-plx:12021] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages

If you think there’s a problem with the machine, how should I diagnose what is wrong?

Thanks.