Error running simple MPI code with OpenMPI from PGI installe

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi PGI Users!

When I try to run a simple code using the OpenMPI bundled with PGI installer (18.4 community edition) without mpiexec I got the following error:

./test_F08 
[bandera:01209] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file ../../../../../orte/mca/ess/singleton/ess_singleton_module.c at line 458
[bandera:01209] [[INVALID],INVALID] ORTE_ERROR_LOG: A system-required executable either could not be found or was not executable by this user in file ../../../../../orte/mca/ess/singleton/ess_singleton_module.c at line 166
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    orte_init:startup:internal-failure
But I couldn't open the help file:
    /proj/pgi/linux86-64/2018/mpi/openmpi-2.1.2/share/openmpi/help-orte-runtime: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    mpi_init:startup:internal-failure
But I couldn't open the help file:
    /proj/pgi/linux86-64/2018/mpi/openmpi-2.1.2/share/openmpi/help-mpi-runtime.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
*** An error occurred in MPI_Init
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
***    and potentially your MPI job)
[bandera:1209] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!

I do not get this error using other compilers (gcc, Intel) and OpenMPI.

When I use mpiexec the program runs ok.

My workflow (Cmake) expects that simple MPI programs can be executed without mpiexec in order to properly set up the appropriate libraries.

Here is an example code, it works with intel + OpenMPI with and without mpiexec, but it fails with PGI and the bundled OpenMPI without mpiexec:

program main                                                   
    use mpi_f08                                                
    implicit none                                              
    integer :: rank, size, len                                 
    character(len=MPI_MAX_LIBRARY_VERSION_STRING) :: version   
                                                               
    call MPI_INIT()                                            
    call MPI_COMM_RANK(MPI_COMM_WORLD, rank)                   
    call MPI_COMM_SIZE(MPI_COMM_WORLD, size)                   
    call MPI_GET_LIBRARY_VERSION(version, len)                 
                                                               
    print *, "rank:", rank                                     
    print *, "size:",size                                      
    print *, "version: "//version                              
    print *, ' No Errors'                                      
                                                               
    call MPI_FINALIZE()                                        
end

Is there a way to run simple MPI programs with PGI without using mpiexec?

Thank you very much for your help,

Best regards,

Hector

2

Greetings Hector,

I believe you are seeing this error due to the fact that the PGI installer allows users to install PGI compilers and tools anywhere they choose on their filesystems. One drawback to this is that we have no way of knowing at the time we build Open MPI here where the users are going to install the PGI compilers and tools. As you have seen below, /proj/pgi is an internal directory where we stage Open MPI and other packages prior to creating the tarballs for installation. Open MPI normally builds some default paths into its machinery to help it locate components, and so it builds in the /proj/pgi paths when we build it here. We do not really have a way to change that. If we were to force everyone to install Open MPI, say, somewhere under /opt/pgi, then you probably would not see this issue.

However, I believe you can work around this issue by setting the $OPAL_PREFIX environment variable to the path where our Open MPI build ended up getting installed onto your system. For example, if you installed the PGI compilers and tools in /home/hector/pgi, then the correct setting for $OPAL_PREFIX should be /home/hector/pgi/linux86-64/2018/mpi/openmpi-2.1.2. Our wrapper around mpirun/mpiexec sets this variable for you, which is why you do not see this issue when invoking mpirun or mpiexec.

Hope this helps. Let us know if you have any further issues with this.

Regards,

+chris