Compilation issues between Fortran with MPI and CUDA Fortran

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

Hello everyone,

I am trying to compile the following simple code. I have the main code which is in Fortran and uses MPI and on a different file I have one simple subroutine that uses the the cudamemcpy command.

!main.f90

program main2
 use MPI 
 implicit none
 integer:: rank, cpus,ierr

 call MPI_INIT(ierr)
 call MPI_COMM_RANK(MPI_COMM_WORLD,rank,ierr)
 call MPI_COMM_SIZE(MPI_COMM_WORLD,cpus,ierr)

 write(*,*) 'rank', rank,'of',cpus

 call MPI_FINALIZE(ierr)

end

!subroutines.cuf

subroutine cp(a,b)
               use cudafor
               implicit none
               integer:: b,istat
               integer, device:: a

               istat=cudaMemcpy(a,b,1)
        end subroutine cp

The compilation commands I am using are the following:
nvfortran -c subroutines.cuf
mpif90 -c main.f90
mpif90 -o test main.o subroutines.o -lgcc -L/opt/ohpc/pub/apps/cuda/11.0/lib64/ -I/opt/ohpc/pub/apps/cuda/11.0/include/ -lcudart -lcuda

but I am receiving the following error:
subroutines.o: In function cp_': /home/u11/vtsakag/cuda_test/fft/mpi_cuda/subroutines.cuf:7: undefined reference to cudamemcpyi4in_’

which I don’t understand because I am linking with the cuda libraries.

Do you have any ideas how I can fix that compilation issue?
Thank you in advance
VT

1 Like
2

Assuming your mpif90 is using nvfortran, try linking with just:

mpif90 -o test main.o subroutines.o -cuda

The “-cuda” indicates to the compiler driver that you’re using CUDA Fortran and will include all needed CUDA runtime libraries including the CUDA Fortran runtime (which is where your undefined reference is coming from).

Note the files with the “.cuf” extension imply the -cuda flag. But when linking just object files, the driver doesn’t know that you’re using CUDA Fortran unless “-cuda” is added.

Also the other CUDA libraries will be implicitly linked as well when -cuda is uses, so no need to add them to the link line again. Also “-lgcc” shouldn’t be needed either.

% mpif90 -c test.cuf
% mpif90 test.o
test.o: In function `cp_':
test.cuf:23: undefined reference to `cudamemcpyi4in_'
test.o:(.init_array+0x10): undefined reference to `Mcuda_compiled'
% mpif90 test.o -cuda
%
1 Like
3

Thank you Mat, that solved all my issues.

4

On the cluster I am working, it seems that the compilation stage is performed correctly but when trying to run there are some issues. More specifically I am getting the following errors:

Sorry!  You were supposed to get help about:
    ini file:file not found
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/openmpi/2020/192603-rel/share/openmpi/help-mpi-btl-openib.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no device params found
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/openmpi/2020/192603-rel/share/openmpi/help-mpi-btl-openib.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------
--------------------------------------------------------------------------
Sorry!  You were supposed to get help about:
    no cpcs for port
But I couldn't open the help file:
    /proj/nv/libraries/Linux_x86_64/openmpi/2020/192603-rel/share/openmpi/help-mpi-btl-openib-cpc-base.txt: No such file or directory.  Sorry!
--------------------------------------------------------------------------

The errors above appear only if I compile with the -cuda flag. When I tried to compile the main.f90 only with MPI (mpif90 -o test main.f90) everything worked perfectly fine. Could you please assist me on that issue?

Thank you in advance
VT