Mpif90+pgf90 compilation problem

Hello everyone, as the title shows, fortran write MPI parallel program, part of the program I rewritten into cudaFortran program running on the GPU, .cuf, then compile, use mpif90 to compile mpifortran program, pgf90 compile to compile cudafortran program, There is no problem compiling, but the link fails. I want to know how to implement MPI+cudafortran hashing.
The procedure is as follows:

!--------------------------------------------------------
      program hello_world
      include 'mpif.h'
      integer ierr,num_procs,myid
      integer a(2,2),b(2),c(2)
      a=2
      b=1
      c=0
      call MPI_INIT(ierr)
      call MPI_COMM_RANK(MPI_COMM_WORLD,myid,ierr)
      call MPI_COMM_SIZE(MPI_COMM_WORLD,num_procs,ierr)
      print*,"Hello world!I'm process",myid,"out of",
     &       num_procs,"processes."
      call add(a,b,c)
      if(myid.eq.0)then
       print*,'c=',c(1),c(2)
      endif
      call MPI_FINALIZE(ierr)
      end program hello_world



----------------------------------------------------------------------------
      subroutine add(a,b,c)
      use cudafor
      use a_m
      implicit none
      integer a(2,2),b(2),c(2)
      integer,device::a_d(2,2),b_d(2),c_d(2)
      a_d=a
      b_d=b
      c_d=c
      call add1<<<1,2>>>(a_d,b_d,c_d)
      a=a_d
      b=b_d
      c=c_d
      end subroutine add
-------------------------------------------------------------------



     module a_m
      use cudafor
      contains
         attributes(global) subroutine add1(a_d,b_d,c_d)
         implicit none
         integer,device::a_d(2,2)
         integer,device::b_d(2)
         integer,device::c_d(2)
         integer ii
         ii=threadIdx%x
         if(ii.le.2)then
          c_d(ii)=a_d(ii,ii)*b_d(ii)
         endif
         return
         end subroutine add1
      end module a_m
--------------------------------------------------------------

makefile:
CUDA_INSTALL_PATH=/home/myfan/PGI/linux86-64/18.4
MPI_INSTALL_PATH=/opt/mpich2-1.4.1p1

PGF90=$(CUDA_INSTALL_PATH)/bin/pgf90
MPIF90=$(MPI_INSTALL_PATH)/bin/mpif90

LDFLAGS= -L$(CUDA_INSTALL_PATH)/lib
LIB=

FFILES=hello_world.f90
CUFILES=a_m.cuf add.cuf
OBJECTS=hello_world.o a_m.o add.o
EXENAME= test.x

ALL:
$(MPIF90) -c $(FFILES)
$(PGF90) -c $(CUFILES)
$(MPIF90) -o $(EXENAME) $(LDFLAGS) $(LIB) $(OBJECTS)
CLEAN:
rm -f *.o $(exename) *.mod

Compile without problems, link out the problem:

a_m.o: In function a_m_add1_': a_m.cuf:(.text+0x32): undefined reference to cudaSetupArgument’
a_m.cuf:(.text+0x47): undefined reference to cudaSetupArgument' a_m.cuf:(.text+0x5c): undefined reference to cudaSetupArgument’
a_m.cuf:(.text+0x68): undefined reference to cudaLaunch' a_m.o: In function __…acc_cuda_funcreg_constructor_1END’:
a_m.cuf:(.text+0xb8): undefined reference to __pgi_cuda_register_fat_binary' a_m.cuf:(.text+0x10c): undefined reference to cudaRegisterFunction’
a_m.o: In function .STATICS4': a_m.cuf:(.data+0x110): undefined reference to cudadevice
’
a_m.cuf:(.data+0x118): undefined reference to cudafor_' a_m.cuf:(.data+0x120): undefined reference to iso_c_binding
’
a_m.cuf:(.data+0x128): undefined reference to pgi_acc_common_' a_m.o:(.init+0xb): undefined reference to Mcuda_compiled’
add.o: In function add_': (.text+0x2a): undefined reference to pgf90_dev_auto_alloc04’
add.o: In function add_': (.text+0x3f): undefined reference to pgf90_dev_auto_alloc04’
add.o: In function add_': (.text+0x54): undefined reference to pgf90_dev_auto_alloc04’
add.o: In function add_': /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyin’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyin' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyin’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to __pgiLaunchKernel' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyout’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyout' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyout’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_auto_dealloc' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_auto_dealloc’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_auto_dealloc' add.o: In function .C1_379’:
add.cuf:(.data+0x28): undefined reference to cudafor_' add.cuf:(.data+0x38): undefined reference to iso_c_binding_’
add.cuf:(.data+0x40): undefined reference to pgi_acc_common_' add.cuf:(.data+0x48): undefined reference to f90_compiled’
add.o:(.init+0x1): undefined reference to `Mcuda_compiled’
make: *** [ALL] Error 1

It should be a problem with lib. I didn’t put the lib library in my makefile. But which library should be linked specifically, how to get it or not, I want to ask.
And Can someone give me a reference Makefile?

Hi M_yeah,

Which Fortran compiler is your mpif90 configured to use?

You’ll want to make sure that it’s using pgfortran (or pgf90) since different objects compiled by Fortran compilers aren’t inter-operable.

Second, you’ll want to add the flag “-Mcuda” to your link options (LDFLAGS). The compiler knows to implicitly add this when it sees it’s compiling a file with the “.cuf” suffix. But with objected files (.o), it doesn’t know you’re using CUDA Fortran. Using “-Mcuda” will have the compiler implicitly add the correct CUDA Fortran libraries to your link (the symbols you show are all found in the CUDA Fortran runtime libraries).

Hope this helps,
Mat

1 Like

I have successfully solved this problem. The specific approach is:
Compile the parallel program with the mpif90 that comes with the PGI compiler, then compile the .cu file with pgf90, then add -Mcuda at the link.
The specific makefile is as follows:

CUDA_INSTALL_PATH=/home/myfan/Desktop/PGI/linux86-64/18.4
MPI_INSTALL_PATH=/home/myfan/Desktop/PGI/linux86-64/18.4/mpi/openmpi-2.1.2
PGF90=$(CUDA_INSTALL_PATH)/bin/pgf90
MPIF90=$(MPI_INSTALL_PATH)/bin/mpif90
#LINK1= /home/myfan/Desktop/PGI/linux86-64/18.4/lib/libcudafor.a
CUDALIB=-L/home/myfan/Desktop/PGI/linux86-64/2018/cuda/8.0/lib64 -lcudart
FFILES=hello_world.f90
CUFILES=a_m.cuf add.cuf
OBJECTS=hello_world.o a_m.o add.o
EXENAME= test.x

ALL:
        $(MPIF90) -c $(FFILES)
        $(PGF90)  $(CUDALIB) $(CUDAINC) -c $(CUFILES)
        $(MPIF90) $(OBJECTS) -Mcuda $(CUDALIB) $(CUDAINC) -o test.x
.PHONY clean:
clean:
        -rm -rf *.o
        -rm -rf *.mod
        -rm -rf test.x

Thanks a lot~~

I too had this problem. I could solve with your answer. Thank you. I was looking for latest documentation on compilation, linking procedure with options