Mpif90+pgf90 compilation problem

M_yeah · April 29, 2019, 1:11am

Hello everyone, as the title shows, fortran write MPI parallel program, part of the program I rewritten into cudaFortran program running on the GPU, .cuf, then compile, use mpif90 to compile mpifortran program, pgf90 compile to compile cudafortran program, There is no problem compiling, but the link fails. I want to know how to implement MPI+cudafortran hashing.
The procedure is as follows:

!--------------------------------------------------------
      program hello_world
      include 'mpif.h'
      integer ierr,num_procs,myid
      integer a(2,2),b(2),c(2)
      a=2
      b=1
      c=0
      call MPI_INIT(ierr)
      call MPI_COMM_RANK(MPI_COMM_WORLD,myid,ierr)
      call MPI_COMM_SIZE(MPI_COMM_WORLD,num_procs,ierr)
      print*,"Hello world!I'm process",myid,"out of",
     &       num_procs,"processes."
      call add(a,b,c)
      if(myid.eq.0)then
       print*,'c=',c(1),c(2)
      endif
      call MPI_FINALIZE(ierr)
      end program hello_world

----------------------------------------------------------------------------
      subroutine add(a,b,c)
      use cudafor
      use a_m
      implicit none
      integer a(2,2),b(2),c(2)
      integer,device::a_d(2,2),b_d(2),c_d(2)
      a_d=a
      b_d=b
      c_d=c
      call add1<<<1,2>>>(a_d,b_d,c_d)
      a=a_d
      b=b_d
      c=c_d
      end subroutine add
-------------------------------------------------------------------

     module a_m
      use cudafor
      contains
         attributes(global) subroutine add1(a_d,b_d,c_d)
         implicit none
         integer,device::a_d(2,2)
         integer,device::b_d(2)
         integer,device::c_d(2)
         integer ii
         ii=threadIdx%x
         if(ii.le.2)then
          c_d(ii)=a_d(ii,ii)*b_d(ii)
         endif
         return
         end subroutine add1
      end module a_m
--------------------------------------------------------------

makefile:
CUDA_INSTALL_PATH=/home/myfan/PGI/linux86-64/18.4
MPI_INSTALL_PATH=/opt/mpich2-1.4.1p1

PGF90=$(CUDA_INSTALL_PATH)/bin/pgf90
MPIF90=$(MPI_INSTALL_PATH)/bin/mpif90

LDFLAGS= -L$(CUDA_INSTALL_PATH)/lib
LIB=

FFILES=hello_world.f90
CUFILES=a_m.cuf add.cuf
OBJECTS=hello_world.o a_m.o add.o
EXENAME= test.x

ALL:
$(MPIF90) -c $(FFILES)
$(PGF90) -c $(CUFILES)
$(MPIF90) -o $(EXENAME) $(LDFLAGS) $(LIB) $(OBJECTS)
CLEAN:
rm -f .o $(exename) .mod

Compile without problems, link out the problem:

a_m.o: In function a_m_add1_': a_m.cuf:(.text+0x32): undefined reference to cudaSetupArgument’
a_m.cuf:(.text+0x47): undefined reference to cudaSetupArgument' a_m.cuf:(.text+0x5c): undefined reference to cudaSetupArgument’
a_m.cuf:(.text+0x68): undefined reference to cudaLaunch' a_m.o: In function __…acc_cuda_funcreg_constructor_1END’:
a_m.cuf:(.text+0xb8): undefined reference to __pgi_cuda_register_fat_binary' a_m.cuf:(.text+0x10c): undefined reference to cudaRegisterFunction’
a_m.o: In function .STATICS4': a_m.cuf:(.data+0x110): undefined reference to cudadevice’
a_m.cuf:(.data+0x118): undefined reference to cudafor_' a_m.cuf:(.data+0x120): undefined reference to iso_c_binding’
a_m.cuf:(.data+0x128): undefined reference to pgi_acc_common_' a_m.o:(.init+0xb): undefined reference to Mcuda_compiled’
add.o: In function add_': (.text+0x2a): undefined reference to pgf90_dev_auto_alloc04’
add.o: In function add_': (.text+0x3f): undefined reference to pgf90_dev_auto_alloc04’
add.o: In function add_': (.text+0x54): undefined reference to pgf90_dev_auto_alloc04’
add.o: In function add_': /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyin’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyin' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyin’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to __pgiLaunchKernel' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyout’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyout' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_copyout’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_auto_dealloc' /home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_auto_dealloc’
/home/myfan/test/mpi/./add.cuf:7: undefined reference to pgf90_dev_auto_dealloc' add.o: In function .C1_379’:
add.cuf:(.data+0x28): undefined reference to cudafor_' add.cuf:(.data+0x38): undefined reference to iso_c_binding_’
add.cuf:(.data+0x40): undefined reference to pgi_acc_common_' add.cuf:(.data+0x48): undefined reference to f90_compiled’
add.o:(.init+0x1): undefined reference to `Mcuda_compiled’
make: *** [ALL] Error 1

It should be a problem with lib. I didn’t put the lib library in my makefile. But which library should be linked specifically, how to get it or not, I want to ask.
And Can someone give me a reference Makefile?

MatColgrove · April 29, 2019, 2:34pm

Hi M_yeah,

Which Fortran compiler is your mpif90 configured to use?

You’ll want to make sure that it’s using pgfortran (or pgf90) since different objects compiled by Fortran compilers aren’t inter-operable.

Second, you’ll want to add the flag “-Mcuda” to your link options (LDFLAGS). The compiler knows to implicitly add this when it sees it’s compiling a file with the “.cuf” suffix. But with objected files (.o), it doesn’t know you’re using CUDA Fortran. Using “-Mcuda” will have the compiler implicitly add the correct CUDA Fortran libraries to your link (the symbols you show are all found in the CUDA Fortran runtime libraries).

Hope this helps,
Mat

M_yeah · May 2, 2019, 2:09am

I have successfully solved this problem. The specific approach is:
Compile the parallel program with the mpif90 that comes with the PGI compiler, then compile the .cu file with pgf90, then add -Mcuda at the link.
The specific makefile is as follows:

CUDA_INSTALL_PATH=/home/myfan/Desktop/PGI/linux86-64/18.4
MPI_INSTALL_PATH=/home/myfan/Desktop/PGI/linux86-64/18.4/mpi/openmpi-2.1.2
PGF90=$(CUDA_INSTALL_PATH)/bin/pgf90
MPIF90=$(MPI_INSTALL_PATH)/bin/mpif90
#LINK1= /home/myfan/Desktop/PGI/linux86-64/18.4/lib/libcudafor.a
CUDALIB=-L/home/myfan/Desktop/PGI/linux86-64/2018/cuda/8.0/lib64 -lcudart
FFILES=hello_world.f90
CUFILES=a_m.cuf add.cuf
OBJECTS=hello_world.o a_m.o add.o
EXENAME= test.x

ALL:
        $(MPIF90) -c $(FFILES)
        $(PGF90)  $(CUDALIB) $(CUDAINC) -c $(CUFILES)
        $(MPIF90) $(OBJECTS) -Mcuda $(CUDALIB) $(CUDAINC) -o test.x
.PHONY clean:
clean:
        -rm -rf *.o
        -rm -rf *.mod
        -rm -rf test.x

Thanks a lot~~

sambath.narayanan · July 8, 2020, 3:03pm

I too had this problem. I could solve with your answer. Thank you. I was looking for latest documentation on compilation, linking procedure with options

bsweetma · February 28, 2023, 8:07pm

Hello, very helpful thread. I tried compiling your code in the manner you described. I ran into issue with:
/apps/pgi/linux86-64/18.7/lib/libcudafor91.so: undefined reference to cudaLaunchCooperativeKernel

Full output from ‘make’:
/apps/mpi/openmpi/openmpi-4.0.2/bin/mpif90 -c hello_world.f90
/apps/pgi/linux86-64/18.7/bin/pgf90 -L/apps/pgi/linux86-64/2018/cuda/8.0/lib64 -lcudart -c a_m.cuf add.cuf
a_m.cuf:
add.cuf:
/apps/mpi/openmpi/openmpi-4.0.2/bin/mpif90 hello_world.o a_m.o add.o -Mcuda -L/apps/pgi/linux86-64/2018/cuda/8.0/lib64 -lcudart -o test.x
/apps/pgi/linux86-64/18.7/lib/libcudafor91.so: undefined reference to `cudaLaunchCooperativeKernel’
pgacclnk: child process exit status 1: /usr/bin/ld
make: *** [Makefile:16: ALL] Error 2

Any help/guidance to get through this error would be helpful.
Thank you.

MatColgrove · February 28, 2023, 9:56pm

If I remember correctly, “cudaLaunchCooperativeKernel” wasn’t added until CUDA 9.0, but you’re explicitly linking with CUDA 8.0.

You should try removing “-L/apps/pgi/linux86-64/2018/cuda/8.0/lib64 -lcudart” and let the compiler link against the correct CUDA runtime library.

If you do need to override the default CUDA version used (this is determined by the CUDA driver installed on the compilation system and likely 9.0 in this case), then use the flag “-Mcuda=cuda8.0”.

bsweetma · March 1, 2023, 10:43am

Hi MatColgrove. Thank you for your advice! I simply commented this line in the Makefile:
#CUDALIB=-L/apps/pgi/linux86-64/2018/cuda/8.0/lib64 -lcudart

And voila, it worked!
I also tried updating the library reference to 9.0 and it also worked [to confirm that “you were remembering correctly” ;-) ]
CUDALIB=-L/apps/pgi/linux86-64/2018/cuda/9.0/lib64 -lcudart

Thanks again for such a quick reply!

Topic		Replies	Views
Mpif90 and nfvortran compatibility issues nvc, nvc++ and nvfortran	1	2901	May 3, 2021
MPICH linking failing Legacy PGI Compilers	12	12601	October 25, 2013
Compilation issues between Fortran with MPI and CUDA Fortran nvc, nvc++ and nvfortran	3	1548	March 2, 2021
runtime error when use mpi, cuda fortran and CULA together Legacy PGI Compilers	2	3876	July 21, 2011
PGI worksation and MPICH2 Legacy PGI Compilers	4	6814	June 21, 2011
mpif90+cudaC Mixed compilation problems Legacy PGI Compilers	4	1076	November 22, 2019
Cuda library link error Legacy PGI Compilers	2	5836	March 30, 2017
Compiling fortran project using PVF 13.9 and Cuda CC5.5 Legacy PGI Compilers	3	8065	November 27, 2013
compiling with mpif90 and pgf90 Legacy PGI Compilers	4	4641	February 19, 2010
f90 linking problem Legacy PGI Compilers	4	16272	February 23, 2005

Mpif90+pgf90 compilation problem

ALL: $(MPIF90) -c $(FFILES) $(PGF90) -c $(CUFILES) $(MPIF90) -o $(EXENAME) $(LDFLAGS) $(LIB) $(OBJECTS) CLEAN: rm -f *.o $(exename) *.mod

Related topics

ALL:
$(MPIF90) -c $(FFILES)
$(PGF90) -c $(CUFILES)
$(MPIF90) -o $(EXENAME) $(LDFLAGS) $(LIB) $(OBJECTS)
CLEAN:
rm -f .o $(exename) .mod