Compile Nemo Ocean with PGI + ACC to CUDA

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi,

We are a group of students from Technical University of Sofia, Bulgaria. We are trying to run the NEMO Ocean project on CUDA by using the PGI Accelerator placing directives in the .F90 fortran sources. We have nested “for” loops which we want to send to the CUDA cores for processing.

  1. The directives we are using (a code snippet containing nested loops):

!$acc kernels loop
DO jk = 2, jpkm1
DO jj = 2, jpjm1
DO ji = fs_2, fs_jpim1 ! vector opt.
zwt(ji,jj,jk) = zwt(ji,jj,jk) + ah_wslp2(ji,jj,jk)
END DO
END DO
END DO


2. We have compiled the file containing the loops with command:
pgfortran -Mcuda -c file.F90.
The files which we include (USE in Fortran source) we compile to .mod with the following command: pgfortran -shared -fPIC include_file.F90

  1. At this point we have the “file.o” + “file.mod”.

  2. After step 2 and 3 we compile NEMO Ocean with the following configuration file by USING gfortran compiler:

%NCDF_INC -I/usr/local/netcdf/include
%NCDF_LIB -L/usr/local/netcdf/lib -lnetcdf -lnetcdff
%FC gfortran
%FCFLAGS -I/usr/include/mpich2 -L/usr/lib/mpich2/lib -lmpich -lmpl -fdefault-real-8 -O3 -funroll-all-loops -fcray-pointer
%FFLAGS %FCFLAGS
%LD gfortran
%LDFLAGS -I/usr/include/mpich2 -L/usr/lib/mpich2/lib -lmpich -lmpl -L/usr/local/netcdf/lib -lnetcdf -lnetcdff
%FPPFLAGS -I/usr/include/mpich2 -L/usr/lib/mpich2/lib -lmpich -lmpl -P -C -traditional
%AR ar
%ARFLAGS -rs
%MK gmake
%USER_INC %NCDF_INC
%USER_LIB %NCDF_LIB

Note: The command we use to compile NEMO is:
./makenemo -m gfortran_linux_modify -r ORCA2_LIM -n MY_ORCA2_LIM -j 4

  1. After the successful compilation of nemo a BUILD folder is generated. The folder contains all the “.o” and “.mod” files of NEMO.

  2. Then we replace the successfuly compiled files which contain the nested for loops with the .o and .mod files produced by pgfortran (from step 3).

  3. We comment the lines of NEMO Ocean’s Make file so they are not being generated when we try to compile with the “modified” (step 3) files.

  4. We compile assuring (step 7) that the new .o files will be used by pgfortran on new compilation. The following ERROR appears:

lib__fcm__nemo.a(trazdf.o): In function __trazdf_MOD_tra_zdf': trazdf.F90: (.text + 0x15f2): Undefined reference to __trazdf_imp_MOD_tra_zdf_imp
trazdf.F90: (.text + 0x1818): Undefined reference to `__trazdf_imp_MOD_tra_zdf_imp
Collect2: error: ld returned 1 exist status
fcm_internal load failed (256)

\

  • We would like to know if this is the right way (logically) to execute a separated fortran code (step 3) on the CUDA cores ?

  • Is our approach correct i.e.:
    Compile only the files containing Accelerator directives with pgfortran, or compile the entire NEMO Ocean project with pgfortran ?

Best Regards,
Technical University of Sofia,
Team of students

P.S. The university intends to buy a license for pgfortran if we manage to complete the project. We are using trial versions for now.

2

Compile only the files containing Accelerator directives with pgfortran, or compile the entire NEMO Ocean project with pgfortran ?

Since you’re using F90 modules and presumably allocatable arrays, you’ll need to use the same compiler for all files. Alternatively, you can write F77 code to bridge between the F90 codes, but this is often more trouble than it’s worth.

Is there a reason my you can’t compiler NEMO with PGI?

  • Mat
3

Hi, Mat!
Thank you for your answer! We do not have a solid reason to compile a single file with pgfortran, and than the entire product using gfortran. We thought this was the correct way to go.
So we listened to your advice and compiled the entire product using pgfortran.
Before that we compiled mpich with pg compilers.
Config file for NEMO Ocean this time around is:


%NCDF_INC -I/home/kalin/installed/netcdf/include
%NCDF_LIB -L/home/kalin/installed/netcdf/lib -lnetcdf -lnetcdff
%FC pgfortran
%FCFLAGS -I/home/kalin/installed/mpich/include -L/home/kalin/installed/mpich/lib -lmpich -lmpl -shared -fPIC
%FFLAGS %FCFLAGS
%LD pgfortran
%LDFLAGS -I/home/kalin/installed/mpich/include -L/home/kalin/installed/mpich/lib -lmpich -lmpl -L/home/kalin/installed/netcdf/lib -lnetcdf -lnetcdff
%FPPFLAGS -I/home/kalin/installed/mpich/include -L/home/kalin/installed/mpich/lib -lmpich -lmpl -P -C -traditional
%AR ar
%ARFLAGS -rs
%MK gmake
%USER_INC %NCDF_INC
%USER_LIB %NCDF_LIB

This is the compile time error it gives me:

PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 52)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_1d
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 67)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_2d
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 88)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_3d
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 105)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_2d_a
PGF90-S-0155-Could not resolve generic procedure mpp_sum (/home/kalin/monday_NEMOGCM/NEMOGCM/CONFIG/MY_ORCA2_LIM-pg-3/BLD/ppsrc/nemo/lib_fortran.f90: 127)
  0 inform,   0 warnings,   1 severes, 0 fatal for glob_sum_3d_a
fcm_internal compile failed (512)

Lines containing errors are as follows:

glob_sum_2d = SUM( ptab(:,:)*tmask_i(:,:) )
      IF( lk_mpp )   CALL mpp_sum( glob_sum_2d )

We declare these before hand:

 
INTERFACE glob_sum
      MODULE PROCEDURE glob_sum_2d, glob_sum_3d,glob_sum_2d_a, glob_sum_3d_a 
END INTERFACE
   

   
   FUNCTION glob_sum_2d( ptab ) 
      REAL(wp), INTENT(in), DIMENSION(:,:) ::   ptab          ! input 2D array
      REAL(wp)                             ::   glob_sum_2d   ! global masked sum
      glob_sum_2d = SUM( ptab(:,:)*tmask_i(:,:) )
      IF( lk_mpp )   CALL mpp_sum( glob_sum_2d )
      
   END FUNCTION glob_sum_2d

As we were going through the forum it became apparent that there is a bug in the compiler itself, but as it turned out, it was affecting older versions.
Should we put some additional arguments to the compiler line, in order to make it go?
The thing is when we use gfortran everything goes well, not so with pgfortran (giving the error I mentioned).
We shouldn‘t be changing any NEMO Ocean Code, should we? Any ideas as of what our next step should be?

4

PGF90-S-0155-Could not resolve generic procedure mpp_sum

This error indicates that the compiler can’t resolve which mpp_sum to use. Can you post the interface block from mpp_sum? Does one of the routines take a “REAL(wp)” data type as an argument?

  • Mat
5 
INTERFACE mpp_sum
      MODULE PROCEDURE mpp_sum_a2s, mpp_sum_as, mpp_sum_ai, mpp_sum_s, mpp_sum_i
 END INTERFACE
   
   
   SUBROUTINE mpp_sum_as( parr, kdim, kcom )      ! Dummy routine
      REAL   , DIMENSION(:) :: parr
      INTEGER               :: kdim
      INTEGER, OPTIONAL     :: kcom 
      WRITE(*,*) 'mpp_sum_as: You should not have seen this print! error?', kdim, parr(1), kcom
   END SUBROUTINE mpp_sum_as

   SUBROUTINE mpp_sum_a2s( parr, kdim, kcom )      ! Dummy routine
      REAL   , DIMENSION(:,:) :: parr
      INTEGER               :: kdim
      INTEGER, OPTIONAL     :: kcom 
      WRITE(*,*) 'mpp_sum_a2s: You should not have seen this print! error?', kdim, parr(1,1), kcom
   END SUBROUTINE mpp_sum_a2s

   SUBROUTINE mpp_sum_ai( karr, kdim, kcom )      ! Dummy routine
      INTEGER, DIMENSION(:) :: karr
      INTEGER               :: kdim
      INTEGER, OPTIONAL     :: kcom 
      WRITE(*,*) 'mpp_sum_ai: You should not have seen this print! error?', kdim, karr(1), kcom
   END SUBROUTINE mpp_sum_ai

   SUBROUTINE mpp_sum_s( psca, kcom )            ! Dummy routine
      REAL                  :: psca
      INTEGER, OPTIONAL     :: kcom 
      WRITE(*,*) 'mpp_sum_s: You should not have seen this print! error?', psca, kcom
   END SUBROUTINE mpp_sum_s

   SUBROUTINE mpp_sum_i( kint, kcom )            ! Dummy routine
      integer               :: kint
      INTEGER, OPTIONAL     :: kcom 
      WRITE(*,*) 'mpp_sum_i: You should not have seen this print! error?', kint, kcom
   END SUBROUTINE mpp_sum_i

We do not understand fortran so good, so what do you think about the snipped ?

6

We do not understand fortran so good, so what do you think about the snipped ?

A generic interface allows you to call a routine with a single name, but the compiler will resolve the call to one of the defined listed procedures based upon the arguments used. Here, all the mpp routines have at least two arguments but the actual call to mpp_sum uses only one. Hence, the compiler isn’t able to match the call with the actual routine.

Though, I find it odd that these routines are just dummy routines not expected to be called. Is this the correct interface?

  • Mat
7

We thank you for your time and effort to help us! We managed to overcome the errors so far and came across yet more of them. For the time being we are doing fine on our own, but we really hope we will be able to use your expertise in the future.