Do concurrents in openacc parallel regions not parallelising properly when compiling with -stdpar=gpu

Accelerated Computing HPC Compilers Legacy PGI Compilers
1

Hi,

This is a follow-up query on using openacc directives with multi-index do concurrent loops (Do concurrent + openacc pragmas parallelism behavior).

I noticed that if I have a do concurrent in an openacc parallel region and compiling with both -acc=gpu and -stdpar=gpu, the compiler doesn’t properly parallelise the inner loop despite having collapse(2). E.g. see below MRE.

! loop_example.f90
program loop_example
  implicit none

  integer, parameter :: ni = 10, nj = 10, nk = 5
  real :: a(ni, nj, nk), b(ni, nj)
  integer :: i, j, k

  ! Initialize
  a = 1.0

  ! Regular nested do loops: outer k reduction into 2D array
  b = 0.0
  !$acc parallel
  !$loop seq
  do k = 1, nk
    !$acc loop collapse(2)
    do j = 1, nj
      do i = 1, ni
        b(i, j) = b(i, j) + a(i, j, k) * k
      enddo
    enddo
  enddo
  !$acc end parallel

  ! Do concurrent: outer k reduction into 2D array
  b = 0.0
  !$acc parallel
  !$acc loop seq
  do k = 1, nk
    !$acc loop collapse(2)
    do concurrent (j = 1:nj, i = 1:ni)
      b(i, j) = b(i, j) + a(i, j, k) * k
    enddo
  enddo
  !$acc end parallel

end program loop_example

compiling with nvfortran -acc=gpu -Minfo=all loop_example.f90 -stdpar=gpu I get Minfo output:

loop_example:
     13, Generating implicit firstprivate(k)
         Generating NVIDIA GPU code
         15, !$acc loop seq
         17, !$acc loop gang, vector(96) collapse(2) ! blockidx%x threadidx%x
         18,   ! blockidx%x threadidx%x collapsed
     27, Generating NVIDIA GPU code
         29, !$acc loop seq
         31, !$acc loop seq

You can see that loop on 31 doesn’t get parallelized at all and the non do concurrent loop does. When compiling without -stdpar=gpu:

     27, Generating NVIDIA GPU code
         29, !$acc loop seq
         31,   ! blockidx%x threadidx%x collapsed
             !$acc loop gang, vector(96) collapse(2) ! blockidx%x threadidx%x

the loop is parallelised properly.

This is with nvhpc 26.1.

2

I’m not sure why you switched from that sub-forum (nvhpc/nvfortran) to this one (GPU-accelerated libraries). I think you’re going to get better help posting on the nvhpc forum. OpenACC support is part of nvhpc.

3

My mistake! I’ve changed the sub forum to the correct one

4

Hi edoy,

You’ll want to use “kernels” here instead of “parallel”. “parallel” only can be applied to explicit loops. For implicit loops like array syntax or do concurrent, “kernels” allows the compiler to “discover” the parallelism.

With “kernels”:

% nvfortran -c acc -stdpar -Minfo test.F90
test.F90:
loop_example:
     10, Memory set idiom, array assignment replaced by call to pgf90_mset4
     13, Memory zero idiom, array assignment replaced by call to pgf90_mzero4
     14, Generating implicit firstprivate(k)
         Generating NVIDIA GPU code
         16, !$acc loop seq
         18, !$acc loop gang, vector(96) collapse(2) ! blockidx%x threadidx%x
             Interchanging generated strip mine loop outwards
         19,   ! blockidx%x threadidx%x collapsed
             Interchanging generated strip mine loop outwards
     14, Generating implicit copyin(a(:,:,:)) [if not already present]
         Generating implicit copy(b(:,:)) [if not already present]
     16, Loop carried dependence due to exposed use of b(:,:) prevents parallelization
     27, Memory zero idiom, array assignment replaced by call to pgf90_mzero4
     28, Generating implicit copy(b(:,:)) [if not already present]
         Generating implicit copyin(a(:,:,:)) [if not already present]
     30, Loop carried dependence due to exposed use of b(:,:) prevents parallelization
     32, Loop is parallelizable
         Generating NVIDIA GPU code
         30, !$acc loop seq
         32,   ! blockidx%x threadidx%x auto-collapsed
             !$acc loop gang, vector(128) collapse(2) ! blockidx%x threadidx%x

If possible, I’d also recommend you interchange the loops so the “k” loop is innermost.

! loop_example.f90
program loop_example
  implicit none

  integer, parameter :: ni = 10, nj = 10, nk = 5
  real :: a(ni, nj, nk), b(ni, nj)
  integer :: i, j, k

  ! Initialize
  a = 1.0

  ! Regular nested do loops: outer k reduction into 2D array
  b = 0.0
  !$acc parallel
  !$acc loop collapse(2)
  do j = 1, nj
     do i = 1, ni
       do k = 1, nk
        b(i, j) = b(i, j) + a(i, j, k) * k
      enddo
    enddo
  enddo
  !$acc end parallel

  ! Do concurrent: outer k reduction into 2D array
  b = 0.0
  !$acc kernels
  !$acc loop collapse(2)
  do concurrent (j = 1:nj, i = 1:ni)
     do k = 1, nk
       b(i, j) = b(i, j) + a(i, j, k) * k
     enddo
  enddo
  !$acc end kernels

end program loop_example

% nvfortran -c acc -stdpar -Minfo test.F90
test.F90:
loop_example:
     10, Memory set idiom, array assignment replaced by call to pgf90_mset4
     13, Memory zero idiom, array assignment replaced by call to pgf90_mzero4
     14, Generating NVIDIA GPU code
         16, !$acc loop gang, vector(96) collapse(2) ! blockidx%x threadidx%x
         17,   ! blockidx%x threadidx%x collapsed
         18, !$acc loop seq
     14, Generating implicit copyin(a(:,:,:)) [if not already present]
         Generating implicit copy(b(:,:)) [if not already present]
     17, Generating implicit firstprivate(k)
     18, Complex loop carried dependence of b prevents parallelization
         Loop carried dependence of b prevents parallelization
         Loop carried backward dependence of b prevents vectorization
     26, Memory zero idiom, array assignment replaced by call to pgf90_mzero4
     27, Generating implicit copy(b(:,:)) [if not already present]
         Generating implicit copyin(a(:,:,:)) [if not already present]
     29, Loop is parallelizable
     30, Complex loop carried dependence of b prevents parallelization
         Loop carried dependence of b prevents parallelization
         Loop carried backward dependence of b prevents vectorization
         Inner sequential loop scheduled on accelerator
         Generating NVIDIA GPU code
         29, !$acc loop gang, vector(128) collapse(2) ! blockidx%x threadidx%x
               ! blockidx%x threadidx%x auto-collapsed
         30, !$acc loop seq

Hope this helps,
Mat

5

Thanks Mat. I wanted to avoid putting k inside as it hurts the CPU performance (array accesses are strided rather than sequential) - we’re trying to keep a single codebase etc. and don’t want to hurt existing CPU production runs while we’re porting.

Would there be a way to get the parallelism discovered in the do concurrent with k outside?

6

oh nvm I missed this part. as you said, kernels instead of parallel works - i should’ve tried that first.

7

The first example output was with “k” on the outer loop, i.e. I just replaced “parallel” with “kernels” and the do concurrent was paralelized.