Illegal address during kernel execution with OpenACC and WHERE clause only when using -O1 or -O0

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

Hi, I think I found yet another interesting bug. To me, it is a quite valid program. Not a priority though, since the workaround would be to expand the implicit loops.

program bug_opt
  implicit none
  
  real :: temp_max
  
  integer :: i, j
  integer, parameter :: rows = 50, cols = 50
  real :: matrix(rows, cols)
  real :: result(rows, cols)
  
  matrix = 1.
  
  !$acc data copy(matrix, result)
  
  temp_max = 1.

  !$acc kernels
  where(matrix(:,:) > 2.)
    matrix(:,:) = 0.
  end where
  !$acc end kernels

  !$acc parallel loop collapse(2)
  do j = 1, cols
    do i = 1, rows
      temp_max = abs(matrix(i, j))
      result(i, j) = temp_max * 2.
    end do
  end do
  !$acc end parallel loop
   
  !$acc end data
  
  print *, "Max value:", maxval(result)
  
end program bug_opt

Note: The latest nvfortran version available to me is 25.7.

$ nvfortran -O3 -acc=gpu main.f90 && ./a.out
 Max value:    2.000000

$ nvfortran -O2 -acc=gpu main.f90 && ./a.out
 Max value:    2.000000    

$ nvfortran -O1 -acc=gpu main.f90 && ./a.out
Failing in Thread:1
Accelerator Fatal Error: call to cuStreamSynchronize returned error 700 (CUDA_ERROR_ILLEGAL_ADDRESS): Illegal address during kernel execution
 File: /path/to/main.f90
 Function: bug_opt:1
 Line: 20

$ nvfortran -O0 -acc=gpu main.f90 && ./a.out
Failing in Thread:1
Accelerator Fatal Error: call to cuStreamSynchronize returned error 700 (CUDA_ERROR_ILLEGAL_ADDRESS): Illegal address during kernel execution
 File: /path/to/main.f90
 Function: bug_opt:1
 Line: 20

Comparing the logs from the compiler, I wonder which kernel on line 20 compiler is referring to, and/or why two kernels are generated for the WHERE clause.

image
image1652×538 94.4 KB

2

Hmm. I can definitely reproduce the weird behavior. Though, I see it more as a race condition - if I run 5 times, I get the right answer 2 or 3 times at “-O1”, but the illegal kernel message the other executions, whereas I don’t see the race at at all at “-O2” or above. It’s also interesting that “-O1” or below defaults to serial kernels for some reason.

Interesting. I opened a report with engineering for this strange behavior - #38238. I’ll get back to you when I hear from them. Thank you for sharing! Please feel free to share any other issues you run into - they all make the compiler better for everyone!