Thanks Mat. I am debugging with compute-sanitizer now. I tried all four tools. It seems like the only one pointing to issues is racecheck. I am consistently getting race conditions in kernels where I am using reduction operations (with or without specifying it in directive). For example:

```
!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
! Compute inviscid timestep
!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
!$acc parallel loop gang vector collapse(4) default(present)
do b = 1,nblocks
do k = 1,nkmax
do j = 1,njmax
do i = 1,nimax
if ((i .le. ni(b)) .and. (j .le. nj(b)) .and. (k .le. nk(b))) then
u_contra = prim_c(i,j,k,b,2)*idx_c(i,j,k,b) + &
& prim_c(i,j,k,b,3)*idy_c(i,j,k,b) + &
& prim_c(i,j,k,b,4)*idz_c(i,j,k,b)
v_contra = prim_c(i,j,k,b,2)*jdx_c(i,j,k,b) + &
& prim_c(i,j,k,b,3)*jdy_c(i,j,k,b) + &
& prim_c(i,j,k,b,4)*jdz_c(i,j,k,b)
w_contra = prim_c(i,j,k,b,2)*kdx_c(i,j,k,b) + &
& prim_c(i,j,k,b,3)*kdy_c(i,j,k,b) + &
& prim_c(i,j,k,b,4)*kdz_c(i,j,k,b)
sound_x = sqrt(gamma_ideal*prim_c(i,j,k,b,5)/prim_c(i,j,k,b,1)) * &
& sqrt(idx_c(i,j,k,b)**2.0d0+idy_c(i,j,k,b)**2.0d0+idz_c(i,j,k,b)**2.0d0)
sound_y = sqrt(gamma_ideal*prim_c(i,j,k,b,5)/prim_c(i,j,k,b,1)) * &
& sqrt(jdx_c(i,j,k,b)**2.0d0+jdy_c(i,j,k,b)**2.0d0+jdz_c(i,j,k,b)**2.0d0)
sound_z = sqrt(gamma_ideal*prim_c(i,j,k,b,5)/prim_c(i,j,k,b,1)) * &
& sqrt(kdx_c(i,j,k,b)**2.0d0+kdy_c(i,j,k,b)**2.0d0+kdz_c(i,j,k,b)**2.0d0)
x_velocity = abs(u_contra)+sound_x
y_velocity = abs(v_contra)+sound_y
z_velocity = abs(w_contra)+sound_z
dt_i = min(dt_i,1.0d0/x_velocity,1.0d0/y_velocity,1.0d0/z_velocity)
end if
end do
end do
end do
end do
!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
! Compute viscous timestep
!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
if (i_eqn .eq. 2) then
if (nondim .eq. 1) then
!$acc parallel loop gang vector collapse(4) reduction(min:dt_v) &
!$acc& default(present)
do b = 1,nblocks
do k = 1,nkmax
do j = 1,njmax
do i = 1,nimax
if ((i .le. ni(b)) .and. (j .le. nj(b)) .and. (k .le. nk(b))) then
norm1 = idx_c(i,j,k,b)**2.0d0+idy_c(i,j,k,b)**2.0d0+idz_c(i,j,k,b)**2.0d0
norm2 = jdx_c(i,j,k,b)**2.0d0+jdy_c(i,j,k,b)**2.0d0+jdz_c(i,j,k,b)**2.0d0
norm3 = kdx_c(i,j,k,b)**2.0d0+kdy_c(i,j,k,b)**2.0d0+kdz_c(i,j,k,b)**2.0d0
dt_v = min(dt_v,0.25d0*(Re/Ma)*(prim_c(i,j,k,b,1)/dyn_visc_c(i,j,k,b))*min(1.0d0/norm1,1.0d0/norm2,1.0d0/norm3))
end if
end do
end do
end do
end do
else if (nondim .eq. 2) then
!$acc parallel loop gang vector collapse(4) reduction(min:dt_v) &
!$acc& default(present)
do b = 1,nblocks
do k = 1,nkmax
do j = 1,njmax
do i = 1,nimax
if ((i .le. ni(b)) .and. (j .le. nj(b)) .and. (k .le. nk(b))) then
norm1 = idx_c(i,j,k,b)**2.0d0+idy_c(i,j,k,b)**2.0d0+idz_c(i,j,k,b)**2.0d0
norm2 = jdx_c(i,j,k,b)**2.0d0+jdy_c(i,j,k,b)**2.0d0+jdz_c(i,j,k,b)**2.0d0
norm3 = kdx_c(i,j,k,b)**2.0d0+kdy_c(i,j,k,b)**2.0d0+kdz_c(i,j,k,b)**2.0d0
dt_v = min(dt_v,0.25d0*Re*(prim_c(i,j,k,b,1)/dyn_visc_c(i,j,k,b))*min(1.0d0/norm1,1.0d0/norm2,1.0d0/norm3))
end if
end do
end do
end do
end do
end if
else if (nondim .eq. 3) then
!$acc parallel loop gang vector collapse(4) reduction(min:dt_v) &
!$acc& default(present)
do b = 1,nblocks
do k = 1,nkmax
do j = 1,njmax
do i = 1,nimax
if ((i .le. ni(b)) .and. (j .le. nj(b)) .and. (k .le. nk(b))) then
norm1 = idx_c(i,j,k,b)**2.0d0+idy_c(i,j,k,b)**2.0d0+idz_c(i,j,k,b)**2.0d0
norm2 = jdx_c(i,j,k,b)**2.0d0+jdy_c(i,j,k,b)**2.0d0+jdz_c(i,j,k,b)**2.0d0
norm3 = kdx_c(i,j,k,b)**2.0d0+kdy_c(i,j,k,b)**2.0d0+kdz_c(i,j,k,b)**2.0d0
dt_v = min(dt_v,0.25d0*Re*(prim_c(i,j,k,b,1)/dyn_visc_c(i,j,k,b))*min(1.0d0/norm1,1.0d0/norm2,1.0d0/norm3))
end if
end do
end do
end do
end do
else
dt_v = 1.0d20
end if
```

Anyways, these don’t seem to be causing issues.

In another module of my code, I am getting a memcheck error. The issue is that the line that compute-sanitizer points to is not very helpful. It is pointing to the line that closes the subroutine, which I can only assume means that the memcheck error is occurring somewhere in the subroutine.

Is there a way I can change this behavior? I tried compiling with debug flags as well to try and catch this but it does not seem to help.

Appreciate any advice.