I encountered an error message of “an illegal memory access” inside a kernel subroutine. I was trying to define a variable on the register memory in order to mitigate the bottleneck of device memory access. The subroutine looks as the following (with some omission):
type photon_pack integer(lg) :: seed integer(sm) :: i, j real(pk) :: energy, x, y, vx, vy end type photon_pack type energy_grid real(pk) :: energy, xgrid, ygrid, dx, dy end type energy_grid attributes(global) subroutine shoot_photon( base_seed, grid, dt, t_end, t_damp ) use cudafor implicit none type(photon_pack) :: ptmp type(energy_grid), device :: grid(:,:) real(pk), device :: dt, t_end, t_damp real(pk) :: vx, vy, vmag, etmp, etrans, tratio integer(lg), device :: base_seed integer(lg) :: blockid, lph, nph, seed integer(sm) :: cur_corr(2), aft_corr(2), istat, itime, ntime, gridsize(2), n, m logical :: move_forw, move_back blockid = BlockIdx%x + GridDim%x * (BlockIdx%y + GridDim%y * (BlockIdx%z - 1) - 1) lph = ThreadIdx%x + BlockDim%x * (blockid - 1) seed = base_seed seed = seed + lph gridsize = size(grid) n = gridsize(1) m = gridsize(2) ntime = t_end/dt tratio = dt/t_damp call init_photon_dev(seed, n, m, ptmp) aft_corr(1) = ptmp%i; aft_corr(2) = ptmp%j do itime = 0, ntime - 1 vx = 1.0; vy = 0.0 ! randomly picked vmag = sqrt(vx*vx + vy*vy) vx = vx * ptmp%energy / vmag vy = vy * ptmp%energy / vmag ptmp%x = ptmp%x + dt*vx ptmp%y = ptmp%y + dt*vy cur_corr = aft_corr ...... [omission] aft_corr(1) = ptmp%i; aft_corr(2) = ptmp%j istat = 0 etmp = ptmp%energy * (1.0_pk - exp(-tratio)) ! ! Illegal memory access if Atomic add is used... ! if (cur_corr(1) == aft_corr(1) .and. cur_corr(2) == aft_corr(2)) then istat = atomicadd(grid(cur_corr(1),cur_corr(2))%energy, etmp) else etmp = etmp / 2.0_pk istat = atomicadd(grid(cur_corr(1),cur_corr(2))%energy, etmp) istat = atomicadd(grid(aft_corr(1),aft_corr(2))%energy, etmp) end if ! ! Illegal memory access if any of the elements of ptmp is modified? ! etrans = ptmp%energy - etmp ptmp%energy = etrans end do return end subroutine shoot_photon
The purpose is to add the variables “etmp” into a device-memory variable “grid%energy” using atomic add. This caused the code to fail with an error of an illegal memory access. If I commented out the atomic part, the code could run (without doing anything); but if I add another two lines, the code also failed at the second line (“ptmp%energy = etrans”).
It seems to me that the subroutine failed when
- attempting to add variables from register memory to device memory (?), or
- attempting to copy values back to device memory.
Could you please let me know if any more options can be used to understand the origin of the error? Please also let me know if I can send in the whole code (a short one in one file) in case that helps.
Thank you very much for your help!