Nvfortran with acc: update is attempted on wrong gpu

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
1

Dear all,

I encountered a problem using update device statements (of variables that are in a module). It appears that the update is attempting to update the wrong gpu (1 higher than I want to).

I could reproduce the problem in a small code consisting of 3 parts.

main.f90:

program PHANTOM_SN
use SN_data
use gpu_transfers
implicit none

!$acc set device_num(0)

SN_set%N = 2
call setup_SN_quadrature()
call transfer_SN_set_to_gpu()

end program PHANTOM_SN

SN_data.f90:

module SN_data
implicit none

type SN_set_type
integer :: N
integer :: no_dirs
real, allocatable, dimension(:) :: weights
end type SN_set_type

type(SN_set_type), target :: SN_set

!$acc declare create(SN_set)
!$acc declare create(SN_set%no_dirs)
!$acc declare create(SN_set%weights)

contains

subroutine setup_SN_quadrature()
implicit none
integer :: N

N = SN_set%N
SN_set%no_dirs = 8
allocate(SN_set%weights(SN_set%no_dirs))
SN_set%weights = 1.0

end subroutine setup_SN_quadrature

end module SN_data

gpu_transfers.f90:

module gpu_transfers
implicit none

contains

subroutine transfer_SN_set_to_gpu()
use SN_data
implicit none

!$acc update device(SN_set)
!$acc update device(SN_set%no_dirs)
!$acc update device(SN_set%weights)
print *,‘Got to this point all right’

end subroutine transfer_SN_set_to_gpu

end module gpu_transfers

I compile with: nvfortran -C -cpp -g -acc -Minfo=accel SN_data.f90 gpu_transfers.f90 main.f90

The result I get is:

upload CUDA data file=/home/dlathouwers/BitBucket_codes/bug/src/gpu_transfers.f90 function=transfer_sn_set_to_gpu line=10 device=0 threadid=1 variable=sn_set bytes=160
upload CUDA data file=/home/dlathouwers/BitBucket_codes/bug/src/gpu_transfers.f90 function=transfer_sn_set_to_gpu line=11 device=0 threadid=1 variable=sn_set%no_dirs bytes=4
Present table dump for device[1]: NVIDIA Tesla GPU 0, compute capability 8.6, threadid=1
Hint: specify 0x800 bit in NV_ACC_DEBUG for verbose info.
host:0x406540 device:0x7fbf32a00000 size:160 presentcount:0+1 line:-1 name:_sn_data_16
FATAL ERROR: data in update device clause was not found on device 1: name=sn_set%weights(:)
file:/home/dlathouwers/BitBucket_codes/bug/src/gpu_transfers.f90 transfer_sn_set_to_gpu line:12

I am using version 24.3.

Can you reproduce the error?
Any ideas?

Thanks.
Dan.

2

Hi Dan,

The problem here is that “SN_set%weights” isn’t getting created on the device. Per the standard, allocatables in a “declare create” should create the device copy of the array when the array is allocated on the host. I’m not sure if it’s a compiler bug, or if this doesn’t apply to allocatables that are members of a derived type.

The work around is to put weights in to a unstructured data region. Since it’s parent is in a declare create, it’s globally accessible as well, so no need to add it to the “declare”.

module SN_data
implicit none

type SN_set_type
integer :: N
integer :: no_dirs
real, allocatable, dimension(:) :: weights
end type SN_set_type

type(SN_set_type), target :: SN_set

!$acc declare create(SN_set)

contains

subroutine setup_SN_quadrature()
implicit none
integer :: N


N = SN_set%N
SN_set%no_dirs = 8
allocate(SN_set%weights(SN_set%no_dirs))
SN_set%weights = 1.0
!$acc enter data copyin(SN_set%weights)

end subroutine setup_SN_quadrature

end module SN_data

% nvfortran -cpp -g -acc -Minfo=accel SN_data.f90 gpu_transfers.f90 main.f90 -gpu=deepcopy ; a.out
SN_data.f90:
setup_sn_quadrature:
     24, Generating enter data copyin(sn_set%weights(:))
gpu_transfers.f90:
transfer_sn_set_to_gpu:
     11, Generating update device(sn_set%no_dirs)
     12, Generating update device(sn_set%weights(:))
main.f90:
 Got to this point all right

-Mat

3

Thank you for the workaround.
My thoughts were that the standard would allow for what I was doing, but this works for me.

Note: gfortran also chokes on the declare create statements other than the ‘top one’.

Dan