Issue with !$acc declare create


In a program I am writing I am using mostly global variables which has been working great with !$acc directives. However, I have recently added a module to the code that declares and allocates/deallocates it’s own set of variables. This separate module still uses the global variables module. I have been getting the following error when running the code:

NVFORTRAN-W-1054-Module variables used in acc routine need to be in acc declare create() - y_wm_n$dev

This error still occurs after writing !$acc declare create(y_wm_n). I am not sure how to proceed since I am using the declare create directive. For reference, here is how the variables is declared and created:

 integer, parameter :: dp = kind(1.0d0)
 integer :: n_wm  = 25
!$acc declare create(n_wm) 
real(dp), dimension(:), allocatable :: y_wm_n
!$acc declare create(y_wm_n)

The (dp) attribute is set in a separate module but works globally for all variables.

I appreciate any and all help!

I feel I was not really clear on this so I will provide more information. I am writing a computational fluid dynamics solver. I have accelerated one of the subroutines using OpenACC. In this subroutine, I am attempting to call another subroutine, such that the structure of this subroutine/call is:

module fluxes
use equilibriumWallModel

subroutine viscous_fluxes

!$acc parallel loop gang vector collapse(3) default(present)
do k = 0,nk
    do j = 0,nj
        do i = 0,ni
            call ewm(i,j,k)
        end do
    end do
end do

end subroutine viscous_fluxes

end module fluxes

The subroutine viscous_fluxes is inside of a module called fluxes (as shown above) and uses the equilibriumWallModel module so that it can call the needed subroutine (ewm is part of equilibriumWallModel). Now, besides for using global variables that the entire solver uses, the equilibriumWallModel module has it’s own variables and associated memory allocation subroutines. The global variables that equilibriumWallModel uses as well as the ones only associated with equilibriumWallModel are the ones that the compiler says I must use acc declare create for. I am confused about this as well since I am using the managed flag when compiling the code.

Anyways, all subroutines inside the equilibriumWallModel module have the !$acc routine directive at the top of the subroutine. None need to be parallelized, all can be done sequentially. However, there seems to be an issue with calling ewm from inside of the collapsed loop in subroutine viscous_fluxes. I tried making all equilibriumWallModel module variables global instead and also using !$acc declare create for all of them. This at least had the code run but would crash immediately. It seems that the output values of the ewm subroutine have a value of zero, which is unlikely/nonphysical - so there may be some memory transfer issue, I really don’t know.

If there is any other information needed I would be more than happy to provide it. I appreciate any help.

Hi Nathan,

Can you provide a minimal reproducing example or, if not, show how “ewm” is defined?

Normally I’d expect the warning message to include something like “arr$sd”, which is the array’s section descriptor, rather than “arr$dev”, which is the device pointer. Hence there’s something unique about your program that’s causing the difference but I’m not sure what. A reproducing example will help.


Hi Mat,

Thanks for your response.

  ! subroutine ewm                                                                                                                                                                                          
  ! Calls needed subroutines for equilibrium wall model                                                                                                                                                     
  subroutine ewm(iind,kind,bind)                                                                                                                                                                            
    !$acc routine                                                                                                                                                                                           
    implicit none                                                                                                                                                                                           
    integer :: i,j,k,b                                                                                                                                                                                      
    integer :: iind,kind,bind                                                                                                                                                                               
    if (initial3 .eq. .true.) then                                                                                                                                                                          
       initial3 = .false.                                                                                                                                                                                   
       call ewmGrid                                                                                                                                                                                         
       call ewmInitialConditions                                                                                                                                                                            
    end if                                                                                                                                                                                                  
    call ewmAlgorithm(iind,kind,bind)                                                                                                                                                                       
  end subroutine ewm                                                                                                                                                                                        

This is essentially a subroutine which calls all the needed subroutines for something. So for example, here is ewmGrid:

  ! subroutine ewmGrid                                                                                                                                                                                      
  ! Creates grid for equilibrium wall model                                                                                                                                                                 
  subroutine ewmGrid                                                                                                                                                                                        
    !$acc routine                                                                                                                                                                                           
    implicit none                                                                                                                                                                                           
    integer  :: j,l                                                                                                                                                                                         
    real(dp) :: beta                                                                                                                                                                                        
    beta = 0.0001d0                                                                                                                                                                                         
    do l = 1,100000                                                                                                                                                                                         
       ! Face locations                                                                                                                                                                                     
       do j = -1,n_wm                                                                                                                                                                                       
          y_wm_n(j) = dy_w*(r_stretch**j-1.0d0)/(r_stretch-1.0d0)                                                                                                                                           
       end do                                                                                                                                                                                               
       ! Centroid locations                                                                                                                                                                                 
       do j = 0,n_wm                                                                                                                                                                                        
          y_wm_c(j) = 0.5d0*(y_wm_n(j)+y_wm_n(j-1))                                                                                                                                                         
       end do                                                                                                                                                                                               
       ! Guarantee that top face is at h_wm                                                                                                                                                                 
       if (abs(y_wm_n(n_wm)-y_c(1,gp_wm,1,1)) .le. 1.0d-10) then                                                                                                                                            
          dy_w = dy_w + beta*(-y_wm_n(n_wm)+y_c(1,gp_wm,1,1))                                                                                                                                               
       end if                                                                                                                                                                                               
    end do                                                                                                                                                                                                  
    ! open(unit = 1, file = 'y_c.dat', status = 'replace')                                                                                                                                                  
    ! do j = 0,n_wm                                                                                                                                                                                         
    !    write(1,*) j,y_wm_c(j)                                                                                                                                                                             
    ! end do                                                                                                                                                                                                
    ! close(1)                                                                                                                                                                                              
  end subroutine ewmGrid                                                                                                                                                                                    

Here is how I am declaring all of the variables used in the various ewm subroutines:

! module equilibriumWallModel                                                                                                                                                                                                                                                                                                                                                                                                                                               
! Contains subroutines for equilibrium wall model                                                                                                                                                                                                                                                                                                                                                                                                                           
module equilibriumWallModel                                                                                                                                                                                                                                                                                                                                                                                                                                                 
  use precision                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  use globalVariables                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  use ieee_arithmetic                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  use misc                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
  implicit none                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  ! Integers                                                                                                                                                                                                                                                                                                                                                                                                                                                                
  integer :: n_wm  = 25                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  integer :: gp_wm = 5                                                                                                                                                                                                                                                                                                                                                                                                                                                      
  !$acc declare create(n_wm,gp_wm)                                                                                                                                                                                                                                                                                                                                                                                                                                          
  ! Reals                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
  real(dp) :: dy_w      = 0.001d0                                                                                                                                                                                                                                                                                                                                                                                                                                           
  real(dp) :: r_stretch = 1.016d0                                                                                                                                                                                                                                                                                                                                                                                                                                           
  !$acc declare create(dy_w,r_stretch)                                                                                                                                                                                                                                                                                                                                                                                                                                      
  real(dp) :: u_bc = 0.0d0                                                                                                                                                                                                                                                                                                                                                                                                                                                  
  real(dp) :: w_bc = 0.0d0                                                                                                                                                                                                                                                                                                                                                                                                                                                  
  real(dp) :: t_bc = 4.5d0                                                                                                                                                                                                                                                                                                                                                                                                                                                  
  !$acc declare create(u_bc,w_bc,t_bc)                                                                                                                                                                                                                                                                                                                                                                                                                                      
  real(dp) :: suth_star                                                                                                                                                                                                                                                                                                                                                                                                                                                     
  !$acc declare create(suth_star)                                                                                                                                                                                                                                                                                                                                                                                                                                           
  real(dp) :: aplus = 17.0d0                                                                                                                                                                                                                                                                                                                                                                                                                                                
  real(dp) :: vk    = 0.41d0                                                                                                                                                                                                                                                                                                                                                                                                                                                
  !$acc declare create(aplus,vk)                                                                                                                                                                                                                                                                                                                                                                                                                                            
  real(dp) :: k_ref  = 7.5d-3                                                                                                                                                                                                                                                                                                                                                                                                                                               
  real(dp) :: cp_ref = 1001.0d0                                                                                                                                                                                                                                                                                                                                                                                                                                             
  real(dp) :: c3     = 2.495d0*10.0d0**(-3)                                                                                                                                                                                                                                                                                                                                                                                                                                 
  real(dp) :: c4     = 194.0d0                                                                                                                                                                                                                                                                                                                                                                                                                                              
  real(dp) :: Pr_t   = 0.9d0                                                                                                                                                                                                                                                                                                                                                                                                                                                
  !$acc declare create(k_ref,cp_ref,c3,c4,Pr_t)                                                                                                                                                                                                                                                                                                                                                                                                                             
  real(dp) :: du_dy_n                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  real(dp) :: dt_dy_n                                                                                                                                                                                                                                                                                                                                                                                                                                                       
  !$acc declare create(du_dy_n,dt_dy_n)                                                                                                                                                                                                                                                                                                                                                                                                                                     
  real(dp) :: t_wm_n                                                                                                                                                                                                                                                                                                                                                                                                                                                        
  real(dp) :: k_wm                                                                                                                                                                                                                                                                                                                                                                                                                                                          
  real(dp) :: cp_wm                                                                                                                                                                                                                                                                                                                                                                                                                                                         
  !$acc declare create(t_wm_n,k_wm,cp_wm)                                                                                                                                                                                                                                                                                                                                                                                                                                   
  real(dp) :: old_tau_wall                                                                                                                                                                                                                                                                                                                                                                                                                                                  
  real(dp) :: new_tau_wall                                                                                                                                                                                                                                                                                                                                                                                                                                                  
  !$acc declare create(old_tau_wall,new_tau_wall)                                                                                                                                                                                                                                                                                                                                                                                                                           
  real(dp) :: old_q_wall                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  real(dp) :: new_q_wall                                                                                                                                                                                                                                                                                                                                                                                                                                                    
  !$acc declare create(old_q_wall,new_q_wall)                                                                                                                                                                                                                                                                                                                                                                                                                               
  ! Real arrays                                                                                                                                                                                                                                                                                                                                                                                                                                                             
  real(dp), dimension(:), allocatable :: y_wm_n                                                                                                                                                                                                                                                                                                                                                                                                                             
  real(dp), dimension(:), allocatable :: y_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                             
  !$acc declare create(y_wm_n,y_wm_c)                                                                                                                                                                                                                                                                                                                                                                                                                                       
  real(dp), dimension(:), allocatable :: u_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                             
  real(dp), dimension(:), allocatable :: w_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                             
  real(dp), dimension(:), allocatable :: vel_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                           
  real(dp), dimension(:), allocatable :: t_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                             
  !$acc declare create(u_wm_c,w_wm_c,vel_wm_c,t_wm_c)                                                                                                                                                                                                                                                                                                                                                                                                                       
  real(dp), dimension(:), allocatable :: mu_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                            
  real(dp), dimension(:), allocatable :: mu_wm_n                                                                                                                                                                                                                                                                                                                                                                                                                            
  !$acc declare create(mu_wm_c,mu_wm_n)                                                                                                                                                                                                                                                                                                                                                                                                                                     
  real(dp), dimension(:), allocatable :: p_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                             
  real(dp), dimension(:), allocatable :: rho_wm_c                                                                                                                                                                                                                                                                                                                                                                                                                           
  real(dp), dimension(:), allocatable :: rho_wm_n                                                                                                                                                                                                                                                                                                                                                                                                                           
  !$acc declare create(p_wm_c,rho_wm_c,rho_wm_n)                                                                                                                                                                                                                                                                                                                                                                                                                            
  real(dp), dimension(:), allocatable :: yplus                                                                                                                                                                                                                                                                                                                                                                                                                              
  real(dp), dimension(:), allocatable :: eddy_visc                                                                                                                                                                                                                                                                                                                                                                                                                          
  real(dp), dimension(:), allocatable :: visc_total                                                                                                                                                                                                                                                                                                                                                                                                                         
  !$acc declare create(yplus,eddy_visc,visc_total)                                                                                                                                                                                                                                                                                                                                                                                                                          
  real(dp), dimension(:), allocatable :: eps                                                                                                                                                                                                                                                                                                                                                                                                                                
  !$acc declare create(eps)                                                                                                                                                                                                                                                                                                                                                                                                                                                 
  real(dp), dimension(:), allocatable :: a_m                                                                                                                                                                                                                                                                                                                                                                                                                                
  real(dp), dimension(:), allocatable :: b_m                                                                                                                                                                                                                                                                                                                                                                                                                                
  real(dp), dimension(:), allocatable :: c_m                                                                                                                                                                                                                                                                                                                                                                                                                                
  real(dp), dimension(:), allocatable :: rhs_m                                                                                                                                                                                                                                                                                                                                                                                                                              
  !$acc declare create(a_m,b_m,c_m,rhs_m)                                                                                                                                                                                                                                                                                                                                                                                                                                   
  real(dp), dimension(:), allocatable :: a_t                                                                                                                                                                                                                                                                                                                                                                                                                                
  real(dp), dimension(:), allocatable :: b_t                                                                                                                                                                                                                                                                                                                                                                                                                                
  real(dp), dimension(:), allocatable :: c_t                                                                                                                                                                                                                                                                                                                                                                                                                                
  real(dp), dimension(:), allocatable :: rhs_t                                                                                                                                                                                                                                                                                                                                                                                                                              
  !$acc declare create(a_t,b_t,c_t,rhs_t)                             


[all other ewm subroutines]

end module equilibriumWallModel

I can provide every subroutine needed for ewm to run independently if this is not enough.


Thanks Natan, but unfortunately I’m not seeing anything unusual here. Though I’m wondering if “ewm” and “ewmGrid” are in the equilibriumWallModel module as well? Also, why isn’t the compiler complaining about “y_c” in addition to “y_wm_n”, but then I don’t see “y_c” declared in the module so not sure what it is.

If you’re unable to create a minimal reproducer, would I be able to access the full source so I can investigate? If it’s not public, please direct message me, or I can send you an email to arrange how I can access the code.


Hi Mat,

Yes, both ewm and ewmGrid are in the equilibriumWallModel module. y_c is a global variable. The compiler would complain about certain variables first, and then I would use !$acc declare create for those. Then, it would complain about others until I used !$acc declare create for all of them. I will send you a direct message so we can investigate further.

Thank you very much,