# OpenACC - Parallelization of small sub-loops present inside large main loops

**URL:** <https://forums.developer.nvidia.com/t/openacc-parallelization-of-small-sub-loops-present-inside-large-main-loops/192906>\
**Category:** CUDA NVCC Compiler\
**Created:** [October 25, 2021, 1:15am UTC](https://forums.developer.nvidia.com/t/openacc-parallelization-of-small-sub-loops-present-inside-large-main-loops/192906 "2021-10-25T01:15:39Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hemanthgrylls](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hemanthgrylls/32/200802_2.png) [@hemanthgrylls](https://forums.developer.nvidia.com/u/hemanthgrylls)\
**Post date:** [October 25, 2021, 1:15am UTC](https://forums.developer.nvidia.com/t/openacc-parallelization-of-small-sub-loops-present-inside-large-main-loops/192906/1 "2021-10-25T01:15:39Z")

</div>

Hi,

I am trying to parallelize a kernel with the following configuration:

```auto
integer :: i,j,k,nbl
double precision :: A(NI,NJ,NK,nblocks), rho,c
double precision, dimension(-2:3) :: Wp, Wp2

    !$acc parallel loop gang vector collapse(4) private(Wp) default(present)
	DO nbl = 1,nblocks
	DO k = 1,NK
	DO j = 1,NJ
	DO i = 1,NI
		
		rho = i*j
		c = j*k
		
		!--------------------------------------------------------------------------------------------
		
		!$acc loop seq
		DO II=-2,3
			
			Wp(II) = rho-c
		
		ENDDO

        rho = i
		c = k

        !$acc loop seq
		DO II=-2,3
			
			Wp2(II) = rho*Wp(II,1) +c
		
		ENDDO

        A(i,j,k,nbl) = SUM(Wp(-2:3)) + SUM(Wp2(-2:3))
		
	ENDDO
	ENDDO
	ENDDO
	ENDDO

```

There is good amount of parallelism in the exterior loops at i,j,k,nbl levels. But the small inner loops (with index II) are still running serially on each thread. Is there a way to parallelize these loops (with index II) as well?

With an aim of distributing the parallelism, I have changed the dirictive  
‘!$acc parallel loop gang vector collapse(4) private(Wp) default(present)’ to  
‘!$acc parallel loop gang vector\_length(32) collapse(4) private(Wp) default(present)’ along with a change of inner loops directives from ‘!$acc loop seq’ to ‘!$acc loop vector’ but the performance deteriorated with that.

Also, am I correct in saying: ‘performance from parallelizing the inner sub-loops becomes insignificant with increasing values of NI,NJ and NK since the device will be already be fully occupied with the outer four loops itself’?

---

<div class="post-metadata">

**Author:** ![Yuki\_Ni](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/yuki_ni/32/14047_2.png) [@Yuki\_Ni](https://forums.developer.nvidia.com/u/Yuki_Ni)\
**Post date:** [October 25, 2021, 3:13pm UTC](https://forums.developer.nvidia.com/t/openacc-parallelization-of-small-sub-loops-present-inside-large-main-loops/192906/2 "2021-10-25T15:13:31Z")

</div>

Please follow the instruction here to report us a ticket [Getting Help with CUDA NVCC Compiler](https://forums.developer.nvidia.com/t/getting-help-with-cuda-nvcc-compiler/183638) with the exact compile command line .

---

<div class="post-metadata">

**Author:** ![hemanthgrylls](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hemanthgrylls/32/200802_2.png) [@hemanthgrylls](https://forums.developer.nvidia.com/u/hemanthgrylls)\
**Post date:** [October 25, 2021, 10:12pm UTC](https://forums.developer.nvidia.com/t/openacc-parallelization-of-small-sub-loops-present-inside-large-main-loops/192906/3 "2021-10-25T22:12:28Z")

</div>

Compiler: pgfortran (21.9)  
Flags: -fast -acc -ta=tesla:managed -Minfo=accel  
Cuda toolkit version: 11.4

I don’t have any bugs or issues to report. But I am looking for help to optimize the loop scheduling to get faster run speed.