Hi everyone,
I’m trying to compare the execution time of my code between CPU version and GPU one. The code is written in Fortran90 and compiled with mpif90 under nvhpc23.1. The GPU acceleration is performed with OpenACC and it is a multi GPU code, where a single CPU is set with a single GPU. In total I’m running with 2 CPUs and 2 GPUs and therefore my launch command is:
mpirun -np 2 ./program
The code is very complex since it is a CFD code and I’m reporting only the part that I’m accelerating. This is the kinetic part where I calculate the reaction rate of the species involved in the calculation. Now I’m only transferring data forward and back from CPUs to GPUs and viceversa to estimate the time of this data movement. Therefore the code is very simple because inside the triple loop there is almost nothing. The code is the following:
SUBROUTINE kinetics
USE global_mod, ONLY: TFlame, NsMAX, num_zones, zones, MINi, MAXi, MINj, MAXj, MINk, MAXk
USE common_alloc
USE openacc
USE common_mpi!, ONLY: my_gpu, device_type, my_id
INTEGER, VALUE :: B, i, j, k, l, ll, s, icomp
!$acc declare create(B, i, j, k, l, ll, s, icomp)
DOUBLE PRECISION, VALUE :: OM(NUMSP), Yi_ijk(NsMAX), Xir(NsMAX)
!$acc declare create(OM(:), Yi_ijk(:), Xir(:))
DOUBLE PRECISION, VALUE :: T_ijk, p_ijk, dens, Rgst,S_y, Wmix_ijk, D_hp,rho_ijk
!$acc declare create(T_ijk, p_ijk, dens, Rgst,S_y, Wmix_ijk, D_hp,rho_ijk)
DOUBLE PRECISION, VALUE :: VREAZ(1000)
!$acc declare create(VREAZ(:))
INTEGER, VALUE :: NsMAT
INTEGER, VALUE :: iR, CC, NUMDICOMP, ISPECIE
INTEGER, VALUE :: NUMDICOMPF, NUMDICOMPB, REV, TERZOCORPO
!$acc declare create(NsMAT, iR, CC, NUMDICOMP, ISPECIE, NUMDICOMPF, NUMDICOMPB, REV, TERZOCORPO)
DOUBLE PRECISION, VALUE :: Kchem, Kchem_B, PRODF, PRODB, KEQ
!$acc declare create(Kchem, Kchem_B, PRODF, PRODB, KEQ)
DOUBLE PRECISION, VALUE :: NIF, NIB, NII!, omF(NUMSP),omB(NUMSP)
!$acc declare create(NIF, NIB, NII)
DOUBLE PRECISION, VALUE :: SOMMAH,SOMMAS,TEM
DOUBLE PRECISION, VALUE :: app
!DOUBLE PRECISION, POINTER, CONTIGUOUS :: A_app(:,:)
DOUBLE PRECISION, VALUE :: Y(NsMAX)
!$acc declare create(SOMMAH,SOMMAS,TEM,app,Y(:))
DOUBLE PRECISION, VALUE :: tmpsum,tmpsum2
!$acc declare create(tmpsum,tmpsum2)
!$acc declare copyin(my_gpu)
!$acc declare create(MINi(:),MAXi(:),MINj(:),MAXj(:),MINk(:),MAXk(:))
integer(c_size_t) :: free_mem, total_mem
!$acc declare create(free_mem, total_mem)
DOUBLE PRECISION :: start_time, end_time, elapsed_time
start_time = MPI_WTIME()
omega = 0.
!$acc enter data copyin(omega(:,:,:,:))
!$acc enter data copyin(Yi(:,:,:,:),T(:,:,:),p(:,:,:),rho(:,:,:))
!$acc update device(TFlame,NUMREAZ,NUMDISPREAZF,NUMDISPREAZB,FLAGREAZ,FLAGM,TAB1F,TAB3F,TAB1B,TAB3B,FORD3,MASS,RORD3,DHF,DSF,REACT_PLOG,TAB2,TAB2M,TROE,SRI,NUMDISPM,NUMSP,ALFAM)
!$acc update device(MINi,MAXi,MINj,MAXj,MINk,MAXk)
!$acc update device(omega)
!$acc loop seq
do k= MINk(BBB)-(Ghost-1), MAXk(BBB)+(Ghost-1)
do j= MINj(BBB)-(Ghost-1), MAXj(BBB)+(Ghost-1)
do i= MINi(BBB)-(Ghost-1), MAXi(BBB)+(Ghost-1)
p_ijk = p(i,j,k)
T_ijk = T(i,j,k)
Yi_ijk = Yi(:,i,j,k)+ 1.0d-20
end do ! Loop su i
end do ! Loop su j
end do ! Loop su k
!$acc update self(omega)
!$acc exit data delete(p,T,Yi,rho,omega)
call MPI_BARRIER(MPI_COMM_WORLD,ierr)
end_time = MPI_WTIME()
elapsed_time = end_time - start_time
if (my_id == 0) then
print *, "Elapsed time (seconds): ", elapsed_time
end if
END SUBROUTINE kinetics
When running with GPUs I compile the entire code with the following flags:
mpif90 -r8 -acc=gpu,noautopar -target=gpu -gpu=cc80,managed -Mpreprocess -Mfree -Mextend -Munixlogical -Mbyteswapio -traceback -Mchkstk -Mnostack_arrays -Mnofprelaxed -Mnofpapprox -Minfo=accel
while when I run the code with the CPUs only I compile in this way:
mpif90 -r8 -Mpreprocess -Mfree -Mextend -Munixlogical -Mbyteswapio -traceback -Mchkstk -Mnostack_arrays -Mnofprelaxed -Mnofpapprox
The issue that I found is that when I run the code (with a 2D dimensional case with a grid of 2000x2000 cells) the GPU code is faster than the CPU one!!! Since inside the loop there is nothing, the GPU code should be slower because of the data movement. The code need 370 seconds per iteration and with the CPU the part that I’m estimating last about 2.46 seconds. With the GPU code the overall time is always around 370 seconds (for real is less) but the triple loop last only 0.2 seconds. This is not possible. In this code that you see the acceleration is not performed since is !$acc loop seq but I also tried with !$acc parallel loop gang vector collapse(3) and the result is the same (indeed the time is different but alway less than CPU execution).
This is very strange but I’m sure I did something wrong. Thank you all!
-Matteo