# pgf90 + openacc & managed memory / um-evaluation package

**URL:** <https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481>\
**Category:** Legacy PGI Compilers (archived)\
**Created:** [May 20, 2015, 9:28am UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481 "2015-05-20T09:28:08Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![escj](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@escj](https://forums.developer.nvidia.com/u/escj)\
**Post date:** [May 20, 2015, 9:28am UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/1 "2015-05-20T09:28:08Z")

</div>

Hello , I’ve downloaded the pgi/15.4 + managed/um-evaluation package .

Some library/link-flag are missing in the pgf90 flag when using -ta=telsa: **managed**.

> pgf90 -ta=host,tesla:\> **managed** \> hello\_openacc.f90 -o hello\_openacc\_managed  
> /tmp/pgf90KZJSN8umD\_5.o: In function `hello': /home/escj/dir_OPENACC/./hello_openacc.f90:3: undefined reference to `pgf90\_man\_alloc04’  
> /home/escj/dir\_OPENACC/./hello\_openacc.f90:7: undefined reference to `__pgi_uacc_cuda_launchk2' /tmp/pgf90KZJSN8umD_5.o: In function `…cuda\_fortran\_constructor\_1’:  
> /home/escj/dir\_OPENACC/./hello\_openacc.f90:15: undefined reference to `__pgi_cuda_register_fat_binary' /home/escj/dir_OPENACC/./hello_openacc.f90:15: undefined reference to `\_\_cudaRegisterFunction’  
> pgacclnk: child process exit status 1: /usr/bin/ld

Adding -Mcuda , solve the problem

> pgf90 \> **-Mcuda** \> -ta=host,tesla:managed hello\_openacc.f90 -o hello\_openacc\_managed

After this, is work correctly  
( but performance are very poor for example in a STREAM benchmark, as I understood data are copied in/out at every launch kernel )

Bye

Juan

---

<div class="post-metadata">

**Author:** ![MatColgrove](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/matcolgrove/32/14047_2.png) [@MatColgrove](https://forums.developer.nvidia.com/u/MatColgrove)\
**Post date:** [May 20, 2015, 4:07pm UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/2 "2015-05-20T16:07:21Z")

</div>

Hi Juan,

Yes, to use Unified Memory with Fortran, you do need to add the “-Mcuda”.

> but performance are very poor for example in a STREAM benchmark, as I understood data are copied in/out at every launch kernel

Not unexpected since UM is meant to help with the initial porting effort. However, the data only gets copied went it’s been modified. So if you’re touching the data on the host and the device, then yes it will get copied back and forth with each kernel launch. If you only touch the data on either the host or device, then it doesn’t get copied.

- Mat

---

<div class="post-metadata">

**Author:** ![escj](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@escj](https://forums.developer.nvidia.com/u/escj)\
**Post date:** [May 22, 2015, 12:46pm UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/3 "2015-05-22T12:46:38Z")

</div>

Hello Mat .

1. ;-) The option _-Mcuda_ was not indicated in the PGI news  
[Account Login | PGI](http://www.pgroup.com/lit/articles/insider/v6n2a4.htm)  
(neither in the README coming with the um-eval package )

2. ;-) I found the problem of the poor performance of my STREAM benchmark …

I have 2 TITAN cards on my test machine !

> pgaccelinfo -short  
> 0 GeForce GTX TITAN  
> 1 GeForce GTX TITAN

Setting **CUDA\_VISIBLE\_DEVICES=0** speedup the performance near the optimal one .

Without CUDA\_VISIBLE\_DEVICES

```plaintext
stream_pgi154_acc_ompi175_manage
...
Function Rate (MB/s) Avg time Min time Max time
Copy: 17502.01565974 ...
Scale: 17508.98335551 ...
Add: 11480.43242445 ...
Triad: 11383.99349557 ...

```

With **CUDA\_VISIBLE\_DEVICES=0**

```plaintext
CUDA_VISIBLE_DEVICES=0 stream_pgi154_acc_ompi175_manage
Function Rate (MB/s) Avg time Min time Max time
Copy: 222708.45205104 
Scale: 221631.04772747
Add: 222303.20011646
Triad: 222423.12092569

```

So good for a first try !

Bye

Juan

---

<div class="post-metadata">

**Author:** ![escj](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@escj](https://forums.developer.nvidia.com/u/escj)\
**Post date:** [May 22, 2015, 1:03pm UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/4 "2015-05-22T13:03:13Z")

</div>

One question left

With GPU DIRECT aware MPI , does the **!$acc host\_data use\_device**  
work/must be used for managed array ?

otherwise how the system/library know which of the host/device version of the array must be transferred via MPI ?

Bye Juan

---

<div class="post-metadata">

**Author:** ![MatColgrove](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/matcolgrove/32/14047_2.png) [@MatColgrove](https://forums.developer.nvidia.com/u/MatColgrove)\
**Post date:** [May 22, 2015, 7:11pm UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/5 "2015-05-22T19:11:41Z")

</div>

Hi Juan,

I’m not sure. I wouldn’t think using “host\_data” is needed, but question if CUDA UM works with GPU Direct.

Let me as some folks and get back to you.

- Mat

---

<div class="post-metadata">

**Author:** ![MatColgrove](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/matcolgrove/32/14047_2.png) [@MatColgrove](https://forums.developer.nvidia.com/u/MatColgrove)\
**Post date:** [May 22, 2015, 10:51pm UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/6 "2015-05-22T22:51:59Z")

</div>

Hi Juan,

Here’s the response I got:

> If managed memory is used host\_data use\_device is not necessary (it would not change the value of the pointer anyway). However the MPI implementation needs to be CUDA-aware and aware of unified memory. CUDA-aware builds of OpenMPI 1.8.5, which was released a couple of days ago, is aware of unified memory. If he needs to run on with another or older CUDA-aware MPI implementation he needs to make sure that CUDA IPC (GPUDirect P2P) and GPUDirect RDMA is disable, e.g. for OpenMPI this can be achieved with:
> 
> mpiexec --mca btl\_smcuda\_use\_cuda\_ipc 0 --mca btl\_smcuda\_use\_cuda\_ipc\_same\_gpu 0
> 
> (Explicitly disabling GPUDirect RDMA is not necessary with OpenMPI because it is off by default).

Hope this helps,  
Mat

---

<div class="post-metadata">

**Author:** ![escj](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@escj](https://forums.developer.nvidia.com/u/escj)\
**Post date:** [June 15, 2015, 8:18am UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/7 "2015-06-15T08:18:16Z")

</div>

Hello Mat .

With the last pgi/15.5 , I have tested the -ta:tesla: **managed** memory with openmpi1.8.5 CUDA-aware ( and also with mvapich2-1.a-gdr) .

Apparently activating the flag **managed** doesn’t inhibited the **host\_data** clause which generate segmentation fault .

Here is an **hello\_manage.f90** example :

=\> Process 0 send 1 to process 1 on GPU . Proc 1 write the result on CPU

> program hello\_managed
> 
> implicit none  
> include ‘mpif.h’
> 
> integer, parameter :: n=256  
> real , allocatable, dimension(:) :: send\_buf,recv\_buf  
> integer :: npe,mype,ierr
> 
> call mpi\_init(ierr)  
> call mpi\_comm\_rank(mpi\_comm\_world, mype, ierr)  
> call mpi\_comm\_size(mpi\_comm\_world, npe, ierr)
> 
> if (npe.ne.2) STOP ‘run with 2 MPI task only’
> 
> allocate( send\_buf(n),recv\_buf(n) )
> 
> !$acc data create(send\_buf,recv\_buf)
> 
> !$acc kernels  
> send\_buf=1.0  
> recv\_buf=-999.0  
> !$acc end kernels
> 
> !$acc host\_data use\_device(send\_buf,recv\_buf)  
> if ( mype .eq. 0 ) then  
> call MPI\_Send(send\_buf,n,MPI\_REAL,1,0,mpi\_comm\_world, ierr)  
> else  
> call MPI\_Recv(recv\_buf,n,MPI\_REAL,0,0,mpi\_comm\_world,mpi\_status\_ignore, ierr)  
> endif  
> !$acc end host\_data
> 
> if ( mype .eq. 1 ) then  
> !$acc update host(recv\_buf(n:n))  
> print\*,‘mype=’,mype,’ recv\_buf(n) \<must be 1\> =',recv\_buf(n)  
> end if
> 
> !$acc end data
> 
> call mpi\_finalize(ierr)
> 
> end program hello\_managed

=\> Compiled without the managed flag , it work correctly \<=\> thank’s to MPI CUDA-aware

> mpif90 -ta=tesla:cuda6.5 hellompi\_managed.f90 -o hellompi\_tesla  
> mpirun -np 2 hellompi\_tesla  
> mype= 1 recv\_buf(n) \<must be 1\> = \> **1.000000**

=\> With the **managed** flag the code crash

> ## mpif90 -Mcuda -ta=tesla:cuda6.5,\> **managed** \> hellompi\_managed.f90 -o hellompi\_managed mpirun -np 2 hellompi\_managed
> 
> ## mpirun noticed that process rank 0 with PID 6225 on node n370 exited on \> **signal 11 (Segmentation fault)**\> .

=\> Removing the **host\_data** make the code run correctly with the managed flag :

> mpif90 -Mcuda -ta=tesla:cuda6.5,managed hellompi\_managed.f90 -o hellompi\_managed\_no\_hostdata  
> mpirun -np 2 hellompi\_managed\_no\_hostdata  
> mype= 1 recv\_buf(n) \<must be 1\> = \> **1.000000**

=\> Last remark, the **update** clause is not deactivated to ( by **create** clause are ) , as shown by the -Minfo=acc

> mpif90 -Mcuda -ta=tesla:cuda6.5,\> **managed** \> -Minfo=acc hellompi\_managed.f90 -o hellompi\_managed  
> hello\_managed:  
> 21, Loop is parallelizable  
> Accelerator kernel generated  
> 21, !$acc loop gang, vector(128) ! blockidx%x threadidx%x  
> 34, \> **Generating update host** \> (recv\_buf(256)

Bye Juan .

---

<div class="post-metadata">

**Author:** ![MatColgrove](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/matcolgrove/32/14047_2.png) [@MatColgrove](https://forums.developer.nvidia.com/u/MatColgrove)\
**Post date:** [June 15, 2015, 11:27pm UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/8 "2015-06-15T23:27:22Z")

</div>

Thanks Juan.

I know in the 15.1 compiler we had a similar issue with host\_data, but I thought we had fixed it in 15.2. Let me try and reproduce the problem here. I may be a different issue or we missed a case.

- Mat

---

<div class="post-metadata">

**Author:** ![MatColgrove](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/matcolgrove/32/14047_2.png) [@MatColgrove](https://forums.developer.nvidia.com/u/MatColgrove)\
**Post date:** [June 16, 2015, 8:58pm UTC](https://forums.developer.nvidia.com/t/pgf90-openacc-managed-memory-um-evaluation-package/134481/9 "2015-06-16T20:58:22Z")

</div>

Hi Juan,

I was able to recreate the issue when using OpenMPI 1.8.5 built with RDMA and CUDA 6.5. Interestingly, when I switched to using MPICH2 gdr with CUDA 7.0, the test passed.

I’m still tracking things down, but it could be a CUDA 6.5 or PGI issue. Not sure which. My next step is to build OpenMPI with CUDA 7.0 and see what happens.

- Mat
