Halo Exchange updates MPI with OpenACC

Accelerated Computing HPC Compilers nvc, nvc++ and nvfortran
, ,
1

Hi,

I am trying to run OpenACC directives on an in-house solver, I have a problem with the halo exchange updates I am using Nvidia SDK HPC version 22.3 with managed flag.
In the function below when I add the OpenACC directives and then compute the updates makes the codes so slow.

2022-06-12_19-59-40
2022-06-12_19-59-401349×771 368 KB

When adding the Pragmas for porting the mpi communication to GPU as seen below it’s not doing nothing and the performance is still the same.

2022-06-12_20-02-28
2022-06-12_20-02-28965×583 211 KB

Is this because I am using managed flag so it’s not working this way? or I am missing something?

2

Correct. It’s a known limitation in several of the MPI implementations that what passing in CUDA Unified Memory pointers (i.e. “managed”), MPI doesn’t know if the memory is dirty or not, so ends up falling back to host side communication. In order to get the benefit of using CUDA Aware MPI and GPU direct communication, you’ll want to use OpenACC data regions to manage your data rather than use UM.

-Mat