Reference makfile for MPI+OpenACC+CUDA code

Dear PGI-community,

on our DGX2 system many users want to develop codes which use MPI, OpenACC and CUDA within a single project. I want to create a reference makefile for the users.

We are using the following setup:
PGI compiler 18.10
OpenMPI (compiled with PGI compiler 18.10)
CUDA 10.0

I created the following makefile, with this makefile I was able to compile a small project. But I have few questions.

  1. I always need to pass the -ta=tesla:nordc option to files which contain OpenACC code? If I remove this option I get linking error at the linking stage.
  2. Invoking the nvcc wrapper compiler at the linking stage is the correct way to link all files together?
  3. The CUDA toolkit version which will be used by the PGI compiler is 10.0 because the CUDA driver version is 10.0?

Thank you for your help!

CC   = mpic++
CCU  = nvcc -ccbin=mpic++

PROG = run

OBJS = main.o func.o
OPTS =  -ta=tesla:cc70  -acc -Minfo=accel -Minfo -Mcuda

%.o : %.c
        ${CC} ${OPTS} -ta=tesla:nordc -c ${CFLAGS} $<
%.o : %.cu
        ${CCU} -Xcompiler "${OPTS}" -c ${CUFLAGS} $<

all : ${PROG}
${PROG} : ${OBJS}
        ${CCU} -Xcompiler "${OPTS}" -o $@ ${OBJS}

Hi Peter85,

  1. I always need to pass the -ta=tesla:nordc option to files which contain OpenACC code? If I remove this option I get linking error at the linking stage.

By default, PGI compiles code with RDC enabled so we can link device code and allow for cross-file device routine calling and used of global device data. However, nvcc does not set RDC by default and why you’re needing to disable it with PGI. To work together, you must compile both with or without RDC.

Can you try adding “-rdc=true” to you NVCC compilation flags and remove “-ta=tesla:nordc”?

  1. Invoking the nvcc wrapper compiler at the linking stage is the correct way to link all files together?

You’ll want to use PGI to link. In addition to using RDC (we invoke the device linker, nvlink, when RDC is enabled), the compiler will also implicitly generate the CUDA registration code during the link step for the OpenACC kernels.

Also, we implicitly add the CUDA runtime libraries to the link when using the “-Mcuda” flag so you shouldn’t need to add any additional libraries. When using other CUDA libraries, such as cuBLAS, I’d suggest using the PGI flag “-Mcudalib[=cublas|cufft|curand|cusolver|cusparse]” and the appropriate libraries will be added.

  1. The CUDA toolkit version which will be used by the PGI compiler is 10.0 because the CUDA driver version is 10.0?

As of the 18.9 compilers, we will check during compilation as to the which driver is install and use the CUDA version appropriate for that driver. There are other options as well such as specifying the CUDA version as a command line option or setting the CUDA_HOME environment variable. Full details can be found at: https://www.pgroup.com/resources/docs/19.3/x86/pvf-release-notes/index.htm#cuda-toolkit-versions

Hope this helps,
Mat

Hi Mat,

thank you for your answer!
I have changed the makefile according to your suggestions:

  1. I removed -ta=tesla:nordc and added -rdc=true
  2. Instead of using NVCC for linking I use the PGI compiler

With the changes I can compile and run the program successfully.

Thank you for your help!

CC   = mpic++
CCU  = nvcc -ccbin=mpic++

OBJS = main.o func.o
OPTS = -ta=tesla:cc70 -acc -Minfo=accel -Minfo

%.o : %.c
        ${CC} ${OPTS} -c $<
%.o : %.cu
        ${CCU} -rdc=true -Xcompiler "${OPTS}" -c  $<

all : ${PROG}
${PROG} : ${OBJS}
        ${CC} -Mcuda ${OPTS} -o $@ ${OBJS}