Multi-GPU with OpenMP

Accelerated Computing CUDA CUDA Programming and Performance
1

Hello all,

I am trying to develop a program which uses multiple GPU’s independently with CUDA.
I was wondering whether I could activate multiple devices within a
parallel OpenMP for loop. The basic structure of the program in mind is :

#pragma omp parallel for 
for (device = 0; device < deviceCount; device++){
cudaSetDevice(device);
// do GPU-intensive work 
}

I would appreciate if you could give me an idea whether this is possible or not.

Thanks

2

Yes that will work. Also, if you have more “stuff” to do than you have devices, you can use something like:

#pragma omp parallel for num_threads(deviceCount)
for(int i = 0; i < stuff.size(); i++)
{
     const int myDeviceId = omp_get_thread_num();
     cudaSetDevice(myDeviceId)
     kernel<<<>>>(stuff[i])
}
1 Like
3

There is a cudaOpenMP sample code that demonstrates using multiple GPUs with OpenMP

4

Thank you both!

I have just adapted the code for OpenMP and it worked just fine.