Multi-GPU with OpenMP

Hello all,

I am trying to develop a program which uses multiple GPU’s independently with CUDA.
I was wondering whether I could activate multiple devices within a
parallel OpenMP for loop. The basic structure of the program in mind is :

#pragma omp parallel for 
for (device = 0; device < deviceCount; device++){
cudaSetDevice(device);
// do GPU-intensive work 
}

I would appreciate if you could give me an idea whether this is possible or not.

Thanks

Yes that will work. Also, if you have more “stuff” to do than you have devices, you can use something like:

#pragma omp parallel for num_threads(deviceCount)
for(int i = 0; i < stuff.size(); i++)
{
     const int myDeviceId = omp_get_thread_num();
     cudaSetDevice(myDeviceId)
     kernel<<<>>>(stuff[i])
}

There is a cudaOpenMP sample code that demonstrates using multiple GPUs with OpenMP

Thank you both!

I have just adapted the code for OpenMP and it worked just fine.