# Sharing 1 GPU betwenn MPI tasks work fine with 4 mpi tasks but cudaMalloc "unknown error" wi

**URL:** <https://forums.developer.nvidia.com/t/sharing-1-gpu-betwenn-mpi-tasks-work-fine-with-4-mpi-tasks-but-cudamalloc-unknown-error-wi/8822>\
**Category:** CUDA Programming and Performance\
**Created:** [April 9, 2009, 2:52pm UTC](https://forums.developer.nvidia.com/t/sharing-1-gpu-betwenn-mpi-tasks-work-fine-with-4-mpi-tasks-but-cudamalloc-unknown-error-wi/8822 "2009-04-09T14:52:57Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![matlerouge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@matlerouge](https://forums.developer.nvidia.com/u/matlerouge)\
**Post date:** [April 9, 2009, 2:52pm UTC](https://forums.developer.nvidia.com/t/sharing-1-gpu-betwenn-mpi-tasks-work-fine-with-4-mpi-tasks-but-cudamalloc-unknown-error-wi/8822/1 "2009-04-09T14:52:57Z")

</div>

Hello,

I have an application that use GPU (tesla 1060). This application calls only “simple” cuda function, such as cudaMalloc, cudamemcpy or kernel execution.

If I “connect” 4 MPI task to one GPU (so, I call cudaSetDevice(0) on each MPI task), my application is executed without problem.

But if I try with more of 4 tasks, for exemple 6 tasks, I get error from each MPI task when I do cudaMalloc (unknown error).

The memory used is very small, so it is not a problem of GPU memory saturation .

Is the number of unix process that acces to one GPU limited ?

Thanks

Matthieu

---

<div class="post-metadata">

**Author:** ![seibert](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/seibert/32/520688_2.png) [@seibert](https://forums.developer.nvidia.com/u/seibert)\
**Post date:** [April 9, 2009, 7:24pm UTC](https://forums.developer.nvidia.com/t/sharing-1-gpu-betwenn-mpi-tasks-work-fine-with-4-mpi-tasks-but-cudamalloc-unknown-error-wi/8822/2 "2009-04-09T19:24:47Z")

</div>

I’ve have 8 processes all use one GPU, so I don’t think 4 is the limit.

---

<div class="post-metadata">

**Author:** ![jack](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jack/32/12069_2.png) [@jack](https://forums.developer.nvidia.com/u/jack)\
**Post date:** [April 9, 2009, 7:37pm UTC](https://forums.developer.nvidia.com/t/sharing-1-gpu-betwenn-mpi-tasks-work-fine-with-4-mpi-tasks-but-cudamalloc-unknown-error-wi/8822/3 "2009-04-09T19:37:29Z")

</div>

I haven’t really done much with MPI, so I don’t know the specifics of it, but perhaps you could make some sort of singleton worker class that handles the GPU functions. This means the object only gets created once, no matter how many threads are using it. Then you could have a ‘task’ queue that each process can put a ‘task’ into (i.e. a kernel call of some kind, and any associated memory writes/reads) and the worker class will execute them in order.

---

<div class="post-metadata">

**Author:** ![matlerouge](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@matlerouge](https://forums.developer.nvidia.com/u/matlerouge)\
**Post date:** [April 10, 2009, 7:46am UTC](https://forums.developer.nvidia.com/t/sharing-1-gpu-betwenn-mpi-tasks-work-fine-with-4-mpi-tasks-but-cudamalloc-unknown-error-wi/8822/4 "2009-04-10T07:46:25Z")

</div>

> [@](#):
>
> I’ve have 8 processes all use one GPU, so I don’t think 4 is the limit.

Ok and your process performs cudalMalloc from each MPI task ? Do you use some kind of synchronisation (semaphore…) when you access to the GPU ?

---

<div class="post-metadata">

**Author:** ![seibert](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/seibert/32/520688_2.png) [@seibert](https://forums.developer.nvidia.com/u/seibert)\
**Post date:** [April 10, 2009, 12:49pm UTC](https://forums.developer.nvidia.com/t/sharing-1-gpu-betwenn-mpi-tasks-work-fine-with-4-mpi-tasks-but-cudamalloc-unknown-error-wi/8822/5 "2009-04-10T12:49:26Z")

</div>

> [@](#):
>
> Ok and your process performs cudalMalloc from each MPI task ? Do you use some kind of synchronisation (semaphore…) when you access to the GPU ?

These are not MPI processes. I am just running 8 copies of the same program. The processes don’t talk to each other, so there is no semaphore or any other synchronization.