# GPU stalls due to stream synchronization -- even when idle?

**URL:** <https://forums.developer.nvidia.com/t/gpu-stalls-due-to-stream-synchronization-even-when-idle/191761>\
**Category:** CUDA Programming and Performance\
**Created:** [October 11, 2021, 2:35pm UTC](https://forums.developer.nvidia.com/t/gpu-stalls-due-to-stream-synchronization-even-when-idle/191761 "2021-10-11T14:35:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tbesard](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/tbesard/32/27383_2.png) [@tbesard](https://forums.developer.nvidia.com/u/tbesard)\
**Post date:** [October 11, 2021, 2:35pm UTC](https://forums.developer.nvidia.com/t/gpu-stalls-due-to-stream-synchronization-even-when-idle/191761/1 "2021-10-11T14:35:55Z")

</div>

I’m seeing some strange stalls in my application at the point where I’m synchronizing my stream in order to flush asynchronous memory frees (prompted to by a failed asynchronous allocation):

 ![image](https://global.discourse-cdn.com/nvidia/original/3X/6/2/6202e2ea6b24faeaf109d9bc1f2ead4131fb5f63.png)

I’m not sure why `cuStreamSynchronize` takes so much time here, as the GPU has become idle much sooner. There’s no other streams active, the screenshot shows all that matters. Most of the stall (i.e. after the last kernel on this stream has finished, but before `cuStreamSynchronize` has returned) is spent doing some `ioctl`. Is this the async memory manager compacting memory, or anything like that? But then I would have expected the samples to point to `libcuda`, and not to the kernel.

---

<div class="post-metadata">

**Author:** ![hhoffman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@hhoffman](https://forums.developer.nvidia.com/u/hhoffman)\
**Post date:** [November 18, 2021, 4:48pm UTC](https://forums.developer.nvidia.com/t/gpu-stalls-due-to-stream-synchronization-even-when-idle/191761/2 "2021-11-18T16:48:26Z")

</div>

What you have described is consistent with the stream synchronize releasing gpu memory back to the OS on behalf of the stream ordered allocator. You should be able to confirm that your stream ordered allocation pool’s reserved memory dropped during this synchronize.

---

<div class="post-metadata">

**Author:** ![tbesard](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/tbesard/32/27383_2.png) [@tbesard](https://forums.developer.nvidia.com/u/tbesard)\
**Post date:** [November 19, 2021, 6:56am UTC](https://forums.developer.nvidia.com/t/gpu-stalls-due-to-stream-synchronization-even-when-idle/191761/3 "2021-11-19T06:56:13Z")

</div>

Thanks. Is there a way to reduce the cost of these? That is, I guess I don’t really require that memory is given back to the OS as my application is still very much running and going to be allocating again right away. Does that require me to set CU\_MEMPOOL\_ATTR\_RELEASE\_THRESHOLD, or what’s the recommendation here?

---

<div class="post-metadata">

**Author:** ![hhoffman](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@hhoffman](https://forums.developer.nvidia.com/u/hhoffman)\
**Post date:** [November 19, 2021, 8:15pm UTC](https://forums.developer.nvidia.com/t/gpu-stalls-due-to-stream-synchronization-even-when-idle/191761/4 "2021-11-19T20:15:51Z")

</div>

> [@tbesard](#):
>
> CU\_MEMPOOL\_ATTR\_RELEASE\_THRESHOLD

Yes. If you set CU\_MEMPOOL\_ATTR\_RELEASE\_THRESHOLD above the memory used in the pool, the pool will not release its memory.

You can set CU\_MEMPOOL\_ATTR\_RELEASE\_THRESHOLD to  
UINT64\_MAX. This will avoid releasing any pages to the os during the synchronization call. If you do so, you may want to look at the cuMemPoolTrimTo api to release memory. The CUDA driver may still release the memory from the pool to avoid allocation failures of other cuda apis in the same process, so cuMemPoolTrimTo would only be needed if multiple processes are trying to use the vidmem resources.