# Understanding the meaning of localMemoryPoolUtilizedSize in CUDA with cudaMallocAsync and cudaFreeAsync **URL:** \ **Category:** Profiling Linux Targets\ **Tags:** cuda\ **Created:** [March 27, 2023, 11:36pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700 "2023-03-27T23:36:06Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![tommyecguitar](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tommyecguitar](https://forums.developer.nvidia.com/u/tommyecguitar)\ **Post date:** [March 27, 2023, 11:36pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/1 "2023-03-27T23:36:06Z")
I am trying to optimize my code using `cudaMallocAsync` and `cudaFreeAsync` . After profiling with Nsight Systems, it appears that these operations are using the local memory pool. While `cudaMemPoolTrimTo` is able to release the `localMemoryPoolSize` , the `localMemoryPoolUtilizedSize` continues to increase. My question is, “What does `localMemoryPoolUtilizedSize` represent?” ![image](https://global.discourse-cdn.com/nvidia/original/3X/a/f/aff4312d1bf49a85ff86390e93a5d2ad64dbab60.png) As an experiment, I checked the GPU memory using the resource monitor in the task manager, but I did not observe any continuous increase in memory usage, similar to what was observed with the `localMemoryPoolUtilizedSize` in the profiling results. ---
**Author:** ![aktcse14](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aktcse14](https://forums.developer.nvidia.com/u/aktcse14)\ **Post date:** [February 9, 2026, 7:46pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/2 "2026-02-09T19:46:32Z")
> [@tommyecguitar](#): > > localMemoryPoolUtilizedSize Hi, Hope you are doing well. Currently we are trying to visualize, time vs GPU memory usage And we are kind of getting these data from nsys. We have the same question? If you have known the detail please let us know. ---
**Author:** ![hwilper](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/hwilper/32/14043_2.png) [@hwilper](https://forums.developer.nvidia.com/u/hwilper)\ **Post date:** [February 9, 2026, 8:11pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/3 "2026-02-09T20:11:51Z")
@skottapalli can you comment? ---
**Author:** ![skottapalli](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@skottapalli](https://forums.developer.nvidia.com/u/skottapalli)\ **Post date:** [February 9, 2026, 9:25pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/4 "2026-02-09T21:25:27Z")
@aktcse14 and @tommyecguitar - could you please share a nsys-rep file that shows this behavior? I need to take a look to see if this is expected or a bug. ---
**Author:** ![tommyecguitar](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tommyecguitar](https://forums.developer.nvidia.com/u/tommyecguitar)\ **Post date:** [February 9, 2026, 10:29pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/5 "2026-02-09T22:29:54Z")
Thank you for replying to my question from 3 years ago. I don’t have the profiler data anymore. But I must have been using CUDA 11.4 or 12.x when I posted this topic. The Nsight version should be appropriate for that. @skottapalli Do you have any nsys data? ---
**Author:** ![skottapalli](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@skottapalli](https://forums.developer.nvidia.com/u/skottapalli)\ **Post date:** [February 9, 2026, 10:43pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/6 "2026-02-09T22:43:09Z")
It looks like the team missed answering to your message from 3 years ago. It popped up again today when another customer asked a question on this thread. I am not sure what you mean by nsys data. The nsys-rep file is the report created by nsys and it will help me understand if the behavior you saw is a bug in nsys. If you run into this problem again, please share the report file. According to the CUDA docs, the `cudaMemPoolTrimTo `- Releases memory back to the OS until the pool contains fewer than minBytesToKeep reserved bytes, or there is no more memory that the allocator can safely release. The allocator cannot release OS allocations that back outstanding asynchronous allocations. The OS allocations may happen at different granularity from the user allocations. Allocations that have not been freed count as outstanding. Allocations that have been asynchronously freed but whose completion has not been observed on the host (eg. by a synchronize) can count as outstanding The localMemoryPoolUtilizedSize at a given point in time reported by nsys is the amount of memory that is in use at that time. The outstanding allocations that have not been freed or that have been freed asynchronously count towards the utilizedSize. ---
**Author:** ![tommyecguitar](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tommyecguitar](https://forums.developer.nvidia.com/u/tommyecguitar)\ **Post date:** [February 9, 2026, 11:20pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/7 "2026-02-09T23:20:49Z")
Sorry, I intended to reply to @aktcse14 , but I mentioned @skottapalli . Since you were kind enough to respond, I would like to ask a follow-up question to confirm my understanding. If memory freed by cudaFreeAsync is not synchronized using cudaDeviceSynchronize, cudaStreamSynchronize, or similar calls, it may be treated as an outstanding allocation and therefore appear in localMemoryPoolUtilizedSize. Is my understanding correct that once a synchronization point is reached somewhere, those memory frees are observed on the host side and will then be reflected in localMemoryPoolUtilizedSize? Also, from the perspective of the GPU’s physical memory, does the continuously increasing localMemoryPoolUtilizedSize not put pressure on future memory allocations? ---
**Author:** ![aktcse14](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aktcse14](https://forums.developer.nvidia.com/u/aktcse14)\ **Post date:** [February 10, 2026, 5:39am UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/8 "2026-02-10T05:39:32Z")
Thank you for the reply. @tommyecguitar, @skottapalli @hwilper I am not totally sure. What I am trying to do is plot ( x axis time, y axis -GPU utilization) for some reason there is no simple support for that. Although nsys seems to be collecting everything I am using this as a guide: [User Guide — nsight-systems 2025.1 documentation](https://docs.nvidia.com/nsight-systems/2025.1/UserGuide/index.html#cuda-gpu-memory-allocation-graph) the 2026 version has a recipe as: nsys recipe gpu\_vram\_usage\_trace --input [report file path] > **[Post-Collection Analysis Guide — Nsight Systems](https://docs.nvidia.com/nsight-systems/AnalysisGuide/index.html)** ---
**Author:** ![aktcse14](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aktcse14](https://forums.developer.nvidia.com/u/aktcse14)\ **Post date:** [February 10, 2026, 5:45am UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/9 "2026-02-10T05:45:31Z")
Not totally sure when the cached allocation is freed from GPU. I was trying to know the difference between these 2, `localMemoryPoolUtilizedSize` ,`localMemoryTotal`. Thank you. ---
**Author:** ![skottapalli](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@skottapalli](https://forums.developer.nvidia.com/u/skottapalli)\ **Post date:** [February 10, 2026, 12:31pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/10 "2026-02-10T12:31:48Z")
> [@tommyecguitar](#): > > Is my understanding correct that once a synchronization point is reached somewhere, those memory frees are observed on the host side and will then be reflected in localMemoryPoolUtilizedSize? I believe that is true. > What I am trying to do is plot ( x axis time, y axis -GPU utilization) > for some reason there is no simple support for that. > Although nsys seems to be collecting everything @aktcse14 - please see the screenshot below. You should be able to see the blue graph in the timeline view. It is an aggregate row which roughly represents CUDA GPU utilization in terms of CUDA kernel coverage. Is this what you are looking for? ![image](https://global.discourse-cdn.com/nvidia/original/4X/e/1/9/e1987f6278f9b0446066539bbf147170b19b176e.jpeg) ---
**Author:** ![skottapalli](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@skottapalli](https://forums.developer.nvidia.com/u/skottapalli)\ **Post date:** [February 10, 2026, 12:36pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/11 "2026-02-10T12:36:07Z")
> from the perspective of the GPU’s physical memory, does the continuously increasing localMemoryPoolUtilizedSize not put pressure on future memory allocations? The increasing localMemoryPoolUtilizedSize may be due to the asynchronous memory allocations that were scheduled, but not executed yet. If so, those will not be released by cudaMemoryPoolTrimTo and they will be treated as outstanding i.e. it counts towards the memory in use from the memory pool. If you are trying to do future memory allocations from the pool, then it would reduce the memory available for such allocations, yes. ---
**Author:** ![aktcse14](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aktcse14](https://forums.developer.nvidia.com/u/aktcse14)\ **Post date:** [February 10, 2026, 5:56pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/12 "2026-02-10T17:56:58Z")
@skottapalli Thank you for the reply, but what I am getting is this It has nothing in the top GPU A100 row. ![image](https://global.discourse-cdn.com/nvidia/original/4X/9/3/5/935ce2fe438369fdef7975c72df127a5df303a6d.png) ---
**Author:** ![skottapalli](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@skottapalli](https://forums.developer.nvidia.com/u/skottapalli)\ **Post date:** [February 10, 2026, 6:13pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/13 "2026-02-10T18:13:30Z")
Can you share the report file? You might want to scroll down to find the CUDA kernels and see the aggregate row for it. Your screenshot shows GPU metrics only and the top-level row for it. ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [February 24, 2026, 6:14pm UTC](https://forums.developer.nvidia.com/t/understanding-the-meaning-of-localmemorypoolutilizedsize-in-cuda-with-cudamallocasync-and-cudafreeasync/247700/14 "2026-02-24T18:14:07Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.