# NITROS image buffer reuse on Jetson with PREEMPT\_RT: custom cache vs. CUDA memory pool

**URL:** <https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728>\
**Category:** Isaac ROS\
**Tags:** cuda\
**Created:** [September 30, 2026, 3:27am UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728 "2026-09-30T03:27:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![xoy232](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xoy232/32/542446_2.png) [@xoy232](https://forums.developer.nvidia.com/u/xoy232)\
**Post date:** [September 30, 2026, 3:27am UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728/1 "2026-09-30T03:27:38Z")

</div>

Hello NVIDIA team and community,

I would appreciate some guidance on a GPU memory reuse change in our NITROS image pipeline. I am trying to understand the intended design and whether we should use an existing mechanism before maintaining a custom implementation.

**Environment and test scope**

- Jetson AGX Orin

- Jetson Linux / L4T R36.4.3, CUDA 12.6

- Kernel: `5.15.148-rt-tegra`, `PREEMPT_RT`

- ROS 2 Humble

- Isaac ROS / NITROS version and commit: [please insert exact version/commit]

- System configuration: three RealSense cameras—front-top, front-bottom, and rear—with separate camera containers.

- Evaluated path: **front-top camera only** , using **1280 × 720 RGB8 at 30 FPS**.

- The front-top container was restricted to **logical CPU 5** (`CpusetCpus=5`), shared by the camera and other processes inside that container.

The relevant pipeline is:

`RealSense → sensor_msgs::msg::Image → NITROS image adapter → H.264 encoder`

The change concerns GPU image allocation in the ROS-to-NITROS adapter, rather than the encoder’s output pool. The results below do not represent a controlled three-camera comparison or validation of the change across all cameras.

**Observation and motivation**

Our saved pre-change adapter calls `cudaMallocAsync` for each incoming image and `cudaFreeAsync` from its cleanup callback.

CPU profiling showed samples in this allocation path, including underlying Jetson `nvmap` allocation work. We investigated whether retaining completed image allocations could reduce this overhead during continuous streaming.

I understand that `cudaMallocAsync` already uses a CUDA memory pool. However, we did **not** capture the effective pool release threshold during the original measurement. We therefore cannot establish whether pool configuration or synchronization behavior caused the observed overhead.

The saved baseline may also contain local changes, so I do not want to attribute this behavior to an unmodified upstream release without confirming that.

**Experimental change**

We added a cache at the image adapter boundary:

- Reuse an idle buffer only when its byte size matches the requested image size.

- Call `cudaMalloc` when no matching buffer is available.

- Hold the allocation through a shared-ownership lease attached to the image buffer’s external deleter.

- Preserve the existing `NitrosBuffer` GPU-event cleanup mechanism. The intended behavior is to recycle only after all registered GPU reads and writes have completed.

- Retain up to **64 MiB of idle buffers per process/device** , freeing excess buffers. This does not limit live frames or total process memory.

- Keep the host-to-device image copy.

The intent is to reduce repeated allocation work while preserving the existing buffer lifetime requirements.

**Preliminary evidence and limitations**

Short profiling captures on **logical CPU 5** recorded:

| Stack path | Before: 3,465 samples | After: 3,121 samples |
| --- | --- | --- |
| `cudaMallocAsync` | 19.77% | 0 samples |
| NITROS image `convert_to_custom` | 21.39% | 2.60% |
| `nvmap_alloc_handle`, all sources | 18.41% | 2.02% |

These are inclusive stack-sample percentages on the sampled CPU, not allocation counts or whole-system CPU utilization. Overlapping paths should not be added together. Replacing the adapter’s async allocation calls also means their disappearance alone does not establish a performance benefit.

The captures were taken at different times on a shared system, and another camera-path change was also present. We have not completed an isolated comparison against a properly configured CUDA async pool, a comparison with a non-RT kernel, or full multi-camera endurance testing.

**Questions**

1. **Preferred mechanism:** For fixed-resolution image streaming, should we first investigate the existing CUDA pool and `cudaMemPoolAttrReleaseThreshold`, or is there a NITROS-provided image/buffer pool we should use? Which settings and synchronization points would you check first?

2. **Ownership and correctness:** Is retaining an allocation through the external deleter compatible with the intended `NitrosBuffer` lifecycle? What should we verify for safe recycling with multiple subscribers, different CUDA streams, and node shutdown?

3. **RT kernel and CPU affinity:** Are there additional considerations for allocation, cleanup, or profiling under `PREEMPT_RT` with a camera container restricted to one logical CPU? Would you recommend checking kernel configuration or relaxing CPU affinity before drawing conclusions about allocator overhead? We have not established either as the cause.

4. **Tradeoffs and validation:** What drawbacks should we evaluate compared with the native async allocator—for example, synchronization on cache misses, memory retention across separate camera processes, or interference with other GPU workloads? Which measurements would make the comparison most useful?

I have read the [CUDA stream-ordered allocator explanation](https://developer.nvidia.com/blog/using-cuda-stream-ordered-memory-allocator-part-1) and the discussion on [GPU memory management with NITROS nodes](https://forums.developer.nvidia.com/t/gpu-memory-management-with-isaac-ros-nitros-nodes/335398). I am still learning how these mechanisms fit together in a running NITROS pipeline.

If using or configuring the existing mechanisms is the better approach, I would be happy to simplify our implementation. Any corrections or pointers would be greatly appreciated.

Thank you for your time and guidance.

---

<div class="post-metadata">

**Author:** ![vchuang](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/vchuang/32/436710_2.png) [@vchuang](https://forums.developer.nvidia.com/u/vchuang)\
**Post date:** [October 2, 2026, 4:52am UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728/2 "2026-10-02T04:52:04Z")

</div>

Hello @xoy232,

Welcome to the Isaac ROS forum and thanks for sharing your findings.  
We can offer some general guidance:

1. **Preferred mechanism:** We recommend evaluating the existing CUDA async pool and its release threshold first. Synchronization can affect when the pool returns memory to the system. A NITROS buffer pool may be another option if one is available in your build.
2. **Ownership:** Before recycling a buffer through an external deleter, ensure all GPU reads and writes have finished. Pay particular attention to multiple subscribers, different CUDA streams, and cleanup during shutdown.
3. **PREEMPT\_RT and affinity:** The current measurements do not establish that either causes the allocation overhead. Testing with less restrictive CPU affinity may help identify contention on logical CPU 5
4. **Tradeoffs:** Compare the custom cache with a configured CUDA async pool under the same workload. Measure allocation counts, retained memory, frame latency, and dropped frames, including a sustained run with all three cameras. If the existing pool meets your goals, simplifying the implementation would be a reasonable choice.

---

<div class="post-metadata">

**Author:** ![xoy232](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xoy232/32/542446_2.png) [@xoy232](https://forums.developer.nvidia.com/u/xoy232)\
**Post date:** [October 2, 2026, 5:51am UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728/3 "2026-10-02T05:51:49Z")

</div>

Hello,

Thank you for the guidance. This helps clarify the next steps.

I will first confirm the exact NITROS revision and the effective CUDA async pool settings, including the release threshold. I will then compare the original configuration, a configured async pool, and the custom cache under the same workload, starting with one camera before extending the test to all three.

I will evaluate CPU affinity separately so that changes in CPU contention are not mixed with changes in allocation behavior. I also understand that safe recycling still needs to be verified across GPU streams, multiple subscribers, and shutdown; the current profiling results do not establish that by themselves.

One clarification: could you point me to the relevant NITROS buffer pool API or an example, if available? I will check whether it applies to our exact revision before making further changes.

I appreciate your suggestion to prefer the existing mechanisms if they meet our requirements. I will follow up with the configuration details and a controlled comparison once available.

Thank you again for your help.

---

<div class="post-metadata">

**Author:** ![vchuang](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/vchuang/32/436710_2.png) [@vchuang](https://forums.developer.nvidia.com/u/vchuang)\
**Post date:** [October 6, 2026, 7:41am UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728/4 "2026-10-06T07:41:09Z")

</div>

Thank you for the update. I should clarify my earlier suggestion about a NITROS buffer pool.

The public Isaac ROS 4.6 source provides [CUDAMemoryPool](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros/blob/release-4.6/isaac_ros_nitros/include/isaac_ros_nitros/types/cuda_memory_pool.hpp), a fixed-size block pool with usage guidance in the header, and [NitrosImage::from\_pool()](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros/blob/release-4.6/isaac_ros_nitros_type/isaac_ros_nitros_image_type/include/isaac_ros_nitros_image_type/nitros_image.hpp) for acquiring image storage from it.  
However, Isaac ROS 4.6 targets ROS 2 Jazzy ( [Getting Started — Isaac ROS](https://nvidia-isaac-ros.github.io/v/release-4.6/getting_started/index.html) ), so these references do not establish compatibility with your Humble build.

If you’re using Isaac ROS 3.2, the relevant reusable pool is GXF’s [nvidia::gxf::BlockMemoryPool](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_nitros/blob/release-3.2/isaac_ros_gxf/gxf/core/include/gxf/std/block_memory_pool.hpp). Image storage can use it through gxf::VideoBuffer::resize() or resizeCustom() with a GXF allocator handle. However, the stock 3.2 image adapter explicitly selects UnboundedAllocator. Using BlockMemoryPool in that path would therefore require coordinated source and configuration changes, this might not be an existing adapter pool setting.

---

<div class="post-metadata">

**Author:** ![xoy232](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xoy232/32/542446_2.png) [@xoy232](https://forums.developer.nvidia.com/u/xoy232)\
**Post date:** [October 6, 2026, 9:34am UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728/5 "2026-10-06T09:34:41Z")

</div>

Thank you for the clarification and the source references. They were very helpful.

We reviewed the upstream 3.2 and 4.6 implementations and confirmed the distinction you described. We also found that our local Humble integration already includes 4.6-era NITROS code, so I should clarify that it is not an unmodified 3.2 build.

We understand that our custom cache shares the buffer-reuse principle, but differs from the fixed-size CUDAMemoryPool. The default 4.6 image adapter also still uses CUDA async allocation, so the pool API alone does not validate our implementation.

Since we are planning a platform upgrade, we plan to retain the current cache as an interim solution and close this phase of optimization. During the upgrade, we will reassess whether the custom cache is still needed and include GPU buffer lifetime, shutdown behavior, and sustained three-camera operation in the integration validation.

Thank you again for your guidance—it helped us better understand the implementation and define an appropriate scope for the remaining work.

---

<div class="post-metadata">

**Author:** ![jhonwalker000091](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jhonwalker000091](https://forums.developer.nvidia.com/u/jhonwalker000091)\
**Post date:** [October 14, 2026, 2:40pm UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728/6 "2026-10-14T14:40:54Z")

</div>

For this Jetson setup, I’d compare the two approaches based on predictable latency, memory reuse, and synchronization overhead. A custom buffer cache can provide tighter control over allocation and reuse, which may be useful under PREEMPT\_RT, while CUDA memory pools can simplify management and reduce repeated allocation costs. The key is to benchmark both under the actual workload, including buffer lifetime, synchronization, and worst-case latency—not just average throughput. Also, ensure buffers aren’t reused until all GPU operations accessing them have completed.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [October 28, 2026, 2:41pm UTC](https://forums.developer.nvidia.com/t/nitros-image-buffer-reuse-on-jetson-with-preempt-rt-custom-cache-vs-cuda-memory-pool/384728/7 "2026-10-28T14:41:30Z")

</div>

Closed on October 28, 2026, 2:41pm UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
