NITROS image buffer reuse on Jetson with PREEMPT_RT: custom cache vs. CUDA memory pool

Robotics & Edge Computing Robotics - Isaac Isaac ROS
1

Hello NVIDIA team and community,

I would appreciate some guidance on a GPU memory reuse change in our NITROS image pipeline. I am trying to understand the intended design and whether we should use an existing mechanism before maintaining a custom implementation.

Environment and test scope

  • Jetson AGX Orin

  • Jetson Linux / L4T R36.4.3, CUDA 12.6

  • Kernel: 5.15.148-rt-tegra, PREEMPT_RT

  • ROS 2 Humble

  • Isaac ROS / NITROS version and commit: [please insert exact version/commit]

  • System configuration: three RealSense cameras—front-top, front-bottom, and rear—with separate camera containers.

  • Evaluated path: front-top camera only, using 1280 Ă— 720 RGB8 at 30 FPS.

  • The front-top container was restricted to logical CPU 5 (CpusetCpus=5), shared by the camera and other processes inside that container.

The relevant pipeline is:

RealSense → sensor_msgs::msg::Image → NITROS image adapter → H.264 encoder

The change concerns GPU image allocation in the ROS-to-NITROS adapter, rather than the encoder’s output pool. The results below do not represent a controlled three-camera comparison or validation of the change across all cameras.

Observation and motivation

Our saved pre-change adapter calls cudaMallocAsync for each incoming image and cudaFreeAsync from its cleanup callback.

CPU profiling showed samples in this allocation path, including underlying Jetson nvmap allocation work. We investigated whether retaining completed image allocations could reduce this overhead during continuous streaming.

I understand that cudaMallocAsync already uses a CUDA memory pool. However, we did not capture the effective pool release threshold during the original measurement. We therefore cannot establish whether pool configuration or synchronization behavior caused the observed overhead.

The saved baseline may also contain local changes, so I do not want to attribute this behavior to an unmodified upstream release without confirming that.

Experimental change

We added a cache at the image adapter boundary:

  • Reuse an idle buffer only when its byte size matches the requested image size.

  • Call cudaMalloc when no matching buffer is available.

  • Hold the allocation through a shared-ownership lease attached to the image buffer’s external deleter.

  • Preserve the existing NitrosBuffer GPU-event cleanup mechanism. The intended behavior is to recycle only after all registered GPU reads and writes have completed.

  • Retain up to 64 MiB of idle buffers per process/device, freeing excess buffers. This does not limit live frames or total process memory.

  • Keep the host-to-device image copy.

The intent is to reduce repeated allocation work while preserving the existing buffer lifetime requirements.

Preliminary evidence and limitations

Short profiling captures on logical CPU 5 recorded:

Stack path Before: 3,465 samples After: 3,121 samples
cudaMallocAsync 19.77% 0 samples
NITROS image convert_to_custom 21.39% 2.60%
nvmap_alloc_handle, all sources 18.41% 2.02%

These are inclusive stack-sample percentages on the sampled CPU, not allocation counts or whole-system CPU utilization. Overlapping paths should not be added together. Replacing the adapter’s async allocation calls also means their disappearance alone does not establish a performance benefit.

The captures were taken at different times on a shared system, and another camera-path change was also present. We have not completed an isolated comparison against a properly configured CUDA async pool, a comparison with a non-RT kernel, or full multi-camera endurance testing.

Questions

  1. Preferred mechanism: For fixed-resolution image streaming, should we first investigate the existing CUDA pool and cudaMemPoolAttrReleaseThreshold, or is there a NITROS-provided image/buffer pool we should use? Which settings and synchronization points would you check first?

  2. Ownership and correctness: Is retaining an allocation through the external deleter compatible with the intended NitrosBuffer lifecycle? What should we verify for safe recycling with multiple subscribers, different CUDA streams, and node shutdown?

  3. RT kernel and CPU affinity: Are there additional considerations for allocation, cleanup, or profiling under PREEMPT_RT with a camera container restricted to one logical CPU? Would you recommend checking kernel configuration or relaxing CPU affinity before drawing conclusions about allocator overhead? We have not established either as the cause.

  4. Tradeoffs and validation: What drawbacks should we evaluate compared with the native async allocator—for example, synchronization on cache misses, memory retention across separate camera processes, or interference with other GPU workloads? Which measurements would make the comparison most useful?

I have read the CUDA stream-ordered allocator explanation and the discussion on GPU memory management with NITROS nodes. I am still learning how these mechanisms fit together in a running NITROS pipeline.

If using or configuring the existing mechanisms is the better approach, I would be happy to simplify our implementation. Any corrections or pointers would be greatly appreciated.

Thank you for your time and guidance.

2

Hello @xoy232,

Welcome to the Isaac ROS forum and thanks for sharing your findings.
We can offer some general guidance:

  1. Preferred mechanism: We recommend evaluating the existing CUDA async pool and its release threshold first. Synchronization can affect when the pool returns memory to the system. A NITROS buffer pool may be another option if one is available in your build.
  2. Ownership: Before recycling a buffer through an external deleter, ensure all GPU reads and writes have finished. Pay particular attention to multiple subscribers, different CUDA streams, and cleanup during shutdown.
  3. PREEMPT_RT and affinity: The current measurements do not establish that either causes the allocation overhead. Testing with less restrictive CPU affinity may help identify contention on logical CPU 5
  4. Tradeoffs: Compare the custom cache with a configured CUDA async pool under the same workload. Measure allocation counts, retained memory, frame latency, and dropped frames, including a sustained run with all three cameras. If the existing pool meets your goals, simplifying the implementation would be a reasonable choice.
3

Hello,

Thank you for the guidance. This helps clarify the next steps.

I will first confirm the exact NITROS revision and the effective CUDA async pool settings, including the release threshold. I will then compare the original configuration, a configured async pool, and the custom cache under the same workload, starting with one camera before extending the test to all three.

I will evaluate CPU affinity separately so that changes in CPU contention are not mixed with changes in allocation behavior. I also understand that safe recycling still needs to be verified across GPU streams, multiple subscribers, and shutdown; the current profiling results do not establish that by themselves.

One clarification: could you point me to the relevant NITROS buffer pool API or an example, if available? I will check whether it applies to our exact revision before making further changes.

I appreciate your suggestion to prefer the existing mechanisms if they meet our requirements. I will follow up with the configuration details and a controlled comparison once available.

Thank you again for your help.

4

Thank you for the update. I should clarify my earlier suggestion about a NITROS buffer pool.

The public Isaac ROS 4.6 source provides CUDAMemoryPool, a fixed-size block pool with usage guidance in the header, and NitrosImage::from_pool() for acquiring image storage from it.
However, Isaac ROS 4.6 targets ROS 2 Jazzy ( Getting Started — Isaac ROS ), so these references do not establish compatibility with your Humble build.

If you’re using Isaac ROS 3.2, the relevant reusable pool is GXF’s nvidia::gxf::BlockMemoryPool. Image storage can use it through gxf::VideoBuffer::resize() or resizeCustom() with a GXF allocator handle. However, the stock 3.2 image adapter explicitly selects UnboundedAllocator. Using BlockMemoryPool in that path would therefore require coordinated source and configuration changes, this might not be an existing adapter pool setting.

5

Thank you for the clarification and the source references. They were very helpful.

We reviewed the upstream 3.2 and 4.6 implementations and confirmed the distinction you described. We also found that our local Humble integration already includes 4.6-era NITROS code, so I should clarify that it is not an unmodified 3.2 build.

We understand that our custom cache shares the buffer-reuse principle, but differs from the fixed-size CUDAMemoryPool. The default 4.6 image adapter also still uses CUDA async allocation, so the pool API alone does not validate our implementation.

Since we are planning a platform upgrade, we plan to retain the current cache as an interim solution and close this phase of optimization. During the upgrade, we will reassess whether the custom cache is still needed and include GPU buffer lifetime, shutdown behavior, and sustained three-camera operation in the integration validation.

Thank you again for your guidance—it helped us better understand the implementation and define an appropriate scope for the remaining work.