Depth alignement in Realsensee D435 triggering CUDA 700 Memory failure

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
1

Hello everyone !

I am using a Jetson Orin Nano Development board connected to 4 realsense D435 cameras. I get teh 700 CUDA memory error intermittently when I run an infiniteit while loop while calling the segmentaton inference model. I noticed that the error is caused by teh align.process(frames) line which converts teh depth image shape (originally: 1080, 1920) to match teh color image shape (1080, 1920, 3). What could I do to prvent it from happening ?

2

*** Please note that this reply is generated by LLM automatically ***

The issue you’re experiencing with the Jetson Orin Nano Development board and the 700 CUDA memory error is likely related to the memory allocation and deallocation process. The error is occurring intermittently when you run an infinite while loop while calling the segmentation inference model, specifically at the alignprocessframes line.

This line is converting the depth image shape from (1080, 1920) to match the color image shape (1080, 1920, 3). This process requires additional memory allocation, which might be causing the error.

To prevent this error from happening, you can try the following:

  1. Memory Optimization: Review your code and optimize memory allocation and deallocation. Ensure that you’re not allocating too much memory at once, and that you’re properly deallocating memory when it’s no longer needed.
  2. Batching: Consider batching your inference requests to reduce the frequency of memory allocation and deallocation. This can help reduce the likelihood of the error occurring.
  3. Model Optimization: Optimize your segmentation inference model to reduce its memory requirements. This might involve reducing the model’s complexity, using a more efficient architecture, or pruning unnecessary weights.
  4. CUDA Memory Management: Ensure that you’re properly managing CUDA memory using cudaMalloc, cudaFree, and cudaMemcpy. Make sure to synchronize your CUDA streams and events to avoid memory access conflicts.
  5. Jetson Orin Nano Specific: As you’re using the Jetson Orin Nano Development board, you might want to explore the board’s specific memory management features, such as the jetson-io tool, to optimize memory allocation and deallocation.

Comparing your issue to similar topics in the provided context:

  • The topic “[RCE] ISR PHY 0 CIL_A 0x88” discusses an issue with the Jetson Orin Nano and camera-related errors. While the specific error is different, the discussion around memory management and optimization might be relevant to your issue.
  • The topic “Capturing Image from custom sensor fails with 4-csi lanes” discusses an issue with capturing images from a custom sensor on the Jetson Orin NX. While the specific error is different, the discussion around memory management and optimization might be relevant to your issue.

In summary, to prevent the 700 CUDA memory error from occurring, you should focus on optimizing memory allocation and deallocation, batching inference requests, optimizing your segmentation inference model, and properly managing CUDA memory. Additionally, exploring Jetson Orin Nano specific memory management features might be beneficial.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

How can I optimise memory allocation & deallocation ?

6

Hi,

Error 700 is cudaErrorIllegalAddress
It means the kernel is making an illegal, out-of-bounds access.

Please double-check your source to see if any issue in the GPU implementation.

Thanks.

7

Hi,

Thsi is the error I am getting. It occurs only during teh color-depth alignmnet call from teh Realsense SDK & does not appear when I remove teh line. ERROR: 2025-09-29 19:55:24.409947520 [E:onnxruntime:Default, cuda_call.cc:123 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=65 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
2025-09-29 19:55:24.410040128 [E:onnxruntime:Default, cuda_call.cc:123 CudaCall] CUDA failure 700: an illegal memory access was encountered ; GPU=0 ; hostname=metazet-desktop ; file=/opt/onnxruntime/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_));

8

Hi,

It occurs only during teh color-depth alignmnet call from teh Realsense SDK & does not appear when I remove teh line.

According to this, it’s recommended to check with RealSense team.

Thanks