Jetson Orin + DeepStream 7.1: Random cudaErrorIllegalAddress crash with YOLOv9 and USB Camera

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, , ,
1

• Hardware Platform (Jetson / GPU)
Jetson Orin 16 GB

• DeepStream Version
7.1

• JetPack Version (valid for Jetson only)
6.1

• TensorRT Version
10.3.0.30

• Linux Version
36.4.0

• Issue Type (questions, new requirements, bugs)
Bug

• How to Reproduce the Issue (for bugs: include sample app used, config files, command line, etc.)
I’m using DeepStream Python bindings to run a YOLOv9 model on the Jetson Orin. The pipeline ingests video from a USB webcam and outputs an RTSP stream. Depending on the model variant and power settings, the application crashes after 1–15 minutes with a cudaErrorIllegalAddress error.

I’ve attached:

For model conversion, I followed the ONNX export instructions here:

What I’ve Tried (but still experiencing cudaErrorIllegalAddress)

  • Cloned the setup on identical hardware
  • Swapped the SSD
  • Tested with both JetPack 6.1 and 6.2
  • Ran DeepStream 7.1 both natively and in the official 7.1-triton-multiarch Docker container
  • Tested on Linux versions including and up to 36.4.0

I initially used JetPack 6.2, but encountered a similar issue mentioned here:

That thread suggests DeepStream 7.1 may not be fully compatible with JetPack 6.2, so I downgraded to JetPack 6.1 — but the issue persists.

This setup is fairly standard, and I’m at a loss as to why the RTSP stream keeps crashing with this CUDA illegal memory access error. Any guidance or insights would be greatly appreciated. I’m happy to provide further details as needed.

jetson@jetson:~/Documents/Video-Pipeline/usb_in_rtsp_out_ai$ python3 usb_in_rtsp_out_ai.py -i /dev/video0 
Creating Pipeline 
 
Creating Source 
 
Creating Video Converter 

Creating H264 Encoder
Is it Integrated GPU? : 1
Creating H264 rtppay
Playing cam %s  /dev/video0
Adding elements to Pipeline 

Linking elements in the Pipeline 


 *** DeepStream: Launched RTSP Streaming at rtsp://droneIP:8554/wx-video ***


Starting pipeline 

Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Initialized
Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
0:00:00.518013357  7655 0xaaaaf644a890 INFO                 nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/home/jetson/Documents/Video-Pipeline/usb_in_rtsp_out_ai/model_b1_gpu0_fp32.engine
Implicit layer support has been deprecated
INFO: [Implicit Engine Info]: layers num: 0

0:00:00.518094606  7655 0xaaaaf644a890 INFO                 nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /home/jetson/Documents/Video-Pipeline/usb_in_rtsp_out_ai/model_b1_gpu0_fp32.engine
0:00:00.530466405  7655 0xaaaaf644a890 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:config_infer_primary_yoloV9.txt sucessfully
NvMMLiteOpen : Block : BlockType = 4 
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
sys:1: Warning: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'
H264: Profile = 66 Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000 
NvVideo: bBlitMode is set to TRUE 
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform_copy.cpp:341: => Failed in mem copy

0:02:00.128778418  7655 0xffff2c001d40 ERROR                nvinfer gstnvinfer.cpp:1267:get_converted_buffer:<primary-inference> cudaMemset2DAsync failed with error cudaErrorIllegalAddress while converting buffer
0:02:00.128859669  7655 0xffff2c001d40 WARN                 nvinfer gstnvinfer.cpp:1576:gst_nvinfer_process_full_frame:<primary-inference> error: Buffer conversion failed
Error: gst-stream-error-quark: Buffer conversion failed (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1576): gst_nvinfer_process_full_frame (): /GstPipeline:pipeline0/GstNvInfer:primary-inference
nvstreammux: Successfully handled EOS for source_id=0
0:02:00.134368403  7655 0xffff2c001d40 ERROR                nvinfer gstnvinfer.cpp:1267:get_converted_buffer:<primary-inference> cudaMemset2DAsync failed with error cudaErrorIllegalAddress while converting buffer
0:02:00.134410709  7655 0xffff2c001d40 WARN                 nvinfer gstnvinfer.cpp:1576:gst_nvinfer_process_full_frame:<primary-inference> error: Buffer conversion failed
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)
[ERROR] 2025-05-05 23:31:55 Error destroying cuda device: %��\��
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)
....
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)
[ERROR] 2025-05-05 23:31:55 Error destroying cuda device: ���\��
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)

terminate called after throwing an instance of 'nv::cuda::RuntimeException'
  what():  cudaErrorIllegalAddress: 
Aborted (core dumped)