Jetson Orin + DeepStream 7.1: Random cudaErrorIllegalAddress crash with YOLOv9 and USB Camera

amar8 · May 6, 2025, 1:13am

• Hardware Platform (Jetson / GPU)
Jetson Orin 16 GB

• DeepStream Version
7.1

• JetPack Version (valid for Jetson only)
6.1

• TensorRT Version
10.3.0.30

• Linux Version
36.4.0

• Issue Type (questions, new requirements, bugs)
Bug

• How to Reproduce the Issue (for bugs: include sample app used, config files, command line, etc.)
I’m using DeepStream Python bindings to run a YOLOv9 model on the Jetson Orin. The pipeline ingests video from a USB webcam and outputs an RTSP stream. Depending on the model variant and power settings, the application crashes after 1–15 minutes with a cudaErrorIllegalAddress error.

I’ve attached:

The full terminal output with the error message
The Python script I’m running DeepStream-Python-Script.txt (15.5 KB)

For model conversion, I followed the ONNX export instructions here:

github.com/marcoslucianops/DeepStream-Yolo

docs/YOLOv9.md

master

# YOLOv9 usage

**NOTE**: The yaml file is not required.

* [Convert model](#convert-model)
* [Compile the lib](#compile-the-lib)
* [Edit the config_infer_primary_yoloV9 file](#edit-the-config_infer_primary_yolov9-file)
* [Edit the deepstream_app_config file](#edit-the-deepstream_app_config-file)
* [Testing the model](#testing-the-model)

##

### Convert model

#### 1. Download the YOLOv9 repo and install the requirements

```
git clone https://github.com/WongKinYiu/yolov9.git
cd yolov9
pip3 install -r requirements.txt

This file has been truncated. show original

What I’ve Tried (but still experiencing cudaErrorIllegalAddress)

Cloned the setup on identical hardware
Swapped the SSD
Tested with both JetPack 6.1 and 6.2
Ran DeepStream 7.1 both natively and in the official 7.1-triton-multiarch Docker container
Tested on Linux versions including and up to 36.4.0

I initially used JetPack 6.2, but encountered a similar issue mentioned here:

That thread suggests DeepStream 7.1 may not be fully compatible with JetPack 6.2, so I downgraded to JetPack 6.1 — but the issue persists.

This setup is fairly standard, and I’m at a loss as to why the RTSP stream keeps crashing with this CUDA illegal memory access error. Any guidance or insights would be greatly appreciated. I’m happy to provide further details as needed.

jetson@jetson:~/Documents/Video-Pipeline/usb_in_rtsp_out_ai$ python3 usb_in_rtsp_out_ai.py -i /dev/video0 
Creating Pipeline 
 
Creating Source 
 
Creating Video Converter 

Creating H264 Encoder
Is it Integrated GPU? : 1
Creating H264 rtppay
Playing cam %s  /dev/video0
Adding elements to Pipeline 

Linking elements in the Pipeline 


 *** DeepStream: Launched RTSP Streaming at rtsp://droneIP:8554/wx-video ***


Starting pipeline 

Opening in BLOCKING MODE 
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
[NvMultiObjectTracker] Initialized
Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
0:00:00.518013357  7655 0xaaaaf644a890 INFO                 nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:2092> [UID = 1]: deserialized trt engine from :/home/jetson/Documents/Video-Pipeline/usb_in_rtsp_out_ai/model_b1_gpu0_fp32.engine
Implicit layer support has been deprecated
INFO: [Implicit Engine Info]: layers num: 0

0:00:00.518094606  7655 0xaaaaf644a890 INFO                 nvinfer gstnvinfer.cpp:684:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2195> [UID = 1]: Use deserialized engine model: /home/jetson/Documents/Video-Pipeline/usb_in_rtsp_out_ai/model_b1_gpu0_fp32.engine
0:00:00.530466405  7655 0xaaaaf644a890 INFO                 nvinfer gstnvinfer_impl.cpp:343:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:config_infer_primary_yoloV9.txt sucessfully
NvMMLiteOpen : Block : BlockType = 4 
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
sys:1: Warning: g_object_get_is_valid_property: object class 'GstUDPSrc' has no property named 'pt'
H264: Profile = 66 Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000 
NvVideo: bBlitMode is set to TRUE 
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform_copy.cpp:341: => Failed in mem copy

0:02:00.128778418  7655 0xffff2c001d40 ERROR                nvinfer gstnvinfer.cpp:1267:get_converted_buffer:<primary-inference> cudaMemset2DAsync failed with error cudaErrorIllegalAddress while converting buffer
0:02:00.128859669  7655 0xffff2c001d40 WARN                 nvinfer gstnvinfer.cpp:1576:gst_nvinfer_process_full_frame:<primary-inference> error: Buffer conversion failed
Error: gst-stream-error-quark: Buffer conversion failed (1): /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1576): gst_nvinfer_process_full_frame (): /GstPipeline:pipeline0/GstNvInfer:primary-inference
nvstreammux: Successfully handled EOS for source_id=0
0:02:00.134368403  7655 0xffff2c001d40 ERROR                nvinfer gstnvinfer.cpp:1267:get_converted_buffer:<primary-inference> cudaMemset2DAsync failed with error cudaErrorIllegalAddress while converting buffer
0:02:00.134410709  7655 0xffff2c001d40 WARN                 nvinfer gstnvinfer.cpp:1576:gst_nvinfer_process_full_frame:<primary-inference> error: Buffer conversion failed
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
CUDA Runtime error cudaFreeHost(host_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:78
CUDA Runtime error cudaFree(device_) # an illegal memory access was encountered, code = cudaErrorIllegalAddress [ 700 ] in file /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvll_osd/memory.hpp:79
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)
[ERROR] 2025-05-05 23:31:55 Error destroying cuda device: %��\��
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)
....
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)
[ERROR] 2025-05-05 23:31:55 Error destroying cuda device: ���\��
[WARN ] 2025-05-05 23:31:55 (cudaErrorIllegalAddress)

terminate called after throwing an instance of 'nv::cuda::RuntimeException'
  what():  cudaErrorIllegalAddress: 
Aborted (core dumped)

yuweiw · May 8, 2025, 2:00am

Could you run the command below to check the memory usage when you are runing your sample?

sudo pip3 install jetson-stats
jtop

You can narrow down by following the several steps below.

Run our deepstream_test_1_usb.py directly and see if there is same issue.
Replace the model in the sample with your yolov9 and see if there is same issue.
Add the rtsp out in the sample and see if there is same issue.

Topic		Replies	Views
AGX Orin Crashing on random intervals DeepStream SDK deepstream	2	33	April 14, 2025
Pipeline crashes after an illegal memory access DeepStream SDK gstreamer , jetson , deepstream	7	96	February 19, 2025
gStreamer python face detection pipeline crashes after running a while DeepStream SDK gstreamer , python , deepstream	3	41	February 11, 2025
AI NVR configuration issues with DeepStream 7.1 and YOLOv8 on Jetson Orin NX Metropolis Microservices for Jetson jetson , deepstream , jetson-platform-services	8	149	January 13, 2025
TensorRT and DeepStream 7.1 running problems on USB Camera DeepStream SDK jetson , deepstream	11	139	December 23, 2024
Deepstreem installation fails DeepStream SDK gstreamer , python	16	991	October 14, 2022
Unstable deepstream rtsp pipeline - CUDA runtime error DeepStream SDK jetson , deepstream	7	45	April 1, 2025
Deepstream python code - Failed in mem copy (Segmentation fault) DeepStream SDK deepstream	5	27	May 12, 2025
ERROR: [TRT]: 10: Could not find any implementation for node /0/model.24/Range DeepStream SDK	12	487	July 9, 2024
Nvbufsurface: Failed to create EGLImage & ERROR from primary_gie: Failed to set buffer pool to active DeepStream SDK	12	244	September 23, 2024

Jetson Orin + DeepStream 7.1: Random cudaErrorIllegalAddress crash with YOLOv9 and USB Camera

Related topics