I’m working on a DeepStream pipeline on a Jetson Orin NX (16GB), but I keep running into an issue where the application fails due to memory copy errors after running for a period of time (sometimes after 1 minute, sometimes after 10). I’m hoping someone can help me identify what might be going wrong.
Environment
• Hardware Platform (Jetson / GPU) Jetson Orin NX (16GB) • DeepStream Version 7.1 • JetPack Version (valid for Jetson only) 6.2 • TensorRT Version 10.3.0.30 • Issue Type( questions, new requirements, bugs) bugs
My guess is that the buffer’s memory is being accessed after it has been released by GStreamer. I’m not sure how you’re obtaining the buffer, but you might try using gst_buffer_ref to increase the buffer’s reference count. This ensures the buffer isn’t prematurely freed. Just make sure to call gst_buffer_unref once you’re done with it to avoid memory leaks.
Some other things you can try:
Replace appsink with fakesink to verify that the illegal access is located in your appsink.
Set nvbuf-memory-type=0, you should use this unless there is a memory issue.
Try using cuda-memcheck ./your_app.
Enable GStreamer debug to get more information export GST_DEBUG=3.
I’ll definitely try replacing appsink with fakesink to check whether the illegal access is happening downstream. Also planning to run cuda-memcheck ./app and will report back with any specific findings.
As for gst_buffer_ref(), I’m not manually accessing buffers in the pipeline — it’s all handled through GStreamer elements — but I’ll inspect any interaction around appsink just in case.
For nvbuf-memory-type=0, I did test it and enabled GST_DEBUG=3. Unfortunately, I still got the following errors:
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform_copy.cpp:438 => Failed in mem copy
cudaErrorIllegalAddress
cuDCFFrameTransformTexture.cu 693
gstnvtracker: Low-level tracker lib returned error 1
gst_nvinfer_process_full_frame: Buffer conversion failed
Have you only set this one memory-related parameter for the nvvideoconvert in the pipeline? You don’t need to set any other memory-related parameters for other plugins in the pipeline.
2.Could you narrow down the scope by reducing the number of plugins? Like replacing the v4l2src to the filesrc, delete the nvinfer and nvtracker plugins, etc…
Yes, I’ve only set the copy-hw=2 parameter on the nvvideoconvert element before nvinfer. There are no other memory-related parameters set for any other plugins in the pipeline.
I’ve tried narrowing down the pipeline by removing plugins. Through this process, it seems the issue might be related to the appsink. Interestingly, even when I replace appsink with autovideosink, the problem still persists.
There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks