gStreamer python face detection pipeline crashes after running a while

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Jetson Orin 4012, NVIDIA Jetson Orin NX Bundle, 8x 2GHz, 16GB DDR5
• DeepStream Version
Container: deepstream:7.1-triton-multiarch
• JetPack Version (valid for Jetson only)
see Container: deepstream:7.1-triton-multiarch
• TensorRT Version
see Container: deepstream:7.1-triton-multiarch
• NVIDIA GPU Driver Version (valid for GPU only)
Container: deepstream:7.1-triton-multiarch
• Issue Type( questions, new requirements, bugs)
Bug/Crash
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
-Run docker container
-Create and run following gStreamer pipeline in python:
gst-launch-1.0 v4l2src device=/dev/video0 !
queue max-size-buffers=3 leaky=downstream max-size-time=500000000 !
videorate !
video/x-raw, framerate=30/1 !
nvvideoconvert src-crop=0:0:1920:1080 !
m.sink_0 nvstreammux name=m batch-size=1 live-source=1 width=1280 height=720 !
nvinfer config-file-path=configs/facedetect.yml !
nvvideoconvert !
fakesink
-Set pipeline state to PLAYING and wait for crash to happen (usually a few minutes)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

When running above gStreamer pipeline for a while i always encounter the following error, leading to crashing the pipeline, after a while.

/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform_copy.cpp:341: => Failed in mem copy

ERROR: Failed to make stream wait on event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
0:01:20.167368168   129 0xaaaad7abf1f0 WARN                 nvinfer gstnvinfer.cpp:1418:gst_nvinfer_input_queue_loop:<nvinfer> error: Failed to queue input batch for inferencing
10.02. 10:51:47.591 - ERROR: [129 FacenetPipeline] Error: gst-stream-error-quark: Failed to queue input batch for inferencing (1), /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1418): gst_nvinfer_input_queue_loop (): /GstPipeline:facenet-pipeline/GstNvInfer:nvinfer

After which a lot of illegal memory address accesses are reported:

ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::61] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::61] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::94] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::61] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: cudaStreamDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaStreamDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaStreamDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress

Does the pipeline need another queue plugin somewhere to fix the “Failed to queue input batch for inferencing” error?

When catching this behaviour, trying to restart the pipeline when the error is happening the output i get is:

10.02. 10:51:47.668 - INFO: [129 FacenetPipeline] Restarting pipeline..
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
INFO: [Implicit Engine Info]: layers num: 3
0   INPUT  kFLOAT input_1         3x416x736       
1   OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46         
2   OUTPUT kFLOAT output_cov/Sigmoid 1x26x46         

Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.

after which the application freezes.

Currently thats a huge blocker for me and i would appreciate any advice.

3

What’s the Jetpack version on your device? Have you ever tried running the pipeline directly on the host without using docker?

4

Since we dont want to go through the Deepstream installation process, we would like to keep the application dockerized. Thats why i am not able to run the pipeline on the host system easily…

I found multiple ways to check for JetPack version on my system:
Running

cat /etc/nv_tegra_release

on host & inside docker gives:

# R36 (release), REVISION: 4.3, GCID: 38968081, BOARD: generic, EABI: aarch64, DATE: Wed Jan  8 01:49:37 UTC 2025
# KERNEL_VARIANT: oot
TARGET_USERSPACE_LIB_DIR=nvidia
TARGET_USERSPACE_LIB_DIR_PATH=usr/lib/aarch64-linux-gnu/nvidia

Running

dpkg-query --show nvidia-l4t-core

on host gives:

nvidia-l4t-core 36.4.3-20250107174145

in docker gives:

dpkg-query: no packages found matching nvidia-l4t-core

Running

dpkg -l | grep 'nvidia-l4t-core'

on host gives:

ii  nvidia-l4t-core                                             36.4.3-20250107174145                       arm64        NVIDIA Core Package

Running

sudo apt-cache show nvidia-jetpack

on host gives

Source: nvidia-jetpack (6.2)
Version: 6.2+b77
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.2+b77), nvidia-jetpack-dev (= 6.2+b77)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.2+b77_arm64.deb
Size: 29298
SHA256: 70553d4b5a802057f9436677ef8ce255db386fd3b5d24ff2c0a8ec0e485c59cd
SHA1: 9deab64d12eef0e788471e05856c84bf2a0cf6e6
MD5sum: 4db65dc36434fe1f84176843384aee23
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Package: nvidia-jetpack
Source: nvidia-jetpack (6.1)
Version: 6.1+b123
Architecture: arm64
Maintainer: NVIDIA Corporation
Installed-Size: 194
Depends: nvidia-jetpack-runtime (= 6.1+b123), nvidia-jetpack-dev (= 6.1+b123)
Homepage: http://developer.nvidia.com/jetson
Priority: standard
Section: metapackages
Filename: pool/main/n/nvidia-jetpack/nvidia-jetpack_6.1+b123_arm64.deb
Size: 29312
SHA256: b6475a6108aeabc5b16af7c102162b7c46c36361239fef6293535d05ee2c2929
SHA1: f0984a6272c8f3a70ae14cb2ca6716b8c1a09543
MD5sum: a167745e1d88a8d7597454c8003fa9a4
Description: NVIDIA Jetpack Meta Package
Description-md5: ad1462289bdbc54909ae109d1d32c0a8

Running

dpkg -l | grep -i 'jetpack'

on host gives:

ii  nvidia-jetpack-dev                                          6.2+b77                                     arm64        NVIDIA Jetpack dev Meta Package
ii  nvidia-jetpack-runtime                                      6.2+b77                                     arm64        NVIDIA Jetpack runtime Meta Package

While troubleshooting i simplified the gstreamer pipeline quit a bit and so far the crash seems not to happed within the following pipeline:

gst-launch-1.0 v4l2src device=/dev/video0 !
      nvvideoconvert src-crop=0:0:1920:1080 !\n
      m.sink_0 nvstreammux name=m batch-size=1 live-source=1 width=1280 height={self.camera_resolution_height} !      
nvinfer config-file-path=configs/facedetect.yml !
      nvvideoconvert !
      fakesink

I dont know, if you wanna look into this, but from my side the problem seems to be resolved.

1 Like
5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.