Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
Jetson Orin 4012, NVIDIA Jetson Orin NX Bundle, 8x 2GHz, 16GB DDR5
• DeepStream Version
Container: deepstream:7.1-triton-multiarch
• JetPack Version (valid for Jetson only)
see Container: deepstream:7.1-triton-multiarch
• TensorRT Version
see Container: deepstream:7.1-triton-multiarch
• NVIDIA GPU Driver Version (valid for GPU only)
Container: deepstream:7.1-triton-multiarch
• Issue Type( questions, new requirements, bugs)
Bug/Crash
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
-Run docker container
-Create and run following gStreamer pipeline in python:
gst-launch-1.0 v4l2src device=/dev/video0 !
queue max-size-buffers=3 leaky=downstream max-size-time=500000000 !
videorate !
video/x-raw, framerate=30/1 !
nvvideoconvert src-crop=0:0:1920:1080 !
m.sink_0 nvstreammux name=m batch-size=1 live-source=1 width=1280 height=720 !
nvinfer config-file-path=configs/facedetect.yml !
nvvideoconvert !
fakesink
-Set pipeline state to PLAYING and wait for crash to happen (usually a few minutes)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)
When running above gStreamer pipeline for a while i always encounter the following error, leading to crashing the pipeline, after a while.
/dvs/git/dirty/git-master_linux/nvutils/nvbufsurftransform/nvbufsurftransform_copy.cpp:341: => Failed in mem copy
ERROR: Failed to make stream wait on event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
0:01:20.167368168 129 0xaaaad7abf1f0 WARN nvinfer gstnvinfer.cpp:1418:gst_nvinfer_input_queue_loop:<nvinfer> error: Failed to queue input batch for inferencing
10.02. 10:51:47.591 - ERROR: [129 FacenetPipeline] Error: gst-stream-error-quark: Failed to queue input batch for inferencing (1), /dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinfer/gstnvinfer.cpp(1418): gst_nvinfer_input_queue_loop (): /GstPipeline:facenet-pipeline/GstNvInfer:nvinfer
After which a lot of illegal memory address accesses are reported:
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::61] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::61] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: [TRT]: 1: [cudaDriverHelpers.cpp::operator()::94] Error Code 1: Cuda Driver (an illegal memory access was encountered)
ERROR: [TRT]: 1: [defaultAllocator.cpp::deallocate::61] Error Code 1: Cuda Runtime (an illegal memory access was encountered)
ERROR: cudaStreamDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaStreamDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaStreamDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaEventDestroy failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFreeHost failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: cudaFree failed, cuda err_no:700, err_str:cudaErrorIllegalAddress
Does the pipeline need another queue plugin somewhere to fix the “Failed to queue input batch for inferencing” error?
When catching this behaviour, trying to restart the pipeline when the error is happening the output i get is:
10.02. 10:51:47.668 - INFO: [129 FacenetPipeline] Restarting pipeline..
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
INFO: [Implicit Engine Info]: layers num: 3
0 INPUT kFLOAT input_1 3x416x736
1 OUTPUT kFLOAT output_bbox/BiasAdd 4x26x46
2 OUTPUT kFLOAT output_cov/Sigmoid 1x26x46
Setting min object dimensions as 16x16 instead of 1x1 to support VIC compute mode.
after which the application freezes.
Currently thats a huge blocker for me and i would appreciate any advice.