Intermittent failure with below error, when executing deepstream application

[pradeep.labheswar]

==========================================================
gstname= video/x-raw
features= <Gst.CapsFeatures object at 0x7f7ef1ad6ac8 (GstCapsFeatures at 0x240e2e0)>
ERROR: nvdsinfer_context_impl.cpp:1573 Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:00:07.976354253 12316 0x1f9d0a0 WARN nvinfer gstnvinfer.cpp:2021:gst_nvinfer_output_loop: error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:00:07.976425277 12316 0x1f9d0a0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1599> [UID = 1]: Tried to release an unknown outputBatchID
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: safeContext.cpp (184) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)
ERROR: nvdsinfer_context_impl.cpp:1573 Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:00:07.976496193 12316 0x1f9d0a0 WARN nvinfer gstnvinfer.cpp:2021:gst_nvinfer_output_loop: error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Cuda failure: status=700 in CreateTextureObj at line 2902
nvbufsurftransform.cpp:2703: => Transformation Failed -2

0:00:07.976539113 12316 0x1f9d0a0 WARN nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1599> [UID = 1]: Tried to release an unknown outputBatchID
ERROR: …/nvdsinfer/nvdsinfer_func_utils.cpp:33 [TRT]: FAILED_EXECUTION: std::exception
ERROR: nvdsinfer_backend.cpp:287 Failed to enqueue inference batch
ERROR: nvdsinfer_context_impl.cpp:1533 Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:00:07.976654265 12316 0x1f9d0f0 WARN nvinfer gstnvinfer.cpp:1225:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
ERROR: nvdsinfer_context_impl.cpp:334 Failed to make stream wait on event, cuda err_no:700, err_str:cudaErrorIllegalAddress
ERROR: nvdsinfer_context_impl.cpp:1509 Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
0:00:07.976729114 12316 0x1f9d0f0 WARN nvinfer gstnvinfer.cpp:1225:gst_nvinfer_input_queue_loop: error: Failed to queue input batch for inferencing
Error: gst-stream-error-quark: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR (1): gstnvinfer.cpp(2021): gst_nvinfer_output_loop (): /GstPipeline:pipeline0/GstNvInfer:primary-inference
Exiting app

Cuda failure: status=700
nvbufsurface: Error(-1) in releasing cuda memory

• T4 GPU
• DeepStream Version: 5.1, using python application
• TensorRT 7.2.2
• CUDA 11.1
• NVIDIA GPU Driver Version (460.32)
• Issue Type(bugs)
• How to reproduce the issue ? (Its Intermittent)
• Core Pipeline: [streammux → nvinfer → nvtracker → streamdemux]

[Amycao]

HI,
I try to mimic your case, streammux → nvinfer → nvtracker → streamdemux, but can not repro your issue. please specify more how to repro your issue.

config-file-path=/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/config_infer_primary.txt batch-size=2 ! nvtracker ll-lib-file = /opt/nvidia/deepstream/deepstream/lib/libnvds_nvdcf.so ll-config-file=/opt/nvidia/deepstream/deepstream-6.0/samples/configs/deepstream-app/tracker_config.yml tracker-width=640 tracker-height=384 ! streamdemux. videotestsrc pattern=0 ! nvvideoconvert ! “video/x-raw(memory:NVMM), format=NV12” ! streammux.sink_0 videotestsrc pattern=1 ! nvvideoconvert ! “video/x-raw(memory:NVMM),format=NV12” ! streammux.sink_1 streamdemux.src_0 ! queue ! nvvideoconvert ! “video/x-raw(memory:NVMM), format=I420” ! nvegltransform ! nveglglessink sync=false streamdemux.src_1 ! queue ! fakesink sync=false
Setting pipeline to PAUSED …