ERROR from sink_sub_bin_encoder9: Device '/dev/nvhost-msenc' failed during initialization

• Hardware Platform: GeForce RTX 2080 Ti *4
• DeepStream Version: 5.0
• TensorRT Version: 7.0
• NVIDIA GPU Driver Version: 450.51

When I run multi instances more than 3 on different gpus ( 2 instances per gpu) and set RTSPStreaming sink ( per instance occupies one rtsp port and udp port ) it returns error:

ERROR from sink_sub_bin_encoder9: Device '/dev/nvhost-msenc' failed during initialization
Debug info: gstv4l2object.c(4052): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin9/nvv4l2h264enc:sink_sub_bin_encoder9:
Call to S_FMT failed for YM12 @ 1280x720: Unknown error -1
** INFO: <bus_callback:187>: Pipeline running

KLT Tracker Init
KLT Tracker Init
KLT Tracker Init
ERROR from sink_sub_bin_encoder9: Device '/dev/nvhost-msenc' failed during initialization
Debug info: gstv4l2object.c(4052): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin9/nvv4l2h264enc:sink_sub_bin_encoder9:
Call to S_FMT failed for YM12 @ 1280x720: Unknown error -1
Sat Aug 15 15:05:13 2020
**PERF: 26.13 (5.84)    0.95 (0.86)     0.75 (0.68)     20.52 (10.05)   25.88 (5.34)
Sat Aug 15 15:05:24 2020
**PERF: 0.40 (2.30)     0.00 (0.61)     0.00 (0.38)     0.00 (4.75)     0.61 (2.14)
nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
nvbufsurface: NvBufSurfaceCopy: failed to copy
ERROR in BufSurfacecopy 
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (719) unspecified launch failure.
Cuda failure: status=719 in CreateTextureObj at line 2496
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (719) unspecified launch failure.
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (719) unspecified launch failure.
Cuda failure: status=719 in CreateTextureObj at line 2496
ERROR from sink_sub_bin_encoder3: Failed to process frame.
0:01:00.259808415 19091 0x7fffb409d450 ERROR                nvinfer gstnvinfer.cpp:1075:get_converted_buffer:<primary_gie> cudaMemset2DAsync failed with error cudaErrorLaunchFailure while converting buffer
nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
nvbufsurface: NvBufSurfaceCopy: failed to copy
Debug info: gstv4l2videoenc.c(1220): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin3/nvv4l2h264enc:sink_sub_bin_encoder3:
Maybe be due to not enough memory or failing driver
nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
nvbufsurface: NvBufSurfaceCopy: failed to copy
0:01:00.259864154 19091 0x7fffb409d450 WARN                 nvinfer gstnvinfer.cpp:1335:gst_nvinfer_process_full_frame:<primary_gie> error: Buffer conversion failed
ERROR in BufSurfacecopy 
Cuda failure: status=719 in CreateTextureObj at line 2496
ERROR: nvdsinfer_context_impl.cpp:1448 Failed to synchronize on cuda copy-coplete-event, cuda err_no:719, err_str:cudaErrorLaunchFailure
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
ERROR: nvdsinfer_context_impl.cpp:1448 Failed to synchronize on cuda copy-coplete-event, cuda err_no:719, err_str:cudaErrorLaunchFailure
ERROR from primary_gie: Buffer conversion failed
0:01:00.260029703 19091 0x7fffb42dc0f0 WARN                 nvinfer gstnvinfer.cpp:1983:gst_nvinfer_output_loop:<primary_gie> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:01:00.260078826 19091 0x7ff82c497230 WARN                 nvinfer gstnvinfer.cpp:1983:gst_nvinfer_output_loop:<primary_gie> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: gstnvinfer.cpp(1335): gst_nvinfer_process_full_frame (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
0:01:00.260175611 19091 0x7fffb42dc0f0 WARN                 nvinfer gstnvinfer.cpp:599:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1482> [UID = 1]: Tried to release an outputBatchID which is already with the context
ERROR from primary_gie: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Cuda failure: status=719 in CreateTextureObj at line 2513
Cuda failure: status=719 in CreateTextureObj at line 2496
0:01:00.260242583 19091 0x7ff82c497230 WARN                 nvinfer gstnvinfer.cpp:599:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1482> [UID = 1]: Tried to release an outputBatchID which is already with the context
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (400) invalid resource handle.
Debug info: gstnvinfer.cpp(1983): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR: nvdsinfer_context_impl.cpp:1448 Failed to synchronize on cuda copy-coplete-event, cuda err_no:719, err_str:cudaErrorLaunchFailure
0:01:00.260379364 19091 0x7ff82c497230 WARN                 nvinfer gstnvinfer.cpp:1983:gst_nvinfer_output_loop:<primary_gie> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:01:00.260423636 19091 0x7ff82c497230 WARN                 nvinfer gstnvinfer.cpp:599:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1482> [UID = 1]: Tried to release an outputBatchID which is already with the context
ERROR from sink_sub_bin_encoder5: Failed to process frame.
Debug info: gstv4l2videoenc.c(1220): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin5/nvv4l2h264enc:sink_sub_bin_encoder5:
Maybe be due to not enough memory or failing driver
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (400) invalid resource handle.
ERROR from primary_gie: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: gstnvinfer.cpp(1983): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR from primary_gie: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
Debug info: gstnvinfer.cpp(1983): gst_nvinfer_output_loop (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (400) invalid resource handle.
ERROR: ../nvdsinfer/nvdsinfer_func_utils.cpp:31 [TRT]: ../rtSafe/safeContext.cpp (133) - Cudnn Error in configure: 7 (CUDNN_STATUS_MAPPING_ERROR)
ERROR: ../nvdsinfer/nvdsinfer_func_utils.cpp:31 [TRT]: FAILED_EXECUTION: std::exception
ERROR: nvdsinfer_backend.cpp:290 Failed to enqueue inference batch
ERROR: nvdsinfer_context_impl.cpp:1408 Infer context enqueue buffer failed, nvinfer error:NVDSINFER_TENSORRT_ERROR
0:01:00.276213984 19091 0x7ff85400a630 WARN                 nvinfer gstnvinfer.cpp:1188:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
ERROR: nvdsinfer_context_impl.cpp:333 Failed to make stream wait on event, cuda err_no:719, err_str:cudaErrorLaunchFailure
ERROR: nvdsinfer_context_impl.cpp:1384 Preprocessor transform input data failed., nvinfer error:NVDSINFER_CUDA_ERROR
0:01:00.276333914 19091 0x7ff85400a630 WARN                 nvinfer gstnvinfer.cpp:1188:gst_nvinfer_input_queue_loop:<primary_gie> error: Failed to queue input batch for inferencing
** Deepstream: Quitting dreamdeck video analytic...**
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (400) invalid resource handle.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
0:01:00.316539088 19091 0x7ff85400a680 ERROR                nvinfer gstnvinfer.cpp:1075:get_converted_buffer:<primary_gie> cudaMemset2DAsync failed with error cudaErrorLaunchFailure while converting buffer
0:01:00.316579126 19091 0x7ff85400a680 WARN                 nvinfer gstnvinfer.cpp:1335:gst_nvinfer_process_full_frame:<primary_gie> error: Buffer conversion failed
0:01:00.326343616 19091 0x7ffb0cfbb4f0 ERROR                nvinfer gstnvinfer.cpp:1075:get_converted_buffer:<primary_gie> cudaMemset2DAsync failed with error cudaErrorLaunchFailure while converting buffer
0:01:00.329907793 19091 0x7ffb0cfbb4f0 WARN                 nvinfer gstnvinfer.cpp:1335:gst_nvinfer_process_full_frame:<primary_gie> error: Buffer conversion failed
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=719 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (709) context is destroyed.
Cuda failure: status=46 in CreateTextureObj at line 2496
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (46) all CUDA-capable devices are busy or unavailable.
nvbufsurftransform.cpp(2369) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (46) all CUDA-capable devices are busy or unavailable.
free(): corrupted unsorted chunks

When I run 3 instances it runs well. But up to 4, also there shows fps indicated the last instance is inferring well, I can’t get the last instance’s rtsp address image showed.

How to solve it?

RTX 2080 has limitation with hardware video encoder. It can only support 3 streams encoding.

You need to use software video encoder when there are more than 3 streams.

1 Like