Using deepstream to open two configuration files to run at the same time. The two configuration files run 8 routes on two different GPUs(Tesla T4) respectively. After running for about an hour, the following crash problems appear:
1:29:45.645769500 3996 0x7f71040066d0 ERROR nvinfer gstnvinfer.cpp:1103:get_converted_buffer:<primary_gie> cudaMemset2DAsync failed with error cudaErrorLaunchFailure while converting buffer
nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
1:29:45.645814033 3996 0x7f71040066d0 WARN nvinfer gstnvinfer.cpp:1363:gst_nvinfer_process_full_frame:<primary_gie> error: Buffer conversion failed
nvbufsurface: NvBufSurfaceCopy: failed to copy
Cuda failure: status=719 in CreateTextureObj at line 2572
nvbufsurftransform.cpp:2624: => Transformation Failed -2
ERROR in BufSurfacecopy
nvbufsurftransform.cpp(2793) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (719) unspecified launch failure.
nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
nvbufsurface: NvBufSurfaceCopy: failed to copy
ERROR in BufSurfacecopy
ERROR from primary_gie: Buffer conversion failed
Debug info: gstnvinfer.cpp(1363): gst_nvinfer_process_full_frame (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR from sink_sub_bin_encoder8: Failed to process frame.
Debug info: gstv4l2videoenc.c(1224): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline/GstBin:processing_bin_7/GstBin:sink_bin/GstBin:sink_sub_bin8/nvv4l2h265enc:sink_sub_bin_encoder8:
Maybe be due to not enough memory or failing driver
ERROR from sink_sub_bin_encoder2: Failed to process frame.
Debug info: gstv4l2videoenc.c(1224): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline/GstBin:processing_bin_1/GstBin:sink_bin/GstBin:sink_sub_bin2/nvv4l2h265enc:sink_sub_bin_encoder2:
Maybe be due to not enough memory or failing driver
./run.sh: line 14: 3996 Segmentation fault (core dumped) ./zzmj_automation_app_zhu