Deepstream program crash problem

Using deepstream to open two configuration files to run at the same time. The two configuration files run 8 routes on two different GPUs(Tesla T4) respectively. After running for about an hour, the following crash problems appear:

1:29:45.645769500 3996 0x7f71040066d0 ERROR nvinfer gstnvinfer.cpp:1103:get_converted_buffer:<primary_gie> cudaMemset2DAsync failed with error cudaErrorLaunchFailure while converting buffer
nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
1:29:45.645814033 3996 0x7f71040066d0 WARN nvinfer gstnvinfer.cpp:1363:gst_nvinfer_process_full_frame:<primary_gie> error: Buffer conversion failed
nvbufsurface: NvBufSurfaceCopy: failed to copy
Cuda failure: status=719 in CreateTextureObj at line 2572
nvbufsurftransform.cpp:2624: => Transformation Failed -2
ERROR in BufSurfacecopy
nvbufsurftransform.cpp(2793) : getLastCudaError() CUDA error : Recevied NvBufSurfTransformError_Execution_Error : (719) unspecified launch failure.
nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
nvbufsurface: NvBufSurfaceCopy: failed to copy
ERROR in BufSurfacecopy
ERROR from primary_gie: Buffer conversion failed
Debug info: gstnvinfer.cpp(1363): gst_nvinfer_process_full_frame (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie
ERROR from sink_sub_bin_encoder8: Failed to process frame.
Debug info: gstv4l2videoenc.c(1224): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline/GstBin:processing_bin_7/GstBin:sink_bin/GstBin:sink_sub_bin8/nvv4l2h265enc:sink_sub_bin_encoder8:
Maybe be due to not enough memory or failing driver
ERROR from sink_sub_bin_encoder2: Failed to process frame.
Debug info: gstv4l2videoenc.c(1224): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline/GstBin:processing_bin_1/GstBin:sink_bin/GstBin:sink_sub_bin2/nvv4l2h265enc:sink_sub_bin_encoder2:
Maybe be due to not enough memory or failing driver
./run.sh: line 14: 3996 Segmentation fault (core dumped) ./zzmj_automation_app_zhu

deepstream 版本为5.1
deepstream 版本为5.1

Deepstream version is 5.1

Sorry for the late response, have you managed to resolve the problem or it’s still an issue to support?

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Hey, which app are you using when you meet the issue? Could you provide the more info here?