GPUassert: the launch timed out and was terminated

jetson aga dev kit
jetpack 4.3

I am suffering from random crashes - using GST_DEBUG=3 I have got the following logging errors

The crashes happen randomly - normally about12-15 hours before it crashes, on this occasion 28 hours

deepstream runs fine until the crash happens, what’s happening?


GPUassert: the launch timed out and was terminated /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvdcf/src/modules/cuDCF/ 377

28:09:25.594245376 e[332m22507e[00m   0x55bc5e2e80 e[31;01mERROR  e[00m e[00m             nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier>e[00m NvDsInferContext[UID 1]:queueInputBatch(): cudaMemcpyAsync for output buffers failed (cudaErrorLaunchTimeout)

28:09:25.594301792 e[332m22507e[00m   0x55bc5e2e80 e[33;01mWARN   e[00m e[00m             nvinfer gstnvinfer.cpp:1098:gst_nvinfer_input_queue_loop:<primary_gie_classifier>e[00m error: Failed to queue input batch for inferencing

Can you updated to latest JetPack and Deepstream? Deepstream4.0 is not supported now. DeepStream Getting Started | NVIDIA Developer

Hi Fiona,

I can’t update at the moment.

Can you at least tell me what this error means please

Thanks chris

The error shows there is cuda memory copy failure.