GPUassert: the launch timed out and was terminated

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

jetson aga dev kit
jetpack 4.3
deepstream4

I am suffering from random crashes - using GST_DEBUG=3 I have got the following logging errors

The crashes happen randomly - normally about12-15 hours before it crashes, on this occasion 28 hours

deepstream runs fine until the crash happens, what’s happening?

thanks

GPUassert: the launch timed out and was terminated /dvs/git/dirty/git-master_linux/deepstream/sdk/src/utils/nvdcf/src/modules/cuDCF/cudaCropScaleInTexture2D.cu 377

28:09:25.594245376 e[332m22507e[00m   0x55bc5e2e80 e[31;01mERROR  e[00m e[00m             nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier>e[00m NvDsInferContext[UID 1]:queueInputBatch(): cudaMemcpyAsync for output buffers failed (cudaErrorLaunchTimeout)

28:09:25.594301792 e[332m22507e[00m   0x55bc5e2e80 e[33;01mWARN   e[00m e[00m             nvinfer gstnvinfer.cpp:1098:gst_nvinfer_input_queue_loop:<primary_gie_classifier>e[00m error: Failed to queue input batch for inferencing
3

Can you updated to latest JetPack and Deepstream? Deepstream4.0 is not supported now. https://developer.nvidia.com/deepstream-getting-started

4

Hi Fiona,

I can’t update at the moment.

Can you at least tell me what this error means please

Thanks chris

5

The error shows there is cuda memory copy failure.