Failure in Releasing CUDA memory

• Hardware Platform (Jetson / GPU): dGPU
• DeepStream Version: 5.0
• NVIDIA GPU Driver Version (valid for GPU only): RTX2080Ti

Hi,
I am running deepstream-app in DS5.0 development docker container. Occasionally I run across this error which causes the pipeline to crash. (This happens very randomly. Sometimes around 5 min after starting the pipeline and sometimes almost after half a day).
I monitor the GPU usage throughout and it always has 7GB+ of free memory. So it’s weird that the error says it could be due to not enough memory.
As for the failing driver message I’m not sure what it means exactly because my other CUDA dependent programs are running well without any error.

This is my config fileconfig.txt (2.7 KB)

NOTE: I have removed the URI from this config file. They are RTSP URLs of different IP Cameras

Following is the error message received before crash:

nvbufsurface: NvBufSurfaceSysToHWCopy: failed in mem copy
nvbufsurface: NvBufSurfaceCopy: failed to copy
ERROR in BufSurfacecopy
ERROR from sink_sub_bin_encoder1: Failed to process frame.
Debug info: gstv4l2videoenc.c(1224): gst_v4l2_video_enc_handle_frame (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/nvv4l2h265enc:sink_sub_bin_encoder1:
Maybe be due to not enough memory or failing driver
Cuda failure: status=719 in CreateTextureObj at line 2555
nvbufsurftransform.cpp:2624: => Transformation Failed -2
Cuda failure: status=719 in CreateTextureObj at line 2555
nvbufsurftransform.cpp:2624: => Transformation Failed -2

UPDATE:
Original Config: NVIDA Driver 450.51.06, CUDA 11.0
Downgraded to: NVIDIA Driver 440.95.01, CUDA 10.2.
The program has been running successfully for the past few hours without any errors.