Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress

[Jack_Xuq]

My environment is:

xxxx:/home/Download/deepstream_python_apps/apps/deepstream-imagedata-multistream# deepstream-app --version-all
deepstream-app version 5.1.0
DeepStreamSDK 5.1.0
CUDA Driver Version: 11.2
CUDA Runtime Version: 11.1
TensorRT Version: 7.2
cuDNN Version: 8.0
libNVWarp360 Version: 2.0.1d3

My pipeline diagram is:

My question is the same as Intermittent failure with below error, when executing deepstream application

The error lies in the tracked l_user=batch_meta.batch_user_meta_list, as long as there is data, it will report an error as CudaError

[Jack_Xuq]

this is my error log:

Frames will be saved in  frame
Creating Pipeline

Creating streamux

.......... 1
Creating source_bin  0

Creating source bin
source-bin-00
Creating Pgie

Creating nvvidconv1

Creating filter1

Creating tiler

Creating nvvidconv

Creating nvosd

Creating EGLSink

Atleast one of the sources is live
Adding elements to Pipeline

Linking elements in the Pipeline

Now playing...
1 :  rtsp://127.0.0.1:554/live/ch
Starting pipeline

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
gstnvtracker: Optional NvMOT_ProcessPast not implemented
gstnvtracker: Optional NvMOT_RemoveStreams not implemented
gstnvtracker: Batch processing is OFF
WARNING: ../nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: TensorRT was linked against cuDNN 8.1.0 but loaded cuDNN 8.0.4
WARNING: ../nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: TensorRT was linked against cuBLAS/cuBLAS LT 11.3.0 but loaded cuBLAS/cuBLAS LT 11.2.1
WARNING: ../nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: TensorRT was linked against cuDNN 8.1.0 but loaded cuDNN 8.0.4
WARNING: ../nvdsinfer/nvdsinfer_func_utils.cpp:36 [TRT]: TensorRT was linked against cuBLAS/cuBLAS LT 11.3.0 but loaded cuBLAS/cuBLAS LT 11.2.1
0:00:13.487786227 10402      0x25f1c10 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1702> [UID = 1]: deserialized trt engine from :/home/Download/deepstream_python_apps/apps/deepstream-imagedata-multistream/model_trt.engine
INFO: ../nvdsinfer/nvdsinfer_model_builder.cpp:685 [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT input_0         3x640x640
1   OUTPUT kFLOAT output_0        8400x15

0:00:13.487903092 10402      0x25f1c10 INFO                 nvinfer gstnvinfer.cpp:619:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1806> [UID = 1]: Use deserialized engine model: /home/Download/deepstream_python_apps/apps/deepstream-imagedata-multistream/model_trt.engine
0:00:13.489660223 10402      0x25f1c10 INFO                 nvinfer gstnvinfer_impl.cpp:313:notifyLoadModelStatus:<primary-inference> [UID 1]: Load new model:config_infer_primary_yoloV5.txt sucessfully
Decodebin child added: source


**PERF:  {'stream0': 0.0}

Decodebin child added: decodebin0

Decodebin child added: rtph264depay0

Decodebin child added: h264parse0

Decodebin child added: capsfilter0

Decodebin child added: nvv4l2decoder0

In cb_newpad

KLT Tracker Init
ERROR: nvdsinfer_context_impl.cpp:1573 Failed to synchronize on cuda copy-coplete-event, cuda err_no:700, err_str:cudaErrorIllegalAddress
0:00:18.318243659 10402       0xf628f0 WARN                 nvinfer gstnvinfer.cpp:2021:gst_nvinfer_output_loop:<primary-inference> error: Failed to dequeue output from inferencing. NvDsInferContext error: NVDSINFER_CUDA_ERROR
0:00:18.318454891 10402       0xf628f0 WARN                 nvinfer gstnvinfer.cpp:616:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::releaseBatchOutput() <nvdsinfer_context_impl.cpp:1607> [UID = 1]: Tried to release an outputBatchID which is already with the context
Cuda failure: status=700 in CreateTextureObj at line 2902
nvbufsurftransform.cpp:2703: => Transformation Failed -2

Segmentation fault (core dumped)

this is my tracker config:

[tracker]
tracker-width=1920
tracker-height=1080
gpu-id=0
# ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
ll-lib-file = /opt/nvidia/deepstream/deepstream/lib/libnvds_mot_klt.so
# ll-config-file=config_tracker_NvDCF_perf.yml
enable-past-frame=1
enable-batch-process=1
display-tracking-id=1

[Amycao]

@Jack_Xuq

The error lies in the tracked l_user=batch_meta.batch_user_meta_list, as long as there is data, it will report an error as CudaError

How about disable the tracker?

[Jack_Xuq]

Then there is no problem at all. I have tested more than 10 rtsp channels. There are some small problems, but they have been solved so far. After adding tracking, it will not work, but I need tracking to improve the missing boxes of infer.

[Amycao]

How about change KLT to DCF or IOU tracker?

[Jack_Xuq]

Yes, I tried, but I still couldn’t

[Amycao]

Is it possible for you to upgrade to latest deepstream version?

[Jack_Xuq]

oh, deepstream 6.1 is ok, why?

[Jack_Xuq]

Isn’t libnvds_nvmultiobjecttracker.so a fusion of several algorithms?

[Amycao]

Yes, but you can specify config to use which tracker algorithm.

[Jack_Xuq]

But I don’t see how to specify which tracker to use in Python. My code is:

        # Set properties of tracker
        config = configparser.ConfigParser()
        config.read('dstest_imagedata_config.txt')
        config.sections()

        for key in config['tracker']:
            if key == 'tracker-width':
                tracker_width = config.getint('tracker', key)
                tracker.set_property('tracker-width', tracker_width)
            if key == 'tracker-height':
                tracker_height = config.getint('tracker', key)
                tracker.set_property('tracker-height', tracker_height)
            if key == 'gpu-id':
                tracker_gpu_id = config.getint('tracker', key)
                tracker.set_property('gpu_id', tracker_gpu_id)
            if key == 'll-lib-file':
                tracker_ll_lib_file = config.get('tracker', key)
                tracker.set_property('ll-lib-file', tracker_ll_lib_file)
            if key == 'll-config-file':
                tracker_ll_config_file = config.get('tracker', key)
                tracker.set_property('ll-config-file', tracker_ll_config_file)
            if key == 'enable-batch-process':
                tracker_enable_batch_process = config.getint('tracker', key)
                tracker.set_property('enable_batch_process', tracker_enable_batch_process)
            if key == 'enable-past-frame':
                tracker_enable_past_frame = config.getint('tracker', key)
                tracker.set_property('enable_past_frame', tracker_enable_past_frame)
            if key == 'display-tracking-id':
                tracker_enable_tracker_id = config.getint('tracker', key)
                tracker.set_property('display-tracking-id', tracker_enable_tracker_id)

Ll lib file is libnvds_ nvmultiobjecttracker.so,It’s in the above configuration file. I don’t seem to specify it?

[Jack_Xuq]

This is my blog. I don’t understand how 6.1 tracker integrates 5.1 and solves my bug. If there is more information on this, can you provide me with some information to improve my notes?

deepstream学习笔记（四）：跟踪模块tracker接入与rtsp流异常问题解决_submarineas的博客-CSDN博客_rtsp连接异常处理

[Jack_Xuq]

Oh, I can see that deepstream 5.1 divides so into four parts, while deepstream 6.1 divides YML into four parts. The tracker item and code I configured above are based on the modification of deepstream 5.1, and the default parameters of YML are directly commented out. As for this bug, I think we can study it again. Thank you for your answer

[Amycao]

ll-config-file= to specify which tacker algorithm config file used, check config_tracker_NvDCF_perf.yml in test2 sample.