Incompatible TensorRT engine(int8) with deepstream

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I have trained yolov4 model using tao framework, from which i got hdf5 model. In the next step, i converted it to .onnx format, next I have converted the onnx model to .engine.int8 format. The last step gave me 3 files -
a) .engine.int8 file
b)cal.bin
c)cal.tensorfile

I have done the whole process till now following the yolo_v4.ipynb file from this link -https://api.ngc.nvidia.com/v2/resources/nvidia/tao/tao-getting-started/versions/5.0.0/zip -O tao-getting-started_5.0.0.zip

Now, i want t use the engine.int8 model file with the calibration file in deepstream framework for inferencing.
For this purpose i have have modified the config_infer_primary.txt file found in deepstream sample config folder. The config files are attached.
Copy_config_infer_primary.txt (4.1 KB)
test_config.txt (4.8 KB)

Now I’m getting error after performing the inferencing using (deepstream-app -c test_config.txt), where test_config.txt is the main config file, which refers to config_infer_primary.txt. The error I’m facing is this -

root@93ad828e3e73:/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app# deepstream-app -c test_config.txt
WARNING: ../nvdsinfer/nvdsinfer_model_builder.cpp:1482 Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/../../models/yolo_v4/export/yolov4_resnet18_epoch_060_trt.engine.int8 open error
0:00:01.702177941   676 0x5ff2a46dfc70 WARN                 nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1888> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/../../models/yolo_v4/export/yolov4_resnet18_epoch_060_trt.engine.int8 failed
0:00:01.720822021   676 0x5ff2a46dfc70 WARN                 nvinfer gstnvinfer.cpp:643:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1993> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/../../models/yolo_v4/export/yolov4_resnet18_epoch_060_trt.engine.int8 failed, try rebuild
0:00:01.720851303   676 0x5ff2a46dfc70 INFO                 nvinfer gstnvinfer.cpp:646:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1914> [UID = 1]: Trying to create engine from model files
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:860 failed to build network since there is no model file matched.
ERROR: ../nvdsinfer/nvdsinfer_model_builder.cpp:799 failed to build network.
0:00:02.181828711   676 0x5ff2a46dfc70 ERROR                nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1934> [UID = 1]: build engine file failed
0:00:02.200702378   676 0x5ff2a46dfc70 ERROR                nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2020> [UID = 1]: build backend context failed
0:00:02.200777885   676 0x5ff2a46dfc70 ERROR                nvinfer gstnvinfer.cpp:640:gst_nvinfer_logger:<primary_gie> NvDsInferContext[UID 1]: Error in NvDsInferContextImpl::initialize() <nvdsinfer_context_impl.cpp:1257> [UID = 1]: generate backend failed, check config file settings
0:00:02.201002504   676 0x5ff2a46dfc70 WARN                 nvinfer gstnvinfer.cpp:846:gst_nvinfer_start:<primary_gie> error: Failed to create NvDsInferContext instance
0:00:02.201017821   676 0x5ff2a46dfc70 WARN                 nvinfer gstnvinfer.cpp:846:gst_nvinfer_start:<primary_gie> error: Config file path: /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/Copy_config_infer_primary.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
** ERROR: <main:716>: Failed to set pipeline to PAUSED
Quitting
ERROR from primary_gie: Failed to create NvDsInferContext instance
Debug info: gstnvinfer.cpp(846): gst_nvinfer_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInfer:primary_gie:
Config file path: /opt/nvidia/deepstream/deepstream-6.1/samples/configs/deepstream-app/Copy_config_infer_primary.txt, NvDsInfer Error: NVDSINFER_CONFIG_FAILED
App run failed
3

Is the “yolov4_resnet18_epoch_060_trt_2.engine.int8” file generated in the same GPU and the same CUDA version and TensorRT version as the DeepStream environment you are running?

This log means the engine file is generated in different environment as to the environment you are running the deepstream-app.

4

I am using an AWS vm which has Nvidia GPU.
I have generated the engine file in that only.
To be more specific I have used the Jupyter notebook provided by Nvidia to generate the engine file, that runs on the aws vm.
And then to use the Deepstream Framework I have docker pulled the deepstream image and made a docker conatiner . I have exported the engine file to that deepstream container and then used that engine file inside the docker conatiner .

So, to conclude my TAO framework that is used to generate the engine and the Deepstream dokcer container , both are in the same AWS vm only,

5

This is a docker which may not have the same TensorRT version as the DeepStream docker.

Even DeepStream can generate TenorRT engine from the onnx and cal files too, we are not sure whether the cal file is generated with the same TensorRT version as the DeepStream docker. The cal file may impact the inferencing accuracy, please consult in the Latest Intelligent Video Analytics/TAO Toolkit topics - NVIDIA Developer Forums forum for how to align the TensorRT version.

The DeepStream compatibility is listed in Installation — DeepStream documentation 6.4 documentation

6

Okay, let me try that and get back to you.
Till then please don’t close this topic/chat.