Using TLT model in Driveworks for inference from camera or video

I have trained a model(Resnet) using my own dataset using the Transfer Learning toolkit. I want to use the model for inference from a video and camera using the Driveworks SDK(sample_object_detector). Running the sample_object_detector with the trt file generated from the tlt-converter i am getting the following error:-
The engine plan file is not compatible with this version of TensorRT, expecting library version 5.1.4 got 5.1.5, please rebuild." this error which i will fix.
What would be a possible fix to this problem? Or my approach of deploying wrong here?

Thanks in advance for you help!

Regards.

Which platform will you run inference, x86 or Jetson?
If Jetson, what is the CUDA/Cudnn/TRT version of it?
$ dpkg -l |grep cuda

More, where did you download the tlt-converter?

I will run the inference on a DRIVE AGX Xavier which i believe is jetson. The version of CUDA is as follows:-

ii  cuda-command-line-tools-10-2            10.2.19-1                                  arm64        CUDA command-line tools
ii  cuda-compiler-10-2                      10.2.19-1                                  arm64        CUDA compiler
ii  cuda-cudart-10-2                        10.2.19-1                                  arm64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-10-2                    10.2.19-1                                  arm64        CUDA Runtime native dev links, headers
ii  cuda-cufft-10-2                         10.2.19-1                                  arm64        CUFFT native runtime libraries
ii  cuda-cuobjdump-10-2                     10.2.19-1                                  arm64        CUDA cuobjdump
ii  cuda-cupti-10-2                         10.2.19-1                                  arm64        CUDA profiling tools runtime libs.
ii  cuda-curand-10-2                        10.2.19-1                                  arm64        CURAND native runtime libraries
ii  cuda-cusolver-10-2                      10.2.19-1                                  arm64        CUDA solver native runtime libraries
ii  cuda-cusparse-10-2                      10.2.19-1                                  arm64        CUSPARSE native runtime libraries
ii  cuda-driver-dev-10-2                    10.2.19-1                                  arm64        CUDA Driver native dev stub library
ii  cuda-gdb-10-2                           10.2.19-1                                  arm64        CUDA-GDB
ii  cuda-gpu-library-advisor-10-2           10.2.19-1                                  arm64        CUDA GPU Library Advisor.
ii  cuda-license-10-2                       10.2.19-1                                  arm64        CUDA licenses
ii  cuda-memcheck-10-2                      10.2.19-1                                  arm64        CUDA-MEMCHECK
ii  cuda-misc-headers-10-2                  10.2.19-1                                  arm64        CUDA miscellaneous headers
ii  cuda-npp-10-2                           10.2.19-1                                  arm64        NPP native runtime libraries
ii  cuda-nvcc-10-2                          10.2.19-1                                  arm64        CUDA nvcc
ii  cuda-nvdisasm-10-2                      10.2.19-1                                  arm64        CUDA disassembler
ii  cuda-nvgraph-10-2                       10.2.19-1                                  arm64        NVGRAPH native runtime libraries
ii  cuda-nvprof-10-2                        10.2.19-1                                  arm64        CUDA Profiler tools
ii  cuda-nvprune-10-2                       10.2.19-1                                  arm64        CUDA nvprune
ii  cuda-nvrtc-10-2                         10.2.19-1                                  arm64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-10-2                     10.2.19-1                                  arm64        NVRTC native dev links, headers
ii  cuda-nvtx-10-2                          10.2.19-1                                  arm64        NVIDIA Tools Extension
ii  cuda-repo-ubuntu1804-10-2-local-10.2.19 1.0-1                                      arm64        cuda repository configuration files

I used the tlt-converter provided with the tlt docker on the same machine i trained the model in.

How about cudnn and tensorRT version?

More, for Jetson platform, the tlt-converter version is not the same as the one inside the docker.
See Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation

  • cuDNN 7.5.1.14
  • TensorRT version 5.1.4.2

These came preinstalled with the AGX and I found the version on the documentation.

I tried running tlt-converter on the AGX but I am getting the following error:-

./tlt-converter: error while loading shared libraries: libnvinfer.so.7: cannot open shared object file: No such file or directory

Moreover I have looked for the library libnvinfer.so.7, which the AGX does not have, however it has libnvinfer.so.5. Currently the environment variable TRT_LIB_PATH is set to the destination where the libnvinfer.so.5 library is.

For Jetson platform, there are different version of tlt-converter.
If the trt5 is installed, please download from https://developer.nvidia.com/tlt-converter-trt51
If the trt7 is installed, please download from https://developer.nvidia.com/tlt-converter-trt71

BTW, you can use below command to check the cudnn and trt version.
$ dpkg -l |grep cudnn

$ dpkg -l |grep tensor

Since I have trt version 5.1.4.2 I ran the tlt converter from the first link provided

The files I have in the same directory are as follows:-

ls -lt
total 60116
-rw-r--r-- 1 nvidia nvidia   841935 May 18 11:49 resnet18_detector.etlt
-rw-r--r-- 1 nvidia nvidia 60638270 May 18 11:49 calibration.tensor
-rw-r--r-- 1 nvidia nvidia     4120 May 18 11:49 calibration.bin
-rw-rw-r-- 1 nvidia nvidia     1131 Feb 18 13:36 Readme.txt
-rw-rw-r-- 1 nvidia nvidia    58256 Aug  2  2019 tlt-converter

On running the following command:-

./tlt-converter ./resnet18_detector.etlt -k #mykey -c ./calibration.bin -o output_cov/Sigmoid,output_bbox/BiasAdd -d 3,384,1248 -i nchw -m 64 -t int8 -e ./resnet.trt -b 4

I am getting the following error:-

./tlt-converter ./resnet18_detector.etlt -k #mykey -c ./calibration.bin -o output_cov/Sigmoid,output_bbox/BiasAdd -d 3,384,1248 -i nchw -m 64 -t int8 -e ./resnet.trt -b 4
[ERROR] UffParser: Unsupported number of graph 0
[ERROR] Failed to parse uff model
[ERROR] Network must have at least one output
[ERROR] Unable to create engine
Segmentation fault (core dumped)

I am trying to convert on the AGX.

Hi SKoyal,
If you were using Jetpack 4.4 to install cuda/trt/cudnn into xavier, the cuda version will be 10.2.89, cudnn version will be 8.0, trt version will be 7.1.

Due to the old cudnn/trt version, I suggest you run with 1.0.1 tlt docker instead of 2.0_dp tlt docker.

BTW, please check with the tips in TLT Converter UffParser: Unsupported number of graph 0 - #3 by jefflgaol too.

tlt-converter worked. It was the key issue. Thank you.

However, I still would want to use the model in Driveworks SDK using ./sample_object_detector_tracker. When I am using the ./sample_object_detector_tracker on the generated TRT it is giving me the following error:-

./sample_object_detector_tracker --input-type=video --video=/home/nvidia/sk/test1.h264 --tensorRT_model=/home/nvidia/sk/tlt-converter-trt5.1/resnet.trt
[20-05-2020 10:55:59] Platform: Detected DDPX - Tegra A
[20-05-2020 10:55:59] TimeSource: monotonic epoch time offset is 1589956455391598
[20-05-2020 10:55:59] PTP Time is available from NVPPS Driver
[20-05-2020 10:55:59] Platform: number of GPU devices detected 1
[20-05-2020 10:55:59] Platform: currently selected GPU device integrated ID 0
[20-05-2020 10:55:59] SDK: Resources mounted from .././data/
[20-05-2020 10:55:59] SDK: Create NvMediaDevice
[20-05-2020 10:55:59] SDK: use EGL display as provided
[20-05-2020 10:56:00] TimeSource: monotonic epoch time offset is 1589956455391598
[20-05-2020 10:56:00] PTP Time is available from NVPPS Driver
[20-05-2020 10:56:00] Initialize DriveWorks SDK v2.2.3136
[20-05-2020 10:56:00] Release build with GNU 7.3.1 from heads/buildbrain-branch-0-gca7b4b26e65 against Drive PDK v5.1.6.1
[20-05-2020 10:56:00] Initialize DriveWorks VisualizationSDK v2.2.3136
[20-05-2020 10:56:00] Initialize DriveWorksGL SDK v2.2.3136
[20-05-2020 10:56:00] SensorFactory::createSensor() -> camera.virtual, camera-group=a,camera-index=0,camera-type=ar0231-rccb-bae-sf3324,input-type=video,offscreen=0,profiling=1,slave=0,tensorRT_model=/home/nvidia/sk/tlt-converter-trt5.1/resnet.trt,video=/home/nvidia/sk/test1.h264
[20-05-2020 10:56:00] CameraVirtualNvMedia: no seek table found at /home/nvidia/sk/test1.h264.seek, seeking is not available.
SimpleCamera: Camera image: 1920x1080
Camera image with 1920x1080 at 60 FPS
[20-05-2020 10:56:00] StreamConsumerGL: successfully initialized
[20-05-2020 10:56:00] StreamProducerCUDA: successfully initialized
[20-05-2020 10:56:00] DNN: TensorRT model file has wrong magic number. Please ensure that the model has been created by TensorRT_optimization tool in DriveWorks. The model might be incompatible.
[20-05-2020 10:56:00] DNN: TensorRT model file has wrong magic number. Please ensure that the model has been created by TensorRT_optimization tool in DriveWorks. The model might be incompatible.
[20-05-2020 10:56:00] Glob Size is 2176512 bytes.
[20-05-2020 10:56:00] Added linear block of size 122683392
[20-05-2020 10:56:00] Added linear block of size 122683392
[20-05-2020 10:56:00] Added linear block of size 30670848
[20-05-2020 10:56:05] Deserialize required 5020426 microseconds.
[20-05-2020 10:56:05] DNN: Metadata json file could not be found. Metadata has been filled with default values. Please place <network_filename>.json in the same directory as the network file if custom metadata is needed.
[20-05-2020 10:56:05] Driveworks exception thrown: DW_INVALID_ARGUMENT: blobName not found

terminate called after throwing an instance of 'std::runtime_error'
  what():  [2020-05-20 10:56:05] DW Error DW_INVALID_ARGUMENT executing DW function:
 dwDNN_getOutputIndex(&m_cvgIdx, coverageBlobName, m_dnn)
 at /dvs/git/dirty/gitlab-master_av/dw/sdk/samples/dnn/sample_object_detector_tracker/main.cpp:316
Aborted (core dumped)

Checking the documentation for TensorRT Optimizer Tool it is mentioned that “The NVIDIA® DriveWorks TensorRT Optimizer Tool enables optimization for a given Caffe, UFF or ONNX model using TensorRT.”. The tlt does not produce either of the three model types. Is there a converter that can convert my trt to either of these model types?

I am trying to avoid using deepstream because I do not want to upgrade the dependencies or flash my AGX during the installation of the deepstream as I other teams with existing dependencies are working on it and their work may get affected.

I do not focus on Driveworks SDK support. And also from tlt user guide Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation , it just ensures the Deepstream inference runs smoothly. Not sure the status for “sample_object_detector_tracker” in the Driveworks SDK.

Thank you for the help. I will repost my problems with different tags for other reach!