LPRNet gives different results when running on Jetson

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

• Hardware Platform Jetson
• DeepStream Version 6.3
• JetPack Version 5.1
• TensorRT Version 5.1
• Issue Type: questions

LPRNet I have trained on custom dataset produces different results when running on Jetson.

In my experiment I would process video on Jetson and save crops from object detector, and then run inference in my training environment. In some cases results are the same, while in others model outputs a wrong character. Sometimes it would drop one character or give wrong output for one or two characters.
For example:
1200-0
On Jetson I get MLY0005, while on Docker I get correct value of MLY9905

I train LPRNet on nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5 image, use it to export to onnx and then transfer it to Jetson where it is automatically converted to tlt once I start the app.

What could be the reason for this behavior and how to get my model to give better results?

5

Hi @nikola2
Do you mean the outputs are fine when run inference in docker nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5 but sometimes wrong when run inference in Jetson device?

6

Yes, exactly

7

For Jetson, may I know what is the tensorrt version?
And how did you convert onnx file to tensorrt engine, with trtexec, or tao-converter, or deepstream-tao-apps or your own script?

8

Like I said, TensorRT version is 5.1.

I think I used trtexec, I would just write location of onnx and engine file and application would automatically convert it:

  onnx-file: ../weights/LP/LPR/lprnet_03.05.2024/lprnet_epoch-070.onnx
  model-engine-file: ../weights/LP/LPR/lprnet_03.05.2024/lprnet_epoch-070.onnx_b16_gpu0_fp16.engine

For conversion to onnx I used thsi command withing docker:

lprnet export -m results/weights/lprnet_epoch-070.hdf5 -e experiment.txt -k nvidia_tlt
9

May I know which application did you use?

10

My own python application I built from deepstream-test3 example.

11

To narrow down, suggest you to run trtexec to convert as well.
Refer to https://docs.nvidia.com/tao/tao-toolkit/text/trtexec_integration/trtexec_lprnet.html.

12

I converted with this command and issue still persists

/usr/src/tensorrt/bin/trtexec --onnx=weights/LP/LPR/lprnet_03.05.2024/lprnet_epoch-070.onnx --maxShapes=image_input:32x3x48x96 --minShapes=image_input:4x3x48x96 --optShapes=image_input:16x3x48x96 --fp16 --saveEngine=weights/LP/LPR/lprnet_03.05.2024/lprnet_test.engine
13

The TensorRT version is a little old.
Could you confirm it? You can run $dpkg -l |grep cuda in your Jetson device?

More, could you update to latest Jetpack?

14

Jtop says this is my version, but here’s the full dpkg output:

ii  cuda-11-4                                  11.4.19-1                             arm64        CUDA 11.4 meta-package
ii  cuda-cccl-11-4                             11.4.298-1                            arm64        CUDA CCCL
ii  cuda-command-line-tools-11-4               11.4.19-1                             arm64        CUDA command-line tools
ii  cuda-compiler-11-4                         11.4.19-1                             arm64        CUDA compiler
ii  cuda-cudart-11-4                           11.4.298-1                            arm64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-11-4                       11.4.298-1                            arm64        CUDA Runtime native dev links, headers
ii  cuda-cuobjdump-11-4                        11.4.315-1                            arm64        CUDA cuobjdump
ii  cuda-cupti-11-4                            11.4.298-1                            arm64        CUDA profiling tools runtime libs.
ii  cuda-cupti-dev-11-4                        11.4.298-1                            arm64        CUDA profiling tools interface.
ii  cuda-cuxxfilt-11-4                         11.4.300-1                            arm64        CUDA cuxxfilt
ii  cuda-documentation-11-4                    11.4.298-1                            arm64        CUDA documentation
ii  cuda-driver-dev-11-4                       11.4.298-1                            arm64        CUDA Driver native dev stub library
ii  cuda-gdb-11-4                              11.4.298-1                            arm64        CUDA-GDB
ii  cuda-libraries-11-4                        11.4.19-1                             arm64        CUDA Libraries 11.4 meta-package
ii  cuda-libraries-dev-11-4                    11.4.19-1                             arm64        CUDA Libraries 11.4 development meta-package
ii  cuda-nvcc-11-4                             11.4.315-1                            arm64        CUDA nvcc
ii  cuda-nvdisasm-11-4                         11.4.298-1                            arm64        CUDA disassembler
ii  cuda-nvml-dev-11-4                         11.4.298-1                            arm64        NVML native dev links, headers
ii  cuda-nvprune-11-4                          11.4.298-1                            arm64        CUDA nvprune
ii  cuda-nvrtc-11-4                            11.4.300-1                            arm64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-11-4                        11.4.300-1                            arm64        NVRTC native dev links, headers
ii  cuda-nvtx-11-4                             11.4.300-1                            arm64        NVIDIA Tools Extension
ii  cuda-profiler-api-11-4                     11.4.298-1                            arm64        CUDA Profiler API
ii  cuda-runtime-11-4                          11.4.19-1                             arm64        CUDA Runtime 11.4 meta-package
ii  cuda-samples-11-4                          11.4.300-1                            arm64        CUDA example applications
ii  cuda-sanitizer-11-4                        11.4.298-1                            arm64        CUDA Sanitizer
ii  cuda-toolkit-11-4                          11.4.19-1                             arm64        CUDA Toolkit 11.4 meta-package
ii  cuda-toolkit-11-4-config-common            11.4.298-1                            all          Common config package for CUDA Toolkit 11.4.
ii  cuda-toolkit-11-config-common              11.4.298-1                            all          Common config package for CUDA Toolkit 11.
ii  cuda-toolkit-config-common                 11.4.298-1                            all          Common config package for CUDA Toolkit.
ii  cuda-tools-11-4                            11.4.19-1                             arm64        CUDA Tools meta-package
ii  cuda-visual-tools-11-4                     11.4.19-1                             arm64        CUDA visual tools
ii  graphsurgeon-tf                            8.5.2-1+cuda11.4                      arm64        GraphSurgeon for TensorRT package
ii  libcudnn8                                  8.6.0.166-1+cuda11.4                  arm64        cuDNN runtime libraries
ii  libcudnn8-dev                              8.6.0.166-1+cuda11.4                  arm64        cuDNN development libraries and headers
ii  libcudnn8-samples                          8.6.0.166-1+cuda11.4                  arm64        cuDNN samples
ii  libnvinfer-bin                             8.5.2-1+cuda11.4                      arm64        TensorRT binaries
ii  libnvinfer-dev                             8.5.2-1+cuda11.4                      arm64        TensorRT development libraries and headers
ii  libnvinfer-plugin-dev                      8.5.2-1+cuda11.4                      arm64        TensorRT plugin libraries
ii  libnvinfer-plugin8                         8.5.2-1+cuda11.4                      arm64        TensorRT plugin libraries
ii  libnvinfer-samples                         8.5.2-1+cuda11.4                      all          TensorRT samples
ii  libnvinfer8                                8.5.2-1+cuda11.4                      arm64        TensorRT runtime libraries
ii  libnvonnxparsers-dev                       8.5.2-1+cuda11.4                      arm64        TensorRT ONNX libraries
ii  libnvonnxparsers8                          8.5.2-1+cuda11.4                      arm64        TensorRT ONNX libraries
ii  libnvparsers-dev                           8.5.2-1+cuda11.4                      arm64        TensorRT parsers libraries
ii  libnvparsers8                              8.5.2-1+cuda11.4                      arm64        TensorRT parsers libraries
ii  nvidia-cuda                                5.1-b147                              arm64        NVIDIA CUDA Meta Package
ii  nvidia-cuda-dev                            5.1-b147                              arm64        NVIDIA CUDA dev Meta Package
ii  nvidia-l4t-cuda                            35.2.1-20230124153320                 arm64        NVIDIA CUDA Package
ii  python3-libnvinfer                         8.5.2-1+cuda11.4                      arm64        Python 3 bindings for TensorRT
ii  python3-libnvinfer-dev                     8.5.2-1+cuda11.4                      arm64        Python 3 development package for TensorRT
ii  tensorrt                                   8.5.2.2-1+cuda11.4                    arm64        Meta package for TensorRT
ii  uff-converter-tf                           8.5.2-1+cuda11.4                      arm64        UFF converter for TensorRT package

I’m not sure, how do I update Jetpack?

15

I think I have newest Jetpack installed.

sudo apt-get upgrade nvidia-jetpack

Reading package lists… Done
Building dependency tree
Reading state information… Done
nvidia-jetpack is already the newest version (5.1-b147).

16

To narrow down, can you generate a tensorrt engine in tao-deploy docker instead.
nvcr.io/nvidia/tao/tao-toolkit:5.2.0-deploy
Please run the docker in a dgpu machine instead of Jetson device.

You can use tao deploy lprnet gen_trt_engine command mentioned in tao_tutorials/notebooks/tao_launcher_starter_kit/lprnet/lprnet.ipynb at main · NVIDIA/tao_tutorials · GitHub.
Or docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tao/tao-toolkit:5.2.0-deploy /bin/bash, and then generate engine in it. And run inference inside the docker.

17

Since I’m using training on a shared server I will make a request for their team to prepare tao-deploy, but this might take some time.
Is there something else I can try while waiting?

Also, during conversion to tensorrt engine I get this warning:

Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.

18

Could you try more in Jetson devices?
experiment1: generate fp32 engine
experiment2: generate engine with --maxShapes=image_input:1x3x48x96 --minShapes=image_input:1x3x48x96 --optShapes=image_input:1x3x48x96

19

Just to confirm I generate fp32 engine just by dropping --fp16 flag?
If that’s the case, I did the following experiments:

  1. I generated fp32 engine - no change, model makes the same mistakes
  2. I generated fp32 engine with max/min/opt shapes as suggested - no change, model makes the same mistakes
  3. I generated fp16 engine with max/min/opt shapes as suggested - no change, model makes the same mistakes
20

May I know how did you run inference in Jetson devices?

Please try to use GitHub - NVIDIA-AI-IOT/deepstream_lpr_app: Sample app code for LPR deployment on DeepStream.

21

I have video of still images I feed to my deepstream application, with YoloV5 as my primary detector and LPRNet as secondary classifier. After inference, crops are saved in a folder. These crops I later upload to server where I can do inference with LPRNet.

22

I am afraid there is something mismatching in preprocessing.

To narrow down, suggest you to install tao-deploy in Jetson device.

Please run with below steps. The tao-deploy can work on Jetson(For example, Orin).

docker run --runtime=nvidia -it --rm -v nvcr.io/nvidia/l4t-tensorrt:r8.5.2.2-devel /bin/bash                                                      

Then, inside the docker,

$apt-get update
$apt install libopenmpi-dev
$pip install nvidia_tao_deploy==5.0.0.423.dev0
$pip install https://urm.nvidia.com/artifactory/sw-eff-pypi/nvidia-eff-tao-encryption/0.1.7/nvidia_eff_tao_encryption-0.1.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
$pip install https://urm.nvidia.com/artifactory/sw-eff-pypi/nvidia-eff/0.6.2/nvidia_eff-0.6.2-py38-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

$lprnet --help

BTW, the source code can be found in tao_deploy/nvidia_tao_deploy/cv/lprnet/scripts/inference.py at main · NVIDIA/tao_deploy · GitHub.

23

I have tried running docker but I keep getting

docker: invalid reference format.
See ‘docker run --help’.

I don’t understand what I’m missing.