Tao-deploy on Orin AGX CLI Error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
,
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc) : Orin AGX
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) : ALL
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here) : 4.0
• Training spec file(If have, please share here)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

I need to convert an .etlt to an TensorRT engine to run on Orin AGX, so I followed this documentation in Installing TAO Deploy through wheel section, the installation went fine without any errors.
But when I try to run the sample command:

detectnet_v2 gen_trt_engine -e /path/to/experiment/spec.txt \
    -m /path/to/etlt/file \
    -k $KEY \
    --data_type fp16 \
    --engine_file /path/to/engine/file

I got this error:

/usr/local/lib/python3.8/dist-packages/pytransform_vax_001219/_pytransform_vax_001219.so: cannot open shared object file: No such file or directory

So, I tried to add that path to LD_LIBRARY_PATH with

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/python3.8/dist-packages/pytransform_vax_001219

and I’m still facing the same error

Here’s the content of docker file I used to create the environment:

FROM nvcr.io/nvidia/l4t-tensorrt:r8.5.2.2-devel
RUN apt update
RUN apt --fix-broken install
RUN apt install -y mpich
RUN pip install mpi4py
RUN pip install torchinfo
RUN pip install clearml
RUN pip install segmentation-models-pytorch
RUN pip install transformers
RUN pip install nvidia-tao-deploy

Please help, I don’t know how to proceed further. Any guidance on how to fix this issue is appreciated.

2

To run inference in Jetson Orin, you can use one of below ways.

  1. Run with deepstream. GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream .
    For example, for yolov4, see deepstream_tao_apps/pgie_yolov4_tao_config.txt at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub, you can config etlt and key.
  2. Run with triton application. GitHub - NVIDIA-AI-IOT/tao-toolkit-triton-apps: Sample app code for deploying TAO Toolkit trained models to Triton
3

I’m trying to get a TensorRT engine for my custom application. From the documents, it seems that tao-deploy should work on Jetson platform…
Could Nvidia make a tao-deploy docker-image that supports Jetson platforms? This would be much better the semi-complete/accurate documentation that puts us in confusion when we follow step by step and it doesn’t work…

4

I will check further for “tao-deploy should work on Jetson platform”.
Instead, as we sync in How to properly install tao-deploy ? bash: tao-deploy: command not found - #14 by glingk , could you use tao-converter to generate the tensorrt engine? Actually this is the way previously we used.

5

Yes, currently I’m exploring tao-converter. Could you please point me to a specific examples of Retinanet using tao export (TAO 4.0) of a QAT model and then converting it using tao-converter… The sample TAO notebooks only have samples of tao-deploy…

6

You can refer to this old version of tao user guide.
https://docs.nvidia.com/tao/tao-toolkit-archive/tao-30-2205/text/object_detection/yolo_v4.html#Generating%20an%20Engine%20Using%20tao-converter

Example Commands:

tao-converter -k $KEY  \
              -p Input,1x3x384x1248,8x3x384x1248,16x3x384x1248 \
                    -e /export/trt.fp16.engine \
                    -t fp16 \
                    /ws/yolov4_resnet18_epoch_100.etlt

Download tao-converter from TAO Converter | NVIDIA NGC

8

Good morning,
I’ve got the same @glingk error running detectnet_v2 gen_trt_engine natively on Jetson Orin AGX.

I installed tao-deploy natively on Orin with the following command

python3.8 -m pip install nvidia-tao-deploy

Is in possible to know from NVIDIA which is the “official” way to take an .etlt model file → convert to tensort engine and use it to run inference with TensorRT c++ or python API on Jetson Orin?

It is frustrating to waste days in order to do one of the basic operation expected to do with Orin

9

You can use a previous official way to convert .etlt to tensorrt engine.
Download tao-converter and use it to convert.

10

I already tried to use the tao converter following your post
How to run tlt-converter - Intelligent Video Analytics / TAO Toolkit - NVIDIA Developer Forums

I dowloaded the binary but it does not run on Jetson Orin with Jetpack 5.1.1 since the binary is based on tensorRT 7.x

I tried the following workaround create symbolic links for libnvinfer.so.7, libnvinfer_plugin.so.7, libnvparsers.so.7 linking to the .so.8 version available on Orin. Now the binary works but when I run it with the following

./tlt-converter resnet34_peoplenet_pruned.etlt -k tlt_encode -c PeopleNet_trt.txt -o output_cov/Sigmoid,output_bbox/BiasAdd -d 3,544,960 -i nchw -e peoplenet_int8.engine -m 64 -t int8 -b 64

I got the following errors

[INFO] [MemUsageChange] Init CUDA: CPU +221, GPU +0, now: CPU 249, GPU 3728 (MiB)
[INFO] [MemUsageChange] Init builder kernel library: CPU +303, GPU +427, now: CPU 574, GPU 4174 (MiB)
[ERROR] 2: [logging.cpp::decRefCount::65] Error Code 2: Internal Error (Assertion mRefCount > 0 failed. )
Segmentation fault (core dumped)

I got the same error also using fp32 and with different batch size. The calibration file is taken from officiel repository

nvidia-tao/tao_deploy/specs/PeopleNet at main · NVIDIA-AI-IOT/nvidia-tao · GitHub

Any suggestion?

11

Please download tao-converter from TAO Converter | NVIDIA NGC.

For example, if your orin has TRT8, you can download the one in

image
image2209×126 5.29 KB

12

Hi @Morganh,
thank you for your suggestion. I tried the binary and it works. I successfully converted the model and run the engine with TensorRT (python)

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

15

Please run with below steps. The tao-deploy can work on Orin.

docker run --runtime=nvidia -it --rm -v nvcr.io/nvidia/l4t-tensorrt:r8.5.2.2-devel /bin/bash                                                      

Then, inside the docker,

apt-get update
apt install libopenmpi-dev
pip install nvidia_tao_deploy==5.0.0.423.dev0
pip install https://urm.nvidia.com/artifactory/sw-eff-pypi/nvidia-eff-tao-encryption/0.1.7/nvidia_eff_tao_encryption-0.1.7-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
pip install https://urm.nvidia.com/artifactory/sw-eff-pypi/nvidia-eff/0.6.2/nvidia_eff-0.6.2-py38-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

detectnet_v2 --help