The effect is very poor when converted to trt

• Network Type:
pointpillar

• TAO Version:
4.0.0-pyt

• Training spec file:
pointpillars.ipynb (18.3 MB)

Hello, I have used the pointpillar tutorial in Tao’s Notebook and prepared the kitti dataset, training, optimization, and more according to the tutorial. Generating a. tlt model and reasoning on the validation set is very effective. However, during deployment, generating a trt engine and using it for inference results in poor performance. The following is a comparison of the bird’s-eye view generated during the inference process.

I followed the tutorial instructions completely. Why is there such a large deviation? I hope to receive some suggestions. Thank you.

. tlt model Immediate inference

After converting to trt

How about running inference against /workspace/tao-experiments/pointpillars/retrain/ckpt/checkpoint_epoch_80.trt?

The effect looks good, but this trt engine cannot be used on orin. I used the tao converter tool on orin to convert the. etlt model as input into trt for deployment. The deployment of orin has also been done through tao converter, and the effect is similar to /workspace/tao experiments/pointpillars/trt. fp16. engine. How can I achieve such a requirement? Thank you for your reply!

For trt engine in orin, suggest you to download similar version of tao-converter(TAO Converter | NVIDIA NGC) and generate again to check if it works.

More, for TAO 5.0, you can export to onnx file(previous etlt model), and then use trtexec https://docs.nvidia.com/tao/tao-toolkit/text/trtexec_integration/trtexec_pointpillars.html#trtexec-pointpillars to generate trt engine.
More command can be found in GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC or https://docs.nvidia.com/tao/tao-toolkit/text/point_cloud/pointpillars.html#exporting-the-model

Thank you for your reply!

The Tao I am using is version 4.0.1, and the Tao-converter downloaded from orin (Jetpack 5.0.2, Tensorrt 8.4.1.5) is v3.22.05_trt8.4_ aarch64, these should be fine. I’ll confirm again. But the current problem is that using tao-converter on both x86 and Jetson platforms can result in poor inference performance

The official website tutorial has only recently been updated to 5.0.0. Would it be best for me to use the new version of Tao? This way, I can use the trtexec provided by orin to achieve model transformation, and use the examples in github( https://github.com/NVIDIA-AI-IOT/tao_toolkit_recipes/tree/main/tao_pointpillars/tensorrt_sample )Implement inference

Yes, you can use TAO 5.0.

Please rebuild the plugin as mentioned in https://github.com/NVIDIA-AI-IOT/tao_toolkit_recipes/tree/main/tao_pointpillars/tensorrt_sample ?

I downloaded the latest notebook (getting_started_v5.0.0. zip) with the following information:
dockers:
nvidia/tao/tao-toolkit:
4.0.0-tf2.9.1:
docker_registry: nvcr.io
tasks:
1. classification_tf2
2. efficientdet_tf2
4.0.0-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. augment
2. bpnet
3. classification_tf1
4. detectnet_v2
5. dssd
6. emotionnet
7. efficientdet_tf1
8. faster_rcnn
9. fpenet
10. gazenet
11. gesturenet
12. heartratenet
13. lprnet
14. mask_rcnn
15. multitask_classification
16. retinanet
17. ssd
18. unet
19. yolo_v3
20. yolo_v4
21. yolo_v4_tiny
22. converter
4.0.1-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. mask_rcnn
2. unet
4.0.0-pyt:
docker_registry: nvcr.io
tasks:
1. action_recognition
2. deformable_detr
3. segformer
4. re_identification
5. pointpillars
6. pose_classification
7. n_gram
8. speech_to_text
9. speech_to_text_citrinet
10. speech_to_text_conformer
11. spectro_gen
12. vocoder
13. text_classification
14. question_answering
15. token_classification
16. intent_slot_classification
17. punctuation_and_capitalization
format_version: 2.0
toolkit_version: 4.0.1
published_date: 03/06/2023

What software upgrades do I need to do to upgrade to TAO5.0.0?
The preprocessing commands for datasets in Notebook are as follows:

tao model pointpillars run python $SPECS_DIR/gen_lidar_points.py -p $DATA_DOWNLOAD_DIR/training/velodyne \
                                                           -c $DATA_DOWNLOAD_DIR/training/calib    \
                                                           -i $DATA_DOWNLOAD_DIR/training/image_2  \
                                                           -o $DATA_DOWNLOAD_DIR/train/lidar

There are more ‘model’ parameters than before, and an error has been reported, it seems that the current parameter is not recognized.Removing this parameter resulted in no results after processing the kitti dataset. Perhaps Tao has not been upgraded, how should I upgrade?

Please refer to TAO Toolkit Quick Start Guide - NVIDIA Docs
$ bash setup/quickstart_launcher.sh --upgrade

bash quickstart_launcher.sh --upgrade
INFO: Check requirements
INFO: Checking Python installation
INFO: python3 found.
Python 3.6.13 :: Anaconda, Inc.
INFO: Python version:
INFO: pip3 found.
INFO: Pip version: pip 21.2.2 from /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages/pip (python 3.6)
INFO: Docker found. Checking additional requirements for docker.
INFO: Checking nvidia-docker2 installation
INFO: NGC CLI found.
INFO: NGC CLI 3.23.0
INFO: Requirements check satisfied. Installing TAO Toolkit.
By installing the TAO Toolkit CLI, you accept the terms and conditions of this license: https://developer.nvidia.com/tao-toolkit-software-license-agreement
Would you like to continue? (y/n): y
INFO: EULA accepted.
INFO: Installing TAO Toolkit CLI
INFO: jupyter installation was found.
Selected Jupyter core packages…
IPython : 7.16.3
ipykernel : 5.5.6
ipywidgets : not installed
jupyter_client : 7.1.2
jupyter_core : 4.9.2
jupyter_server : 1.13.1
jupyterlab : 3.2.9
nbclient : 0.5.9
nbconvert : 6.0.7
nbformat : 5.1.3
notebook : 6.4.10
qtconsole : not installed
traitlets : 4.3.3
INFO: TAO Toolkit was found
INFO: Upgrading installed nvidia-tao to the latest version.
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: nvidia-tao in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (4.0.1)
Collecting nvidia-tao
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/8e/0a/5953eb77b0e305a0d846642b5349d1ce0a34a660eb5b57d6a514ca0e62aa/nvidia_tao-5.0.0-py3-none-any.whl (35 kB)
Requirement already satisfied: docker==4.3.1 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (4.3.1)
Requirement already satisfied: certifi in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (2021.5.30)
Requirement already satisfied: urllib3>=1.26.5 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (1.26.15)
Requirement already satisfied: requests in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (2.27.1)
Requirement already satisfied: tabulate==0.8.7 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (0.8.7)
Requirement already satisfied: six==1.15.0 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (1.15.0)
Requirement already satisfied: idna==2.10 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (2.10)
Requirement already satisfied: chardet==3.0.4 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (3.0.4)
Requirement already satisfied: docker-pycreds==0.4.0 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (0.4.0)
Requirement already satisfied: websocket-client==0.57.0 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from nvidia-tao) (0.57.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/lhm/anaconda3/envs/tao/lib/python3.6/site-packages (from requests->nvidia-tao) (2.0.12)
INFO: Configuration of the TAO Toolkit Instance

dockers:
nvidia/tao/tao-toolkit:
4.0.0-tf2.9.1:
docker_registry: nvcr.io
tasks:
1. classification_tf2
2. efficientdet_tf2
4.0.0-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. augment
2. bpnet
3. classification_tf1
4. detectnet_v2
5. dssd
6. emotionnet
7. efficientdet_tf1
8. faster_rcnn
9. fpenet
10. gazenet
11. gesturenet
12. heartratenet
13. lprnet
14. mask_rcnn
15. multitask_classification
16. retinanet
17. ssd
18. unet
19. yolo_v3
20. yolo_v4
21. yolo_v4_tiny
22. converter
4.0.1-tf1.15.5:
docker_registry: nvcr.io
tasks:
1. mask_rcnn
2. unet
4.0.0-pyt:
docker_registry: nvcr.io
tasks:
1. action_recognition
2. deformable_detr
3. segformer
4. re_identification
5. pointpillars
6. pose_classification
7. n_gram
8. speech_to_text
9. speech_to_text_citrinet
10. speech_to_text_conformer
11. spectro_gen
12. vocoder
13. text_classification
14. question_answering
15. token_classification
16. intent_slot_classification
17. punctuation_and_capitalization
format_version: 2.0
toolkit_version: 4.0.1
published_date: 03/06/2023

This operation has been done before and seems to have remained unchanged.

Perhaps I should create a new Python environment and reinstall and deploy TAO5.0.0

It still doesn’t work. I deleted the previous image tao toolkit: 4.0.0 pyt. I created a new Python environment and completed the installation and upgrade through the setup in TAO 5.0.0. However, when I executed tao info --verbose, the same version was still displayed as before.The command with the “model” parameter cannot be recognized properly.

Could you run below?
!pip3 install --upgrade nvidia-tao

Refer to
https://github.com/NVIDIA/tao_tutorials/blob/b7ea82a06f1b866018947f73185b5f48abe5a693/notebooks/tao_launcher_starter_kit/pointpillars/pointpillars.ipynb

This upgrade command has been executed and is displayed as follows:

Looking in indexes: Simple Index
Requirement already satisfied: nvidia-tao in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (4.0.1)
Collecting nvidia-tao
Using cached https://pypi.tuna.tsinghua.edu.cn/packages/8e/0a/5953eb77b0e305a0d846642b5349d1ce0a34a660eb5b57d6a514ca0e62aa/nvidia_tao-5.0.0-py3-none-any.whl (35 kB)
Requirement already satisfied: idna==2.10 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (2.10)
Requirement already satisfied: docker-pycreds==0.4.0 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (0.4.0)
Requirement already satisfied: six==1.15.0 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (1.15.0)
Requirement already satisfied: websocket-client==0.57.0 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (0.57.0)
Requirement already satisfied: docker==4.3.1 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (4.3.1)
Requirement already satisfied: urllib3<2.0.0,>=1.26.15 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (1.26.16)
Requirement already satisfied: chardet==3.0.4 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (3.0.4)
Requirement already satisfied: requests in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (2.27.1)
Requirement already satisfied: certifi in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (2021.5.30)
Requirement already satisfied: tabulate==0.8.7 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from nvidia-tao) (0.8.7)
Requirement already satisfied: charset-normalizer~=2.0.0 in /home/lhm/anaconda3/envs/tao5/lib/python3.6/site-packages (from requests->nvidia-tao) (2.0.12)

The upgrade was not successful

Executing ‘pip3 install nvidia tao==5.0.0’ in the current Python environment, installation failed:

Could you please share the full log when you run
$ pip3 install --upgrade nvidia-tao

Please try to run
$ pip3 uninstall nvidia-tao

then check if tao is uninstalled.
$ tao info --verbose

Then install again.
$ pip3 install nvidia-tao

If not working, please follow TAO Toolkit Quick Start Guide - NVIDIA Docs

#### Installing TAO Launcher

Once you have installed the required pre-requisites.

  1. Install the CLI launcher via the quick start script downloaded with the getting_started NGC package from here
bash setup/quickstart_launcher.sh --install
  1. You can also use this script to update the launcher to the latest version of TAO Toolkit by running the following command
bash setup/quickstart_launcher.sh --upgrade

The result is the same.

Sometimes it can be directly installed as 4.0.1, and sometimes during setup, ngccli_ Linux.zip will be downloaded, and I will choose to replace it all. However, there will be a message stating that Jupyter cannot be found. When installing Jupyter, it will get stuck as follows. I don’t know if this is the key to the problem.

Perhaps my pip source needs to be changed? Can you provide me with a suitable PIP source?

I still want to ask why the TRT engine generated with tao converter has poor performance, and how to avoid this situation?

Thank you very much for your patient answers recently!

As mentioned above, please follow TAO Toolkit Quick Start Guide - NVIDIA Docs and retry.

#### Installing TAO Launcher

Once you have installed the required pre-requisites.

  1. Install the CLI launcher via the quick start script downloaded with the getting_started NGC package from here
bash setup/quickstart_launcher.sh --install
  1. You can also use this script to update the launcher to the latest version of TAO Toolkit by running the following command
bash setup/quickstart_launcher.sh --upgrade