TAO-Converter TRT engine inference results is blank

Accelerated Computing Intelligent Video Analytics TAO Toolkit
, ,
1

Sorry for this question, I’ve looked around the forum and wasn’t able to fix this issue yet.

I’ve trained a retinanet model via the TAO framework and tested it’s inferencing & it was able to produce a satisfactory result

However, when I attempted to export the .tlt model & further convert it to a tensorRT engine & tested inferencing on the tensorRT engine the results are empty.
Potential problems that I’m contemplating:

  1. What is the actual image preprocessing before feeding it to the network? Where can I find it?
  2. Maybe I’m not converting the model correctly?

Here’s the command I used to export TLT to ETLT (On workstation with RTX2070 super):

export DOCKER_REGISTRY="nvcr.io"
export DOCKER_NAME="nvidia/tao/tao-toolkit"
export DOCKER_TAG="4.0.0-tf1.15.5"
export DOCKER_CONTAINER=$DOCKER_REGISTRY/$DOCKER_NAME:$DOCKER_TAG

export KEY="nvidia_tlt"
export EXPERIMENT_NAME="mobilenet_v2_qat_2"
export SPEC_FILE="retinanet_train_mobilenet_v2.txt"
export EXPORT_EPOCH="022"
docker run -it --rm --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /home/developer/apps/tao-training/workspace:/workspace \
$DOCKER_CONTAINER \
retinanet export   -o /workspace/retinanet/models/$EXPERIMENT_NAME/export/retinanet_mobilenet_v2_epoch_$EXPORT_EPOCH.etlt \
                         -e /workspace/retinanet/specs/$SPEC_FILE \
                         -m /workspace/retinanet/models/$EXPERIMENT_NAME/weights/retinanet_mobilenet_v2_epoch_$EXPORT_EPOCH.tlt \
                         -k $KEY \
                          --data_type int8 \
                          --cal_cache_file /workspace/retinanet/models/$EXPERIMENT_NAME/export/cal.bin \
                          --cal_json_file /workspace/retinanet/models/$EXPERIMENT_NAME/export/cal.json

Here’s the command I used to convert ETLT to tensorRT engine (Run On Orin AGX using nvcr.io/nvidia/l4t-ml:r35.2.1-py3 as base image) :

./tao-converter -k $KEY \
            -d 3,544,736 \
            -o NMS \
            -m 1 \
            -t int8 \
            -c /workspaces/convert_to_trt/etlt_models/sorter_object_detection/retinanet/tao_etlt_retinanet_mobilenet_v2_int8/cal.bin \
            -e /workspaces/convert_to_trt/etlt_models/sorter_object_detection/retinanet/tao_etlt_retinanet_mobilenet_v2_int8/retinanet_mobilenet_v2_epoch_022.engine \
            /workspaces/convert_to_trt/etlt_models/sorter_object_detection/retinanet/tao_etlt_retinanet_mobilenet_v2_int8/retinanet_mobilenet_v2_epoch_022.etlt

retinanet_train_mobilenet_v2.txt (1.7 KB)
retinanet_mobilenet_v2_epoch_022.tlt (38.2 MB)

Original_Branch_AllLights_5
Original_Branch_AllLights_51920×1440 211 KB

TAO_inference_Branch_AllLights_5
TAO_inference_Branch_AllLights_51920×1440 230 KB

retinanet_mobilenet_v2_epoch_022.etlt (20.8 MB)
cal.bin (2.3 KB)
retinanet_mobilenet_v2_epoch_022.engine (13.6 MB)

3

To narrow down, could you please generate fp16 or fp32 tensorrt engine and try again?

4

I tried with fp32 and still empty inferences :(

./tao-converter -k $KEY \
            -d 3,544,736 \
            -o NMS \
            -e /workspaces/convert_to_trt/etlt_models/sorter_object_detection/retinanet/tao_etlt_retinanet_mobilenet_v2_no_data_type/retinanet_mobilenet_v2_epoch_022.engine \
            /workspaces/convert_to_trt/etlt_models/sorter_object_detection/retinanet/tao_etlt_retinanet_mobilenet_v2_no_data_type/retinanet_mobilenet_v2_epoch_022.etlt
5

How did you run inference?

6

Here’s the code is use to run inference
test_detection_trt_engine.py (4.7 KB)

7

Could you try to run inference with the tao deploy docker?
Refer to RetinaNet with TAO Deploy - NVIDIA Docs

8

@Morganh I’m not sure what the game plan is here… I need to be able to run inference on a Jetson Platform using python (Orin AGX), and I believe that tao deploy doesn’t work on Jetson Platform yet…
Can you please elaborate on the thought process of troubleshooting this issue, what could have gone wrong and any reference examples/documents that could help?
My initial thought was that either

  1. tao-export was not ran correctly → as I don’t understand how the model dimensions get set, where does the command get it from?
  2. tao-convert was not ran correctly
  3. What is the correct preprocessing step before running inference → the only thing I found was to make sure the image is CHW → Is there a proper example that demonstrates the preprocessing? Is is dependent on the backbone/encoder ?
9

Officially, there are two kinds of ways to run inference outside tao docker.
One is deepstream_tao_apps. Another is triton_tao_apps.
For retinanet, you can refer to GitHub - NVIDIA-AI-IOT/deepstream_tao_apps: Sample apps to demonstrate how to deploy models trained with TAO on DeepStream, the preprocess is https://github.com/NVIDIA-AI-IOT/deepstream_tao_apps/blob/master/configs/retinanet_tao/pgie_retina_tao_config.txt#L25-L27

Also, you can refer to be below.
https://github.com/NVIDIA-AI-IOT/tao-toolkit-triton-apps/blob/ae6b5ec41c3a9651957c4dddfc262a43f47e263c/tao_triton/python/entrypoints/tao_client.py#L323

The postprocessing is in https://github.com/NVIDIA-AI-IOT/tao-toolkit-triton-apps/blob/ae6b5ec41c3a9651957c4dddfc262a43f47e263c/tao_triton/python/postprocessing/retinanet_postprocessor.py

10

Thank you for the pointer. Got it working :)

11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.