Hello everyone
I’m encountering an issue while trying to convert a GazeNet model using tao-converter in the TAO Toolkit environment. The conversion fails with an error about dynamic shapes and missing optimization profiles, and I’m struggling to identify the correct input names for the model. I’d appreciate any guidance or suggestions to resolve this.
Environment
GPU: NVIDIA RTX A2000 12GB
-
Model: GazeNet (deployable v1.0), downloaded using ngc registry model download-version nvidia/tao/gazenet:deployable_v1.0
-
TAO Toolkit Version: 4.0.1-TensorFlow (build), as per the output of my Docker container
-
Docker Image: nvcr.io/nvidia/tao/tao-toolkit:4.0.1-tf1.15.5
-
Container Setup: Running with flags --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864
-
Container Path: /workspace/model_dir
-
Model File: gazenet_vdeployable_v1.0/model.etlt
Issue
I’m trying to convert the model.etlt file to test out how it works using the tao-converter tool. Initially, I ran the following command:
tao-converter -k nvidia_tlt model.etlt
This resulted in the following error:
[INFO] [MemUsageChange] Init CUDA: CPU +20, GPU +0, now: CPU 32, GPU 1033 (MiB)
[INFO] [MemUsageChange] Init builder kernel library: CPU +549, GPU +118, now: CPU 636, GPU 1151 (MiB)
[INFO] ----------------------------------------------------------------
[INFO] Input filename: /tmp/fileKqmGDT
[INFO] ONNX IR version: 0.0.5
[INFO] Opset version: 10
[INFO] Producer name: tf2onnx
[INFO] Producer version: 1.6.3
[INFO] Domain:
[INFO] Model version: 0
[INFO] Doc string:
[INFO] ----------------------------------------------------------------
[WARNING] parsers/onnx/onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[INFO] Detected input dimensions from the model: (-1, 1, 224, 224)
[ERROR] Model has dynamic shape but no optimization profile specified.
Aborted (core dumped)
I understand the error indicates that the model has dynamic shapes (likely the batch size, as seen in the detected input dimensions (-1, 1, 224, 224)), and I need to specify optimization profiles using the -p option. GazeNet typically has four inputs: face, left eye, right eye (each 224x224), and facegrid (25x25). Based on the documentation, I assumed the inputs are color images (3 channels) for face and eyes, and 1 channel for facegrid, except the error log shows (1, 224, 224), suggesting grayscale inputs, which might indicate a mismatch.
I modified my command to include input dimensions and optimization profiles, assuming input names as “face”, “left_eye”, “right_eye”, and “facegrid”:
tao-converter model.etlt -k nvidia_tlt -d 3,224,224 -d 3,224,224 -d 3,224,224 -d 1,25,25 -o output_cov/Sigmoid -o output_bbox/BiasAdd -e gazenet.trt -m 1 -t fp16 -p face,1x3x224x224,1x3x224x224,1x3x224x224 -p left_eye,1x3x224x224,1x3x224x224,1x3x224x224 -p right_eye,1x3x224x224,1x3x224x224,1x3x224x224 -p facegrid,1x1x25x25,1x1x25x25,1x1x25x25
However this resulted in another error:
[ERROR] Wrong input name specified in -p, please double check.
Aborted (core dumped)
It seems the input names I assumed (“face”, “left_eye”, “right_eye”, “facegrid”) are incorrect. can anyone help me please