Unet tao-convert error: Network must have at least one output

Please provide the following information when requesting support.

• Hardware RTX3090, RTX2070
• Network Type unet
• TLT Version
Configuration of the TAO Toolkit Instance
dockers: [‘nvidia/tao/tao-toolkit-tf’, ‘nvidia/tao/tao-toolkit-pyt’, ‘nvidia/tao/tao-toolkit-lm’]
format_version: 2.0
toolkit_version: 3.22.02
published_date: 02/28/2022
• Training spec file
unet_retrain_vgg_6S1100.txt (1.5 KB)

• How to reproduce the issue ?

Using the unet notebook with my own custom data (color images 704 X 1280), it trains, prunes, and retrains, evaluates, and does inference, all well.

Exporting the model with the command tao unet export produces a working tensorRT engine that I can use on C++ on the same machine.


!tao unet export --gpu_index=$GPU_INDEX -m $USER_EXPERIMENT_DIR/retrain/weights/model_retrained.tlt \
               -k $KEY \
               -e $SPECS_DIR/unet_retrain_vgg_6S1100.txt  \
               -o $USER_EXPERIMENT_DIR/export/tao.fp32_6S003A01.etlt \
               --data_type fp32 \
               --engine_file $USER_EXPERIMENT_DIR/export/tao.fp32_S6003A1.engine \
               --max_batch_size 1 \
               --batch_size 1

So far so good…

But using the tao converter command produces an error:


!tao converter -k $KEY  \
               -e $USER_EXPERIMENT_DIR/export/trt.fp16.6S003A1.engine -t fp16 \
               -d 3,704,1280 \
               -p input_1,1x3x704x1280,4x3x704x1280,16x3x704x1280 \
               $USER_EXPERIMENT_DIR/retrain/weights/model_retrained.etlt

Results in

2022-08-23 23:56:15,569 [INFO] root: Registry: [‘nvcr.io’]
2022-08-23 23:56:15,734 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
[INFO] [MemUsageChange] Init CUDA: CPU +536, GPU +0, now: CPU 757, GPU 2134 (MiB)
[ERROR] UffParser: Unsupported number of graph 0
[ERROR] Failed to parse the model, please check the encoding key to make sure it’s correct
[ERROR] 4: [network.cpp::validate::2411] Error Code 4: Internal Error (Network must have at least one output)
[ERROR] Unable to create engine
2022-08-23 23:56:18,133 [INFO] tlt.components.docker_handler.docker_handler: Stopping container

I need to run the model on another computer with a different GPU (RTX2070 8GB), but same CUDA, cudNN, and tensorRT versions, and therefore need to convert the model on that computer, which is giving the same error message there, so I was using the tao unet notebook as a testbed for running the tao-converter utility on the other computer…

The output layer of the network is

softmax_1 (Softmax) (None, 704, 1280, 6) 0 permute_1[0][0]

so I added that and run the converter as:


!tao converter -k $KEY  \
               -e $USER_EXPERIMENT_DIR/export/trt.fp16.6S003A1.engine -t fp16 \
               -d 3,704,1280 \
               -i nchw \
               -p input_1,1x3x704x1280,1x3x704x1280,1x3x704x1280 \
               -o softmax_1,0x704x1280x6 \
               $USER_EXPERIMENT_DIR/retrain/weights/model_retrained.etlt

Which results in another error: Number of optimization profiles does not match model input node number:

2022-08-24 00:28:01,952 [INFO] root: Registry: [‘nvcr.io’]
2022-08-24 00:28:02,119 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
[INFO] [MemUsageChange] Init CUDA: CPU +536, GPU +0, now: CPU 542, GPU 2414 (MiB)
[INFO] ----------------------------------------------------------------
[INFO] Input filename: /tmp/fileW8SZFK
[INFO] ONNX IR version: 0.0.0
[INFO] Opset version: 0
[INFO] Producer name:
[INFO] Producer version:
[INFO] Domain:
[INFO] Model version: 0
[INFO] Doc string:
[INFO] ----------------------------------------------------------------
[ERROR] Number of optimization profiles does not match model input node number.
2022-08-24 00:28:04,421 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Thanks in advance for the help!

Can you retry with explicit env?
For example,

  • modify $KEY to the explicit key.
  • modify $USER_EXPERIMENT_DIR to the explicit path.

Exactly the same result:


!tao converter -k nvidia_tlt  \
               -e /media/david/DATA/MP/6S003A1/export/trt.fp16.6S003A1.engine -t fp16 \
               -d 3,704,1280 \
               -i nchw \
               -p input_1,1x3x704x1280,1x3x704x1280,2x3x704x1280 \
               -o softmax_1,-1x704x1280x6 \
               /media/david/DATA/MP/6S003A1/retrain/weights/model_retrained.etlt

2022-08-24 14:17:25,165 [INFO] root: Registry: [‘nvcr.io’]
2022-08-24 14:17:25,332 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
[INFO] [MemUsageChange] Init CUDA: CPU +536, GPU +0, now: CPU 542, GPU 2689 (MiB)
[INFO] ----------------------------------------------------------------
[INFO] Input filename: /tmp/fileXVOkGx
[INFO] ONNX IR version: 0.0.0
[INFO] Opset version: 0
[INFO] Producer name:
[INFO] Producer version:
[INFO] Domain:
[INFO] Model version: 0
[INFO] Doc string:
[INFO] ----------------------------------------------------------------
[ERROR] Number of optimization profiles does not match model input node number.
2022-08-24 14:17:27,590 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

If I try anything other than 3 optimization profiles I get

Please provide three optimization profiles via -p <input_name>,<min_shape>,<opt_shape>,<max_shape>, where each shape has x as delimiter, e.g., NxC, NxCxHxW, NxCxDxHxW, etc.

Could you run below and share the result?
! tao converter run ls /media/david/DATA/MP/6S003A1/retrain/weights/model_retrained.etlt

According to above log, please double check if the key is correct.
Refer to Creating a Real-Time License Plate Detection and Recognition App | NVIDIA - #6 by Taistu

The problem in unrelated to the key value.

The error I’m getting is more related to problems opening the source model file than with the key itself. Although if the key is wrong, that is a problem with loading the source model file.

I realized that after using any random name and resulted in the same error.

The solution is with the extension of the model, etlt. I couldn’t get tao converter to read a tlt model file, so
I used tao export to produce the etlt model file, and imputed that to tao converter…

I transferred to the second computer, ran the tao-converter tool, and plugged the generated engine into my C++ program, and its all working…

So, in the end, I fell victim to very bad error handling in tao…

Many thanks!!

So, may I know you make it working after replacing the .etlt model, right?
Usually, if the .etlt model is wrong or not available, there is the log as mentioned above.

As I wrote…

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.