Please provide the following information when requesting support.
• Hardware RTX3090, RTX2070
• Network Type unet
• TLT Version
Configuration of the TAO Toolkit Instance
dockers: [‘nvidia/tao/tao-toolkit-tf’, ‘nvidia/tao/tao-toolkit-pyt’, ‘nvidia/tao/tao-toolkit-lm’]
format_version: 2.0
toolkit_version: 3.22.02
published_date: 02/28/2022
• Training spec file
unet_retrain_vgg_6S1100.txt (1.5 KB)
• How to reproduce the issue ?
Using the unet notebook with my own custom data (color images 704 X 1280), it trains, prunes, and retrains, evaluates, and does inference, all well.
Exporting the model with the command tao unet export produces a working tensorRT engine that I can use on C++ on the same machine.
!tao unet export --gpu_index=$GPU_INDEX -m $USER_EXPERIMENT_DIR/retrain/weights/model_retrained.tlt \
-k $KEY \
-e $SPECS_DIR/unet_retrain_vgg_6S1100.txt \
-o $USER_EXPERIMENT_DIR/export/tao.fp32_6S003A01.etlt \
--data_type fp32 \
--engine_file $USER_EXPERIMENT_DIR/export/tao.fp32_S6003A1.engine \
--max_batch_size 1 \
--batch_size 1
So far so good…
But using the tao converter command produces an error:
!tao converter -k $KEY \
-e $USER_EXPERIMENT_DIR/export/trt.fp16.6S003A1.engine -t fp16 \
-d 3,704,1280 \
-p input_1,1x3x704x1280,4x3x704x1280,16x3x704x1280 \
$USER_EXPERIMENT_DIR/retrain/weights/model_retrained.etlt
Results in
2022-08-23 23:56:15,569 [INFO] root: Registry: [‘nvcr.io’]
2022-08-23 23:56:15,734 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
[INFO] [MemUsageChange] Init CUDA: CPU +536, GPU +0, now: CPU 757, GPU 2134 (MiB)
[ERROR] UffParser: Unsupported number of graph 0
[ERROR] Failed to parse the model, please check the encoding key to make sure it’s correct
[ERROR] 4: [network.cpp::validate::2411] Error Code 4: Internal Error (Network must have at least one output)
[ERROR] Unable to create engine
2022-08-23 23:56:18,133 [INFO] tlt.components.docker_handler.docker_handler: Stopping container
I need to run the model on another computer with a different GPU (RTX2070 8GB), but same CUDA, cudNN, and tensorRT versions, and therefore need to convert the model on that computer, which is giving the same error message there, so I was using the tao unet notebook as a testbed for running the tao-converter utility on the other computer…
The output layer of the network is
softmax_1 (Softmax) (None, 704, 1280, 6) 0 permute_1[0][0]
so I added that and run the converter as:
!tao converter -k $KEY \
-e $USER_EXPERIMENT_DIR/export/trt.fp16.6S003A1.engine -t fp16 \
-d 3,704,1280 \
-i nchw \
-p input_1,1x3x704x1280,1x3x704x1280,1x3x704x1280 \
-o softmax_1,0x704x1280x6 \
$USER_EXPERIMENT_DIR/retrain/weights/model_retrained.etlt
Which results in another error: Number of optimization profiles does not match model input node number:
2022-08-24 00:28:01,952 [INFO] root: Registry: [‘nvcr.io’]
2022-08-24 00:28:02,119 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
[INFO] [MemUsageChange] Init CUDA: CPU +536, GPU +0, now: CPU 542, GPU 2414 (MiB)
[INFO] ----------------------------------------------------------------
[INFO] Input filename: /tmp/fileW8SZFK
[INFO] ONNX IR version: 0.0.0
[INFO] Opset version: 0
[INFO] Producer name:
[INFO] Producer version:
[INFO] Domain:
[INFO] Model version: 0
[INFO] Doc string:
[INFO] ----------------------------------------------------------------
[ERROR] Number of optimization profiles does not match model input node number.
2022-08-24 00:28:04,421 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
Thanks in advance for the help!