Please provide the following information when requesting support.
• Hardware RTX3090
• Network Type unet / vgg16
• TLT Version
dockers: [‘nvidia/tao/tao-too lkit-tf’, ‘nvidia/tao/tao-toolkit-pyt’, ‘nvidia/tao/tao-toolkit-lm’]
format_version: 2.0
toolkit_version: 3.22.02
published_date: 02/28/2022
• Training spec file(If have, please share here)
unet_train_vgg_6S.txt (1.5 KB)
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
After training an initial model at 512X512 images, there are precision issues for the actual location of specific classified pixels.
The actual camera frame is 1280X720, so we created a new data set using the actual camera. All images and masks are png, 1280X720.
As per documentation on tao unet https://docs.nvidia.com/tao/tao-toolkit/text/semantic_segmentation/unet.html#creating-a-configuration-file,
| model_input_height | int | – | The model input height dimension of the model, which should be a multiple of 16. | >100 |
|---|---|---|---|---|
| model_input_width | int | – | The model input width dimension of the model, which should be a multiple of 16. | >100 |
and my spec file has:
model_config {
model_input_width: 1280
model_input_height: 720
model_input_channels: 3
num_layers: 16
all_projections: true
arch: “vgg”
use_batch_norm: False
training_precision {
backend_floatx: FLOAT32
}
}
However training with
!tao unet train --gpus=1 --gpu_index=$GPU_INDEX
-e $SPECS_DIR/unet_train_vgg_6S.txt
-r $USER_EXPERIMENT_DIR/unpruned
-m $USER_EXPERIMENT_DIR/pretrained_vgg16/vgg_16.hdf5
-n model
-k $KEY
Results in error:
ValueError: A
Concatenatelayer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 512, 46, 80), (None, 512, 45, 80)]
2022-05-24 21:20:21,149 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
The complete training log attached
tao unet train log 2022 05 24.txt (33.5 KB)