Please provide the following information when requesting support.
• Hardware V100 (cloud)
• Network Type UNET
• TLT Version - Azure Nvidia GPU Optimized VMI v21.04.1
• Training spec file
random_seed: 42
model_config {
model_input_width: 572
model_input_height: 572
model_input_channels: 3
num_layers: 18
all_projections: true
arch: “resnet”
use_batch_norm: False
training_precision {
backend_floatx: FLOAT32
}
}
training_config {
batch_size: 3
epochs: 50
log_summary_steps: 10
checkpoint_interval: 1
loss: “cross_dice_sum”
learning_rate:0.0001
regularizer {
type: L2
weight: 2e-5
}
optimizer {
adam {
epsilon: 9.99999993923e-09
beta1: 0.899999976158
beta2: 0.999000012875
}
}
}
dataset_config {
dataset: “cityscapes”
augment: False
input_image_type: “color”
train_images_path:“/home/lumen/semantic_cityscapes/data/images/train”
train_masks_path:“/home/lumen/semantic_cityscapes/data/masks/train”
val_images_path:“/home/lumen/semantic_cityscapes/data/images/val”
val_masks_path:“/home/lumen/semantic_cityscapes/data/masks/val”
test_images_path:“/home/lumen/semantic_cityscapes/data/images/test”
data_class_config {
target_classes {
name: “driveable”
mapping_class: “driveable”
label_id: 1
}
target_classes {
name: “nondriveable”
mapping_class: “nondriveable”
label_id: 2
}
target_classes {
name: “other”
mapping_class: “other”
label_id: 0
}
}
}
More Info:
I have restructured the cityscapes dataset into 3 classes and for TAO format, as shown here Data Annotation Format — TAO Toolkit 3.22.05 documentation
When I am trying to run the
!tao unet train --gpus=1 --gpu_index=$GPU_INDEX
-e $SPECS_DIR/unet_train_resnet_unet_cityscapes.txt
-r $USER_EXPERIMENT_DIR/trained
-m $USER_EXPERIMENT_DIR/pretrained_resnet18/pretrained_semantic_segmentation_vresnet18/resnet_18.hdf5
-n model_cityscapes
-k $KEY
command I get the following errors
2022-06-30 22:26:13,841 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1
2022-06-30 22:26:13,841 [INFO] iva.unet.model.utilities: Label Id 2: Train Id 2
2022-06-30 22:26:13,841 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0
2022-06-30 22:26:13,843 [INFO] root: [Errno 2] No such file or directory: ‘/home/lumen/semantic_cityscapes/data/images/train’
Traceback (most recent call last):
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py”, line 519, in
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py”, line 515, in main
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py”, line 502, in main
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py”, line 378, in run_experiment
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py”, line 265, in train_unet
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py”, line 89, in init
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py”, line 208, in get_images_masks_lists
File “/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py”, line 169, in read_data_image_dir
FileNotFoundError: [Errno 2] No such file or directory: ‘/home/lumen/semantic_cityscapes/data/images/train’
2022-06-30 22:26:15,470 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
even though when running !ls /home/lumen/semantic_cityscapes/data/images/train all the images are listed. I am running the notebook from the root folder, and the dataset in the path above is not permission restricted.
P.S. I cannot seem to format the post (buttons don’t do anything) in order for it to look more comprehensible.