Unet Inference Error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi.
I get this error when inference with unet TLT and TAO. This is inference log:

Total params: 86,617,858
Trainable params: 86,096,002
Non-trainable params: 521,856
__________________________________________________________________________________________________
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py", line 412, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py", line 408, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py", line 318, in run_experiment
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py", line 278, in infer_unet
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py", line 205, in run_inference_tlt
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py", line 142, in visualize_masks
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/inference.py", line 86, in overlay_seg_image
**AttributeError: 'NoneType' object has no attribute 'shape'**

This is while I have been using unet for almost 2 months and I have not got this error yet. This is my spec file:

random_seed: 42
model_config {
  model_input_width: 640
  model_input_height: 640
  model_input_channels: 3
  num_layers: 101
  all_projections: true
  arch: "resnet"
  freeze_blocks: 0
  freeze_blocks: 1
  use_batch_norm: True
  training_precision {
    backend_floatx: FLOAT32
  }
}

training_config {
  batch_size: 2
  epochs: 500
  log_summary_steps: 499
  checkpoint_interval: 5
  loss: "cross_dice_sum"
  learning_rate:0.0002
  regularizer {
    type: L2
    weight: 3e-09
  }
  optimizer {
    adam {
      epsilon: 9.99999993923e-09
      beta1: 0.899999976158
      beta2: 0.999000012875
    }
  }
}
dataset_config {
  dataset: "custom"
  augment: False
  augmentation_config {
    spatial_augmentation {
    hflip_probability : 0.5
    vflip_probability : 0.5
    crop_and_resize_prob : 0.5
    }
    brightness_augmentation {
      delta: 0.2
    }
  }
  input_image_type: "color"
  train_images_path: "/workspace/tlt/results/tlt_unet_corrosion1000_resnet101/unpruned_model/corrosion_1000_temp/train/images/"
  train_masks_path: "/workspace/tlt/results/tlt_unet_corrosion1000_resnet101/unpruned_model/corrosion_1000_temp/train/masks"

  val_images_path: "/workspace/tlt/results/tlt_unet_corrosion1000_resnet101/unpruned_model/corrosion_1000_temp/val/images"
  val_masks_path: "/workspace/tlt/results/tlt_unet_corrosion1000_resnet101/unpruned_model/corrosion_1000_temp/val/masks"

  test_images_path: "/workspace/tlt/results/2/"

  data_class_config {
    target_classes {
      name: "background"
      mapping_class: "background"
      label_id: 0
    }
    target_classes {
      name: "foreground"
      mapping_class: "foreground"
      label_id: 255
    }  
  }
}
3

Please check if there is something changing in your dataset.

5

It is not a problem when I inference the images one by one, but when, as always, I give all of the images as input, it gives an error. But I do not get the same error with other models like mask rcnn or yolo_v4

6

Could you change above to
test_images_path: "/workspace/tlt/results/2 "

7

Hi.
I changed but the problem was not solved

8

Can you change to “/workspace/tlt/results/tlt_unet_corrosion1000_resnet101/unpruned_model/corrosion_1000_temp/val/images” and try again?

9

I changed and the inference was successful.

10

So,
test_images_path: "/workspace/tlt/results/2 " ==> error
test_images_path: “/workspace/tlt/results/tlt_unet_corrosion1000_resnet101/unpruned_model/corrosion_1000_temp/val/images” ==> successful

Please check the difference between these two image folders.

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.