Yolov4 image_mean error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi I am using TAO v4.0 with the following augmentation config:
augmentation_config {
hue: 0.1
saturation: 1.5
exposure:1.5
vertical_flip:0.5
horizontal_flip: 0.5
jitter: 0.3
output_width: 512
output_height: 288
output_channel: 3
randomize_input_shape_period: 0
mosaic_prob: 0.5
mosaic_min_ratio: 0.1
image_mean {
key: ‘b’
value: 103.9
}
image_mean {
key: ‘g’
value: 116.8
}
image_mean {
key: ‘r’
value: 123.7
}
}

I am getting the following error at the first epoch:
INFO: Starting Training Loop.
Epoch 1/80
INFO: ufunc ‘add’ did not contain a loop with signature matching types (dtype(‘<U32’), dtype(‘<U32’)) → dtype(‘<U32’)
Traceback (most recent call last):
File “</usr/local/lib/python3.6/dist-packages/iva/yolo_v4/scripts/train.py>”, line 3, in
File “”, line 152, in
File “”, line 707, in return_func
File “”, line 695, in return_func
File “”, line 148, in main
File “”, line 133, in main
File “”, line 83, in run_experiment
File “”, line 731, in train
File “”, line 232, in fit_generator
File “”, line 308, in tensorboard_images
File “”, line 282, in summary_from_images
File “”, line 219, in summary_from_image
numpy.core._exceptions.UFuncTypeError: ufunc ‘add’ did not contain a loop with signature matching types (dtype(‘<U32’), dtype(‘<U32’)) → dtype(‘<U32’)
Telemetry data couldn’t be sent, but the command ran successfully.
[WARNING]: <urlopen error [Errno -2] Name or service not known>
Execution status: FAIL
2023-02-14 12:12:07,867 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

NOTE: If I remove image_mean values from the augmentation_config training runs okay, so it musth be a problem with image_means. Can you help me with this please?
Thanks

3

You are training with your own dataset, right? How about running with default yolov4 notebook with public KITTI dataset?

4

I was trying to investigate why the images on TensorBoard are greyscale, can confirm that having visualisation and image_mean both set causes the error above.

5

Ahh that makes sense! I actually came across this error investigating exactly the same thing. Did you @douglas.l figure out why images are greyscale? Thank you so much

6

No, not yet, very interested to know why though!

7

@douglas.l May I know how did you set image_mean for greyscale image?

8

@Morganh sorry for confusion, I didn’t set it for greyscale, I copied the example from the documentation:

  image_mean {
    key: 'b'
    value: 103.9
  }
  image_mean {
    key: 'g'
    value: 116.8
  }
  image_mean {
    key: 'r'
    value: 123.7
  }

Unfortunately this doesn’t work, it produces the same error that @benedetta.delfino saw.

I was expecting that the images shown on the TensorBoard would be the augmented images in colour, so that I can see the effect of hue augmentation etc. Instead the augmented images are (mostly!) in greyscale. Surprisingly, it turns out that they are not always greyscale, sometimes I see patches of red and green in the images, but I don’t understand why that sometimes happens. Perhaps the augmentation example images are produced after mean normalisation? Ideally I’d like to see the augmented images before mean normalisation.

1 Like
9

Could you please share the full training spec file and full training log? Thanks.

10

Training log:
training_log.txt (51.6 KB)
Spec file:
spec.txt (2.1 KB)
hope this helps!

11

Hi ,
Sorry for late reply. I cannot reproduce any error with KITTI dataset.
Please try to download KITTI dataset which is mentioned in official notebook(TAO Toolkit Quick Start Guide - NVIDIA Docs) and check again. Thanks.

My spec

root@4291224:/workspace/tlt-experiments/experiment_kitti/yolo_v4# cat spec.txt
random_seed: 42
yolov4_config {
  big_anchor_shape: "[(114.94, 60.67), (159.06, 114.59), (297.59, 176.38)]"
  mid_anchor_shape: "[(42.99, 31.91), (79.57, 31.75), (56.80, 56.93)]"
  small_anchor_shape: "[(15.60, 13.88), (30.25, 20.25), (20.67, 49.63)]"
  box_matching_iou: 0.25
  matching_neutral_box_iou: 0.5
  arch: "resnet"
  nlayers: 18
  arch_conv_blocks: 2
  loss_loc_weight: 1.0
  loss_neg_obj_weights: 1.0
  loss_class_weights: 1.0
  label_smoothing: 0.0
  big_grid_xy_extend: 0.05
  mid_grid_xy_extend: 0.1
  small_grid_xy_extend: 0.2
  freeze_bn: false
  #freeze_blocks: 0
  force_relu: false
}
training_config {
  visualizer {
      enabled: true #False
      num_images: 3
  }
  batch_size_per_gpu: 8
  num_epochs: 80
  enable_qat: false
  checkpoint_interval: 10
  learning_rate {
    soft_start_cosine_annealing_schedule {
      min_learning_rate: 1e-7
      max_learning_rate: 1e-4
      soft_start: 0.3
    }
  }
  regularizer {
    type: L1
    weight: 3e-5
  }
  optimizer {
    adam {
      epsilon: 1e-7
      beta1: 0.9
      beta2: 0.999
      amsgrad: false
    }
  }
  model_ema: true
}
eval_config {
  average_precision_mode: SAMPLE
  batch_size: 8
  matching_iou_threshold: 0.5
}
nms_config {
  confidence_threshold: 0.001
  clustering_iou_threshold: 0.5
  force_on_cpu: true
  top_k: 200
}
augmentation_config {
  hue: 0.1
  saturation: 1.5
  exposure:1.5
  vertical_flip:0
  horizontal_flip: 0.5
  jitter: 0.3
  output_width: 1248
  output_height: 384
  output_channel: 3
  randomize_input_shape_period: 0
  mosaic_prob: 0.5
  mosaic_min_ratio:0.2
  image_mean {
    key: 'b'
    value: 103.9
  }
  image_mean {
    key: 'g'
    value: 116.8
  }
  image_mean {
    key: 'r'
    value: 123.7
  }
}
dataset_config {
  data_sources {
    tfrecords_path: "/workspace/tlt-experiments/data/kitti_dataset/tfrecords/kitti_trainval/*"
    image_directory_path: "/workspace/tlt-experiments/data/kitti_dataset/training"
  }
  include_difficult_in_training: true
  image_extension: "png"
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  #validation_data_sources: {
  #    tfrecords_path: "/workspace/tao-experiments/data/val/tfrecords/val-fold*"
  #    image_directory_path: "/workspace/tao-experiments/data/val"
  #}
  validation_fold: 0
}
12

@Morganh I will try the default kitti dataset. In the meantime, could you show what the images in tensorboard look like for you for the default kitti dataset? Thank you

13

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

See below.

image
image1920×1131 92.1 KB

Refer to steps in Visualizing Training - NVIDIA Docs