Wrong validation mAP

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi,
I trained the YOLOV4 model and validation mAP became 0.44 , but when I inference on the validation images, no object is detected. So how is the mAP is 0.44?

2

How did you get the validation mAP, is it using "tao yolo_v4 evaluate xxx " ?
And for inference, did you use "tao yolo_v4 inference xxx "?

3

Yes, I used TAO evaluate and inference tasks

4

Can you share the training spec file?

5

Here is the spec file:
final_spec.txt (2.3 KB)

6

Please deduce the threshold in the command line when you run "tao yolo_v4 inference xxx ".

7

I set the threshold to 0.2, but an object with score of 0.2 was detected in only one of the images. So how is the mAP is 0.44?

8

Several comments here.

  1. Did you use latest TAO? If yes, I find that your training spec is not compatible. For example, loss_loc_weight, loss_neg_obj_weights and loss_class_weights are not correct.
  2. Could you please share the full training log? I need to check the loss and whether the mAP is 0.44.
  3. And for inference, you mention that “an object with score of 0.2 was detected in only one of the images”. How many images did you run inference? Do you mean only one image can detect bbox and its score is 0.2?
9

Yes I used the latest TAO version. But my training spec file is for the previous version and has the default values of the previous version. I will update the spec file and try again.

Here is the training log:
yolov4_training_log_resnet18.csv (3.5 KB)

inference was run on 20 images.
“Do you mean only one image can detect bbox and its score is 0.2?” yes

10

Yes, please update the spec file and retry.

For the validation 0.44 result, can you run
"tao yolo_v4 evaluate xxx " against your validation set to double confirm?

validation_data_sources: {
label_directory_path: “/workspace/results/yolov4_json/unpruned_model/mask_temp/val/kitti_labels”
image_directory_path: “/workspace/results/yolov4_json/unpruned_model/mask_temp/val/images”

How many images in this validation set? Can you run "tao yolo_v4 inference xxx " against it as well?

11

  • I update the spec file and the results were much better.

  • I run evaluate and inference tasks on previous model with mAP 0.44 to double check and get the same results as before.

  • There are 20 images in the validation dataset.

12

What’s the new mAP? Can you share the training spec and full training log?

13

New mAP is about 0.52
Inference results on validation dataset also were better.
Here is spec file:
final_spec.txt (2.3 KB)
and here is training log:
yolov4_training_log_resnet18.csv (4.0 KB)

14

How many training images?
Also did you set correct anchor_shapes?

From all the mAP results, I am afraid it is still vibrating and the loss did not decrease to a better area.

It is need to finetune some parameters, such as learning rate, epoch, bs, etc.

15

Thanks for your help.

  • There are 70 images.
  • I used TAO kmeans task to generate anchor shapes.
16

Suggest to add more images. And also you can finetune parameters as mentioned above.

1 Like
17

Thank you