NVIDIA TAO - detectnet_v2 - 0mAP problem

[gglrthiru]

whenever i train the detectnet_v2 architecture in visdrone dataset, I get 0mAP problem. I thought that this is because of wrong class_maps but, the classmaps are fine. The model isn’t learning anything. I have provided the link to drive where you can find files: tao_mounts.json, model_log, training_spec file

• Hardware RTX 2070 SUPER

• Network Type: DetectNet_v2

• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here) 3.21.08

• Training spec file(If have, please share here): issues - Google Drive

• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

1. Download the visdrone dataset and convert it to tfrecords using dataset_convert tool
2. After converting, train the detectnet_v2 model on it, this causes a 0mAP problem and the model doesn’t learn anything

I followed the steps from the NVIDIA TAO documentation to train the detectnet_v2 model

[Morganh]

See DetectNet_v2 - NVIDIA Docs
For detectnet_v2 network,

The train tool does not support training on images of multiple resolutions. However, the dataloader does support resizing images to the input resolution defined in the specification file. This can be enabled by setting the enable_auto_resize parameter to true in the augmentation_config module of the spec file.

Does visdrone dataset have training images of multiple resolutions ?

[gglrthiru]

yeah it has images of multiple resolutions but i have enabled auto_resize parameter

[Morganh]

For detectnet_v2 network, the train tool does not support training on images of multiple resolutions.
Please resize images/labels offline to the same resolution.

[gglrthiru]

actually auto_resize does this right, IDK correct me if I’m wrong!

[gglrthiru]

I tried this way too. But, the model didn’t learn anything and the same problem persists

[Morganh]

Seems that the objects in visdrone dataset are very small. In this case, please modify evaluation_box_config and run evaluation again.

  evaluation_box_config {
    key: "car"
    value {
      minimum_height: 20
      maximum_height: 9999
      minimum_width: 10
      maximum_width: 9999
    }
  }

  evaluation_box_config {
    key: "pedestrian"
    value {
      minimum_height: 20
      maximum_height: 9999
      minimum_width: 10
      maximum_width: 9999
    }
  }

to

  evaluation_box_config {
    key: "car"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }

  evaluation_box_config {
    key: "pedestrian"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }

[gglrthiru]

No, I tried it just now and I got the same results, the model didn’t learn anything

The loss will be very less from the start and after finishing, if we check the status.json file, it will show 0mAP and 0 precision in all objects that the model was trained.

[Morganh]

Seems that the training is normal but evaluation is not.
Please change minimum_bounding_box_height to a lower value too. For example,

postprocessing_config {
  target_class_config {
    key: "car"
    value {
      clustering_config {
        clustering_algorithm: NMS
        coverage_threshold: 0.005
        nms_iou_threshold: 0.5
        nms_confidence_threshold: 0.5
        minimum_bounding_box_height: 4
      }
    }
  }

[gglrthiru]

I changed it but still the same problem. I have a doubt, does the box coords in the label txt files needs to be normalized.

This is the output after changing the post_process_config, still the model doesnt learn.

[gglrthiru]

The status.json file report for first two epochs

[Morganh]

As mentioned above, please resize the images and also labels.
After done, please check if the bboxes are correct.