Mean average precision of 0.00 for detectnet_v2 using Tao Toolkit

[user82614]

Please provide the following information when requesting support.

• Hardware - 4x A6000 (I’m only using one for this use case)
• Network Type Detectnet_v2
• Training spec file (attached)

detectnet_v2_train_resnet18_kitti.txt (5.9 KB)
0.txt (441 Bytes)

Please find attached the training config file, example of a picture and associated label file of my use case.

The image size is 1920 x 1080 p

I’m currently training a detectnetv2 with resnset18 model. I’ve got about 3300 images to train with.

After a certain amount of epochs, it runs a evaluation and i keep getting mAP of 0 for each class (i have a loss of 7.9 after 20 epochs but I still expected to get something back for the mAP).

Validation cost: 0.004761
Mean average_precision (in %): 0.0000

class name      average precision (in %)
------------  --------------------------
cracked                                0
offset                                 0
standard                               0

Can someone please take a look at the files and please let me know if I have done something wrong. I looked at other posts similar to this and followed their solutions but didn’t get anywhere, so I am not sure if I am missing something obvious.

Thank You!

[Morganh]

Can you upload the training log and tfrecord generation log?

[user82614]

Sure, thanks

tfrecord_log (5.3 KB)
training_log (180.8 KB)

Just as a consideration which came to my mind, I noticed the images produced by replicator is png with 4 channels (RGBA)… whilst all the other images used by the nvidia tutorials use 3 channels (RGB)… does this matter with using Tao Toolkit?

[Morganh]

Please use 3 channels images.
More ,could you change as below if trigger new training?

    zoom_min: 0.1  **(set to 1.0)**
    zoom_max: 0.1   **(set to 1.0)**
  }
}

And, if trigger new evaluation only, please try below.

postprocessing_config {
  target_class_config {
    key: "standard"
    value {
      clustering_config {
        clustering_algorithm: DBSCAN
        dbscan_confidence_threshold: 0.9  **(try lower value)**
        coverage_threshold: 0.00499999988824
        dbscan_eps: 0.20000000298
        dbscan_min_samples: 0.0500000007451
        minimum_bounding_box_height: 20   **(set to lower value)**

[user82614]

Hi Morganh,

I’ve changed the images to 3 channel (RGB) and changed the zoom augmentation values - it worked! The model now trains with synthetic data!

Thanks for your help. I hope this is useful for other developers in this forum.