Zero average accuracy on custom dataset

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Getting 0 average accuracy on resnet_34 model.

• Hardware (T4)
• Network Type (Detectnet_v2/Resnet_34)
Annotation file example:(
Cam1_0000.txt (79 Bytes))
• Tao toolkit: 4.0.1
• Training spec file(
detectnet_v2_train_resnet18_kitti.txt (3.3 KB)
) (3.2 KB))

Snippet of Evaluation output:

eval
eval873×293 129 KB

Tfrecord log file:
tfrecords.txt (5.0 KB)

Train log file:
train_log.txt (224.5 KB)

Input image dimension:
Width - 640
height - 480

I just want only one class person to be trained and evaluated.

2

Please change

  target_class_mapping {
    key: "Person"
    value: "Person"
  }

to

  target_class_mapping {
    key: "person"
    value: "person"
  }
3

We are trying to detect a person using peoplenet model in missionary environment where the person should get detected when he is near and up to some distance away from camera and should get detect some percent when far from camera.
We have a dataset of 27,000 images.

So how to fine tune the training parameters of peoplenet training config to get the higher accuracy of the model.
detectnet_v2_train_resnet18_kitti.txt (3.3 KB)

4

So, the zero AP issue is fixed, right?
Could you share latest training log?
And where did you download below model?
pretrained_model_file: "/workspace/tao-experiments/detectnet_v2/pretrained_resnet34/peoplenet_vtrainable_unencrypted_v2.6/model.hdf5".

5

Yes zero value error is fixed the model is showing a precision value but it is comparatively low.

First, we had around 43,000 images trained for 200 epoch it had give 48% mAP score for the class “person”. Since, in the older Forums it had mentioned to keep the input image dimension for all the ones but we had different dimension for other 15,000. So, we had removed that.

But now after removing it count had changed to 27,000 but the mAP score is not increased but decreased to 24%.

Training log:
trainlog.txt (2.5 MB)

Training config .txt:
detectnet_v2_train_resnet18_kitti.txt (3.3 KB)

The pretrained weights were downloaded from the below line:

In this we had selectively downloaded trainable_unencrypted_v2.6 pretrained weights file.

6

Please modify below and retry.

  • output_image_width: 960
    output_image_height: 544
  • maximum_height: 9999
    maximum_width: 9999
  • lower batch_size_per_gpu. For example, 4.
7

We retrained the model after making mentioned changes with the dataset with same dimensions image dataset but the MAP was 11 percent, then we added the images with different dimensions with those same dimension dataset and it contained around 27000 images and retrained the model and the MAP was 50 percent.

How can we increase the accuracy of the model.

8

For your current training you can try the following experimentation steps.

  • setting your batch_size_per_gpu to 2,4 or 8 at the most
  • set enable_autoweighting: False

Also, as an initial experiment, it may also help to set the learning rate config parameters to max_learning_rate: 5e-5, min_learning_rate: 5e-7 and disable regularization by setting regularizer_type: NO_REG
Disabling regularization will effect pruning, but as a first step for increasing accuracy, that will be a good starting point.

9

Also, for your dataset is there only person class object, or do you have other objects in your dataset?

10

We have only one class ‘person’ object in dataset.

11

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

So, adding the dataset can help improve the mAP.

Besides above hints(such as lower bs, changing learning rate, etc), please try to use deeper backbones to train.

More, if possible, could you share an example of what the dataset looks like? And are the images all the same resolution?

12

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.