We followed the link provided below to train a custom model using EfficientDet_tf2: TAO Toolkit Getting Started | NVIDIA NGC
We successfully trained the model using the COCO dataset provided in the EfficientDet_tf2 notebook. During inference, the model correctly detects objects.
However, when we train the model with our own data, the training completes successfully, but it does not detect any objects.
Here are the details of our custom dataset training:
- Number of images for training: 1396
- We created annotation files as mentioned in the following link: Data Annotation Format - NVIDIA Docs The only change we made is setting the segmentation field to blank [ ] since we only have bounding boxes. While creating TF records, we set include_masks to False.
- Training time: 1 hour
- num_epochs: 300
- num_examples_per_epoch: 349
- learning_rate: 0.8
- image_size: ‘512x512’
- num_classes: 2
- Actual image resolution is 2592x1944 and are approximately 1.6MB in size
- Instance type: AWS g4dn.2xlarge, which has 4 GPUs
Apart from the above details, we kept almost all settings as they were.
We would appreciate any insights or suggestions regarding the issue we are facing. Thank you.
train.json (2.4 MB)