Refer to
https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/text/faqs.html
Distribute the dataset class: How do I balance the weight between classes if the dataset has significantly higher samples for one class versus another?
To account for imbalance, increase the class_weight for classes with fewer samples. You can also try disabling enable_autoweighting; in this case initial_weight is used to control cov/regression weighting. It is important to keep the number of samples of different classes balanced, which helps improve mAP.