tlt-train: Criteria to save models during training

Hi,

  1. Is there a way to get tlt-train to save only those epochs which result in an improved mAP (assuming DetectNetV2 training) ?

  2. In the Getting Started Guide: https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#trainer, the ‘Supported Values’ explanation for “checkpoint_interval” doesn’t seem to make sense.

Could this be a typo?
If not, can we use this to tell the tlt-train how to save epochs?

Some light on this would be helpful!

Thanks, as always!

Hello Pushkar,
Thank you so much for using TLT. This is indeed a typo. Yes, you may use this parameter to tell tlt-train the interval at which you want to save intermediate models while training.

Thanks,

Hi Praveen,

Thanks for the clarification.

It would be great to know if it is possible to tell tlt-train to only save epochs with improved mAP. And if this is possible, what is the setting in the space file.

Thanks!

Hi Pushkar,
At the moment this isn’t possible with tlt-train.

Thanks!

Got it.

Thanks