How to continue training the model on the new data

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi,
I’m training a model on TAO toolkit using yolo_v3 with resnet18 pre-trained model.
And I got a output model from saving checkpoints.

How to update this neural network model in response to new data(like Continual Learning)?
Following the Tao toolkit guide, I tried to giving the options resume_model_path or pretrained_model_path.
But the AP started zero based not what I trained and got weights before.

Is there any way to continue training the model on the new data?

3

I think you are going to add more new data on existing dataset, right? Or just use the fresh new data?

4

Just new data. Is there any difference of them?

5

For fresh new data, if it is not similar to previous data, it is expected to get low AP during new training. The anchor shape may not match most ground truth boxes in the dataset to help the network learn bounding boxes. You can select some new data which are similar to or part of previous data and try again.

6

Thanks for reply.
Then, Is there other way to continue train with fresh new data ?
How about add more new data on existing dataset ?

7

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks
Yes, you can add new data on the existing dataset and set previous tlt model as the pretrained model. More, for the all dataset, it is better to generate new anchor shapes with kmeans.
More, if training class is added or removed, the AP will be affected.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.