Scene Adaptation

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I have trained an object detection model using TLT; but I currently have new training data from other vantage points; and would like to retrain the model. It says online that TLT supports scene adaptation, but I cannot find anything in the documentation to do this.

3

Hi ishan,
Yes, it works.
For you case, firstly, you need to use tlt-dataset-convert to generate tfrecords based on your new training data. Add it into training spec.

data_sources: {
tfrecords_path: “path to your old tfrecords”
image_directory_path: “path to your old training data”
}
data_sources: {
tfrecords_path: “path to your new tfrecords”
image_directory_path: “path to your new training data”
}

Then, set your previous trained model as pre-trained model, and trigger training.

4

Thanks for your reply.

How would I use my previously trained model as a pre - trained model? What is the reason behind having 2 data sources; how would the training spec know that one of them is the ‘old’ training data? What is the benefit of using this method versus retraining the entire model again from scratch, because the number of epochs stay the same?

Is there any documentation that explains this in more detail?

5

You can trigger the Jupyter notebook inside the docker. Then you can see some examples.

  1. How would I use my previously trained model as a pre - trained model?
    ==> You can refer to the examples/specs/retrain_xxx_spec. It tells you as below.
    For example,

pretrained_model_file: “/workspace/tlt-experiments/experiment_dir_pruned/resnet18_nopool_bn_detectnet_v2_pruned.tlt”

  1. What is the reason behind having 2 data sources
    ==> Because you just want to add some new data.

3.how would the training spec know that one of them is the ‘old’ training data?
==> Because you mention two kinds of tfrecords in the spec

  1. What is the benefit of using this method versus retraining the entire model again from scratch, because the number of epochs stay the same?
    ==> You can also retrain from scratch. But you mention that you already trained a tlt model. So, you can use it as a pre-trained model during your new training. That’s tranfer learning.

  2. Is there any documentation that explains this in more detail?
    https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide

6

Thanks for the detailed answer - under the hood does when using transfer learning does it retrain all the layers or only the last couple ones?

7

Take Faster-rcnn as an example, if the file is a model file(.tlt or .hdf5), TLT faster-rcnn will extract the weights from it and then load the weights for initialization. For a layer which has no weights, retraining will not load any weights.