Wrong inference using trained model in TLT

[leo2105]

Please provide the following information when requesting support.

• Hardware : V100
• Network Type : Detectnet_v2
• TLT Version : I’m using container of TLT 3.0 but commands like TLT 2.0
• Training spec file : trafficcamnet_retrain.txt (6.9 KB)

Hi

I have issues when I make inference using a model trained on detectnet v2 using trafficcamnet as pre-trained model in TLT using a custom dataset (which has the same 4 classes). It looks that TLT randomize the weights of the neural network instead of using the weights of trafficcamnet. That’s shouldnt be happening right?

TrafficCamNet .etlt inference in Deepstream-5.0:

0 epochs - .etlt Model trained using weights of TrafficCamNet inference in Deepstream-5.0

240 epochs - .etlt Model trained using weights of TrafficCamNet inference in Deepstream-5.0

So, It looks that is training from scratch.
The last image after 240 epochs could make sense, because my custom dataset has few images and each image has few bboxes.

Python commands:

cmd = "detectnet_v2 train -e {}/trafficcamnet_retrain.txt  \
                              -r {}/experiment_dir_unpruned \
                              -k tlt_encode \
                              -n resnet18_detector_unpruned \
                              --gpus {}".format(local_specs_tlt_path, local_user_exp_dir_path, nro_gpus)
os.system(cmd)

Thanks in advance.

[Morganh]

In your spec, there is
pretrained_model_file: "/workspace/trafficcamnet/experiment_dir_pruned/trafficcamnet_pruned_2608.tlt"

Is it the model you have pruned?
It is not the official trafficcamnet model , right?

[leo2105]

I just changed the name of this one : https://ngc.nvidia.com/catalog/models/nvidia:tlt_trafficcamnet, I tried many times training just downloading the model from the ngc catalog

[Morganh]

The “pretrained_model_file” parameter defines the path to a pretrained TAO model file. If the load_graph flag is set to false , it is assumed that only the weights of the pretrained model file is to be used. In this case, TAO train constructs the feature extractor graph in the experiment and loads the weights from the pretrained model file that has matching layer names. Thus, transfer learning across different resolutions and domains are supported. For layers that may be absent in the pretrained model, the tool initializes them with random weights and skips the import for that layer.
Please set “load_graph” to True and retry.
https://docs.nvidia.com/tao/tao-toolkit/text/object_detection/detectnet_v2.html#model-config