Retraining Trafficcamnet with custom vehicle dataset

• Hardware (T4/V100/Xavier/Nano/etc) - Tesla T4
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) - Detectnet_v2
• TLT Version - 3.0

Hi,

I have trained a model using custom dataset for classes : car and two-wheelers using trafficcamnet unpruned model as a base. I had some doubts in altering the specs file for training, pruning and retrained the model for which I got support from this post → Retrain TrafficCamNet with custom vehicle dataset using TLT 3.0

I have generated the dataset by grabbing the frames from videos and then annotating it using KITTI labelling tool.

After training the model I tried comparing it with the pretrained trafficcamnet model and the below screenshots where the results:

Custom Model Detections


Pretrained Model Detections


The custom model is not able to detect the car and two-wheelers but not persons and roadsigns. As per my understanding I have used the pretrained model as base and added dataset to increase the accuracy of car detections while retaining the older detection rate and improve it. But here I can only detect the cars and two-wheelers with low-detection rate as compared to pretraind model but not the other classes that are already trained in the pretrained model. How should I train the model in way to retain the exisiting model characteristics (detecting cars, two-wheelers, persons and roadsigns) and also improve the detection rate?

The specs files that I used for training is attached below along with the notebook file:
detectnet_v2_inference_kitti_etlt.txt (2.4 KB)
detectnet_v2_inference_kitti_tlt.txt (1.9 KB)
detectnet_v2_retrain_resnet18_kitti.txt (4.2 KB)
detectnet_v2_tfrecords_kitti_trainval.txt (308 Bytes)
detectnet_v2_train_resnet18_kitti.txt (4.2 KB)
detectnet_v2.ipynb (20.3 MB)

I need help on this.

According to the training spec file, the custom model is trained on two classes (car and two-wheelers). So it can only detect the cars and two-wheelers. In order to detect four classes, need to prepare corresponding dataset and set four classes in training spec for training.
You aims to train a two classes model, right?
Your training dataset(9227 images) contains 29684 cars and only 1818 two-wheelers.
Could you train via

  • set the lr rate very small
    (min_learning_rate: 5e-06
    max_learning_rate: 5e-05 )
  • disable enable_autoweighting
  • set higher class weight (0.2) for two-wheelers class and lower class weight (0.01) for car class

@Morganh

My objective here is to add more custom annotated vehicle data to the exisiting pretrained model (Trafficcamnet) and try to increase its accuracy in detecting vehicles - car and two-wheelers (bicycle class).

How can i achieve this and train a model using the trafficcamnet as base and improve detection accuracy while retaining the detections provided by the pretrained model rather than training a new model with car and two-wheeler classes that might require huge amount of data for training.

Could you shed some light on this?

Thanks in advance.

May I know that

  • Is it public dataset for your custom 960x544 images? I want to run some experiments and also try to reproduce your result.
  • How did you get above inference result of " Pretrained Model Detections" ? You were running “tao detectnet_v2 inference xxx” ,right?

@Morganh

  1. No the dataset is not public, its a proprietary dataset. We have sliced frames from video recording of the area and then converted the images to 960x544 images and annotated them.

  2. I added the pretrained model to the config file in the samples/config/tlt_pretrained_models/ and ran the model with the command : deepstream-app -c <deepstream_app_source1_trafficcamnet> and replaced the model paths to the custom trained model to check out its detections.

Also ran the command in the detectnetv2 notebook file for visulalising the inferences during the model training process :

!tao detectnet_v2 inference -e $SPECS_DIR/detectnet_v2_inference_kitti_tlt.txt
-o $USER_EXPERIMENT_DIR/tlt_infer_testing
-i $DATA_DOWNLOAD_DIR/testing/images
-k $KEY

More, in terms of the result " Custom Model Detections" , is it from your unpruned model or pruned model?

@Morganh

The uploaded test result video recordings are after pruning, retraining the pruned and exporting the model.

Could you export the unpruned model and try to run again with fp32 mode?

@Morganh yes, I’ll do that and share the results.

@Morganh

I exported the unpruned model located in experiment_direxperiment_dir_unpruned/weights and changed the network mode to fp32 in config file and ran the application over video files. The below recordings are the results:


Your training images are similar to the 2nd figure, right?

@Morganh Yes. They are similar to the second recording.

So the training are learning to fit your own images. Please ignore the 1st case.
Could you try to run more experiments mentioned above?

@Morganh Should i follow these?

But my only concern is that why is the model not detecting cars as earlier pretrained model as I’m adding the cars to the same car class not a new one. I want to utilize the pretrained model features as it is already trained on over 1.2M images.

You can try to modify your spec as below and run 10 epochs.

model_config {
load_graph: True
...

soft_start_annealing_schedule {
min_learning_rate: 10e-10
max_learning_rate: 10e-10
soft_start: 0.0
annealing: 0.3
}
...

@Morganh

  1. In which spec file (detectnet_v2_retrain_resnet18_kitti.txt or detectnet_v2_train_resnet18_kitti.txt) should change the following

model_config {
load_graph: True

soft_start_annealing_schedule {
min_learning_rate: 10e-10
max_learning_rate: 10e-10
soft_start: 0.0
annealing: 0.3
}

  1. Should I start from the first step i.e creation of tfrecords then train the model, prune it and retrain it all with 10epoch for the test that you mentioned?

  2. I’m using the trafficcamnet_pretrained_unpruned_v1.0 as the pretrained model in the specs file for train step? Is it correct or should I pruned model as a base?

@Morganh Can you please advise on this so that I can continue the training process.

Some modification for my comments earlier… Could you refer to Tlt3.0 retrain trafficcamnet getting the error when do the evaluation - #26 by Morganh to run evaluation instead of training ?

Actually I also try to set up similar experiments and reproduce your result. If you have similar public dataset, please share with me. Thanks.

@Morganh
Thanks for your prompt reply.

I will change the following in both train and retrain spec files and run a training for 10 epochs

  • set the lr rate very small
    (min_learning_rate: 5e-06
    max_learning_rate: 5e-05 )

  • disable enable_autoweighting

  • set higher class weight (0.2) for two-wheelers class and lower class weight (0.01) for car class

  • Will also modify the retrain spec file that has these below lines

model_config {
load_graph: True

soft_start_annealing_schedule {
min_learning_rate: 10e-10
max_learning_rate: 10e-10
soft_start: 0.0
annealing: 0.3
}

But the only thing here is the minimum and max learning rate, should I set min_learning_rate: 10e-10
max_learning_rate: 10e-10 (or) min_learning_rate: 5e-06 max_learning_rate: 5e-05 ?

Yes, this is the original suggestion to run training. You can try more epochs.

For this setting, it is just running evaluation with the trafficcamnet pretrained model.