DINO Retail Object Detection - Distillation

Accelerated Computing Intelligent Video Analytics TAO Toolkit
,
1

Hi,
I am trying to train a DINO Retail Object Detection using the fan_tiny backbone. I have successfully fine tuned fan_base and fan_small on our retail object dataset using the available trainable models from the NGC page .
Now, I want to see if I can reach similar/close accuracy using fan_tiny and get higher inference speed.

First, I could not find any retail object trainable model fan_tiny in NGC. Is there one I can use or is there none avaialble?

Since I could not find any trainable fan_tiny, I’ve started to run some knowledge distillation using our fine tuned fan_small as the teacher and Pre-trained DINO ImageNet weights fan_hybrid_tiny as the student.
At the moment, it is not converging at all.

Before I share my specs etc…, I just would like to know if that distillation is possible and if I am using the right student tiny model.

Thank you

• Network Type: DINO Retail Object Detection
• TAO Version: 5.5.0

3

Please try to use fan_tiny pretrained model( Pre-trained FAN based ImageNet Classification weights | NVIDIA NGC) to finetune your retail object dataset.

4

That’s the one I’ve used and I mentioned above. But it is not converging at all after 25 epochs and mAP on valid is staying close to 0.

Please have a look at the specs file and log outputs and let me know if you see anything wrong. There are a bunch of errors showing up. But the distillation is starting. There are just couple of epochs in the log but I went until 25 and it was the same.

Thanks

train:
  pretrained_model_path: /workspace/tao-experiments/models/pretrained_dino_image_net/fan_hybrid_tiny.pth
  verbose: True
  num_gpus: 4
  num_nodes: 1
  validation_interval: 1
  checkpoint_interval: 1
  optim:
    lr_backbone: 1e-6
    lr: 1e-5
    lr_steps: [11]
    momentum: 0.9
  num_epochs: 25
dataset:
  train_data_sources:
    - image_dir: /workspace/tao-experiments/data/train
      json_file: /workspace/tao-experiments/data/annotations/instances_train.json
  val_data_sources:
    - image_dir: /workspace/tao-experiments/data/test
      json_file: /workspace/tao-experiments/data/annotations/instances_test.json
  num_classes: 2
  batch_size: 2
  workers: 8
  augmentation:
    fixed_padding: False
model:
  backbone: fan_tiny
  train_backbone: True
  num_feature_levels: 4
  dec_layers: 6
  enc_layers: 6
  num_queries: 900
  num_select: 100
  dropout_ratio: 0.0
  dim_feedforward: 2048
distill:
  teacher:
    backbone: fan_small
    train_backbone: False
    num_feature_levels: 4
    dec_layers: 6
    enc_layers: 6
    num_queries: 900
    dropout_ratio: 0.0
    dim_feedforward: 2048
  pretrained_teacher_model_path: /workspace/tao-experiments/models/trainable_retail_object_detection_binary_v2.2.1.3/dino_model_epoch_011.pth
  bindings:
  - teacher_module_name: model.backbone.0.body
    student_module_name: model.backbone.0.body
    criterion: L2
    weight: 1.0
results_dir: /workspace/tao-experiments/results/retail_object_detection/trainings/training10

train10_distill_log.txt (515.0 KB)

5

I mean you can use the pretrained model to run DINO training with your own dataset firstly.

6

yes but there is no fan_tiny pretrained model available. So the only solution to get a fan_tiny is either to train from scratch (but I don’t have enough data) or perform distillation. Am I correct?

7

I suggest you can run DINO training with your dataset and this fan_tiny pretained model. To check the result from DINO network. Not train from Retail Object Detection network.

8

just to be clear are you saying I can train DINO using fan_tiny

model:
  pretrained_backbone_path: "/pretrained_fan_image_net/fan_hybrid_tiny.pth"
  backbone: fan_tiny
  train_backbone: True

and not specifying any train.pretrained_model_path ?

Or should I specify a train.pretrained_model_path with a different backbone like DINO_retail_fan_small?

Thanks

9

Yes. Refer to tao_tutorials/notebooks/tao_launcher_starter_kit/dino/specs/train.yaml at tao_5.5_release · NVIDIA/tao_tutorials · GitHub. But change something as below.

model:
  pretrained_model_path: /path/to/your-fan-tiny-pretrained-model
  backbone: fan_tiny
  train_backbone: True
  num_queries: 900

Above is for DINO training instead of distillation.

For DINO distillation, please refer to tao_tutorials/notebooks/tao_launcher_starter_kit/dino/specs/distill.yaml at tao_5.5_release · NVIDIA/tao_tutorials · GitHub.

11

model doesn’t have a pretrained_model_path, just a pretrained_backbone_path.
I’ve double checked in nvidia_tao_pytorch/cv/dino/config/model.py and anyway I am getting the error:

Error merging 'train_distill.yaml' with schema
Key 'pretrained_model_path' not in 'DINOModelConfig'
  full_key: model.pretrained_model_path
  reference_type=DINOModelConfig
  object_type=DINOModelConfig

so I am not sure what you are suggesting when you said:

But change something as below.

?

Distillation is running at the moment with the following config but loss is pretty high and mAP really low.

model:
  pretrained_backbone_path: "/pretrained_fan_image_net/fan_hybrid_tiny.pth"
  backbone: fan_tiny
  train_backbone: True
12

Checked. You are right. I am misled by the document. Please set pretrained_backbone_path under model section. Refer to tao_pytorch_backend/nvidia_tao_pytorch/cv/dino/config/model.py at dc07b02eb78c2eb868315107892b466496e55a0f · NVIDIA/tao_pytorch_backend · GitHub

13

Thanks, I managed to perform distillation from DINO fan_small to DINO fan_tiny…
mAP went from 0.80 to 0.56. Not sure if it’s because the tiny backbone is really less accurate or if the distillation process is not performant enough.

14

As mentioned above, you can run DINO training instead of using DINO-distillation training. Please run DINO training by using the fan_tiny pretrained model and your own dataset.

15

Actually I did it already and the results are not as great.
By doing DINO training I can’t benefit from the model pre-trained on Nvidia retail objects dataset.Nvidia dataset is much larger than mine.

Do you know if there is any plan to release a DINO retail object detection fan_tiny model?
Thanks

16

Unfortunately there is not model for fan_tiny.
For DINO training, please try to use another trainable https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pretrained_dino_coco/files?version=dino_gc_vit_tiny_trainable_v1.0 to do experiment. The backbone is gc_vit_tiny instead of fan_tiny.

1 Like