Backbone models for RT-DETR

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Where can i find backbone models for RT-DETR? on the documnetation is say model support ResNet, EfficientViT, FAN ..
I can not find them on NGC. I need RESNET, Fansmall, and fficientViT.
Thanks!

3

The supported backbone can be found in tao_pytorch_backend/nvidia_tao_pytorch/cv/rtdetr/model/backbone at release/6.25.10 · NVIDIA/tao_pytorch_backend · GitHub.

  1. For fan_small, please set backbone: fan_small_12_p4_hybrid. The pretrained model is in https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/models/pretrained_fan_classification_imagenet/files?version=fan_hybrid_small.

  2. For resnet_50, you can use below.
    wget https://download.pytorch.org/models/resnet50-0676ba61.pth
    Similar topic is: Unable to reproduce RT-DETR original results on COCO - #12 by vpraveen

  3. .For EfficientViT, you can also set the backbone name according to tao_pytorch_backend/nvidia_tao_pytorch/cv/rtdetr/model/backbone/efficientvit.py at release/6.25.10 · NVIDIA/tao_pytorch_backend · GitHub.
    TAO doesn’t have pretrained commercial weights for effiicientViT released yet.There are several model candidates that were released as part of the EfficientViT release, which you can pick from CVHub520/efficientvit: EfficientViT is a new family of vision models for efficient high-resolution vision. . We are compatible with the model defined there and hence the weights as well. You may try to use EfficientViT-L2 and double check.
    Similar topic: Unable to train RT-DETR using efficientvit_b1 as backbone - #6 by Morganh.

  4. More other backbones can be found in ngc.
    For example,
    pretrained models of ConvNextV2 are in Pretrained ConvNeXtV2 | NVIDIA NGC ,
    pretrained models of C-RADIOv2 are in GPU-optimized AI, Machine Learning, & HPC Software | NVIDIA NGC | NVIDIA NGC

4

Not sure how to train a C-RADIOv2 backbone model because it’s not in the registry.

There’s no valid way to specify it in the specs.yaml because of this.

5

The C-RADIOv2 is set in frozen_fm .

When you run rtdetr train, the model.backbone is used for sure. If you enable frozen_fm, then the training becomes a RT‑DETR student network + frozen foundation model（C‑RADIOv2) . It is a training with a frozen teacher.
More info can be found in RT-DETR — Tao Toolkit.
You can set radio_v2-l or others based on tao_pytorch_backend/nvidia_tao_pytorch/cv/rtdetr/model/backbone/radio.py at release/6.25.11 · NVIDIA/tao_pytorch_backend · GitHub.

When you run rtdetr distill, model.backbone is also needed. By default, you can set the teacher model with distill configs. Or you can set frozen_fm.