Dino training is not successful

Accelerated Computing Intelligent Video Analytics TAO Toolkit
,
1

• Hardware (T4 GPUs/ Linux VM Ubuntu )
• Network Type (Dino)
• TLT Version (toolkit_version: 6.25.10)
• Training spec file(I used the same code from github tao_tutorials dino except changed the dataset and other requisites like yaml file)

Everything seems to be in place in terms of dataset format, yaml file configuration etc., but the model just doesn’t learn anything.

Validation mAP : 0.00015174336616171174

2025-12-19 07:54:23,321 - [TAO Toolkit] - INFO - Validation mAP50 : 0.0007720637125986637
(pl_dino_model.py:277)

Validation mAP50 : 0.0007720637125986637

Even after 25 epochs., what could I potentially be doing wrong?

2

Spec file:

train:
num_gpus: 1
num_nodes: 1
validation_interval: 1
optim:
lr_backbone: 2e-05
lr: 2e-4
lr_steps: [11]
momentum: 0.9
num_epochs: 50
dataset:
train_data_sources:
- image_dir: /data/train
json_file: /data/annotations/train.json
val_data_sources:
- image_dir: /data/val
json_file: /data/annotations/val.json
num_classes: 2
batch_size: 1
workers: 4
augmentation:
fixed_padding: False
model:
backbone: fan_small
train_backbone: True
pretrained_backbone_path: /workspace/tao-broken/dino/pretrained_dino_nvimagenet_vfan_small_hybrid_nvimagenet/fan_small_hybrid_nvimagenet.pth
num_feature_levels: 4
dec_layers: 6
enc_layers: 6
num_queries: 300
num_select: 100
dropout_ratio: 0.0
dim_feedforward: 2048

4

You may need to check if the bbox is correct for the objects.
And double check the json file according to DINO — Tao Toolkit.

DINO expects directories of images for training or validation and annotated JSON files in COCO format.

The category_id from your COCO JSON file should start from 1 because 0 is set as a background class. In addition, dataset.num_classes should be set to max class_id + 1. For instance, even though there are only 80 classes used in COCO, the largest class_id is 90, so dataset.num_classes should be set to 91.

5

{
“id”: 1364,
“file_name”: “Plastic_Broken_Pit_Image (97).JPG”,
“width”: 588,
“height”: 941
},
{
“id”: 1365,
“file_name”: “Plastic_Broken_Pit_Image (98).JPG”,
“width”: 746,
“height”: 936
}
],
“annotations”: [
{
“id”: 1,
“image_id”: 1,
“category_id”: 1,
“bbox”: [
207.0,
182.0,
183.0,
154.0
],
“area”: 28182.0,
“iscrowd”: 0
},
{
“id”: 2,
“image_id”: 1,
“category_id”: 1,
“bbox”: [
415.0,
302.0,
174.0,
156.0
],
“area”: 27144.04,
“iscrowd”: 0
},

My json looks like this, I’ve checked everything I dont know what Im missing the model just doesn’t learn. Please help.

6

Could you share the full training log and the /data/annotations/train.json , /data/annotations/val.json?
If possible, could you share several images? You can share with me via private message for the images. Thanks.

7

{
“id”: 164,
“file_name”: “Plastic_Broken_Pit_Image (49).JPG”,
“width”: 537,
“height”: 903
},
{
“id”: 165,
“file_name”: “Plastic_Broken_Pit_Image (5).JPG”,
“width”: 583,
“height”: 910
},
{
“id”: 166,
“file_name”: “Plastic_Broken_Pit_Image (7).jpg”,
“width”: 800,
“height”: 600
},
{
“id”: 167,
“file_name”: “Plastic_Broken_Pit_Image (8).jpg”,
“width”: 600,
“height”: 800
},
{
“id”: 168,
“file_name”: “Plastic_Broken_Pit_Image (84).JPG”,
“width”: 1218,
“height”: 881
},
{
“id”: 169,
“file_name”: “Plastic_Broken_Pit_Image (86).JPG”,
“width”: 1184,
“height”: 815
},
{
“id”: 170,
“file_name”: “Plastic_Broken_Pit_Image (99).JPG”,
“width”: 1193,
“height”: 717
}
],
“annotations”: [
{
“id”: 1,
“image_id”: 1,
“category_id”: 1,
“bbox”: [
138.0,
558.0,
230.0,
162.0
],
“area”: 37260.03,
“iscrowd”: 0
},
{
“id”: 2,
“image_id”: 1,
“category_id”: 1,
“bbox”: [
113.0,
100.0,
274.0,
166.0
],
“area”: 45483.95,
“iscrowd”: 0
},
{
“id”: 3,
“image_id”: 2,
“category_id”: 1,
“bbox”: [
186.0,
82.0,
167.0,
40.0
],
“area”: 6680.0,
“iscrowd”: 0
},
{
“id”: 4,
“image_id”: 3,
“category_id”: 1,
“bbox”: [
132.0,
189.0,
53.0,
114.0
],
“area”: 6042.0,
“iscrowd”: 0
},
{
“id”: 5,
“image_id”: 4,
“category_id”: 1,
“bbox”: [
522.0,
327.0,
146.0,
93.0
],
“area”: 13578.04,
“iscrowd”: 0
},

Val.json content, I was unable to upload the val.json file

{
“id”: 1362,
“file_name”: “Plastic_Broken_Pit_Image (95).JPG”,
“width”: 1097,
“height”: 711
},
{
“id”: 1363,
“file_name”: “Plastic_Broken_Pit_Image (96).JPG”,
“width”: 1201,
“height”: 911
},
{
“id”: 1364,
“file_name”: “Plastic_Broken_Pit_Image (97).JPG”,
“width”: 588,
“height”: 941
},
{
“id”: 1365,
“file_name”: “Plastic_Broken_Pit_Image (98).JPG”,
“width”: 746,
“height”: 936
}
],
“annotations”: [
{
“id”: 1,
“image_id”: 1,
“category_id”: 1,
“bbox”: [
207.0,
182.0,
183.0,
154.0
],
“area”: 28182.0,
“iscrowd”: 0
},
{
“id”: 2,
“image_id”: 1,
“category_id”: 1,
“bbox”: [
415.0,
302.0,
174.0,
156.0
],
“area”: 27144.04,
“iscrowd”: 0
},
{
“id”: 3,
“image_id”: 2,
“category_id”: 1,
“bbox”: [
202.0,
667.0,
200.0,
67.0
],
“area”: 13400.0,
“iscrowd”: 0
},
{
“id”: 4,
“image_id”: 3,
“category_id”: 1,
“bbox”: [
72.0,
717.0,
421.0,
81.0
],
“area”: 34101.07,
“iscrowd”: 0
},

8

For multi-GPU, change train.num_gpus in train.yaml based on your machine

For multi-node, change train.num_gpus and num_nodes in train.yaml based on your machine
2025-12-19 05:44:57,510 [TAO Toolkit] [INFO] root 160: Registry: [‘nvcr.io’]
2025-12-19 05:44:57,605 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:6.25.10-pyt
2025-12-19 05:44:57,620 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 308: Printing tty value True
2025-12-19 05:45:03,999 - matplotlib - WARNING - mkdir -p failed for path /.config/matplotlib: [Errno 13] Permission denied: ‘/.config’
2025-12-19 05:45:03,999 - matplotlib - WARNING - Matplotlib created a temporary cache directory at /tmp/matplotlib-mv6z0kdt because there was an issue with the default path (/.config/matplotlib); it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2025-12-19 05:45:04,194 - matplotlib.font_manager - INFO - generated new fontManager
/usr/local/lib/python3.12/dist-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
/usr/local/lib/python3.12/dist-packages/timm/models/layers/init.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {name} is deprecated, please import via timm.layers", FutureWarning)
sys:1: UserWarning:
‘train.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See Automatic schema-matching | Hydra for migration instructions.
/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/hydra/hydra_runner.py:110: UserWarning:
‘train.yaml’ is validated against ConfigStore schema with the same name.
This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2.
See Automatic schema-matching | Hydra for migration instructions.
_run_hydra(
/usr/local/lib/python3.12/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
See Changes to job's runtime working directory | Hydra for more information.
ret = run_job(
/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/loggers/api_logging.py:236: UserWarning: Log file already exists at /results/train/status.json
rank_zero_warn(
Seed set to 1234
2025-12-19 05:45:19,104 - [TAO Toolkit] - INFO - Loaded pretrained weights from /workspace/tao-broken/dino/pretrained_dino_nvimagenet_vfan_small_hybrid_nvimagenet/fan_small_hybrid_nvimagenet.pth (backbone.py:239)Train results will be saved at: /results/train
Loaded pretrained weights from /workspace/tao-broken/dino/pretrained_dino_nvimagenet_vfan_small_hybrid_nvimagenet/fan_small_hybrid_nvimagenet.pth
_IncompatibleKeys(missing_keys=[‘out_norm1.weight’, ‘out_norm1.bias’, ‘out_norm2.weight’, ‘out_norm2.bias’, ‘out_norm3.weight’, ‘out_norm3.bias’, ‘learnable_downsample.weight’, ‘learnable_downsample.bias’], unexpected_keys=[‘norm.weight’, ‘norm.bias’, ‘head.fc.weight’, ‘head.fc.bias’]).weight’, ‘norm.bias’, ‘head.fc.weight’, ‘head.fc.bias’]) (backbone.py:240)GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
/usr/local/lib/python3.12/dist-packages/pytorch_lightning/callbacks/model_checkpoint.py:654: Checkpoint directory /results/train exists and is not empty.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
| Name | Type | Params | Mode

0 | model | DINOModel | 48.1 M | train

1 | matcher | HungarianMatcher | 0 | train
2 | criterion | SetCriterion | 0 | train
3 | box_processors | PostProcess | 0 | train

48.1 M Trainable params
0 Non-trainable params
48.1 M Total params
192.400 Total estimated model params size (MB)
601 Modules in train mode
0 Modules in eval mode
Serializing 1365 elements to byte tensors and concatenating them all …
Serialized dataset takes 0.35 MiB
Worker 0 obtains a dataset of length=1365 from its local leader.

Sanity Checking DataLoader 0: 0%| | 0/2 [00:00<?, ?it/s]/usr/local/lib/python3.12/dist-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None warnings.warn(
/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/dino/model/deformable_transformer.py:342: UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is //.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/cuda/jit_utils.cpp:1442.)
level_start_index = torch.cat((spatial_shapes.new_zeros((1, )), spatial_shapes.prod(1).cumsum(0)[:-1]))/usr/local/lib/python3.12/dist-packages/torch/functional.py:539: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/pytorch/pytorch/aten/src/ATen/native/TensorShape.cpp:3611.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Sanity Checking DataLoader 0: 100%|██████████| 2/2 [00:02<00:00, 0.91it/s]2025-12-19 05:45:23,220 - [TAO Toolkit] - INFO - Validation mAP : 0.0
(pl_dino_model.py:276)

Validation mAP : 0.0

2025-12-19 05:45:23,221 - [TAO Toolkit] - INFO - Validation mAP50 : 0.0
(pl_dino_model.py:277)

Validation mAP50 : 0.0

                                                                       Serializing 1365 elements to byte tensors and concatenating them all ...

Serialized dataset takes 0.35 MiB
Worker 0 obtains a dataset of length=1365 from its local leader.

Epoch 0: 100%|██████████| 1365/1365 [24:20<00:00, 0.93it/s, v_num=1, train_loss_step=35.10, lr=0.0002]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: 0%| | 0/170 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/170 [00:00<?, ?it/s]
Validation DataLoader 0: 1%| | 1/170 [00:00<01:20, 2.09it/s]
Validation DataLoader 0: 1%| | 2/170 [00:00<01:20, 2.09it/s]
Validation DataLoader 0: 2%|▏ | 3/170 [00:01<01:19, 2.10it/s]
Validation DataLoader 0: 2%|▏ | 4/170 [00:01<01:15, 2.20it/s]
Validation DataLoader 0: 3%|▎ | 5/170 [00:02<01:13, 2.24it/s]
Validation DataLoader 0: 4%|▎ | 6/170 [00:02<01:13, 2.22it/s]
Validation DataLoader 0: 4%|▍ | 7/170 [00:03<01:14, 2.20it/s]
Validation DataLoader 0: 5%|▍ | 8/170 [00:03<01:13, 2.19it/s]
Validation DataLoader 0: 5%|▌ | 9/170 [00:04<01:13, 2.19it/s]
Validation DataLoader 0: 6%|▌ | 10/170 [00:04<01:12, 2.22it/s]
Validation DataLoader 0: 6%|▋ | 11/170 [00:04<01:11, 2.21it/s]
Validation DataLoader 0: 7%|▋ | 12/170 [00:05<01:11, 2.20it/s]
Validation DataLoader 0: 8%|▊ | 13/170 [00:05<01:11, 2.20it/s]
Validation DataLoader 0: 8%|▊ | 14/170 [00:06<01:11, 2.19it/s]
Validation DataLoader 0: 9%|▉ | 15/170 [00:06<01:10, 2.19it/s]
Validation DataLoader 0: 9%|▉ | 16/170 [00:07<01:10, 2.19it/s]
Validation DataLoader 0: 10%|█ | 17/170 [00:07<01:08, 2.22it/s]
Validation DataLoader 0: 11%|█ | 18/170 [00:08<01:08, 2.21it/s]
Validation DataLoader 0: 11%|█ | 19/170 [00:08<01:08, 2.21it/s]
Validation DataLoader 0: 12%|█▏ | 20/170 [00:09<01:08, 2.20it/s]
Validation DataLoader 0: 12%|█▏ | 21/170 [00:09<01:07, 2.22it/s]
Validation DataLoader 0: 13%|█▎ | 22/170 [00:09<01:06, 2.21it/s]
Validation DataLoader 0: 14%|█▎ | 23/170 [00:10<01:06, 2.22it/s]
Validation DataLoader 0: 14%|█▍ | 24/170 [00:10<01:05, 2.22it/s]
Validation DataLoader 0: 15%|█▍ | 25/170 [00:11<01:05, 2.21it/s]
Validation DataLoader 0: 15%|█▌ | 26/170 [00:11<01:05, 2.21it/s]
Validation DataLoader 0: 16%|█▌ | 27/170 [00:12<01:04, 2.23it/s]
Validation DataLoader 0: 16%|█▋ | 28/170 [00:12<01:03, 2.24it/s]
Validation DataLoader 0: 17%|█▋ | 29/170 [00:12<01:02, 2.24it/s]
Validation DataLoader 0: 18%|█▊ | 30/170 [00:13<01:02, 2.23it/s]
Validation DataLoader 0: 18%|█▊ | 31/170 [00:13<01:01, 2.24it/s]
Validation DataLoader 0: 19%|█▉ | 32/170 [00:14<01:01, 2.23it/s]
Validation DataLoader 0: 19%|█▉ | 33/170 [00:14<01:01, 2.22it/s]
Validation DataLoader 0: 20%|██ | 34/170 [00:15<01:01, 2.23it/s]
Validation DataLoader 0: 21%|██ | 35/170 [00:15<01:00, 2.22it/s]
Validation DataLoader 0: 21%|██ | 36/170 [00:16<01:00, 2.23it/s]
Validation DataLoader 0: 22%|██▏ | 37/170 [00:16<00:59, 2.23it/s]
Validation DataLoader 0: 22%|██▏ | 38/170 [00:17<00:59, 2.23it/s]
Validation DataLoader 0: 23%|██▎ | 39/170 [00:17<00:58, 2.24it/s]
Validation DataLoader 0: 24%|██▎ | 40/170 [00:17<00:58, 2.24it/s]
Validation DataLoader 0: 24%|██▍ | 41/170 [00:18<00:57, 2.23it/s]
Validation DataLoader 0: 25%|██▍ | 42/170 [00:18<00:57, 2.24it/s]
Validation DataLoader 0: 25%|██▌ | 43/170 [00:19<00:56, 2.24it/s]
Validation DataLoader 0: 26%|██▌ | 44/170 [00:19<00:56, 2.23it/s]
Validation DataLoader 0: 26%|██▋ | 45/170 [00:20<00:55, 2.25it/s]
Validation DataLoader 0: 27%|██▋ | 46/170 [00:20<00:55, 2.25it/s]
Validation DataLoader 0: 28%|██▊ | 47/170 [00:20<00:54, 2.25it/s]
Validation DataLoader 0: 28%|██▊ | 48/170 [00:21<00:54, 2.24it/s]
Validation DataLoader 0: 29%|██▉ | 49/170 [00:21<00:53, 2.26it/s]
Validation DataLoader 0: 29%|██▉ | 50/170 [00:22<00:53, 2.26it/s]
Validation DataLoader 0: 30%|███ | 51/170 [00:22<00:52, 2.26it/s]
Validation DataLoader 0: 31%|███ | 52/170 [00:23<00:52, 2.26it/s]
Validation DataLoader 0: 31%|███ | 53/170 [00:23<00:51, 2.26it/s]
Validation DataLoader 0: 32%|███▏ | 54/170 [00:23<00:51, 2.27it/s]
Validation DataLoader 0: 32%|███▏ | 55/170 [00:24<00:50, 2.27it/s]
Validation DataLoader 0: 33%|███▎ | 56/170 [00:24<00:50, 2.27it/s]
Validation DataLoader 0: 34%|███▎ | 57/170 [00:25<00:49, 2.27it/s]
Validation DataLoader 0: 34%|███▍ | 58/170 [00:25<00:49, 2.28it/s]
Validation DataLoader 0: 35%|███▍ | 59/170 [00:25<00:48, 2.28it/s]
Validation DataLoader 0: 35%|███▌ | 60/170 [00:26<00:48, 2.29it/s]
Validation DataLoader 0: 36%|███▌ | 61/170 [00:26<00:47, 2.29it/s]
Validation DataLoader 0: 36%|███▋ | 62/170 [00:27<00:47, 2.29it/s]
Validation DataLoader 0: 37%|███▋ | 63/170 [00:27<00:46, 2.29it/s]
Validation DataLoader 0: 38%|███▊ | 64/170 [00:27<00:46, 2.29it/s]
Validation DataLoader 0: 38%|███▊ | 65/170 [00:28<00:45, 2.29it/s]
Validation DataLoader 0: 39%|███▉ | 66/170 [00:28<00:45, 2.29it/s]
Validation DataLoader 0: 39%|███▉ | 67/170 [00:29<00:44, 2.29it/s]
Validation DataLoader 0: 40%|████ | 68/170 [00:29<00:44, 2.29it/s]
Validation DataLoader 0: 41%|████ | 69/170 [00:30<00:44, 2.29it/s]
Validation DataLoader 0: 41%|████ | 70/170 [00:30<00:43, 2.29it/s]
Validation DataLoader 0: 42%|████▏ | 71/170 [00:30<00:43, 2.29it/s]
Validation DataLoader 0: 42%|████▏ | 72/170 [00:31<00:42, 2.29it/s]
Validation DataLoader 0: 43%|████▎ | 73/170 [00:31<00:42, 2.29it/s]
Validation DataLoader 0: 44%|████▎ | 74/170 [00:32<00:41, 2.30it/s]
Validation DataLoader 0: 44%|████▍ | 75/170 [00:32<00:41, 2.30it/s]
Validation DataLoader 0: 45%|████▍ | 76/170 [00:33<00:40, 2.30it/s]
Validation DataLoader 0: 45%|████▌ | 77/170 [00:33<00:40, 2.29it/s]
Validation DataLoader 0: 46%|████▌ | 78/170 [00:33<00:40, 2.30it/s]
Validation DataLoader 0: 46%|████▋ | 79/170 [00:34<00:39, 2.29it/s]
Validation DataLoader 0: 47%|████▋ | 80/170 [00:34<00:39, 2.30it/s]
Validation DataLoader 0: 48%|████▊ | 81/170 [00:35<00:38, 2.30it/s]
Validation DataLoader 0: 48%|████▊ | 82/170 [00:35<00:38, 2.29it/s]
Validation DataLoader 0: 49%|████▉ | 83/170 [00:36<00:37, 2.29it/s]
Validation DataLoader 0: 49%|████▉ | 84/170 [00:36<00:37, 2.29it/s]
Validation DataLoader 0: 50%|█████ | 85/170 [00:37<00:37, 2.29it/s]
Validation DataLoader 0: 51%|█████ | 86/170 [00:37<00:36, 2.29it/s]
Validation DataLoader 0: 51%|█████ | 87/170 [00:37<00:36, 2.29it/s]
Validation DataLoader 0: 52%|█████▏ | 88/170 [00:38<00:35, 2.29it/s]
Validation DataLoader 0: 52%|█████▏ | 89/170 [00:38<00:35, 2.29it/s]
Validation DataLoader 0: 53%|█████▎ | 90/170 [00:39<00:34, 2.29it/s]
Validation DataLoader 0: 54%|█████▎ | 91/170 [00:39<00:34, 2.29it/s]
Validation DataLoader 0: 54%|█████▍ | 92/170 [00:40<00:34, 2.29it/s]
Validation DataLoader 0: 55%|█████▍ | 93/170 [00:40<00:33, 2.29it/s]
Validation DataLoader 0: 55%|█████▌ | 94/170 [00:41<00:33, 2.29it/s]
Validation DataLoader 0: 56%|█████▌ | 95/170 [00:41<00:32, 2.28it/s]
Validation DataLoader 0: 56%|█████▋ | 96/170 [00:42<00:32, 2.28it/s]
Validation DataLoader 0: 57%|█████▋ | 97/170 [00:42<00:31, 2.28it/s]
Validation DataLoader 0: 58%|█████▊ | 98/170 [00:42<00:31, 2.28it/s]
Validation DataLoader 0: 58%|█████▊ | 99/170 [00:43<00:31, 2.28it/s]
Validation DataLoader 0: 59%|█████▉ | 100/170 [00:43<00:30, 2.28it/s]
Validation DataLoader 0: 59%|█████▉ | 101/170 [00:44<00:30, 2.28it/s]
Validation DataLoader 0: 60%|██████ | 102/170 [00:44<00:29, 2.28it/s]
Validation DataLoader 0: 61%|██████ | 103/170 [00:45<00:29, 2.27it/s]
Validation DataLoader 0: 61%|██████ | 104/170 [00:45<00:29, 2.27it/s]
Validation DataLoader 0: 62%|██████▏ | 105/170 [00:46<00:28, 2.27it/s]
Validation DataLoader 0: 62%|██████▏ | 106/170 [00:46<00:28, 2.27it/s]
Validation DataLoader 0: 63%|██████▎ | 107/170 [00:47<00:27, 2.27it/s]
Validation DataLoader 0: 64%|██████▎ | 108/170 [00:47<00:27, 2.27it/s]
Validation DataLoader 0: 64%|██████▍ | 109/170 [00:48<00:26, 2.26it/s]
Validation DataLoader 0: 65%|██████▍ | 110/170 [00:48<00:26, 2.26it/s]
Validation DataLoader 0: 65%|██████▌ | 111/170 [00:49<00:26, 2.25it/s]
Validation DataLoader 0: 66%|██████▌ | 112/170 [00:49<00:25, 2.25it/s]
Validation DataLoader 0: 66%|██████▋ | 113/170 [00:50<00:25, 2.25it/s]
Validation DataLoader 0: 67%|██████▋ | 114/170 [00:50<00:24, 2.24it/s]
Validation DataLoader 0: 68%|██████▊ | 115/170 [00:51<00:24, 2.24it/s]
Validation DataLoader 0: 68%|██████▊ | 116/170 [00:51<00:24, 2.24it/s]
Validation DataLoader 0: 69%|██████▉ | 117/170 [00:52<00:23, 2.24it/s]
Validation DataLoader 0: 69%|██████▉ | 118/170 [00:52<00:23, 2.24it/s]
Validation DataLoader 0: 70%|███████ | 119/170 [00:53<00:22, 2.24it/s]
Validation DataLoader 0: 71%|███████ | 120/170 [00:53<00:22, 2.24it/s]
Validation DataLoader 0: 71%|███████ | 121/170 [00:54<00:21, 2.24it/s]
Validation DataLoader 0: 72%|███████▏ | 122/170 [00:54<00:21, 2.24it/s]
Validation DataLoader 0: 72%|███████▏ | 123/170 [00:54<00:21, 2.24it/s]
Validation DataLoader 0: 73%|███████▎ | 124/170 [00:55<00:20, 2.24it/s]
Validation DataLoader 0: 74%|███████▎ | 125/170 [00:55<00:20, 2.24it/s]
Validation DataLoader 0: 74%|███████▍ | 126/170 [00:56<00:19, 2.23it/s]
Validation DataLoader 0: 75%|███████▍ | 127/170 [00:56<00:19, 2.23it/s]
Validation DataLoader 0: 75%|███████▌ | 128/170 [00:57<00:18, 2.23it/s]
Validation DataLoader 0: 76%|███████▌ | 129/170 [00:57<00:18, 2.23it/s]
Validation DataLoader 0: 76%|███████▋ | 130/170 [00:58<00:17, 2.23it/s]
Validation DataLoader 0: 77%|███████▋ | 131/170 [00:58<00:17, 2.23it/s]
Validation DataLoader 0: 78%|███████▊ | 132/170 [00:59<00:17, 2.23it/s]
Validation DataLoader 0: 78%|███████▊ | 133/170 [00:59<00:16, 2.23it/s]
Validation DataLoader 0: 79%|███████▉ | 134/170 [01:00<00:16, 2.23it/s]
Validation DataLoader 0: 79%|███████▉ | 135/170 [01:00<00:15, 2.22it/s]
Validation DataLoader 0: 80%|████████ | 136/170 [01:01<00:15, 2.22it/s]
Validation DataLoader 0: 81%|████████ | 137/170 [01:01<00:14, 2.22it/s]
Validation DataLoader 0: 81%|████████ | 138/170 [01:02<00:14, 2.22it/s]
Validation DataLoader 0: 82%|████████▏ | 139/170 [01:02<00:13, 2.22it/s]
Validation DataLoader 0: 82%|████████▏ | 140/170 [01:02<00:13, 2.22it/s]
Validation DataLoader 0: 83%|████████▎ | 141/170 [01:03<00:13, 2.22it/s]
Validation DataLoader 0: 84%|████████▎ | 142/170 [01:03<00:12, 2.22it/s]
Validation DataLoader 0: 84%|████████▍ | 143/170 [01:04<00:12, 2.22it/s]
Validation DataLoader 0: 85%|████████▍ | 144/170 [01:04<00:11, 2.22it/s]
Validation DataLoader 0: 85%|████████▌ | 145/170 [01:05<00:11, 2.22it/s]
Validation DataLoader 0: 86%|████████▌ | 146/170 [01:05<00:10, 2.22it/s]
Validation DataLoader 0: 86%|████████▋ | 147/170 [01:06<00:10, 2.22it/s]
Validation DataLoader 0: 87%|████████▋ | 148/170 [01:06<00:09, 2.21it/s]
Validation DataLoader 0: 88%|████████▊ | 149/170 [01:07<00:09, 2.21it/s]
Validation DataLoader 0: 88%|████████▊ | 150/170 [01:07<00:09, 2.21it/s]
Validation DataLoader 0: 89%|████████▉ | 151/170 [01:08<00:08, 2.21it/s]
Validation DataLoader 0: 89%|████████▉ | 152/170 [01:08<00:08, 2.21it/s]
Validation DataLoader 0: 90%|█████████ | 153/170 [01:09<00:07, 2.21it/s]
Validation DataLoader 0: 91%|█████████ | 154/170 [01:09<00:07, 2.21it/s]
Validation DataLoader 0: 91%|█████████ | 155/170 [01:09<00:06, 2.22it/s]
Validation DataLoader 0: 92%|█████████▏| 156/170 [01:10<00:06, 2.22it/s]
Validation DataLoader 0: 92%|█████████▏| 157/170 [01:10<00:05, 2.21it/s]
Validation DataLoader 0: 93%|█████████▎| 158/170 [01:11<00:05, 2.22it/s]
Validation DataLoader 0: 94%|█████████▎| 159/170 [01:11<00:04, 2.21it/s]
Validation DataLoader 0: 94%|█████████▍| 160/170 [01:12<00:04, 2.22it/s]
Validation DataLoader 0: 95%|█████████▍| 161/170 [01:12<00:04, 2.22it/s]
Validation DataLoader 0: 95%|█████████▌| 162/170 [01:13<00:03, 2.22it/s]
Validation DataLoader 0: 96%|█████████▌| 163/170 [01:13<00:03, 2.22it/s]
Validation DataLoader 0: 96%|█████████▋| 164/170 [01:14<00:02, 2.21it/s]
Validation DataLoader 0: 97%|█████████▋| 165/170 [01:14<00:02, 2.21it/s]
Validation DataLoader 0: 98%|█████████▊| 166/170 [01:15<00:01, 2.21it/s]
Validation DataLoader 0: 98%|█████████▊| 167/170 [01:15<00:01, 2.21it/s]
Validation DataLoader 0: 99%|█████████▉| 168/170 [01:15<00:00, 2.21it/s]
Validation DataLoader 0: 99%|█████████▉| 169/170 [01:16<00:00, 2.21it/s]
Validation DataLoader 0: 100%|██████████| 170/170 [01:16<00:00, 2.21it/s]2025-12-19 06:11:01,528 - [TAO Toolkit] - INFO - Validation mAP : 5.716125742765144e-05
(pl_dino_model.py:276)

Validation mAP : 5.716125742765144e-05

2025-12-19 06:11:01,528 - [TAO Toolkit] - INFO - Validation mAP50 : 0.00034225586857093444
(pl_dino_model.py:277)

Validation mAP50 : 0.00034225586857093444

Epoch 1: 100%|██████████| 1365/1365 [24:30<00:00, 0.93it/s, v_num=1, train_loss_step=34.20, lr=0.0002, val_loss=27.30, train_loss_epoch=44.00]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: 0%| | 0/170 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/170 [00:00<?, ?it/s]
Validation DataLoader 0: 1%| | 1/170 [00:00<01:22, 2.05it/s]
Validation DataLoader 0: 1%| | 2/170 [00:00<01:21, 2.07it/s]
Validation DataLoader 0: 2%|▏ | 3/170 [00:01<01:20, 2.09it/s]
Validation DataLoader 0: 2%|▏ | 4/170 [00:01<01:15, 2.18it/s]
Validation DataLoader 0: 3%|▎ | 5/170 [00:02<01:14, 2.23it/s]
Validation DataLoader 0: 4%|▎ | 6/170 [00:02<01:14, 2.21it/s]
Validation DataLoader 0: 4%|▍ | 7/170 [00:03<01:14, 2.19it/s]
Validation DataLoader 0: 5%|▍ | 8/170 [00:03<01:14, 2.18it/s]
Validation DataLoader 0: 5%|▌ | 9/170 [00:04<01:14, 2.17it/s]
Validation DataLoader 0: 6%|▌ | 10/170 [00:04<01:12, 2.20it/s]
Validation DataLoader 0: 6%|▋ | 11/170 [00:05<01:12, 2.20it/s]
Validation DataLoader 0: 7%|▋ | 12/170 [00:05<01:12, 2.19it/s]
Validation DataLoader 0: 8%|▊ | 13/170 [00:05<01:11, 2.18it/s]
Validation DataLoader 0: 8%|▊ | 14/170 [00:06<01:11, 2.18it/s]
Validation DataLoader 0: 9%|▉ | 15/170 [00:06<01:11, 2.17it/s]
Validation DataLoader 0: 9%|▉ | 16/170 [00:07<01:10, 2.17it/s]
Validation DataLoader 0: 10%|█ | 17/170 [00:07<01:09, 2.21it/s]
Validation DataLoader 0: 11%|█ | 18/170 [00:08<01:09, 2.20it/s]
Validation DataLoader 0: 11%|█ | 19/170 [00:08<01:08, 2.20it/s]
Validation DataLoader 0: 12%|█▏ | 20/170 [00:09<01:08, 2.19it/s]
Validation DataLoader 0: 12%|█▏ | 21/170 [00:09<01:07, 2.21it/s]
Validation DataLoader 0: 13%|█▎ | 22/170 [00:09<01:07, 2.20it/s]
Validation DataLoader 0: 14%|█▎ | 23/170 [00:10<01:06, 2.21it/s]
Validation DataLoader 0: 14%|█▍ | 24/170 [00:10<01:06, 2.21it/s]
Validation DataLoader 0: 15%|█▍ | 25/170 [00:11<01:05, 2.20it/s]
Validation DataLoader 0: 15%|█▌ | 26/170 [00:11<01:05, 2.20it/s]
Validation DataLoader 0: 16%|█▌ | 27/170 [00:12<01:04, 2.22it/s]
Validation DataLoader 0: 16%|█▋ | 28/170 [00:12<01:03, 2.23it/s]
Validation DataLoader 0: 17%|█▋ | 29/170 [00:13<01:03, 2.23it/s]
Validation DataLoader 0: 18%|█▊ | 30/170 [00:13<01:03, 2.21it/s]
Validation DataLoader 0: 18%|█▊ | 31/170 [00:13<01:02, 2.23it/s]
Validation DataLoader 0: 19%|█▉ | 32/170 [00:14<01:02, 2.22it/s]
Validation DataLoader 0: 19%|█▉ | 33/170 [00:14<01:01, 2.21it/s]
Validation DataLoader 0: 20%|██ | 34/170 [00:15<01:01, 2.22it/s]
Validation DataLoader 0: 21%|██ | 35/170 [00:15<01:00, 2.21it/s]
Validation DataLoader 0: 21%|██ | 36/170 [00:16<01:00, 2.22it/s]
Validation DataLoader 0: 22%|██▏ | 37/170 [00:16<00:59, 2.22it/s]
Validation DataLoader 0: 22%|██▏ | 38/170 [00:17<00:59, 2.23it/s]
Validation DataLoader 0: 23%|██▎ | 39/170 [00:17<00:58, 2.23it/s]
Validation DataLoader 0: 24%|██▎ | 40/170 [00:17<00:58, 2.23it/s]
Validation DataLoader 0: 24%|██▍ | 41/170 [00:18<00:57, 2.22it/s]
Validation DataLoader 0: 25%|██▍ | 42/170 [00:18<00:57, 2.23it/s]
Validation DataLoader 0: 25%|██▌ | 43/170 [00:19<00:56, 2.23it/s]
Validation DataLoader 0: 26%|██▌ | 44/170 [00:19<00:56, 2.23it/s]
Validation DataLoader 0: 26%|██▋ | 45/170 [00:20<00:55, 2.24it/s]
Validation DataLoader 0: 27%|██▋ | 46/170 [00:20<00:55, 2.24it/s]
Validation DataLoader 0: 28%|██▊ | 47/170 [00:20<00:54, 2.24it/s]
Validation DataLoader 0: 28%|██▊ | 48/170 [00:21<00:54, 2.24it/s]
Validation DataLoader 0: 29%|██▉ | 49/170 [00:21<00:53, 2.25it/s]
Validation DataLoader 0: 29%|██▉ | 50/170 [00:22<00:53, 2.25it/s]
Validation DataLoader 0: 30%|███ | 51/170 [00:22<00:52, 2.25it/s]
Validation DataLoader 0: 31%|███ | 52/170 [00:23<00:52, 2.25it/s]
Validation DataLoader 0: 31%|███ | 53/170 [00:23<00:51, 2.25it/s]
Validation DataLoader 0: 32%|███▏ | 54/170 [00:23<00:51, 2.26it/s]
Validation DataLoader 0: 32%|███▏ | 55/170 [00:24<00:50, 2.26it/s]
Validation DataLoader 0: 33%|███▎ | 56/170 [00:24<00:50, 2.26it/s]
Validation DataLoader 0: 34%|███▎ | 57/170 [00:25<00:49, 2.26it/s]
Validation DataLoader 0: 34%|███▍ | 58/170 [00:25<00:49, 2.27it/s]
Validation DataLoader 0: 35%|███▍ | 59/170 [00:25<00:48, 2.28it/s]
Validation DataLoader 0: 35%|███▌ | 60/170 [00:26<00:48, 2.28it/s]
Validation DataLoader 0: 36%|███▌ | 61/170 [00:26<00:47, 2.28it/s]
Validation DataLoader 0: 36%|███▋ | 62/170 [00:27<00:47, 2.28it/s]
Validation DataLoader 0: 37%|███▋ | 63/170 [00:27<00:46, 2.28it/s]
Validation DataLoader 0: 38%|███▊ | 64/170 [00:28<00:46, 2.28it/s]
Validation DataLoader 0: 38%|███▊ | 65/170 [00:28<00:45, 2.28it/s]
Validation DataLoader 0: 39%|███▉ | 66/170 [00:28<00:45, 2.29it/s]
Validation DataLoader 0: 39%|███▉ | 67/170 [00:29<00:45, 2.28it/s]
Validation DataLoader 0: 40%|████ | 68/170 [00:29<00:44, 2.28it/s]
Validation DataLoader 0: 41%|████ | 69/170 [00:30<00:44, 2.28it/s]
Validation DataLoader 0: 41%|████ | 70/170 [00:30<00:43, 2.28it/s]
Validation DataLoader 0: 42%|████▏ | 71/170 [00:31<00:43, 2.28it/s]
Validation DataLoader 0: 42%|████▏ | 72/170 [00:31<00:42, 2.28it/s]
Validation DataLoader 0: 43%|████▎ | 73/170 [00:31<00:42, 2.28it/s]
Validation DataLoader 0: 44%|████▎ | 74/170 [00:32<00:41, 2.29it/s]
Validation DataLoader 0: 44%|████▍ | 75/170 [00:32<00:41, 2.29it/s]
Validation DataLoader 0: 45%|████▍ | 76/170 [00:33<00:41, 2.29it/s]
Validation DataLoader 0: 45%|████▌ | 77/170 [00:33<00:40, 2.28it/s]
Validation DataLoader 0: 46%|████▌ | 78/170 [00:34<00:40, 2.29it/s]
Validation DataLoader 0: 46%|████▋ | 79/170 [00:34<00:39, 2.29it/s]
Validation DataLoader 0: 47%|████▋ | 80/170 [00:34<00:39, 2.29it/s]
Validation DataLoader 0: 48%|████▊ | 81/170 [00:35<00:38, 2.29it/s]
Validation DataLoader 0: 48%|████▊ | 82/170 [00:35<00:38, 2.29it/s]
Validation DataLoader 0: 49%|████▉ | 83/170 [00:36<00:38, 2.29it/s]
Validation DataLoader 0: 49%|████▉ | 84/170 [00:36<00:37, 2.29it/s]
Validation DataLoader 0: 50%|█████ | 85/170 [00:37<00:37, 2.28it/s]
Validation DataLoader 0: 51%|█████ | 86/170 [00:37<00:36, 2.29it/s]
Validation DataLoader 0: 51%|█████ | 87/170 [00:38<00:36, 2.28it/s]
Validation DataLoader 0: 52%|█████▏ | 88/170 [00:38<00:35, 2.29it/s]
Validation DataLoader 0: 52%|█████▏ | 89/170 [00:38<00:35, 2.28it/s]
Validation DataLoader 0: 53%|█████▎ | 90/170 [00:39<00:35, 2.28it/s]
Validation DataLoader 0: 54%|█████▎ | 91/170 [00:39<00:34, 2.28it/s]
Validation DataLoader 0: 54%|█████▍ | 92/170 [00:40<00:34, 2.28it/s]
Validation DataLoader 0: 55%|█████▍ | 93/170 [00:40<00:33, 2.28it/s]
Validation DataLoader 0: 55%|█████▌ | 94/170 [00:41<00:33, 2.28it/s]
Validation DataLoader 0: 56%|█████▌ | 95/170 [00:41<00:32, 2.28it/s]
Validation DataLoader 0: 56%|█████▋ | 96/170 [00:42<00:32, 2.28it/s]
Validation DataLoader 0: 57%|█████▋ | 97/170 [00:42<00:32, 2.27it/s]
Validation DataLoader 0: 58%|█████▊ | 98/170 [00:43<00:31, 2.27it/s]
Validation DataLoader 0: 58%|█████▊ | 99/170 [00:43<00:31, 2.27it/s]
Validation DataLoader 0: 59%|█████▉ | 100/170 [00:43<00:30, 2.27it/s]
Validation DataLoader 0: 59%|█████▉ | 101/170 [00:44<00:30, 2.27it/s]
Validation DataLoader 0: 60%|██████ | 102/170 [00:44<00:29, 2.27it/s]
Validation DataLoader 0: 61%|██████ | 103/170 [00:45<00:29, 2.27it/s]
Validation DataLoader 0: 61%|██████ | 104/170 [00:45<00:29, 2.27it/s]
Validation DataLoader 0: 62%|██████▏ | 105/170 [00:46<00:28, 2.27it/s]
Validation DataLoader 0: 62%|██████▏ | 106/170 [00:46<00:28, 2.27it/s]
Validation DataLoader 0: 63%|██████▎ | 107/170 [00:47<00:27, 2.26it/s]
Validation DataLoader 0: 64%|██████▎ | 108/170 [00:47<00:27, 2.26it/s]
Validation DataLoader 0: 64%|██████▍ | 109/170 [00:48<00:27, 2.26it/s]
Validation DataLoader 0: 65%|██████▍ | 110/170 [00:48<00:26, 2.25it/s]
Validation DataLoader 0: 65%|██████▌ | 111/170 [00:49<00:26, 2.25it/s]
Validation DataLoader 0: 66%|██████▌ | 112/170 [00:49<00:25, 2.24it/s]
Validation DataLoader 0: 66%|██████▋ | 113/170 [00:50<00:25, 2.24it/s]
Validation DataLoader 0: 67%|██████▋ | 114/170 [00:50<00:25, 2.24it/s]
Validation DataLoader 0: 68%|██████▊ | 115/170 [00:51<00:24, 2.24it/s]
Validation DataLoader 0: 68%|██████▊ | 116/170 [00:51<00:24, 2.23it/s]
Validation DataLoader 0: 69%|██████▉ | 117/170 [00:52<00:23, 2.23it/s]
Validation DataLoader 0: 69%|██████▉ | 118/170 [00:52<00:23, 2.24it/s]
Validation DataLoader 0: 70%|███████ | 119/170 [00:53<00:22, 2.23it/s]
Validation DataLoader 0: 71%|███████ | 120/170 [00:53<00:22, 2.23it/s]
Validation DataLoader 0: 71%|███████ | 121/170 [00:54<00:21, 2.23it/s]
Validation DataLoader 0: 72%|███████▏ | 122/170 [00:54<00:21, 2.23it/s]
Validation DataLoader 0: 72%|███████▏ | 123/170 [00:55<00:21, 2.23it/s]
Validation DataLoader 0: 73%|███████▎ | 124/170 [00:55<00:20, 2.23it/s]
Validation DataLoader 0: 74%|███████▎ | 125/170 [00:56<00:20, 2.23it/s]
Validation DataLoader 0: 74%|███████▍ | 126/170 [00:56<00:19, 2.23it/s]
Validation DataLoader 0: 75%|███████▍ | 127/170 [00:57<00:19, 2.22it/s]
Validation DataLoader 0: 75%|███████▌ | 128/170 [00:57<00:18, 2.22it/s]
Validation DataLoader 0: 76%|███████▌ | 129/170 [00:58<00:18, 2.22it/s]
Validation DataLoader 0: 76%|███████▋ | 130/170 [00:58<00:17, 2.22it/s]
Validation DataLoader 0: 77%|███████▋ | 131/170 [00:58<00:17, 2.22it/s]
Validation DataLoader 0: 78%|███████▊ | 132/170 [00:59<00:17, 2.22it/s]
Validation DataLoader 0: 78%|███████▊ | 133/170 [00:59<00:16, 2.22it/s]
Validation DataLoader 0: 79%|███████▉ | 134/170 [01:00<00:16, 2.22it/s]
Validation DataLoader 0: 79%|███████▉ | 135/170 [01:00<00:15, 2.21it/s]
Validation DataLoader 0: 80%|████████ | 136/170 [01:01<00:15, 2.21it/s]
Validation DataLoader 0: 81%|████████ | 137/170 [01:01<00:14, 2.21it/s]
Validation DataLoader 0: 81%|████████ | 138/170 [01:02<00:14, 2.21it/s]
Validation DataLoader 0: 82%|████████▏ | 139/170 [01:02<00:13, 2.22it/s]
Validation DataLoader 0: 82%|████████▏ | 140/170 [01:03<00:13, 2.21it/s]
Validation DataLoader 0: 83%|████████▎ | 141/170 [01:03<00:13, 2.22it/s]
Validation DataLoader 0: 84%|████████▎ | 142/170 [01:04<00:12, 2.21it/s]
Validation DataLoader 0: 84%|████████▍ | 143/170 [01:04<00:12, 2.21it/s]
Validation DataLoader 0: 85%|████████▍ | 144/170 [01:05<00:11, 2.21it/s]
Validation DataLoader 0: 85%|████████▌ | 145/170 [01:05<00:11, 2.21it/s]
Validation DataLoader 0: 86%|████████▌ | 146/170 [01:06<00:10, 2.21it/s]
Validation DataLoader 0: 86%|████████▋ | 147/170 [01:06<00:10, 2.21it/s]
Validation DataLoader 0: 87%|████████▋ | 148/170 [01:07<00:09, 2.21it/s]
Validation DataLoader 0: 88%|████████▊ | 149/170 [01:07<00:09, 2.20it/s]
Validation DataLoader 0: 88%|████████▊ | 150/170 [01:08<00:09, 2.21it/s]
Validation DataLoader 0: 89%|████████▉ | 151/170 [01:08<00:08, 2.20it/s]
Validation DataLoader 0: 89%|████████▉ | 152/170 [01:09<00:08, 2.20it/s]
Validation DataLoader 0: 90%|█████████ | 153/170 [01:09<00:07, 2.20it/s]
Validation DataLoader 0: 91%|█████████ | 154/170 [01:09<00:07, 2.20it/s]
Validation DataLoader 0: 91%|█████████ | 155/170 [01:10<00:06, 2.21it/s]
Validation DataLoader 0: 92%|█████████▏| 156/170 [01:10<00:06, 2.21it/s]
Validation DataLoader 0: 92%|█████████▏| 157/170 [01:11<00:05, 2.21it/s]
Validation DataLoader 0: 93%|█████████▎| 158/170 [01:11<00:05, 2.21it/s]
Validation DataLoader 0: 94%|█████████▎| 159/170 [01:12<00:04, 2.21it/s]
Validation DataLoader 0: 94%|█████████▍| 160/170 [01:12<00:04, 2.21it/s]
Validation DataLoader 0: 95%|█████████▍| 161/170 [01:12<00:04, 2.21it/s]
Validation DataLoader 0: 95%|█████████▌| 162/170 [01:13<00:03, 2.21it/s]
Validation DataLoader 0: 96%|█████████▌| 163/170 [01:13<00:03, 2.21it/s]
Validation DataLoader 0: 96%|█████████▋| 164/170 [01:14<00:02, 2.21it/s]
Validation DataLoader 0: 97%|█████████▋| 165/170 [01:14<00:02, 2.20it/s]
Validation DataLoader 0: 98%|█████████▊| 166/170 [01:15<00:01, 2.20it/s]
Validation DataLoader 0: 98%|█████████▊| 167/170 [01:15<00:01, 2.21it/s]
Validation DataLoader 0: 99%|█████████▉| 168/170 [01:16<00:00, 2.21it/s]
Validation DataLoader 0: 99%|█████████▉| 169/170 [01:16<00:00, 2.21it/s]
Validation DataLoader 0: 100%|██████████| 170/170 [01:17<00:00, 2.20it/s]2025-12-19 06:36:52,377 - [TAO Toolkit] - INFO - Validation mAP : 0.00011353408185866579
(pl_dino_model.py:276)

Validation mAP : 0.00011353408185866579

2025-12-19 06:36:52,377 - [TAO Toolkit] - INFO - Validation mAP50 : 0.0006553666887454467
(pl_dino_model.py:277)

Validation mAP50 : 0.0006553666887454467

9

Since I do not have your training images, could you please check if the position of bboxes are correct for the objects?
Usually for the mAP near 0, we need to check the groundtruth firstly.
More, the official notebook of DINO is in tao_tutorials/notebooks/tao_launcher_starter_kit/dino/dino.ipynb at main · NVIDIA/tao_tutorials · GitHub. You can try to run it as a reference to narrow down.

10

I wrote you a message, I have tried everything. I duplicated the notebook dino.ipynb and changed the directories and nothing else. It still doesn’t work for some reason.

11

Hi @vishal.panuganti ,
Well received your message. The ground truth is correct but I am afraid there are some areas which are similar to the ground truth. This will affect the training.
Please double check the training loss. Is it decreasing? Could you please share the full training log and full spec yaml file? Thanks.
Also, did you ever try a larger backbone?