Hi Team,
I am using Nvidia Tao toolkit nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt and trying to train rt-detr-distill model.
I have prepared my dataset in coco format.
I have use below model as a teacher
Resnet50 - Teacher Model Download
!wget https://download.pytorch.org/models/resnet50-0676ba61.pth -O {os.environ[‘PRETRAINED_DIR’]}/resnet50.pth
#Resnet18 - Student Model
!wget https://download.pytorch.org/models/resnet18-f37072fd.pth -O {os.environ[‘PRETRAINED_DIR’]}/resnet18.pth
Train the teacher model (ResNet-50)
!tao model rtdetr train
-e /workspace/tao-experiments/specs/teacher_spec.yaml
results_dir=/workspace/tao-experiments/results/teacher
teacher_spec.yaml
results_dir: /workspace/tao-experiments/results/teacher dataset: train_data_sources: - image_dir: /workspace/tao-experiments/dataset/train/image_2 json_file: /workspace/tao-experiments/dataset/train/train_annotations_coco.json val_data_sources: image_dir: /workspace/tao-experiments/dataset/val/image_2 json_file: /workspace/tao-experiments/dataset/val/val_annotations_coco.json test_data_sources: image_dir: /workspace/tao-experiments/dataset/val/image_2 json_file: /workspace/tao-experiments/dataset/val/val_annotations_coco.json batch_size: 8 workers: 8 num_classes: 4 remap_mscoco_category: false augmentation: multi_scales: - [480, 832] - [512, 896] - [544, 960] - [576, 992] - [608, 1056] train_spatial_size: [544, 960] eval_spatial_size: [544, 960] model: backbone: resnet_50 train_backbone: true pretrained_backbone_path: /workspace/tao-experiments/pretrained_models/resnet50.pth num_queries: 300 dec_layers: 6 enc_layers: 1 return_interm_indices: [1, 2, 3] train: optim: lr: 0.00005 lr_backbone: 0.000005 weight_decay: 0.0001 lr_scheduler: MultiStep lr_steps: [40] lr_decay: 0.1 optimizer: AdamW num_epochs: 80 checkpoint_interval: 1 validation_interval: 1 num_gpus: 1 gpu_ids: [0]
Training Logs
{"date": "5/29/2026", "time": "13:51:45", "status": "STARTED", "verbosity": "INFO", "message": "Starting RT-DETR train."} {"date": "5/29/2026", "time": "13:51:46", "status": "RUNNING", "verbosity": "WARNING", "message": "All batch normalization layers are frozen."} {"date": "5/29/2026", "time": "13:51:46", "status": "RUNNING", "verbosity": "WARNING", "message": "Stage 0 is frozen."} {"date": "5/29/2026", "time": "13:51:46", "status": "RUNNING", "verbosity": "INFO", "message": "Loaded pretrained weights from /workspace/tao-experiments/pretrained_models/resnet50.pth"} {"date": "5/29/2026", "time": "13:51:46", "status": "RUNNING", "verbosity": "INFO", "message": "incompatible keys: <All keys matched successfully>"} {"date": "5/29/2026", "time": "13:51:57", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.0"} {"date": "5/29/2026", "time": "13:51:57", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.0"} {"date": "5/29/2026", "time": "13:52:3", "status": "STARTED", "verbosity": "INFO", "message": "Starting Training Loop."} {"date": "5/29/2026", "time": "15:12:19", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.5221180385320983"} {"date": "5/29/2026", "time": "15:12:19", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.7854342199493163"} {"date": "5/29/2026", "time": "15:12:19", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.5221180385320983", "val_mAP50": "0.7854342199493163", "val_loss": 1.0756627321243286}} {"epoch": 0, "step": 12296, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:16.349406", "time_per_step": "0:00:00.349117", "eta": "4 days, 9:41:31.603112", "date": "5/29/2026", "time": "15:12:20", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.5221180385320983", "val_mAP50": "0.7854342199493163", "val_loss": 1.0756627321243286}} {"date": "5/29/2026", "time": "15:12:20", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 18.43692398071289}} {"date": "5/29/2026", "time": "16:33:6", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.569024587372739", "kpi": {"train_loss": 18.43692398071289}} {"date": "5/29/2026", "time": "16:33:6", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8349913352295117", "kpi": {"train_loss": 18.43692398071289}} {"date": "5/29/2026", "time": "16:33:6", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.569024587372739", "val_mAP50": "0.8349913352295117", "val_loss": 1.0090608596801758}} {"epoch": 1, "step": 24592, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:46.036324", "time_per_step": "0:00:00.350309", "eta": "4 days, 8:59:50.833255", "date": "5/29/2026", "time": "16:33:7", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.569024587372739", "val_mAP50": "0.8349913352295117", "val_loss": 1.0090608596801758}} {"date": "5/29/2026", "time": "16:33:7", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 15.321683883666992}} {"date": "5/29/2026", "time": "17:55:25", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.5993154891310936", "kpi": {"train_loss": 15.321683883666992}} {"date": "5/29/2026", "time": "17:55:25", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8621619627887857", "kpi": {"train_loss": 15.321683883666992}} {"date": "5/29/2026", "time": "17:55:25", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.5993154891310936", "val_mAP50": "0.8621619627887857", "val_loss": 0.9577521085739136}} {"epoch": 2, "step": 36888, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:22:11.647828", "time_per_step": "0:00:00.352452", "eta": "4 days, 9:28:56.882782", "date": "5/29/2026", "time": "17:55:26", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.5993154891310936", "val_mAP50": "0.8621619627887857", "val_loss": 0.9577521085739136}} {"date": "5/29/2026", "time": "17:55:26", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 14.434335708618164}} {"date": "5/29/2026", "time": "19:17:18", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.610289531845289", "kpi": {"train_loss": 14.434335708618164}} {"date": "5/29/2026", "time": "19:17:18", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8671286681746886", "kpi": {"train_loss": 14.434335708618164}} {"date": "5/29/2026", "time": "19:17:18", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.610289531845289", "val_mAP50": "0.8671286681746886", "val_loss": 0.9211843609809875}} {"epoch": 3, "step": 49184, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:51.889364", "time_per_step": "0:00:00.353362", "eta": "4 days, 7:41:43.591682", "date": "5/29/2026", "time": "19:17:19", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.610289531845289", "val_mAP50": "0.8671286681746886", "val_loss": 0.9211843609809875}} {"date": "5/29/2026", "time": "19:17:19", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 13.880502700805664}} {"date": "5/29/2026", "time": "20:38:58", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.618969212906595", "kpi": {"train_loss": 13.880502700805664}} {"date": "5/29/2026", "time": "20:38:58", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8750454562135263", "kpi": {"train_loss": 13.880502700805664}} {"date": "5/29/2026", "time": "20:38:58", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.618969212906595", "val_mAP50": "0.8750454562135263", "val_loss": 0.9226559996604919}} {"epoch": 4, "step": 61480, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:38.873551", "time_per_step": "0:00:00.353585", "eta": "4 days, 6:03:35.516353", "date": "5/29/2026", "time": "20:38:59", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.618969212906595", "val_mAP50": "0.8750454562135263", "val_loss": 0.9226559996604919}} {"date": "5/29/2026", "time": "20:38:59", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 13.511309623718262}} {"date": "5/29/2026", "time": "22:0:47", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6252474117214967", "kpi": {"train_loss": 13.511309623718262}} {"date": "5/29/2026", "time": "22:0:47", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8821282565790435", "kpi": {"train_loss": 13.511309623718262}} {"date": "5/29/2026", "time": "22:0:47", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6252474117214967", "val_mAP50": "0.8821282565790435", "val_loss": 0.9209717512130737}} {"epoch": 5, "step": 73776, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:47.605824", "time_per_step": "0:00:00.353846", "eta": "4 days, 4:52:42.830958", "date": "5/29/2026", "time": "22:0:48", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6252474117214967", "val_mAP50": "0.8821282565790435", "val_loss": 0.9209717512130737}} {"date": "5/29/2026", "time": "22:0:48", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 13.228558540344238}} {"date": "5/29/2026", "time": "23:22:27", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6337671230407278", "kpi": {"train_loss": 13.228558540344238}} {"date": "5/29/2026", "time": "23:22:27", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8895605002274805", "kpi": {"train_loss": 13.228558540344238}} {"date": "5/29/2026", "time": "23:22:27", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6337671230407278", "val_mAP50": "0.8895605002274805", "val_loss": 0.9063936471939087}} {"epoch": 6, "step": 86072, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:38.212336", "time_per_step": "0:00:00.353989", "eta": "4 days, 3:19:29.500515", "date": "5/29/2026", "time": "23:22:28", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6337671230407278", "val_mAP50": "0.8895605002274805", "val_loss": 0.9063936471939087}} {"date": "5/29/2026", "time": "23:22:28", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 13.022134780883789}} {"date": "5/30/2026", "time": "0:43:48", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6402605494387484", "kpi": {"train_loss": 13.022134780883789}} {"date": "5/30/2026", "time": "0:43:48", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8910820554609936", "kpi": {"train_loss": 13.022134780883789}} {"date": "5/30/2026", "time": "0:43:48", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6402605494387484", "val_mAP50": "0.8910820554609936", "val_loss": 0.8948339223861694}} {"epoch": 7, "step": 98368, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:19.913799", "time_per_step": "0:00:00.353858", "eta": "4 days, 1:35:53.793549", "date": "5/30/2026", "time": "0:43:49", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6402605494387484", "val_mAP50": "0.8910820554609936", "val_loss": 0.8948339223861694}} {"date": "5/30/2026", "time": "0:43:49", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.826338768005371}} {"date": "5/30/2026", "time": "2:5:35", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6435123590504284", "kpi": {"train_loss": 12.826338768005371}} {"date": "5/30/2026", "time": "2:5:35", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8934596926711573", "kpi": {"train_loss": 12.826338768005371}} {"date": "5/30/2026", "time": "2:5:35", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6435123590504284", "val_mAP50": "0.8934596926711573", "val_loss": 0.8854935765266418}} {"epoch": 8, "step": 110664, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:45.398673", "time_per_step": "0:00:00.353961", "eta": "4 days, 0:44:43.305787", "date": "5/30/2026", "time": "2:5:36", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6435123590504284", "val_mAP50": "0.8934596926711573", "val_loss": 0.8854935765266418}} {"date": "5/30/2026", "time": "2:5:36", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.668617248535156}} {"date": "5/30/2026", "time": "3:27:6", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6367460661951189", "kpi": {"train_loss": 12.668617248535156}} {"date": "5/30/2026", "time": "3:27:6", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8926260341143254", "kpi": {"train_loss": 12.668617248535156}} {"date": "5/30/2026", "time": "3:27:6", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6367460661951189", "val_mAP50": "0.8926260341143254", "val_loss": 0.8953309059143066}} {"epoch": 9, "step": 122960, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:29.308293", "time_per_step": "0:00:00.353992", "eta": "3 days, 23:04:11.580501", "date": "5/30/2026", "time": "3:27:7", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6367460661951189", "val_mAP50": "0.8926260341143254", "val_loss": 0.8953309059143066}} {"date": "5/30/2026", "time": "3:27:7", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.522625923156738}} {"date": "5/30/2026", "time": "4:49:8", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6440097480570504", "kpi": {"train_loss": 12.522625923156738}} {"date": "5/30/2026", "time": "4:49:8", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.897623675835408", "kpi": {"train_loss": 12.522625923156738}} {"date": "5/30/2026", "time": "4:49:8", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6440097480570504", "val_mAP50": "0.897623675835408", "val_loss": 0.8808030486106873}} {"epoch": 10, "step": 135256, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:22:01.678640", "time_per_step": "0:00:00.354299", "eta": "3 days, 22:19:55.826152", "date": "5/30/2026", "time": "4:49:10", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6440097480570504", "val_mAP50": "0.897623675835408", "val_loss": 0.8808030486106873}} {"date": "5/30/2026", "time": "4:49:10", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.405221939086914}} {"date": "5/30/2026", "time": "6:11:8", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6473311082300923", "kpi": {"train_loss": 12.405221939086914}} {"date": "5/30/2026", "time": "6:11:8", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8969817692198769", "kpi": {"train_loss": 12.405221939086914}} {"date": "5/30/2026", "time": "6:11:8", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6473311082300923", "val_mAP50": "0.8969817692198769", "val_loss": 0.8845046758651733}} {"epoch": 11, "step": 147552, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:57.504861", "time_per_step": "0:00:00.354490", "eta": "3 days, 20:53:10.330523", "date": "5/30/2026", "time": "6:11:10", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6473311082300923", "val_mAP50": "0.8969817692198769", "val_loss": 0.8845046758651733}} {"date": "5/30/2026", "time": "6:11:10", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.288747787475586}} {"date": "5/30/2026", "time": "7:32:59", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6528735569211049", "kpi": {"train_loss": 12.288747787475586}} {"date": "5/30/2026", "time": "7:32:59", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.8996669409101946", "kpi": {"train_loss": 12.288747787475586}} {"date": "5/30/2026", "time": "7:32:59", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6528735569211049", "val_mAP50": "0.8996669409101946", "val_loss": 0.8685032725334167}} {"epoch": 12, "step": 159848, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:45.925552", "time_per_step": "0:00:00.354508", "eta": "3 days, 19:18:17.012009", "date": "5/30/2026", "time": "7:33:0", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6528735569211049", "val_mAP50": "0.8996669409101946", "val_loss": 0.8685032725334167}} {"date": "5/30/2026", "time": "7:33:0", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.178637504577637}} {"date": "5/30/2026", "time": "8:54:56", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6597089249826661", "kpi": {"train_loss": 12.178637504577637}} {"date": "5/30/2026", "time": "8:54:56", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9045669053106418", "kpi": {"train_loss": 12.178637504577637}} {"date": "5/30/2026", "time": "8:54:56", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6597089249826661", "val_mAP50": "0.9045669053106418", "val_loss": 0.8473281264305115}} {"epoch": 13, "step": 172144, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:48.814341", "time_per_step": "0:00:00.354583", "eta": "3 days, 17:59:41.746526", "date": "5/30/2026", "time": "8:54:58", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6597089249826661", "val_mAP50": "0.9045669053106418", "val_loss": 0.8473281264305115}} {"date": "5/30/2026", "time": "8:54:58", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.083407402038574}} {"date": "5/30/2026", "time": "10:16:59", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.65587424750197", "kpi": {"train_loss": 12.083407402038574}} {"date": "5/30/2026", "time": "10:16:59", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9029207462721532", "kpi": {"train_loss": 12.083407402038574}} {"date": "5/30/2026", "time": "10:16:59", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.65587424750197", "val_mAP50": "0.9029207462721532", "val_loss": 0.8598546981811523}} {"epoch": 14, "step": 184440, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:22:01.544775", "time_per_step": "0:00:00.354670", "eta": "3 days, 16:51:40.410407", "date": "5/30/2026", "time": "10:17:0", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.65587424750197", "val_mAP50": "0.9029207462721532", "val_loss": 0.8598546981811523}} {"date": "5/30/2026", "time": "10:17:0", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 12.01944637298584}} {"date": "5/30/2026", "time": "11:38:43", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6627981858186596", "kpi": {"train_loss": 12.01944637298584}} {"date": "5/30/2026", "time": "11:38:43", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9050513178896983", "kpi": {"train_loss": 12.01944637298584}} {"date": "5/30/2026", "time": "11:38:43", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6627981858186596", "val_mAP50": "0.9050513178896983", "val_loss": 0.8591951131820679}} {"epoch": 15, "step": 196736, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:43.190686", "time_per_step": "0:00:00.354702", "eta": "3 days, 15:10:04.203873", "date": "5/30/2026", "time": "11:38:44", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6627981858186596", "val_mAP50": "0.9050513178896983", "val_loss": 0.8591951131820679}} {"date": "5/30/2026", "time": "11:38:44", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.940107345581055}} {"date": "5/30/2026", "time": "13:1:12", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6603939029006994", "kpi": {"train_loss": 11.940107345581055}} {"date": "5/30/2026", "time": "13:1:12", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9043399924443106", "kpi": {"train_loss": 11.940107345581055}} {"date": "5/30/2026", "time": "13:1:12", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6603939029006994", "val_mAP50": "0.9043399924443106", "val_loss": 0.8506278395652771}} {"epoch": 16, "step": 209032, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:22:19.159867", "time_per_step": "0:00:00.354849", "eta": "3 days, 14:26:07.071639", "date": "5/30/2026", "time": "13:1:14", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6603939029006994", "val_mAP50": "0.9043399924443106", "val_loss": 0.8506278395652771}} {"date": "5/30/2026", "time": "13:1:14", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.883407592773438}} {"date": "5/30/2026", "time": "14:23:2", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6528417693466415", "kpi": {"train_loss": 11.883407592773438}} {"date": "5/30/2026", "time": "14:23:2", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9036883390677644", "kpi": {"train_loss": 11.883407592773438}} {"date": "5/30/2026", "time": "14:23:2", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6528417693466415", "val_mAP50": "0.9036883390677644", "val_loss": 0.8590911030769348}} {"epoch": 17, "step": 221328, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:48.138683", "time_per_step": "0:00:00.354876", "eta": "3 days, 12:31:44.598321", "date": "5/30/2026", "time": "14:23:3", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6528417693466415", "val_mAP50": "0.9036883390677644", "val_loss": 0.8590911030769348}} {"date": "5/30/2026", "time": "14:23:3", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.804657936096191}} {"date": "5/30/2026", "time": "15:45:23", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6635034554991441", "kpi": {"train_loss": 11.804657936096191}} {"date": "5/30/2026", "time": "15:45:23", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9052976175109939", "kpi": {"train_loss": 11.804657936096191}} {"date": "5/30/2026", "time": "15:45:23", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6635034554991441", "val_mAP50": "0.9052976175109939", "val_loss": 0.8347188830375671}} {"epoch": 18, "step": 233624, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:22:14.076237", "time_per_step": "0:00:00.354979", "eta": "3 days, 11:36:18.650440", "date": "5/30/2026", "time": "15:45:24", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6635034554991441", "val_mAP50": "0.9052976175109939", "val_loss": 0.8347188830375671}} {"date": "5/30/2026", "time": "15:45:24", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.762237548828125}} {"date": "5/30/2026", "time": "17:7:51", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6609493044519293", "kpi": {"train_loss": 11.762237548828125}} {"date": "5/30/2026", "time": "17:7:51", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9060010158336989", "kpi": {"train_loss": 11.762237548828125}} {"date": "5/30/2026", "time": "17:7:51", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6609493044519293", "val_mAP50": "0.9060010158336989", "val_loss": 0.8547279834747314}} {"epoch": 19, "step": 245920, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:22:26.789391", "time_per_step": "0:00:00.355165", "eta": "3 days, 10:26:47.363434", "date": "5/30/2026", "time": "17:7:53", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6609493044519293", "val_mAP50": "0.9060010158336989", "val_loss": 0.8547279834747314}} {"date": "5/30/2026", "time": "17:7:53", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.691534996032715}} {"date": "5/30/2026", "time": "18:29:12", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.666310439265576", "kpi": {"train_loss": 11.691534996032715}} {"date": "5/30/2026", "time": "18:29:12", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9095973226644418", "kpi": {"train_loss": 11.691534996032715}} {"date": "5/30/2026", "time": "18:29:12", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.666310439265576", "val_mAP50": "0.9095973226644418", "val_loss": 0.8304161429405212}} {"epoch": 20, "step": 258216, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:18.841133", "time_per_step": "0:00:00.355081", "eta": "3 days, 7:57:31.626868", "date": "5/30/2026", "time": "18:29:14", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.666310439265576", "val_mAP50": "0.9095973226644418", "val_loss": 0.8304161429405212}} {"date": "5/30/2026", "time": "18:29:14", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.652144432067871}} {"date": "5/30/2026", "time": "19:50:40", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6656189939835517", "kpi": {"train_loss": 11.652144432067871}} {"date": "5/30/2026", "time": "19:50:40", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9094680266411784", "kpi": {"train_loss": 11.652144432067871}} {"date": "5/30/2026", "time": "19:50:40", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6656189939835517", "val_mAP50": "0.9094680266411784", "val_loss": 0.8465880155563354}} {"epoch": 21, "step": 270512, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:26.488843", "time_per_step": "0:00:00.354962", "eta": "3 days, 6:43:36.352906", "date": "5/30/2026", "time": "19:50:41", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6656189939835517", "val_mAP50": "0.9094680266411784", "val_loss": 0.8465880155563354}} {"date": "5/30/2026", "time": "19:50:41", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.596617698669434}} {"date": "5/30/2026", "time": "21:11:45", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6685049306071078", "kpi": {"train_loss": 11.596617698669434}} {"date": "5/30/2026", "time": "21:11:45", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.908006122182108", "kpi": {"train_loss": 11.596617698669434}} {"date": "5/30/2026", "time": "21:11:45", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6685049306071078", "val_mAP50": "0.908006122182108", "val_loss": 0.842343807220459}} {"epoch": 22, "step": 282808, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:04.185052", "time_per_step": "0:00:00.354846", "eta": "3 days, 5:00:58.547973", "date": "5/30/2026", "time": "21:11:46", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6685049306071078", "val_mAP50": "0.908006122182108", "val_loss": 0.842343807220459}} {"date": "5/30/2026", "time": "21:11:46", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.534299850463867}} {"date": "5/30/2026", "time": "22:34:21", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6716245881509245", "kpi": {"train_loss": 11.534299850463867}} {"date": "5/30/2026", "time": "22:34:21", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9110129813451578", "kpi": {"train_loss": 11.534299850463867}} {"date": "5/30/2026", "time": "22:34:21", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6716245881509245", "val_mAP50": "0.9110129813451578", "val_loss": 0.8275952339172363}} {"epoch": 23, "step": 295104, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:22:34.475419", "time_per_step": "0:00:00.354964", "eta": "3 days, 5:04:10.623480", "date": "5/30/2026", "time": "22:34:22", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6716245881509245", "val_mAP50": "0.9110129813451578", "val_loss": 0.8275952339172363}} {"date": "5/30/2026", "time": "22:34:22", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.501358032226562}} {"date": "5/31/2026", "time": "0:4:6", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6696505502242865", "kpi": {"train_loss": 11.501358032226562}} {"date": "5/31/2026", "time": "0:4:6", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9098934580144709", "kpi": {"train_loss": 11.501358032226562}} {"date": "5/31/2026", "time": "0:4:7", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6696505502242865", "val_mAP50": "0.9098934580144709", "val_loss": 0.8421132564544678}} {"epoch": 24, "step": 307400, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:29:44.174316", "time_per_step": "0:00:00.355159", "eta": "3 days, 10:15:29.587389", "date": "5/31/2026", "time": "0:4:8", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6696505502242865", "val_mAP50": "0.9098934580144709", "val_loss": 0.8421132564544678}} {"date": "5/31/2026", "time": "0:4:8", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.451658248901367}} {"date": "5/31/2026", "time": "1:23:29", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6643446191943254", "kpi": {"train_loss": 11.451658248901367}} {"date": "5/31/2026", "time": "1:23:29", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9082775586607432", "kpi": {"train_loss": 11.451658248901367}} {"date": "5/31/2026", "time": "1:23:29", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6643446191943254", "val_mAP50": "0.9082775586607432", "val_loss": 0.8461251258850098}} {"epoch": 25, "step": 319696, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:19:20.610589", "time_per_step": "0:00:00.354789", "eta": "2 days, 23:24:32.971782", "date": "5/31/2026", "time": "1:23:30", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6643446191943254", "val_mAP50": "0.9082775586607432", "val_loss": 0.8461251258850098}} {"date": "5/31/2026", "time": "1:23:30", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.419159889221191}} {"date": "5/31/2026", "time": "2:43:43", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6742443696967643", "kpi": {"train_loss": 11.419159889221191}} {"date": "5/31/2026", "time": "2:43:43", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9125230633484649", "kpi": {"train_loss": 11.419159889221191}} {"date": "5/31/2026", "time": "2:43:43", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6742443696967643", "val_mAP50": "0.9125230633484649", "val_loss": 0.8321139812469482}} {"epoch": 26, "step": 331992, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:13.298291", "time_per_step": "0:00:00.354574", "eta": "2 days, 22:51:44.809447", "date": "5/31/2026", "time": "2:43:44", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6742443696967643", "val_mAP50": "0.9125230633484649", "val_loss": 0.8321139812469482}} {"date": "5/31/2026", "time": "2:43:44", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.36496639251709}} {"date": "5/31/2026", "time": "4:3:53", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6698590834754928", "kpi": {"train_loss": 11.36496639251709}} {"date": "5/31/2026", "time": "4:3:53", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9120568893726902", "kpi": {"train_loss": 11.36496639251709}} {"date": "5/31/2026", "time": "4:3:53", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6698590834754928", "val_mAP50": "0.9120568893726902", "val_loss": 0.8249950408935547}} {"epoch": 27, "step": 344288, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:09.211634", "time_per_step": "0:00:00.354375", "eta": "2 days, 21:27:59.004951", "date": "5/31/2026", "time": "4:3:55", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6698590834754928", "val_mAP50": "0.9120568893726902", "val_loss": 0.8249950408935547}} {"date": "5/31/2026", "time": "4:3:55", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.345901489257812}} {"date": "5/31/2026", "time": "5:24:14", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6692198743986845", "kpi": {"train_loss": 11.345901489257812}} {"date": "5/31/2026", "time": "5:24:14", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9116486572835705", "kpi": {"train_loss": 11.345901489257812}} {"date": "5/31/2026", "time": "5:24:14", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6692198743986845", "val_mAP50": "0.9116486572835705", "val_loss": 0.8366845846176147}} {"epoch": 28, "step": 356584, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:19.763526", "time_per_step": "0:00:00.354214", "eta": "2 days, 20:16:47.939824", "date": "5/31/2026", "time": "5:24:16", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6692198743986845", "val_mAP50": "0.9116486572835705", "val_loss": 0.8366845846176147}} {"date": "5/31/2026", "time": "5:24:16", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.312235832214355}} {"date": "5/31/2026", "time": "6:44:30", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6675052629221268", "kpi": {"train_loss": 11.312235832214355}} {"date": "5/31/2026", "time": "6:44:30", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9116267100861757", "kpi": {"train_loss": 11.312235832214355}} {"date": "5/31/2026", "time": "6:44:30", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6675052629221268", "val_mAP50": "0.9116267100861757", "val_loss": 0.8423800468444824}} {"epoch": 29, "step": 368880, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:14.992088", "time_per_step": "0:00:00.354075", "eta": "2 days, 18:52:29.604416", "date": "5/31/2026", "time": "6:44:32", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6675052629221268", "val_mAP50": "0.9116267100861757", "val_loss": 0.8423800468444824}} {"date": "5/31/2026", "time": "6:44:32", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.288728713989258}} {"date": "5/31/2026", "time": "8:5:10", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6721899256867852", "kpi": {"train_loss": 11.288728713989258}} {"date": "5/31/2026", "time": "8:5:10", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9137764567687566", "kpi": {"train_loss": 11.288728713989258}} {"date": "5/31/2026", "time": "8:5:10", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6721899256867852", "val_mAP50": "0.9137764567687566", "val_loss": 0.8208468556404114}} {"epoch": 30, "step": 381176, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:38.181029", "time_per_step": "0:00:00.354008", "eta": "2 days, 17:51:10.870413", "date": "5/31/2026", "time": "8:5:11", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6721899256867852", "val_mAP50": "0.9137764567687566", "val_loss": 0.8208468556404114}} {"date": "5/31/2026", "time": "8:5:11", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.24813175201416}} {"date": "5/31/2026", "time": "9:25:50", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6764776971998775", "kpi": {"train_loss": 11.24813175201416}} {"date": "5/31/2026", "time": "9:25:50", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.912604356903841", "kpi": {"train_loss": 11.24813175201416}} {"date": "5/31/2026", "time": "9:25:50", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6764776971998775", "val_mAP50": "0.912604356903841", "val_loss": 0.8219685554504395}} {"epoch": 31, "step": 393472, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:33.969698", "time_per_step": "0:00:00.353912", "eta": "2 days, 16:27:10.545525", "date": "5/31/2026", "time": "9:25:51", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6764776971998775", "val_mAP50": "0.912604356903841", "val_loss": 0.8219685554504395}} {"date": "5/31/2026", "time": "9:25:51", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.22734260559082}} {"date": "5/31/2026", "time": "10:46:33", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6742711129574499", "kpi": {"train_loss": 11.22734260559082}} {"date": "5/31/2026", "time": "10:46:33", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9137127046832944", "kpi": {"train_loss": 11.22734260559082}} {"date": "5/31/2026", "time": "10:46:33", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6742711129574499", "val_mAP50": "0.9137127046832944", "val_loss": 0.8217761516571045}} {"epoch": 32, "step": 405768, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:41.657397", "time_per_step": "0:00:00.353829", "eta": "2 days, 15:12:37.897638", "date": "5/31/2026", "time": "10:46:34", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6742711129574499", "val_mAP50": "0.9137127046832944", "val_loss": 0.8217761516571045}} {"date": "5/31/2026", "time": "10:46:34", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.21201229095459}} {"date": "5/31/2026", "time": "12:7:23", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6788949313153984", "kpi": {"train_loss": 11.21201229095459}} {"date": "5/31/2026", "time": "12:7:23", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9129461405104143", "kpi": {"train_loss": 11.21201229095459}} {"date": "5/31/2026", "time": "12:7:23", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6788949313153984", "val_mAP50": "0.9129461405104143", "val_loss": 0.8213538527488708}} {"epoch": 33, "step": 418064, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:48.351862", "time_per_step": "0:00:00.353777", "eta": "2 days, 13:57:04.185672", "date": "5/31/2026", "time": "12:7:24", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6788949313153984", "val_mAP50": "0.9129461405104143", "val_loss": 0.8213538527488708}} {"date": "5/31/2026", "time": "12:7:24", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.177251815795898}} {"date": "5/31/2026", "time": "13:28:25", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6707730298471505", "kpi": {"train_loss": 11.177251815795898}} {"date": "5/31/2026", "time": "13:28:25", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9136547393875547", "kpi": {"train_loss": 11.177251815795898}} {"date": "5/31/2026", "time": "13:28:25", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6707730298471505", "val_mAP50": "0.9136547393875547", "val_loss": 0.837088942527771}} {"epoch": 34, "step": 430360, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:00.617702", "time_per_step": "0:00:00.353725", "eta": "2 days, 12:45:27.796580", "date": "5/31/2026", "time": "13:28:26", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6707730298471505", "val_mAP50": "0.9136547393875547", "val_loss": 0.837088942527771}} {"date": "5/31/2026", "time": "13:28:26", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.13133716583252}} {"date": "5/31/2026", "time": "14:49:3", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6710377429579707", "kpi": {"train_loss": 11.13133716583252}} {"date": "5/31/2026", "time": "14:49:3", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9139988577060967", "kpi": {"train_loss": 11.13133716583252}} {"date": "5/31/2026", "time": "14:49:3", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6710377429579707", "val_mAP50": "0.9139988577060967", "val_loss": 0.8298373222351074}} {"epoch": 35, "step": 442656, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:36.974133", "time_per_step": "0:00:00.353639", "eta": "2 days, 11:07:06.861874", "date": "5/31/2026", "time": "14:49:4", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6710377429579707", "val_mAP50": "0.9139988577060967", "val_loss": 0.8298373222351074}} {"date": "5/31/2026", "time": "14:49:4", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.131875991821289}} {"date": "5/31/2026", "time": "16:9:10", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6695377751107546", "kpi": {"train_loss": 11.131875991821289}} {"date": "5/31/2026", "time": "16:9:10", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9126787816878199", "kpi": {"train_loss": 11.131875991821289}} {"date": "5/31/2026", "time": "16:9:10", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6695377751107546", "val_mAP50": "0.9126787816878199", "val_loss": 0.8232117295265198}} {"epoch": 36, "step": 454952, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:05.398830", "time_per_step": "0:00:00.353530", "eta": "2 days, 9:23:52.149687", "date": "5/31/2026", "time": "16:9:11", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6695377751107546", "val_mAP50": "0.9126787816878199", "val_loss": 0.8232117295265198}} {"date": "5/31/2026", "time": "16:9:11", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.103382110595703}} {"date": "5/31/2026", "time": "17:29:32", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6719589577768791", "kpi": {"train_loss": 11.103382110595703}} {"date": "5/31/2026", "time": "17:29:32", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9147781202993771", "kpi": {"train_loss": 11.103382110595703}} {"date": "5/31/2026", "time": "17:29:32", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6719589577768791", "val_mAP50": "0.9147781202993771", "val_loss": 0.8275781273841858}} {"epoch": 37, "step": 467248, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:20.713821", "time_per_step": "0:00:00.353429", "eta": "2 days, 8:14:29.980479", "date": "5/31/2026", "time": "17:29:33", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6719589577768791", "val_mAP50": "0.9147781202993771", "val_loss": 0.8275781273841858}} {"date": "5/31/2026", "time": "17:29:33", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.066482543945312}} {"date": "5/31/2026", "time": "18:50:46", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6783705743758174", "kpi": {"train_loss": 11.066482543945312}} {"date": "5/31/2026", "time": "18:50:46", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9151253783491382", "kpi": {"train_loss": 11.066482543945312}} {"date": "5/31/2026", "time": "18:50:46", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6783705743758174", "val_mAP50": "0.9151253783491382", "val_loss": 0.829935610294342}} {"epoch": 38, "step": 479544, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:12.638252", "time_per_step": "0:00:00.353435", "eta": "2 days, 7:29:38.168323", "date": "5/31/2026", "time": "18:50:47", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6783705743758174", "val_mAP50": "0.9151253783491382", "val_loss": 0.829935610294342}} {"date": "5/31/2026", "time": "18:50:47", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.050759315490723}} {"date": "5/31/2026", "time": "20:11:18", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6693501953556581", "kpi": {"train_loss": 11.050759315490723}} {"date": "5/31/2026", "time": "20:11:18", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9144161304723312", "kpi": {"train_loss": 11.050759315490723}} {"date": "5/31/2026", "time": "20:11:18", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6693501953556581", "val_mAP50": "0.9144161304723312", "val_loss": 0.8293554186820984}} {"epoch": 39, "step": 491840, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:31.073617", "time_per_step": "0:00:00.353382", "eta": "2 days, 5:40:42.944698", "date": "5/31/2026", "time": "20:11:19", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6693501953556581", "val_mAP50": "0.9144161304723312", "val_loss": 0.8293554186820984}} {"date": "5/31/2026", "time": "20:11:19", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 11.041130065917969}} {"date": "5/31/2026", "time": "21:31:36", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6862275770627171", "kpi": {"train_loss": 11.041130065917969}} {"date": "5/31/2026", "time": "21:31:36", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9205058236966038", "kpi": {"train_loss": 11.041130065917969}} {"date": "5/31/2026", "time": "21:31:36", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6862275770627171", "val_mAP50": "0.9205058236966038", "val_loss": 0.8044294714927673}} {"epoch": 40, "step": 504136, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:16.941692", "time_per_step": "0:00:00.353315", "eta": "2 days, 4:11:00.725992", "date": "5/31/2026", "time": "21:31:37", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6862275770627171", "val_mAP50": "0.9205058236966038", "val_loss": 0.8044294714927673}} {"date": "5/31/2026", "time": "21:31:37", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 10.64139461517334}} {"date": "5/31/2026", "time": "22:52:34", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.687313496165912", "kpi": {"train_loss": 10.64139461517334}} {"date": "5/31/2026", "time": "22:52:34", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9217833324578335", "kpi": {"train_loss": 10.64139461517334}} {"date": "5/31/2026", "time": "22:52:34", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.687313496165912", "val_mAP50": "0.9217833324578335", "val_loss": 0.8022148609161377}} {"epoch": 41, "step": 516432, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:56.452600", "time_per_step": "0:00:00.353293", "eta": "2 days, 3:15:45.198782", "date": "5/31/2026", "time": "22:52:35", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.687313496165912", "val_mAP50": "0.9217833324578335", "val_loss": 0.8022148609161377}} {"date": "5/31/2026", "time": "22:52:35", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 10.54123306274414}} {"date": "6/1/2026", "time": "0:14:30", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6899576031891844", "kpi": {"train_loss": 10.54123306274414}} {"date": "6/1/2026", "time": "0:14:30", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9222962846825815", "kpi": {"train_loss": 10.54123306274414}} {"date": "6/1/2026", "time": "0:14:30", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6899576031891844", "val_mAP50": "0.9222962846825815", "val_loss": 0.7970926761627197}} {"epoch": 42, "step": 528728, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:54.791749", "time_per_step": "0:00:00.353386", "eta": "2 days, 2:30:47.294695", "date": "6/1/2026", "time": "0:14:32", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6899576031891844", "val_mAP50": "0.9222962846825815", "val_loss": 0.7970926761627197}} {"date": "6/1/2026", "time": "0:14:32", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 10.49298095703125}} {"date": "6/1/2026", "time": "1:35:46", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6868637813144539", "kpi": {"train_loss": 10.49298095703125}} {"date": "6/1/2026", "time": "1:35:46", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9223728937284388", "kpi": {"train_loss": 10.49298095703125}} {"date": "6/1/2026", "time": "1:35:46", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6868637813144539", "val_mAP50": "0.9223728937284388", "val_loss": 0.7968611121177673}} {"epoch": 43, "step": 541024, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:21:14.451125", "time_per_step": "0:00:00.353404", "eta": "2 days, 0:44:40.240488", "date": "6/1/2026", "time": "1:35:47", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6868637813144539", "val_mAP50": "0.9223728937284388", "val_loss": 0.7968611121177673}} {"date": "6/1/2026", "time": "1:35:47", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 10.444878578186035}} {"date": "6/1/2026", "time": "2:56:37", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6904469681415006", "kpi": {"train_loss": 10.444878578186035}} {"date": "6/1/2026", "time": "2:56:37", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9229961271480759", "kpi": {"train_loss": 10.444878578186035}} {"date": "6/1/2026", "time": "2:56:37", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6904469681415006", "val_mAP50": "0.9229961271480759", "val_loss": 0.7901350855827332}} {"epoch": 44, "step": 553320, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:49.947520", "time_per_step": "0:00:00.353371", "eta": "1 day, 23:09:08.163217", "date": "6/1/2026", "time": "2:56:38", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6904469681415006", "val_mAP50": "0.9229961271480759", "val_loss": 0.7901350855827332}} {"date": "6/1/2026", "time": "2:56:38", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 10.449082374572754}} {"date": "6/1/2026", "time": "4:17:38", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6877165669272577", "kpi": {"train_loss": 10.449082374572754}} {"date": "6/1/2026", "time": "4:17:38", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9228422005872915", "kpi": {"train_loss": 10.449082374572754}} {"date": "6/1/2026", "time": "4:17:38", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6877165669272577", "val_mAP50": "0.9228422005872915", "val_loss": 0.8000348806381226}} {"epoch": 45, "step": 565616, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:20:58.575299", "time_per_step": "0:00:00.353355", "eta": "1 day, 21:53:11.560159", "date": "6/1/2026", "time": "4:17:39", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6877165669272577", "val_mAP50": "0.9228422005872915", "val_loss": 0.8000348806381226}} {"date": "6/1/2026", "time": "4:17:39", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 10.427369117736816}} {"date": "6/1/2026", "time": "5:42:42", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.6895492913047768", "kpi": {"train_loss": 10.427369117736816}} {"date": "6/1/2026", "time": "5:42:42", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9228778487139015", "kpi": {"train_loss": 10.427369117736816}} {"date": "6/1/2026", "time": "5:42:42", "status": "RUNNING", "verbosity": "INFO", "message": "Eval metrics generated.", "kpi": {"val_mAP": "0.6895492913047768", "val_mAP50": "0.9228778487139015", "val_loss": 0.7960942387580872}} {"epoch": 46, "step": 577912, "max_epoch": 79, "max_step": 983680, "time_per_epoch": "1:25:03.097942", "time_per_step": "0:00:00.353706", "eta": "1 day, 22:46:42.232074", "date": "6/1/2026", "time": "5:42:43", "status": "RUNNING", "verbosity": "INFO", "message": "Training loop in progress", "kpi": {"val_mAP": "0.6895492913047768", "val_mAP50": "0.9228778487139015", "val_loss": 0.7960942387580872}} {"date": "6/1/2026", "time": "5:42:43", "status": "RUNNING", "verbosity": "INFO", "message": "Train metrics generated.", "kpi": {"train_loss": 10.403894424438477}} {"date": "6/1/2026", "time": "7:21:50", "status": "STARTED", "verbosity": "INFO", "message": "Starting RT-DETR train."} {"date": "6/1/2026", "time": "7:21:50", "status": "RUNNING", "verbosity": "INFO", "message": "Setting resume checkpoint to /workspace/tao-experiments/results/teacher/train/model_epoch_046.pth"} {"date": "6/1/2026", "time": "7:21:50", "status": "RUNNING", "verbosity": "INFO", "message": "Resuming training from checkpoint: /workspace/tao-experiments/results/teacher/train/model_epoch_046.pth"} {"date": "6/1/2026", "time": "7:21:51", "status": "RUNNING", "verbosity": "WARNING", "message": "All batch normalization layers are frozen."} {"date": "6/1/2026", "time": "7:21:51", "status": "RUNNING", "verbosity": "WARNING", "message": "Stage 0 is frozen."} {"date": "6/1/2026", "time": "7:21:51", "status": "RUNNING", "verbosity": "INFO", "message": "Loaded pretrained weights from /workspace/tao-experiments/pretrained_models/resnet50.pth"} {"date": "6/1/2026", "time": "7:21:51", "status": "RUNNING", "verbosity": "INFO", "message": "incompatible keys: <All keys matched successfully>"} {"date": "6/1/2026", "time": "7:22:4", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP : 0.5867447565159093"} {"date": "6/1/2026", "time": "7:22:4", "status": "RUNNING", "verbosity": "INFO", "message": "Validation mAP50 : 0.9001493829705141"} {"date": "6/1/2026", "time": "7:22:10", "status": "STARTED", "verbosity": "INFO", "message": "Starting Training Loop."}
The Training run successful but when I am trying to run distillation process it gives error.
distill_config.yaml
# distill_config.yaml - THIS IS ALL YOU NEED # Student is in 'model' section, Teacher is in 'distill' section results_dir: /workspace/tao-experiments/results/distilled dataset: train_data_sources: - image_dir: /workspace/tao-experiments/dataset/train/image_2 json_file: /workspace/tao-experiments/dataset/train/train_annotations_coco.json val_data_sources: image_dir: /workspace/tao-experiments/dataset/val/image_2 json_file: /workspace/tao-experiments/dataset/val/val_annotations_coco.json test_data_sources: image_dir: /workspace/tao-experiments/dataset/val/image_2 json_file: /workspace/tao-experiments/dataset/val/val_annotations_coco.json batch_size: 32 workers: 8 num_classes: 4 remap_mscoco_category: false augmentation: multi_scales: - [480, 832] - [512, 896] - [544, 960] - [576, 992] - [608, 1056] train_spatial_size: [544, 960] eval_spatial_size: [544, 960] # STUDENT MODEL (defined here - lightweight) model: backbone: resnet_18 train_backbone: true pretrained_backbone_path: /workspace/tao-experiments/pretrained_models/resnet18.pth num_queries: 300 dec_layers: 6 enc_layers: 1 return_interm_indices: [1, 2, 3] train: optim: lr: 0.0001 lr_backbone: 0.00001 weight_decay: 0.0001 lr_scheduler: MultiStep lr_steps: [40] lr_decay: 0.1 optimizer: AdamW num_epochs: 50 checkpoint_interval: 5 validation_interval: 5 num_gpus: 1 gpu_ids: [0] # TEACHER MODEL (defined here - provides knowledge) distill: teacher: backbone: resnet_50 pretrained_teacher_model_path: /workspace/tao-experiments/results/teacher/train/model_epoch_045.pth bindings: - teacher_module_name: srcs student_module_name: srcs criterion: IOU weight: 20
Run distillation to train the student model
# Run distillation to train the student model
!tao model rtdetr distill \
-e /workspace/tao-experiments/specs/distill_config.yaml \
results_dir=/workspace/tao-experiments/results/student
I Get below issue
2026-06-01 13:01:35,854 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io'] 2026-06-01 13:01:35,937 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 360: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt 2026-06-01 13:01:35,959 [TAO Toolkit] [WARNING] nvidia_tao_cli.components.docker_handler.docker_handler 303: Docker will run the commands as root. If you would like to retain your local host permissions, please add the "user":"UID:GID" in the DockerOptions portion of the "/home/smarg/.tao_mounts.json" file. You can obtain your users UID and GID by using the "id -u" and "id -g" commands on the terminal. 2026-06-01 13:01:35,959 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 316: Printing tty value True [2026-06-01 07:31:43,361 - TAO Toolkit - nvidia_tao_core.microservices.utils.job_utils.workflow - INFO] Logging configured at level: INFO /usr/local/lib/python3.12/dist-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) /usr/local/lib/python3.12/dist-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 4.57.6 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models. _warnings.warn( 2026-06-01 07:31:47,491 - [TAO Toolkit] - DEBUG - Successfully loaded backend: modelopt.pytorch (__init__.py:35) [2026-06-01 07:31:47,491 - TAO Toolkit - TAO Toolkit - DEBUG] Successfully loaded backend: modelopt.pytorch 2026-06-01 07:31:47,492 - [TAO Toolkit] - DEBUG - Successfully loaded backend: torchao (__init__.py:47) [2026-06-01 07:31:47,492 - TAO Toolkit - TAO Toolkit - DEBUG] Successfully loaded backend: torchao 2026-06-01 07:31:47,804 - [TAO Toolkit] - DEBUG - Successfully loaded backend: modelopt.onnx (__init__.py:59) [2026-06-01 07:31:47,804 - TAO Toolkit - TAO Toolkit - DEBUG] Successfully loaded backend: modelopt.onnx 2026-06-01 07:31:47,804 - [TAO Toolkit] - DEBUG - Available quantization backends: ['modelopt.pytorch', 'torchao', 'modelopt.onnx'] (__init__.py:70) [2026-06-01 07:31:47,804 - TAO Toolkit - TAO Toolkit - DEBUG] Available quantization backends: ['modelopt.pytorch', 'torchao', 'modelopt.onnx'] 2026-06-01 07:31:54,527 - nvidia_tao_core.microservices.utils.job_utils.workflow - INFO - Logging configured at level: INFO /usr/local/lib/python3.12/dist-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning) sys:1: UserWarning: 'distill_config.yaml' is validated against ConfigStore schema with the same name. This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions. /usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/hydra/hydra_runner.py:110: UserWarning: 'distill_config.yaml' is validated against ConfigStore schema with the same name. This behavior is deprecated in Hydra 1.1 and will be removed in Hydra 1.2. See https://hydra.cc/docs/1.2/upgrades/1.0_to_1.1/automatic_schema_matching for migration instructions. _run_hydra( /usr/local/lib/python3.12/dist-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default. See https://hydra.cc/docs/1.2/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information. ret = run_job( Distill results will be saved at: /workspace/tao-experiments/results/student/distill /usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/loggers/api_logging.py:236: UserWarning: Log file already exists at /workspace/tao-experiments/results/student/distill/status.json rank_zero_warn( 2026-06-01 07:31:59,491 - [TAO Toolkit] - WARNING - All batch normalization layers are frozen. (backbone_base.py:353)All batch normalization layers are frozen. 2026-06-01 07:31:59,492 - [TAO Toolkit] - WARNING - Stage 0 is frozen. (backbone_base.py:365)Stage 0 is frozen. 2026-06-01 07:31:59,550 - [TAO Toolkit] - INFO - Loaded pretrained weights from /workspace/tao-experiments/pretrained_models/resnet18.pth (build_nn_model.py:110)Loaded pretrained weights from /workspace/tao-experiments/pretrained_models/resnet18.pth 2026-06-01 07:31:59,551 - [TAO Toolkit] - INFO - incompatible keys: <All keys matched successfully> (build_nn_model.py:111)incompatible keys: <All keys matched successfully> Error executing job with overrides: ['results_dir=/workspace/tao-experiments/results/student']Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/decorators/workflow.py", line 69, in _func runner(cfg, **kwargs) File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/rtdetr/scripts/distill.py", line 40, in main run_experiment(experiment_config=cfg, File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/rtdetr/scripts/train.py", line 86, in run_experiment pt_model = lightning_module(experiment_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/rtdetr/distillation/distiller.py", line 57, in __init__ super().__init__(experiment_spec, export) File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/distillation/distiller.py", line 35, in __init__ self._build_model(export) File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/rtdetr/distillation/distiller.py", line 142, in _build_model assert v is not None, f"{k_ckpt} doesn't exist in the pretrained teacher model." ^^^^^^^^^^^^^ AssertionError: model.backbone.bn1.num_batches_tracked doesn't exist in the pretrained teacher model. Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace. E0601 07:32:02.048000 287 torch/distributed/elastic/multiprocessing/api.py:880] failed (exitcode: 1) local_rank: 0 (pid: 298) of binary: /usr/bin/python Traceback (most recent call last): File "/usr/local/bin/torchrun", line 7, in <module> sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper return f(*args, **kwargs) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 936, in main run(args) File "/usr/local/lib/python3.12/dist-packages/torch/distributed/run.py", line 927, in run elastic_launch( File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 151, in __call__ return launch_agent(self._config, self._entrypoint, list(args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/distributed/launcher/api.py", line 288, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError: ============================================================ /usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/rtdetr/scripts/distill.py FAILED ------------------------------------------------------------ Failures: <NO_OTHER_FAILURES> ------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2026-06-01_07:32:02 host : 468e36b7b448 rank : 0 (local_rank: 0) exitcode : 1 (pid: 298) error_file: <N/A> traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ 2026-06-01 07:32:02,344 - [TAO Toolkit] - INFO - Sending telemetry data. (entrypoint.py:355) 2026-06-01 07:32:02,344 - [TAO Toolkit] - INFO - Sending telemetry data. (entrypoint.py:355) [2026-06-01 07:32:02,344 - TAO Toolkit - TAO Toolkit - INFO] Sending telemetry data.2026-06-01 07:32:03,662 - [TAO Toolkit] - WARNING - Execution status: FAIL (entrypoint.py:373) 2026-06-01 07:32:03,662 - [TAO Toolkit] - WARNING - Execution status: FAIL (entrypoint.py:373) [2026-06-01 07:32:03,662 - TAO Toolkit - TAO Toolkit - WARNING] Execution status: FAIL 2026-06-01 13:02:05,263 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 381: Stopping container.
Please suggest how to resolve this issue?
