Please provide the following information when requesting support.
• Hardware (T4/V100/Xavier/Nano/etc) RTX4080
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Classification using classification_pyt
Training spec file is as follow.
encryption_key: tlt_encode
results_dir: StAndrew/classification/classification_results
wandb:
enable: False
train:
enable_ema: False
seed: 47
resume_training_checkpoint_path: null
num_nodes: 1
num_gpus: 2
gpu_ids: [0,1]
validation_interval: 5
checkpoint_interval: 5
tensorboard:
enabled: True
optim:
lr: 0.001
optim: "adamw"
policy: "cosine"
weight_decay: 0.05
warmup_epochs: 0
dataset:
dataset: "CLDataset"
root_dir: StAndrew/classification/abscondment
batch_size: 128
workers: 8
num_classes: 3
img_size: 224
augmentation:
random_flip:
vflip_probability: 0
hflip_probability: 0.5
enable: True
random_rotate:
rotate_probability: 0.5
angle_list: [90, 180, 270]
enable: True
random_color:
brightness: 0.4
contrast: 0.4
saturation: 0.4
enable: False
with_scale_random_crop:
enable: False
with_random_crop: False
with_random_blur: False
train_dataset:
images_dir: StAndrew/classification/abscondment/training_set
val_dataset:
images_dir: StAndrew/classification/abscondment/val_set
test_dataset:
images_dir: StAndrew/classification/abscondment/test_set
model:
backbone:
type: "fan_small_12_p4_hybrid"
pretrained_backbone_path: StAndrew/classification/pretrained_fan_hybrid_small/fan_hybrid_large_in22k.pth
freeze_backbone: True
head:
type: "TAOLinearClsHead"
binary: False
topk: [1]
loss:
type: CrossEntropyLoss
label_smooth_val: 0.1
I am going to train using fan_hybrid_large_in22k.pth pretrained model.
I have errors as
Seed set to 47
2025-11-10 07:33:13,335 - [TAO Toolkit] - WARNING - The backbone is frozen. (backbone_base.py:358)The backbone is frozen.
2025-11-10 07:33:13,335 - [TAO Toolkit] - INFO - Loading pretrained weights from StAndrew/classification/pretrained_fan_hybrid_small/fan_hybrid_large_in22k.pth (classifier.py:64)Loading pretrained weights from StAndrew/classification/pretrained_fan_hybrid_small/fan_hybrid_large_in22k.pth
Error executing job with overrides: ['results_dir=/workspace/home/user/tao_tutorials_ver_6_25_9/StAndrew/classification/abscondment_results', 'train.num_epochs=40']Traceback (most recent call last):
File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/decorators/workflow.py", line 72, in _func
raise e
File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/core/decorators/workflow.py", line 51, in _func
runner(cfg, **kwargs)
File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/classification_pyt/scripts/train.py", line 81, in main
run_experiment(
File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/classification_pyt/scripts/train.py", line 49, in run_experiment
model = lightning_module(experiment_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/classification_pyt/model/classifier_pl_model.py", line 112, in __init__
self._build_model(export)
File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/classification_pyt/model/classifier_pl_model.py", line 172, in _build_model
self.model = build_model(experiment_config=self.experiment_spec, export=export)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/nvidia_tao_pytorch/cv/classification_pyt/model/classifier.py", line 66, in build_model
msg = model.load_state_dict(state_dict, strict=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 2588, in load_state_dict
raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for FAN:
size mismatch for norm.bias: copying a param with shape torch.Size([480]) from checkpoint, the shape in current model is torch.Size([384]).). torch.Size([384]).([384, 1536]).36, 1, 3, 3]).
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
2025-11-10 07:33:14,072 - [TAO Toolkit] - WARNING - Telemetry data couldn't be sent, but the command ran successfully. (entrypoint.py:339)
2025-11-10 07:33:14,072 - TAO Toolkit - WARNING - Telemetry data couldn't be sent, but the command ran successfully.
2025-11-10 07:33:14,072 - [TAO Toolkit] - WARNING - [Error]: 'str' object has no attribute 'decode' (entrypoint.py:342)
2025-11-10 07:33:14,072 - TAO Toolkit - WARNING - [Error]: 'str' object has no attribute 'decode'
2025-11-10 07:33:14,072 - [TAO Toolkit] - WARNING - Execution status: FAIL (entrypoint.py:346)
2025-11-10 07:33:14,072 - TAO Toolkit - WARNING - Execution status: FAIL
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=31 encoding='UTF-8'>
/usr/lib/python3.12/tempfile.py:1075: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpsjqrc1v0'>
_warnings.warn(warn_message, ResourceWarning)
What could be wrong?
