Train with my own tlt model

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

I have already trained a tlt model and I want to use it as a pretrained model to train a new model. However, when I try to run it directly from the commands I have given, the following error occurs.

This is my commands:

!tlt-train ssd -e $SPECS_DIR/ssd_train_resnet18_kitti_buffet_1021.txt \
               -r $USER_EXPERIMENT_DIR/test_dir_unpruned \
               -k $KEY \
               -m $USER_EXPERIMENT_DIR/experiment_dir_retrain_3/weights/ssd_resnet18_epoch_300.tlt \
               --gpus 1

And, this is the error:

ValueError: Dimensions must be equal, but are 38 and 43 for 'loss/ssd_predictions_loss/mul' (op: 'Mul') with input shapes: [8,65382,38], [8,65382,43].

I know that the error is related to the number of object classes, because I want to reduce the classes in the new model, but I don’t know how to fix it. Thanks for helping!

3

So, you already trained a tlt model which aims to detect 43 classes?

4

Yes. Then I want to use this trained model as a pre-trained model in TLT2.0.

5

Can you share your training spec file?

6

The architecture is based on resnet-18.
This is my training spec file:

random_seed: 42
ssd_config {
  aspect_ratios_global: "[1.0, 2.0, 0.5, 3.0, 1.0/3.0]"
  scales: "[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]"
  two_boxes_for_ar1: true
  clip_boxes: false
  loss_loc_weight: 0.8
  focal_loss_alpha: 0.25
  focal_loss_gamma: 2.0
  variances: "[0.1, 0.1, 0.2, 0.2]"
  arch: "resnet"
  nlayers: 18
  freeze_bn: false
  freeze_blocks: 0
}
training_config {
  batch_size_per_gpu: 8
  num_epochs: 2000
  enable_qat: false
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 5e-5
    max_learning_rate: 2e-2
    soft_start: 0.1
    annealing: 0.6
    }
  }
  regularizer {
    type: NO_REG
    weight: 3e-9
  }
}
eval_config {
  validation_period_during_training: 10
  average_precision_mode: SAMPLE
  batch_size: 8
  matching_iou_threshold: 0.5
}
nms_config {
  confidence_threshold: 0.01
  clustering_iou_threshold: 0.4
  top_k: 200
}
augmentation_config {
  preprocessing {
    output_image_width: 960
    output_image_height: 540
    output_image_channel: 3
    crop_right: 960
    crop_bottom: 540
    min_bbox_width: 1.0
    min_bbox_height: 1.0
  }
  spatial_augmentation {
    hflip_probability: 0.5
    vflip_probability: 0.0
    zoom_min: 0.7
    zoom_max: 1.8
    translate_max_x: 8.0
    translate_max_y: 8.0
  }
  color_augmentation {
    hue_rotation_max: 25.0
    saturation_shift_max: 0.20000000298
    contrast_scale_max: 0.10000000149
    contrast_center: 0.5
  }
}
dataset_config {
  data_sources: {
    tfrecords_path: "/workspace/tlt_demo/data/tfrecords/kitti_trainval/kitti_trainval*"
    image_directory_path: "/workspace/tlt_demo/data"
  }
  image_extension: "jpg"

  target_class_mapping {
      key: 'class_001'
      value: 'class_001'
  }
  target_class_mapping {
      key: 'class_002'
      value: 'class_002'
  }
  target_class_mapping {
      key: 'class_003'
      value: 'class_003'
  }
  target_class_mapping {
      key: 'class_004'
      value: 'class_004'
  }
  target_class_mapping {
      key: 'class_005'
      value: 'class_005'
  }
  target_class_mapping {
      key: 'class_006'
      value: 'class_006'
  }
  target_class_mapping {
      key: 'class_007'
      value: 'class_007'
  }
  target_class_mapping {
      key: 'class_008'
      value: 'class_008'
  }
  target_class_mapping {
      key: 'class_009'
      value: 'class_009'
  }
  target_class_mapping {
      key: 'class_010'
      value: 'class_010'
  }
  target_class_mapping {
      key: 'class_011'
      value: 'class_011'
  }
  target_class_mapping {
      key: 'class_012'
      value: 'class_012'
  }
  target_class_mapping {
      key: 'class_013'
      value: 'class_013'
  }
  target_class_mapping {
      key: 'class_014'
      value: 'class_014'
  }
  target_class_mapping {
      key: 'class_015'
      value: 'class_015'
  }
  target_class_mapping {
      key: 'class_016'
      value: 'class_016'
  }
  target_class_mapping {
      key: 'class_017'
      value: 'class_017'
  }
  target_class_mapping {
      key: 'class_018'
      value: 'class_018'
  }
  target_class_mapping {
      key: 'class_019'
      value: 'class_019'
  }
  target_class_mapping {
      key: 'class_020'
      value: 'class_020'
  }
  target_class_mapping {
      key: 'class_021'
      value: 'class_021'
  }
  target_class_mapping {
      key: 'class_022'
      value: 'class_022'
  }
  target_class_mapping {
      key: 'class_023'
      value: 'class_023'
  }
  target_class_mapping {
      key: 'class_024'
      value: 'class_024'
  }
  target_class_mapping {
      key: 'class_025'
      value: 'class_025'
  }
  target_class_mapping {
      key: 'class_026'
      value: 'class_026'
  }
  target_class_mapping {
      key: 'class_027'
      value: 'class_027'
  }
  target_class_mapping {
      key: 'class_028'
      value: 'class_028'
  }
  target_class_mapping {
      key: 'class_029'
      value: 'class_029'
  }
  target_class_mapping {
      key: 'class_030'
      value: 'class_030'
  }
  target_class_mapping {
      key: 'class_031'
      value: 'class_031'
  }
  target_class_mapping {
      key: 'class_032'
      value: 'class_032'
  }
  target_class_mapping {
      key: 'class_033'
      value: 'class_033'
  }
  target_class_mapping {
      key: 'class_034'
      value: 'class_034'
  }
  target_class_mapping {
      key: 'class_035'
      value: 'class_035'
  }
  target_class_mapping {
      key: 'class_036'
      value: 'class_036'
  }
  target_class_mapping {
      key: 'class_037'
      value: 'class_037'
  }
  target_class_mapping {
      key: 'class_038'
      value: 'class_038'
  }
validation_fold: 0
}
7

Can you check above tfrecord files only contain 38 classes?

8

Or you can generate new tfrecord files based on your 38 classes dataset.

9

Do you mean kitti_trainval? If so, there are only 38 classes.

I have generated a new tfrecord with the 38 classes dataset before the training.

10

Or is it feasible to use (.tlt) files as pre-trained models? Is there any rules that only (.hdf5) can be used?

11

You are using tlt 2.0. Can you install latest TLT/TAO and try?
Latest docker supports this.

12

More, can you share the full log?

13

Is the latest version tlt3.0? I will try to install latest docker and try it out. Thanks for your advice.

I guess the reason for the error is that my first trained model has 43 dimensions and I put this (.tlt) file into the second process I want to train without processing it, which is only 38 classes.

Here is my full log:

To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
Using TensorFlow backend.
2021-11-23 08:12:22.021970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-11-23 08:12:24.390643: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-11-23 08:12:24.413575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:65:00.0
2021-11-23 08:12:24.413618: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-11-23 08:12:24.414815: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-11-23 08:12:24.415855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-11-23 08:12:24.416171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-11-23 08:12:24.417494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-11-23 08:12:24.418516: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-11-23 08:12:24.421681: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-11-23 08:12:24.422604: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-11-23 08:12:24.422635: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-11-23 08:12:24.765776: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-11-23 08:12:24.765819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      0 
2021-11-23 08:12:24.765825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0:   N 
2021-11-23 08:12:24.767037: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9859 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:65:00.0, compute capability: 7.5)
2021-11-23 08:12:24,768 [INFO] /usr/local/lib/python3.6/dist-packages/iva/ssd/utils/spec_loader.pyc: Merging specification from /workspace/tlt_demo/ssd/tlt_specs/ssd_train_resnet18_kitti_buffet_1021.txt
2021-11-23 08:12:24,774 [INFO] iva.ssd.scripts.train: Loading pretrained weights. This may take a while...
2021-11-23 08:12:24,882 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2021-11-23 08:12:24,882 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2021-11-23 08:12:24,882 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2021-11-23 08:12:24,882 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4
2021-11-23 08:12:24,883 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 303, number of sources: 1, batch size per gpu: 8, steps: 38
2021-11-23 08:12:24,987 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2021-11-23 08:12:25.015316: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:65:00.0
2021-11-23 08:12:25.015357: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2021-11-23 08:12:25.015396: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2021-11-23 08:12:25.015410: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2021-11-23 08:12:25.015422: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2021-11-23 08:12:25.015436: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2021-11-23 08:12:25.015448: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2021-11-23 08:12:25.015462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-11-23 08:12:25.016147: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2021-11-23 08:12:25,200 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1
2021-11-23 08:12:25,205 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2021-11-23 08:12:25,205 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
The shape of this layer does not match original model: conv1
Loading the model as a pruned model.
2021-11-23 08:17:55,231 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2021-11-23 08:17:55,231 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2021-11-23 08:17:55,231 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2021-11-23 08:17:55,231 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 16, io threads: 32, compute threads: 16, buffered batches: 4
2021-11-23 08:17:55,231 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 75, number of sources: 1, batch size per gpu: 8, steps: 10
2021-11-23 08:17:55,257 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2021-11-23 08:17:55,458 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1
2021-11-23 08:17:55,463 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2021-11-23 08:17:55,463 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1607, in _create_c_op
    c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimensions must be equal, but are 38 and 43 for 'loss/ssd_predictions_loss/mul' (op: 'Mul') with input shapes: [8,65382,38], [8,65382,43].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/tlt-train-g1", line 8, in <module>
    sys.exit(main())
  File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/magnet_train.py", line 45, in main
  File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 248, in main
  File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 133, in run_experiment
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 342, in compile
    sample_weight, mask)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_utils.py", line 404, in weighted
    score_array = fn(y_true, y_pred)
  File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/architecture/ssd_loss.py", line 113, in compute_loss
  File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/architecture/ssd_loss.py", line 71, in bce_focal_loss
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_ops.py", line 899, in binary_op_wrapper
    return func(x, y, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_ops.py", line 1206, in _mul_dispatch
    return gen_math_ops.mul(x, y, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6701, in mul
    "Mul", x=x, y=y, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1770, in __init__
    control_input_ops)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1610, in _create_c_op
    raise ValueError(str(e))
ValueError: Dimensions must be equal, but are 38 and 43 for 'loss/ssd_predictions_loss/mul' (op: 'Mul') with input shapes: [8,65382,38], [8,65382,43].
14

Latest is 3.21.08 version. See Transfer Learning Toolkit for Video Streaming Analytics | NVIDIA NGC

Doc:

1 Like
15

Thank for your help, I will try it first.

17

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.