The shape of this layer does not match original model: conv1 Loading the model as a pruned model

Hello ,
Please I try to do the raining with the TLT3 SSD by running this command

tlt ssd train -e /workspace/tlt-experiments/Data/Work/resnet18/config/TrainEvaluator.txt -r /workspace/tlt-experiments/Data/Work/resnet18/data/ResultTrain/weights -k KEY -m /workspace/tlt-experiments/Data/Work/resnet18/pretrained_resnet18/tlt_resnet18_ssd_v1/resnet18.hdf5 --gpus 1

I am getting this error

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2021-05-10 16:56:49,141 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

The shape of this layer does not match original model: conv1
Loading the model as a pruned model.
/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was not compiled. Compile it manually.
warnings.warn('No training configuration found in save file: ’
Initialize optimizer
Traceback (most recent call last):
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 1607, in _create_c_op
c_op = c_api.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Index out of range using input dim 2; input has only 2 dims for ‘loss_1/predictions_loss/strided_slice_2’ (op: ‘StridedSlice’) with input shapes: [?,?], [3], [3], [3] and with computed input tensors: input[3] = <1 1 1>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py”, line 313, in
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py”, line 309, in main
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py”, line 117, in run_experiment
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 342, in compile
sample_weight, mask)
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training_utils.py”, line 404, in weighted
score_array = fn(y_true, y_pred)
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/architecture/ssd_loss.py”, line 131, in compute_loss
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py”, line 802, in _slice_helper
name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/array_ops.py”, line 968, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_array_ops.py”, line 10392, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py”, line 794, in _apply_op_helper
op_def=op_def)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py”, line 513, in new_func
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 3357, in create_op
attrs, op_def, compute_device)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 3426, in _create_op_internal
op_def=op_def)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 1770, in init
control_input_ops)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py”, line 1610, in _create_c_op
raise ValueError(str(e))
ValueError: Index out of range using input dim 2; input has only 2 dims for ‘loss_1/predictions_loss/strided_slice_2’ (op: ‘StridedSlice’) with input shapes: [?,?], [3], [3], [3] and with computed input tensors: input[3] = <1 1 1>.
Traceback (most recent call last):
File “/usr/local/bin/ssd”, line 8, in
sys.exit(main())
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/entrypoint/ssd.py”, line 12, in main
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/entrypoint/entrypoint.py”, line 296, in launch_job
AssertionError: Process run failed.
2021-05-10 18:57:01,719 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

my config file is attached TrainEvaluator.txt (1.5 KB)

I found the same error in this link Error loading 'conv1' when training resnet18_ssd? but it did not solve my problem
THANKS

How did you download /workspace/tlt-experiments/Data/Work/resnet18/pretrained_resnet18/tlt_resnet18_ssd_v1/resnet18.hdf5 ? Can you share the command?

ngc registry model download-version “nvidia/iva/tlt_resnet18_ssd:1”

See https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/open_model_architectures.html#training , please download from https://ngc.nvidia.com/catalog/models/nvidia:tlt_pretrained_object_detection/files?version=resnet18 , i.e,

ngc registry model download-version “nvidia/tlt_pretrained_object_detection:resnet18”

1 Like

Thanks