Please provide the following information when requesting support.
dssd model does not resume
My logs:
print("To run with multigpu, please change --gpus based on the number of available GPUs in your machine.")
> !tao dssd train --gpus 1 --gpu_index=$GPU_INDEX \
> -e $SPECS_DIR/train1.txt \
> -r $USER_EXPERIMENT_DIR/unpruned2 \
> -k $KEY \
> -m $USER_EXPERIMENT_DIR/pretrained_resnet34/pretrained_object_detection_vresnet34/resnet_34.hdf5
To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
2022-02-02 17:06:43,794 [INFO] root: Registry: ['nvcr.io']
.................................
Weights for those layers can not be loaded: ['activation_2', 'add_17', 'add_18', 'add_19', 'add_20', 'add_21', 'add_22', 'add_23', 'add_24', 'add_25', 'add_26', 'add_27', 'add_28', 'add_29', 'add_30', 'add_31', 'add_32']
STOP trainig now and check the pre-train model if this is not expected!
Initialize optimizer
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:121: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.
2022-02-02 14:09:22,847 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:121: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:122: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.
2022-02-02 14:09:22,847 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:122: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:123: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
2022-02-02 14:09:22,848 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:123: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
Input (InputLayer) (6, 3, 300, 300) 0
__________________________________________________________________________________________________
conv1 (Conv2D) (6, 64, 150, 150) 9408 Input[0][0]
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization) (6, 64, 150, 150) 256 conv1[0][0]
__________________________________________________________________________________________________
activation_2 (Activation) (6, 64, 150, 150) 0 bn_conv1[0][0]
__________________________________________________________________________________________________
block_1a_conv_1 (Conv2D) (6, 64, 75, 75) 36864 activation_2[0][0]
__________________________________________________________________________________________________
block_1a_bn_1 (BatchNormalizati (6, 64, 75, 75) 256 block_1a_conv_1[0][0]
__________________________________________________________________________________________________
block_1a_relu_1 (Activation) (6, 64, 75, 75) 0 block_1a_bn_1[0][0]
__________________________________________________________________________________________________
block_1a_conv_2 (Conv2D) (6, 64, 75, 75) 36864 block_1a_relu_1[0][0]
__________________________________________________________________________________________________
block_1a_conv_shortcut (Conv2D) (6, 64, 75, 75) 4096 activation_2[0][0]
__________________________________________________________________________________________________
block_1a_bn_2 (BatchNormalizati (6, 64, 75, 75) 256 block_1a_conv_2[0][0]
__________________________________________________________________________________________________
block_1a_bn_shortcut (BatchNorm (6, 64, 75, 75) 256 block_1a_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_17 (Add) (6, 64, 75, 75) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_1a_relu (Activation) (6, 64, 75, 75) 0 add_17[0][0]
__________________________________________________________________________________________________
block_1b_conv_1 (Conv2D) (6, 64, 75, 75) 36864 block_1a_relu[0][0]
__________________________________________________________________________________________________
block_1b_bn_1 (BatchNormalizati (6, 64, 75, 75) 256 block_1b_conv_1[0][0]
__________________________________________________________________________________________________
block_1b_relu_1 (Activation) (6, 64, 75, 75) 0 block_1b_bn_1[0][0]
__________________________________________________________________________________________________
block_1b_conv_2 (Conv2D) (6, 64, 75, 75) 36864 block_1b_relu_1[0][0]
__________________________________________________________________________________________________
block_1b_conv_shortcut (Conv2D) (6, 64, 75, 75) 4096 block_1a_relu[0][0]
__________________________________________________________________________________________________
block_1b_bn_2 (BatchNormalizati (6, 64, 75, 75) 256 block_1b_conv_2[0][0]
__________________________________________________________________________________________________
block_1b_bn_shortcut (BatchNorm (6, 64, 75, 75) 256 block_1b_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_18 (Add) (6, 64, 75, 75) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_1b_relu (Activation) (6, 64, 75, 75) 0 add_18[0][0]
__________________________________________________________________________________________________
block_1c_conv_1 (Conv2D) (6, 64, 75, 75) 36864 block_1b_relu[0][0]
__________________________________________________________________________________________________
block_1c_bn_1 (BatchNormalizati (6, 64, 75, 75) 256 block_1c_conv_1[0][0]
__________________________________________________________________________________________________
block_1c_relu_1 (Activation) (6, 64, 75, 75) 0 block_1c_bn_1[0][0]
__________________________________________________________________________________________________
block_1c_conv_2 (Conv2D) (6, 64, 75, 75) 36864 block_1c_relu_1[0][0]
__________________________________________________________________________________________________
block_1c_conv_shortcut (Conv2D) (6, 64, 75, 75) 4096 block_1b_relu[0][0]
__________________________________________________________________________________________________
block_1c_bn_2 (BatchNormalizati (6, 64, 75, 75) 256 block_1c_conv_2[0][0]
__________________________________________________________________________________________________
block_1c_bn_shortcut (BatchNorm (6, 64, 75, 75) 256 block_1c_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_19 (Add) (6, 64, 75, 75) 0 block_1c_bn_2[0][0]
block_1c_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_1c_relu (Activation) (6, 64, 75, 75) 0 add_19[0][0]
__________________________________________________________________________________________________
block_2a_conv_1 (Conv2D) (6, 128, 38, 38) 73728 block_1c_relu[0][0]
__________________________________________________________________________________________________
block_2a_bn_1 (BatchNormalizati (6, 128, 38, 38) 512 block_2a_conv_1[0][0]
__________________________________________________________________________________________________
block_2a_relu_1 (Activation) (6, 128, 38, 38) 0 block_2a_bn_1[0][0]
__________________________________________________________________________________________________
block_2a_conv_2 (Conv2D) (6, 128, 38, 38) 147456 block_2a_relu_1[0][0]
__________________________________________________________________________________________________
block_2a_conv_shortcut (Conv2D) (6, 128, 38, 38) 8192 block_1c_relu[0][0]
__________________________________________________________________________________________________
block_2a_bn_2 (BatchNormalizati (6, 128, 38, 38) 512 block_2a_conv_2[0][0]
__________________________________________________________________________________________________
block_2a_bn_shortcut (BatchNorm (6, 128, 38, 38) 512 block_2a_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_20 (Add) (6, 128, 38, 38) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_2a_relu (Activation) (6, 128, 38, 38) 0 add_20[0][0]
__________________________________________________________________________________________________
block_2b_conv_1 (Conv2D) (6, 128, 38, 38) 147456 block_2a_relu[0][0]
__________________________________________________________________________________________________
block_2b_bn_1 (BatchNormalizati (6, 128, 38, 38) 512 block_2b_conv_1[0][0]
__________________________________________________________________________________________________
block_2b_relu_1 (Activation) (6, 128, 38, 38) 0 block_2b_bn_1[0][0]
__________________________________________________________________________________________________
block_2b_conv_2 (Conv2D) (6, 128, 38, 38) 147456 block_2b_relu_1[0][0]
__________________________________________________________________________________________________
block_2b_conv_shortcut (Conv2D) (6, 128, 38, 38) 16384 block_2a_relu[0][0]
__________________________________________________________________________________________________
block_2b_bn_2 (BatchNormalizati (6, 128, 38, 38) 512 block_2b_conv_2[0][0]
__________________________________________________________________________________________________
block_2b_bn_shortcut (BatchNorm (6, 128, 38, 38) 512 block_2b_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_21 (Add) (6, 128, 38, 38) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_2b_relu (Activation) (6, 128, 38, 38) 0 add_21[0][0]
__________________________________________________________________________________________________
block_2c_conv_1 (Conv2D) (6, 128, 38, 38) 147456 block_2b_relu[0][0]
__________________________________________________________________________________________________
block_2c_bn_1 (BatchNormalizati (6, 128, 38, 38) 512 block_2c_conv_1[0][0]
__________________________________________________________________________________________________
block_2c_relu_1 (Activation) (6, 128, 38, 38) 0 block_2c_bn_1[0][0]
__________________________________________________________________________________________________
block_2c_conv_2 (Conv2D) (6, 128, 38, 38) 147456 block_2c_relu_1[0][0]
__________________________________________________________________________________________________
block_2c_conv_shortcut (Conv2D) (6, 128, 38, 38) 16384 block_2b_relu[0][0]
__________________________________________________________________________________________________
block_2c_bn_2 (BatchNormalizati (6, 128, 38, 38) 512 block_2c_conv_2[0][0]
__________________________________________________________________________________________________
block_2c_bn_shortcut (BatchNorm (6, 128, 38, 38) 512 block_2c_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_22 (Add) (6, 128, 38, 38) 0 block_2c_bn_2[0][0]
block_2c_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_2c_relu (Activation) (6, 128, 38, 38) 0 add_22[0][0]
__________________________________________________________________________________________________
block_2d_conv_1 (Conv2D) (6, 128, 38, 38) 147456 block_2c_relu[0][0]
__________________________________________________________________________________________________
block_2d_bn_1 (BatchNormalizati (6, 128, 38, 38) 512 block_2d_conv_1[0][0]
__________________________________________________________________________________________________
block_2d_relu_1 (Activation) (6, 128, 38, 38) 0 block_2d_bn_1[0][0]
__________________________________________________________________________________________________
block_2d_conv_2 (Conv2D) (6, 128, 38, 38) 147456 block_2d_relu_1[0][0]
__________________________________________________________________________________________________
block_2d_conv_shortcut (Conv2D) (6, 128, 38, 38) 16384 block_2c_relu[0][0]
__________________________________________________________________________________________________
block_2d_bn_2 (BatchNormalizati (6, 128, 38, 38) 512 block_2d_conv_2[0][0]
__________________________________________________________________________________________________
block_2d_bn_shortcut (BatchNorm (6, 128, 38, 38) 512 block_2d_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_23 (Add) (6, 128, 38, 38) 0 block_2d_bn_2[0][0]
block_2d_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_2d_relu (Activation) (6, 128, 38, 38) 0 add_23[0][0]
__________________________________________________________________________________________________
block_3a_conv_1 (Conv2D) (6, 256, 19, 19) 294912 block_2d_relu[0][0]
__________________________________________________________________________________________________
block_3a_bn_1 (BatchNormalizati (6, 256, 19, 19) 1024 block_3a_conv_1[0][0]
__________________________________________________________________________________________________
block_3a_relu_1 (Activation) (6, 256, 19, 19) 0 block_3a_bn_1[0][0]
__________________________________________________________________________________________________
block_3a_conv_2 (Conv2D) (6, 256, 19, 19) 589824 block_3a_relu_1[0][0]
__________________________________________________________________________________________________
block_3a_conv_shortcut (Conv2D) (6, 256, 19, 19) 32768 block_2d_relu[0][0]
__________________________________________________________________________________________________
block_3a_bn_2 (BatchNormalizati (6, 256, 19, 19) 1024 block_3a_conv_2[0][0]
__________________________________________________________________________________________________
block_3a_bn_shortcut (BatchNorm (6, 256, 19, 19) 1024 block_3a_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_24 (Add) (6, 256, 19, 19) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_3a_relu (Activation) (6, 256, 19, 19) 0 add_24[0][0]
__________________________________________________________________________________________________
block_3b_conv_1 (Conv2D) (6, 256, 19, 19) 589824 block_3a_relu[0][0]
__________________________________________________________________________________________________
block_3b_bn_1 (BatchNormalizati (6, 256, 19, 19) 1024 block_3b_conv_1[0][0]
__________________________________________________________________________________________________
block_3b_relu_1 (Activation) (6, 256, 19, 19) 0 block_3b_bn_1[0][0]
__________________________________________________________________________________________________
block_3b_conv_2 (Conv2D) (6, 256, 19, 19) 589824 block_3b_relu_1[0][0]
__________________________________________________________________________________________________
block_3b_conv_shortcut (Conv2D) (6, 256, 19, 19) 65536 block_3a_relu[0][0]
__________________________________________________________________________________________________
block_3b_bn_2 (BatchNormalizati (6, 256, 19, 19) 1024 block_3b_conv_2[0][0]
__________________________________________________________________________________________________
block_3b_bn_shortcut (BatchNorm (6, 256, 19, 19) 1024 block_3b_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_25 (Add) (6, 256, 19, 19) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_3b_relu (Activation) (6, 256, 19, 19) 0 add_25[0][0]
__________________________________________________________________________________________________
block_3c_conv_1 (Conv2D) (6, 256, 19, 19) 589824 block_3b_relu[0][0]
__________________________________________________________________________________________________
block_3c_bn_1 (BatchNormalizati (6, 256, 19, 19) 1024 block_3c_conv_1[0][0]
__________________________________________________________________________________________________
block_3c_relu_1 (Activation) (6, 256, 19, 19) 0 block_3c_bn_1[0][0]
__________________________________________________________________________________________________
block_3c_conv_2 (Conv2D) (6, 256, 19, 19) 589824 block_3c_relu_1[0][0]
__________________________________________________________________________________________________
block_3c_conv_shortcut (Conv2D) (6, 256, 19, 19) 65536 block_3b_relu[0][0]
__________________________________________________________________________________________________
block_3c_bn_2 (BatchNormalizati (6, 256, 19, 19) 1024 block_3c_conv_2[0][0]
__________________________________________________________________________________________________
block_3c_bn_shortcut (BatchNorm (6, 256, 19, 19) 1024 block_3c_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_26 (Add) (6, 256, 19, 19) 0 block_3c_bn_2[0][0]
block_3c_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_3c_relu (Activation) (6, 256, 19, 19) 0 add_26[0][0]
__________________________________________________________________________________________________
block_3d_conv_1 (Conv2D) (6, 256, 19, 19) 589824 block_3c_relu[0][0]
__________________________________________________________________________________________________
block_3d_bn_1 (BatchNormalizati (6, 256, 19, 19) 1024 block_3d_conv_1[0][0]
__________________________________________________________________________________________________
block_3d_relu_1 (Activation) (6, 256, 19, 19) 0 block_3d_bn_1[0][0]
__________________________________________________________________________________________________
block_3d_conv_2 (Conv2D) (6, 256, 19, 19) 589824 block_3d_relu_1[0][0]
__________________________________________________________________________________________________
block_3d_conv_shortcut (Conv2D) (6, 256, 19, 19) 65536 block_3c_relu[0][0]
__________________________________________________________________________________________________
block_3d_bn_2 (BatchNormalizati (6, 256, 19, 19) 1024 block_3d_conv_2[0][0]
__________________________________________________________________________________________________
block_3d_bn_shortcut (BatchNorm (6, 256, 19, 19) 1024 block_3d_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_27 (Add) (6, 256, 19, 19) 0 block_3d_bn_2[0][0]
block_3d_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_3d_relu (Activation) (6, 256, 19, 19) 0 add_27[0][0]
__________________________________________________________________________________________________
block_3e_conv_1 (Conv2D) (6, 256, 19, 19) 589824 block_3d_relu[0][0]
__________________________________________________________________________________________________
block_3e_bn_1 (BatchNormalizati (6, 256, 19, 19) 1024 block_3e_conv_1[0][0]
__________________________________________________________________________________________________
block_3e_relu_1 (Activation) (6, 256, 19, 19) 0 block_3e_bn_1[0][0]
__________________________________________________________________________________________________
block_3e_conv_2 (Conv2D) (6, 256, 19, 19) 589824 block_3e_relu_1[0][0]
__________________________________________________________________________________________________
block_3e_conv_shortcut (Conv2D) (6, 256, 19, 19) 65536 block_3d_relu[0][0]
__________________________________________________________________________________________________
block_3e_bn_2 (BatchNormalizati (6, 256, 19, 19) 1024 block_3e_conv_2[0][0]
__________________________________________________________________________________________________
block_3e_bn_shortcut (BatchNorm (6, 256, 19, 19) 1024 block_3e_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_28 (Add) (6, 256, 19, 19) 0 block_3e_bn_2[0][0]
block_3e_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_3e_relu (Activation) (6, 256, 19, 19) 0 add_28[0][0]
__________________________________________________________________________________________________
block_3f_conv_1 (Conv2D) (6, 256, 19, 19) 589824 block_3e_relu[0][0]
__________________________________________________________________________________________________
block_3f_bn_1 (BatchNormalizati (6, 256, 19, 19) 1024 block_3f_conv_1[0][0]
__________________________________________________________________________________________________
block_3f_relu_1 (Activation) (6, 256, 19, 19) 0 block_3f_bn_1[0][0]
__________________________________________________________________________________________________
block_3f_conv_2 (Conv2D) (6, 256, 19, 19) 589824 block_3f_relu_1[0][0]
__________________________________________________________________________________________________
block_3f_conv_shortcut (Conv2D) (6, 256, 19, 19) 65536 block_3e_relu[0][0]
__________________________________________________________________________________________________
block_3f_bn_2 (BatchNormalizati (6, 256, 19, 19) 1024 block_3f_conv_2[0][0]
__________________________________________________________________________________________________
block_3f_bn_shortcut (BatchNorm (6, 256, 19, 19) 1024 block_3f_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_29 (Add) (6, 256, 19, 19) 0 block_3f_bn_2[0][0]
block_3f_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_3f_relu (Activation) (6, 256, 19, 19) 0 add_29[0][0]
__________________________________________________________________________________________________
block_4a_conv_1 (Conv2D) (6, 512, 19, 19) 1179648 block_3f_relu[0][0]
__________________________________________________________________________________________________
block_4a_bn_1 (BatchNormalizati (6, 512, 19, 19) 2048 block_4a_conv_1[0][0]
__________________________________________________________________________________________________
block_4a_relu_1 (Activation) (6, 512, 19, 19) 0 block_4a_bn_1[0][0]
__________________________________________________________________________________________________
block_4a_conv_2 (Conv2D) (6, 512, 19, 19) 2359296 block_4a_relu_1[0][0]
__________________________________________________________________________________________________
block_4a_conv_shortcut (Conv2D) (6, 512, 19, 19) 131072 block_3f_relu[0][0]
__________________________________________________________________________________________________
block_4a_bn_2 (BatchNormalizati (6, 512, 19, 19) 2048 block_4a_conv_2[0][0]
__________________________________________________________________________________________________
block_4a_bn_shortcut (BatchNorm (6, 512, 19, 19) 2048 block_4a_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_30 (Add) (6, 512, 19, 19) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_4a_relu (Activation) (6, 512, 19, 19) 0 add_30[0][0]
__________________________________________________________________________________________________
block_4b_conv_1 (Conv2D) (6, 512, 19, 19) 2359296 block_4a_relu[0][0]
__________________________________________________________________________________________________
block_4b_bn_1 (BatchNormalizati (6, 512, 19, 19) 2048 block_4b_conv_1[0][0]
__________________________________________________________________________________________________
block_4b_relu_1 (Activation) (6, 512, 19, 19) 0 block_4b_bn_1[0][0]
__________________________________________________________________________________________________
block_4b_conv_2 (Conv2D) (6, 512, 19, 19) 2359296 block_4b_relu_1[0][0]
__________________________________________________________________________________________________
block_4b_conv_shortcut (Conv2D) (6, 512, 19, 19) 262144 block_4a_relu[0][0]
__________________________________________________________________________________________________
block_4b_bn_2 (BatchNormalizati (6, 512, 19, 19) 2048 block_4b_conv_2[0][0]
__________________________________________________________________________________________________
block_4b_bn_shortcut (BatchNorm (6, 512, 19, 19) 2048 block_4b_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_31 (Add) (6, 512, 19, 19) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_4b_relu (Activation) (6, 512, 19, 19) 0 add_31[0][0]
__________________________________________________________________________________________________
block_4c_conv_1 (Conv2D) (6, 512, 19, 19) 2359296 block_4b_relu[0][0]
__________________________________________________________________________________________________
block_4c_bn_1 (BatchNormalizati (6, 512, 19, 19) 2048 block_4c_conv_1[0][0]
__________________________________________________________________________________________________
block_4c_relu_1 (Activation) (6, 512, 19, 19) 0 block_4c_bn_1[0][0]
__________________________________________________________________________________________________
block_4c_conv_2 (Conv2D) (6, 512, 19, 19) 2359296 block_4c_relu_1[0][0]
__________________________________________________________________________________________________
block_4c_conv_shortcut (Conv2D) (6, 512, 19, 19) 262144 block_4b_relu[0][0]
__________________________________________________________________________________________________
block_4c_bn_2 (BatchNormalizati (6, 512, 19, 19) 2048 block_4c_conv_2[0][0]
__________________________________________________________________________________________________
block_4c_bn_shortcut (BatchNorm (6, 512, 19, 19) 2048 block_4c_conv_shortcut[0][0]
__________________________________________________________________________________________________
add_32 (Add) (6, 512, 19, 19) 0 block_4c_bn_2[0][0]
block_4c_bn_shortcut[0][0]
__________________________________________________________________________________________________
block_4c_relu (Activation) (6, 512, 19, 19) 0 add_32[0][0]
__________________________________________________________________________________________________
ssd_expand_block_0_conv_0 (Conv (6, 256, 19, 19) 131328 block_4c_relu[0][0]
__________________________________________________________________________________________________
ssd_expand_block_0_relu_0 (ReLU (6, 256, 19, 19) 0 ssd_expand_block_0_conv_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_0_conv_1 (Conv (6, 256, 19, 19) 589824 ssd_expand_block_0_relu_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_0_bn_1 (BatchN (6, 256, 19, 19) 1024 ssd_expand_block_0_conv_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_0_relu_1 (ReLU (6, 256, 19, 19) 0 ssd_expand_block_0_bn_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_1_conv_0 (Conv (6, 128, 19, 19) 32896 ssd_expand_block_0_relu_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_1_relu_0 (ReLU (6, 128, 19, 19) 0 ssd_expand_block_1_conv_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_1_conv_1 (Conv (6, 256, 10, 10) 294912 ssd_expand_block_1_relu_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_1_bn_1 (BatchN (6, 256, 10, 10) 1024 ssd_expand_block_1_conv_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_1_relu_1 (ReLU (6, 256, 10, 10) 0 ssd_expand_block_1_bn_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_2_conv_0 (Conv (6, 64, 10, 10) 16448 ssd_expand_block_1_relu_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_2_relu_0 (ReLU (6, 64, 10, 10) 0 ssd_expand_block_2_conv_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_2_conv_1 (Conv (6, 128, 5, 5) 73728 ssd_expand_block_2_relu_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_2_bn_1 (BatchN (6, 128, 5, 5) 512 ssd_expand_block_2_conv_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_2_relu_1 (ReLU (6, 128, 5, 5) 0 ssd_expand_block_2_bn_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_3_conv_0 (Conv (6, 64, 5, 5) 8256 ssd_expand_block_2_relu_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_3_relu_0 (ReLU (6, 64, 5, 5) 0 ssd_expand_block_3_conv_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_3_conv_1 (Conv (6, 128, 3, 3) 73728 ssd_expand_block_3_relu_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_3_bn_1 (BatchN (6, 128, 3, 3) 512 ssd_expand_block_3_conv_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_3_relu_1 (ReLU (6, 128, 3, 3) 0 ssd_expand_block_3_bn_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_4_conv_0 (Conv (6, 64, 3, 3) 8256 ssd_expand_block_3_relu_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_4_relu_0 (ReLU (6, 64, 3, 3) 0 ssd_expand_block_4_conv_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_4_conv_1 (Conv (6, 128, 2, 2) 73728 ssd_expand_block_4_relu_0[0][0]
__________________________________________________________________________________________________
ssd_expand_block_4_bn_1 (BatchN (6, 128, 2, 2) 512 ssd_expand_block_4_conv_1[0][0]
__________________________________________________________________________________________________
dssd_large_conv_0_block_0 (Conv (6, 128, 3, 3) 147456 ssd_expand_block_3_relu_1[0][0]
__________________________________________________________________________________________________
ssd_expand_block_4_relu_1 (ReLU (6, 128, 2, 2) 0 ssd_expand_block_4_bn_1[0][0]
__________________________________________________________________________________________________
dssd_large_bn_0_block_0 (BatchN (6, 128, 3, 3) 512 dssd_large_conv_0_block_0[0][0]
__________________________________________________________________________________________________
dssd_small_deconv_block_0 (Conv (6, 128, 4, 4) 65536 ssd_expand_block_4_relu_1[0][0]
__________________________________________________________________________________________________
dssd_large_relu_0_block_0 (ReLU (6, 128, 3, 3) 0 dssd_large_bn_0_block_0[0][0]
__________________________________________________________________________________________________
dssd_pad_0 (ZeroPadding2D) (6, 128, 4, 4) 0 dssd_small_deconv_block_0[0][0]
__________________________________________________________________________________________________
dssd_large_conv_1_block_0 (Conv (6, 128, 3, 3) 147456 dssd_large_relu_0_block_0[0][0]
__________________________________________________________________________________________________
dssd_small_conv_block_0 (Conv2D (6, 128, 3, 3) 65536 dssd_pad_0[0][0]
__________________________________________________________________________________________________
dssd_large_bn_1_block_0 (BatchN (6, 128, 3, 3) 512 dssd_large_conv_1_block_0[0][0]
__________________________________________________________________________________________________
dssd_small_bn_block_0 (BatchNor (6, 128, 3, 3) 512 dssd_small_conv_block_0[0][0]
__________________________________________________________________________________________________
dssd_mul_block_0 (Multiply) (6, 128, 3, 3) 0 dssd_large_bn_1_block_0[0][0]
dssd_small_bn_block_0[0][0]
__________________________________________________________________________________________________
dssd_large_conv_0_block_1 (Conv (6, 128, 5, 5) 147456 ssd_expand_block_2_relu_1[0][0]
__________________________________________________________________________________________________
dssd_relu_block_0 (ReLU) (6, 128, 3, 3) 0 dssd_mul_block_0[0][0]
__________________________________________________________________________________________________
dssd_large_bn_0_block_1 (BatchN (6, 128, 5, 5) 512 dssd_large_conv_0_block_1[0][0]
__________________________________________________________________________________________________
dssd_small_deconv_block_1 (Conv (6, 128, 6, 6) 65536 dssd_relu_block_0[0][0]
__________________________________________________________________________________________________
dssd_large_relu_0_block_1 (ReLU (6, 128, 5, 5) 0 dssd_large_bn_0_block_1[0][0]
__________________________________________________________________________________________________
dssd_pad_1 (ZeroPadding2D) (6, 128, 6, 6) 0 dssd_small_deconv_block_1[0][0]
__________________________________________________________________________________________________
dssd_large_conv_1_block_1 (Conv (6, 128, 5, 5) 147456 dssd_large_relu_0_block_1[0][0]
__________________________________________________________________________________________________
dssd_small_conv_block_1 (Conv2D (6, 128, 5, 5) 65536 dssd_pad_1[0][0]
__________________________________________________________________________________________________
dssd_large_bn_1_block_1 (BatchN (6, 128, 5, 5) 512 dssd_large_conv_1_block_1[0][0]
__________________________________________________________________________________________________
dssd_small_bn_block_1 (BatchNor (6, 128, 5, 5) 512 dssd_small_conv_block_1[0][0]
__________________________________________________________________________________________________
dssd_mul_block_1 (Multiply) (6, 128, 5, 5) 0 dssd_large_bn_1_block_1[0][0]
dssd_small_bn_block_1[0][0]
__________________________________________________________________________________________________
dssd_large_conv_0_block_2 (Conv (6, 256, 10, 10) 589824 ssd_expand_block_1_relu_1[0][0]
__________________________________________________________________________________________________
dssd_relu_block_1 (ReLU) (6, 128, 5, 5) 0 dssd_mul_block_1[0][0]
__________________________________________________________________________________________________
dssd_large_bn_0_block_2 (BatchN (6, 256, 10, 10) 1024 dssd_large_conv_0_block_2[0][0]
__________________________________________________________________________________________________
dssd_small_deconv_block_2 (Conv (6, 256, 10, 10) 131072 dssd_relu_block_1[0][0]
__________________________________________________________________________________________________
dssd_large_relu_0_block_2 (ReLU (6, 256, 10, 10) 0 dssd_large_bn_0_block_2[0][0]
__________________________________________________________________________________________________
dssd_pad_2 (ZeroPadding2D) (6, 256, 12, 12) 0 dssd_small_deconv_block_2[0][0]
__________________________________________________________________________________________________
dssd_large_conv_1_block_2 (Conv (6, 256, 10, 10) 589824 dssd_large_relu_0_block_2[0][0]
__________________________________________________________________________________________________
dssd_small_conv_block_2 (Conv2D (6, 256, 10, 10) 589824 dssd_pad_2[0][0]
__________________________________________________________________________________________________
dssd_large_bn_1_block_2 (BatchN (6, 256, 10, 10) 1024 dssd_large_conv_1_block_2[0][0]
__________________________________________________________________________________________________
dssd_small_bn_block_2 (BatchNor (6, 256, 10, 10) 1024 dssd_small_conv_block_2[0][0]
__________________________________________________________________________________________________
dssd_mul_block_2 (Multiply) (6, 256, 10, 10) 0 dssd_large_bn_1_block_2[0][0]
dssd_small_bn_block_2[0][0]
__________________________________________________________________________________________________
dssd_large_conv_0_block_3 (Conv (6, 256, 19, 19) 589824 ssd_expand_block_0_relu_1[0][0]
__________________________________________________________________________________________________
dssd_relu_block_2 (ReLU) (6, 256, 10, 10) 0 dssd_mul_block_2[0][0]
__________________________________________________________________________________________________
dssd_large_bn_0_block_3 (BatchN (6, 256, 19, 19) 1024 dssd_large_conv_0_block_3[0][0]
__________________________________________________________________________________________________
dssd_small_deconv_block_3 (Conv (6, 256, 20, 20) 262144 dssd_relu_block_2[0][0]
__________________________________________________________________________________________________
dssd_large_relu_0_block_3 (ReLU (6, 256, 19, 19) 0 dssd_large_bn_0_block_3[0][0]
__________________________________________________________________________________________________
dssd_pad_3 (ZeroPadding2D) (6, 256, 20, 20) 0 dssd_small_deconv_block_3[0][0]
__________________________________________________________________________________________________
dssd_large_conv_1_block_3 (Conv (6, 256, 19, 19) 589824 dssd_large_relu_0_block_3[0][0]
__________________________________________________________________________________________________
dssd_small_conv_block_3 (Conv2D (6, 256, 19, 19) 262144 dssd_pad_3[0][0]
__________________________________________________________________________________________________
dssd_large_bn_1_block_3 (BatchN (6, 256, 19, 19) 1024 dssd_large_conv_1_block_3[0][0]
__________________________________________________________________________________________________
dssd_small_bn_block_3 (BatchNor (6, 256, 19, 19) 1024 dssd_small_conv_block_3[0][0]
__________________________________________________________________________________________________
dssd_mul_block_3 (Multiply) (6, 256, 19, 19) 0 dssd_large_bn_1_block_3[0][0]
dssd_small_bn_block_3[0][0]
__________________________________________________________________________________________________
dssd_large_conv_0_block_4 (Conv (6, 128, 38, 38) 147456 block_2d_relu[0][0]
__________________________________________________________________________________________________
dssd_relu_block_3 (ReLU) (6, 256, 19, 19) 0 dssd_mul_block_3[0][0]
__________________________________________________________________________________________________
dssd_large_bn_0_block_4 (BatchN (6, 128, 38, 38) 512 dssd_large_conv_0_block_4[0][0]
__________________________________________________________________________________________________
dssd_small_deconv_block_4 (Conv (6, 128, 38, 38) 131072 dssd_relu_block_3[0][0]
__________________________________________________________________________________________________
dssd_large_relu_0_block_4 (ReLU (6, 128, 38, 38) 0 dssd_large_bn_0_block_4[0][0]
__________________________________________________________________________________________________
dssd_pad_4 (ZeroPadding2D) (6, 128, 40, 40) 0 dssd_small_deconv_block_4[0][0]
__________________________________________________________________________________________________
dssd_large_conv_1_block_4 (Conv (6, 128, 38, 38) 147456 dssd_large_relu_0_block_4[0][0]
__________________________________________________________________________________________________
dssd_small_conv_block_4 (Conv2D (6, 128, 38, 38) 147456 dssd_pad_4[0][0]
__________________________________________________________________________________________________
dssd_large_bn_1_block_4 (BatchN (6, 128, 38, 38) 512 dssd_large_conv_1_block_4[0][0]
__________________________________________________________________________________________________
dssd_small_bn_block_4 (BatchNor (6, 128, 38, 38) 512 dssd_small_conv_block_4[0][0]
__________________________________________________________________________________________________
dssd_mul_block_4 (Multiply) (6, 128, 38, 38) 0 dssd_large_bn_1_block_4[0][0]
dssd_small_bn_block_4[0][0]
__________________________________________________________________________________________________
dssd_relu_block_4 (ReLU) (6, 128, 38, 38) 0 dssd_mul_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_0 (Conv2 (6, 128, 38, 38) 16384 dssd_relu_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_1 (Conv2 (6, 128, 19, 19) 32768 dssd_relu_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_2 (Conv2 (6, 128, 10, 10) 32768 dssd_relu_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_3 (Conv2 (6, 128, 5, 5) 16384 dssd_relu_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_4 (Conv2 (6, 128, 3, 3) 16384 dssd_relu_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_5 (Conv2 (6, 128, 2, 2) 16384 ssd_expand_block_4_relu_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_0 (BatchNo (6, 128, 38, 38) 512 ssd_mpred_conv_0_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_1 (BatchNo (6, 128, 19, 19) 512 ssd_mpred_conv_0_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_2 (BatchNo (6, 128, 10, 10) 512 ssd_mpred_conv_0_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_3 (BatchNo (6, 128, 5, 5) 512 ssd_mpred_conv_0_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_4 (BatchNo (6, 128, 3, 3) 512 ssd_mpred_conv_0_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_5 (BatchNo (6, 128, 2, 2) 512 ssd_mpred_conv_0_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_0 (ReLU) (6, 128, 38, 38) 0 ssd_mpred_bn_0_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_1 (ReLU) (6, 128, 19, 19) 0 ssd_mpred_bn_0_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_2 (ReLU) (6, 128, 10, 10) 0 ssd_mpred_bn_0_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_3 (ReLU) (6, 128, 5, 5) 0 ssd_mpred_bn_0_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_4 (ReLU) (6, 128, 3, 3) 0 ssd_mpred_bn_0_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_5 (ReLU) (6, 128, 2, 2) 0 ssd_mpred_bn_0_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_0 (Conv2 (6, 128, 38, 38) 16384 ssd_mpred_relu_0_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_1 (Conv2 (6, 128, 19, 19) 16384 ssd_mpred_relu_0_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_2 (Conv2 (6, 128, 10, 10) 16384 ssd_mpred_relu_0_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_3 (Conv2 (6, 128, 5, 5) 16384 ssd_mpred_relu_0_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_4 (Conv2 (6, 128, 3, 3) 16384 ssd_mpred_relu_0_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_5 (Conv2 (6, 128, 2, 2) 16384 ssd_mpred_relu_0_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_0 (BatchNo (6, 128, 38, 38) 512 ssd_mpred_conv_1_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_1 (BatchNo (6, 128, 19, 19) 512 ssd_mpred_conv_1_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_2 (BatchNo (6, 128, 10, 10) 512 ssd_mpred_conv_1_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_3 (BatchNo (6, 128, 5, 5) 512 ssd_mpred_conv_1_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_4 (BatchNo (6, 128, 3, 3) 512 ssd_mpred_conv_1_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_5 (BatchNo (6, 128, 2, 2) 512 ssd_mpred_conv_1_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_0 (ReLU) (6, 128, 38, 38) 0 ssd_mpred_bn_1_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_1 (ReLU) (6, 128, 19, 19) 0 ssd_mpred_bn_1_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_2 (ReLU) (6, 128, 10, 10) 0 ssd_mpred_bn_1_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_3 (ReLU) (6, 128, 5, 5) 0 ssd_mpred_bn_1_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_4 (ReLU) (6, 128, 3, 3) 0 ssd_mpred_bn_1_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_5 (ReLU) (6, 128, 2, 2) 0 ssd_mpred_bn_1_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_0 (Conv2 (6, 512, 38, 38) 65536 ssd_mpred_relu_1_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_0 (Conv2 (6, 512, 38, 38) 65536 dssd_relu_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_1 (Conv2 (6, 512, 19, 19) 65536 ssd_mpred_relu_1_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_1 (Conv2 (6, 512, 19, 19) 131072 dssd_relu_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_2 (Conv2 (6, 512, 10, 10) 65536 ssd_mpred_relu_1_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_2 (Conv2 (6, 512, 10, 10) 131072 dssd_relu_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_3 (Conv2 (6, 512, 5, 5) 65536 ssd_mpred_relu_1_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_3 (Conv2 (6, 512, 5, 5) 65536 dssd_relu_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_4 (Conv2 (6, 512, 3, 3) 65536 ssd_mpred_relu_1_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_4 (Conv2 (6, 512, 3, 3) 65536 dssd_relu_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_5 (Conv2 (6, 512, 2, 2) 65536 ssd_mpred_relu_1_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_5 (Conv2 (6, 512, 2, 2) 65536 ssd_expand_block_4_relu_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_0 (BatchNo (6, 512, 38, 38) 2048 ssd_mpred_conv_2_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_0 (BatchNo (6, 512, 38, 38) 2048 ssd_mpred_conv_3_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_1 (BatchNo (6, 512, 19, 19) 2048 ssd_mpred_conv_2_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_1 (BatchNo (6, 512, 19, 19) 2048 ssd_mpred_conv_3_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_2 (BatchNo (6, 512, 10, 10) 2048 ssd_mpred_conv_2_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_2 (BatchNo (6, 512, 10, 10) 2048 ssd_mpred_conv_3_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_3 (BatchNo (6, 512, 5, 5) 2048 ssd_mpred_conv_2_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_3 (BatchNo (6, 512, 5, 5) 2048 ssd_mpred_conv_3_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_4 (BatchNo (6, 512, 3, 3) 2048 ssd_mpred_conv_2_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_4 (BatchNo (6, 512, 3, 3) 2048 ssd_mpred_conv_3_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_5 (BatchNo (6, 512, 2, 2) 2048 ssd_mpred_conv_2_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_5 (BatchNo (6, 512, 2, 2) 2048 ssd_mpred_conv_3_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_add_block_0 (Add) (6, 512, 38, 38) 0 ssd_mpred_bn_2_block_0[0][0]
ssd_mpred_bn_3_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_add_block_1 (Add) (6, 512, 19, 19) 0 ssd_mpred_bn_2_block_1[0][0]
ssd_mpred_bn_3_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_add_block_2 (Add) (6, 512, 10, 10) 0 ssd_mpred_bn_2_block_2[0][0]
ssd_mpred_bn_3_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_add_block_3 (Add) (6, 512, 5, 5) 0 ssd_mpred_bn_2_block_3[0][0]
ssd_mpred_bn_3_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_add_block_4 (Add) (6, 512, 3, 3) 0 ssd_mpred_bn_2_block_4[0][0]
ssd_mpred_bn_3_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_add_block_5 (Add) (6, 512, 2, 2) 0 ssd_mpred_bn_2_block_5[0][0]
ssd_mpred_bn_3_block_5[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_0 (ReLU) (6, 512, 38, 38) 0 ssd_mpred_add_block_0[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_1 (ReLU) (6, 512, 19, 19) 0 ssd_mpred_add_block_1[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_2 (ReLU) (6, 512, 10, 10) 0 ssd_mpred_add_block_2[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_3 (ReLU) (6, 512, 5, 5) 0 ssd_mpred_add_block_3[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_4 (ReLU) (6, 512, 3, 3) 0 ssd_mpred_add_block_4[0][0]
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_5 (ReLU) (6, 512, 2, 2) 0 ssd_mpred_add_block_5[0][0]
__________________________________________________________________________________________________
ssd_conf_0 (Conv2D) (6, 12, 38, 38) 55308 ssd_mpred_relu_3_block_0[0][0]
__________________________________________________________________________________________________
ssd_conf_1 (Conv2D) (6, 12, 19, 19) 55308 ssd_mpred_relu_3_block_1[0][0]
__________________________________________________________________________________________________
ssd_conf_2 (Conv2D) (6, 12, 10, 10) 55308 ssd_mpred_relu_3_block_2[0][0]
__________________________________________________________________________________________________
ssd_conf_3 (Conv2D) (6, 12, 5, 5) 55308 ssd_mpred_relu_3_block_3[0][0]
__________________________________________________________________________________________________
ssd_conf_4 (Conv2D) (6, 12, 3, 3) 55308 ssd_mpred_relu_3_block_4[0][0]
__________________________________________________________________________________________________
ssd_conf_5 (Conv2D) (6, 12, 2, 2) 55308 ssd_mpred_relu_3_block_5[0][0]
__________________________________________________________________________________________________
permute_13 (Permute) (6, 38, 38, 12) 0 ssd_conf_0[0][0]
__________________________________________________________________________________________________
permute_14 (Permute) (6, 19, 19, 12) 0 ssd_conf_1[0][0]
__________________________________________________________________________________________________
permute_15 (Permute) (6, 10, 10, 12) 0 ssd_conf_2[0][0]
__________________________________________________________________________________________________
permute_16 (Permute) (6, 5, 5, 12) 0 ssd_conf_3[0][0]
__________________________________________________________________________________________________
permute_17 (Permute) (6, 3, 3, 12) 0 ssd_conf_4[0][0]
__________________________________________________________________________________________________
permute_18 (Permute) (6, 2, 2, 12) 0 ssd_conf_5[0][0]
__________________________________________________________________________________________________
conf_reshape_0 (Reshape) (6, 8664, 1, 2) 0 permute_13[0][0]
__________________________________________________________________________________________________
conf_reshape_1 (Reshape) (6, 2166, 1, 2) 0 permute_14[0][0]
__________________________________________________________________________________________________
conf_reshape_2 (Reshape) (6, 600, 1, 2) 0 permute_15[0][0]
__________________________________________________________________________________________________
conf_reshape_3 (Reshape) (6, 150, 1, 2) 0 permute_16[0][0]
__________________________________________________________________________________________________
conf_reshape_4 (Reshape) (6, 54, 1, 2) 0 permute_17[0][0]
__________________________________________________________________________________________________
conf_reshape_5 (Reshape) (6, 24, 1, 2) 0 permute_18[0][0]
__________________________________________________________________________________________________
mbox_conf (Concatenate) (6, 11658, 1, 2) 0 conf_reshape_0[0][0]
conf_reshape_1[0][0]
conf_reshape_2[0][0]
conf_reshape_3[0][0]
conf_reshape_4[0][0]
conf_reshape_5[0][0]
__________________________________________________________________________________________________
ssd_loc_0 (Conv2D) (6, 24, 38, 38) 110616 ssd_mpred_relu_3_block_0[0][0]
__________________________________________________________________________________________________
ssd_loc_1 (Conv2D) (6, 24, 19, 19) 110616 ssd_mpred_relu_3_block_1[0][0]
__________________________________________________________________________________________________
ssd_loc_2 (Conv2D) (6, 24, 10, 10) 110616 ssd_mpred_relu_3_block_2[0][0]
__________________________________________________________________________________________________
ssd_loc_3 (Conv2D) (6, 24, 5, 5) 110616 ssd_mpred_relu_3_block_3[0][0]
__________________________________________________________________________________________________
ssd_loc_4 (Conv2D) (6, 24, 3, 3) 110616 ssd_mpred_relu_3_block_4[0][0]
__________________________________________________________________________________________________
ssd_loc_5 (Conv2D) (6, 24, 2, 2) 110616 ssd_mpred_relu_3_block_5[0][0]
__________________________________________________________________________________________________
before_softmax_permute (Permute (6, 2, 1, 11658) 0 mbox_conf[0][0]
__________________________________________________________________________________________________
permute_19 (Permute) (6, 38, 38, 24) 0 ssd_loc_0[0][0]
__________________________________________________________________________________________________
permute_20 (Permute) (6, 19, 19, 24) 0 ssd_loc_1[0][0]
__________________________________________________________________________________________________
permute_21 (Permute) (6, 10, 10, 24) 0 ssd_loc_2[0][0]
__________________________________________________________________________________________________
permute_22 (Permute) (6, 5, 5, 24) 0 ssd_loc_3[0][0]
__________________________________________________________________________________________________
permute_23 (Permute) (6, 3, 3, 24) 0 ssd_loc_4[0][0]
__________________________________________________________________________________________________
permute_24 (Permute) (6, 2, 2, 24) 0 ssd_loc_5[0][0]
__________________________________________________________________________________________________
ssd_anchor_0 (AnchorBoxes) (6, 1444, 6, 8) 0 ssd_loc_0[0][0]
__________________________________________________________________________________________________
ssd_anchor_1 (AnchorBoxes) (6, 361, 6, 8) 0 ssd_loc_1[0][0]
__________________________________________________________________________________________________
ssd_anchor_2 (AnchorBoxes) (6, 100, 6, 8) 0 ssd_loc_2[0][0]
__________________________________________________________________________________________________
ssd_anchor_3 (AnchorBoxes) (6, 25, 6, 8) 0 ssd_loc_3[0][0]
__________________________________________________________________________________________________
ssd_anchor_4 (AnchorBoxes) (6, 9, 6, 8) 0 ssd_loc_4[0][0]
__________________________________________________________________________________________________
ssd_anchor_5 (AnchorBoxes) (6, 4, 6, 8) 0 ssd_loc_5[0][0]
__________________________________________________________________________________________________
mbox_conf_softmax_ (Softmax) (6, 2, 1, 11658) 0 before_softmax_permute[0][0]
__________________________________________________________________________________________________
loc_reshape_0 (Reshape) (6, 8664, 1, 4) 0 permute_19[0][0]
__________________________________________________________________________________________________
loc_reshape_1 (Reshape) (6, 2166, 1, 4) 0 permute_20[0][0]
__________________________________________________________________________________________________
loc_reshape_2 (Reshape) (6, 600, 1, 4) 0 permute_21[0][0]
__________________________________________________________________________________________________
loc_reshape_3 (Reshape) (6, 150, 1, 4) 0 permute_22[0][0]
__________________________________________________________________________________________________
loc_reshape_4 (Reshape) (6, 54, 1, 4) 0 permute_23[0][0]
__________________________________________________________________________________________________
loc_reshape_5 (Reshape) (6, 24, 1, 4) 0 permute_24[0][0]
__________________________________________________________________________________________________
anchor_reshape_0 (Reshape) (6, 8664, 1, 8) 0 ssd_anchor_0[0][0]
__________________________________________________________________________________________________
anchor_reshape_1 (Reshape) (6, 2166, 1, 8) 0 ssd_anchor_1[0][0]
__________________________________________________________________________________________________
anchor_reshape_2 (Reshape) (6, 600, 1, 8) 0 ssd_anchor_2[0][0]
__________________________________________________________________________________________________
anchor_reshape_3 (Reshape) (6, 150, 1, 8) 0 ssd_anchor_3[0][0]
__________________________________________________________________________________________________
anchor_reshape_4 (Reshape) (6, 54, 1, 8) 0 ssd_anchor_4[0][0]
__________________________________________________________________________________________________
anchor_reshape_5 (Reshape) (6, 24, 1, 8) 0 ssd_anchor_5[0][0]
__________________________________________________________________________________________________
mbox_conf_softmax (Permute) (6, 11658, 1, 2) 0 mbox_conf_softmax_[0][0]
__________________________________________________________________________________________________
mbox_loc (Concatenate) (6, 11658, 1, 4) 0 loc_reshape_0[0][0]
loc_reshape_1[0][0]
loc_reshape_2[0][0]
loc_reshape_3[0][0]
loc_reshape_4[0][0]
loc_reshape_5[0][0]
__________________________________________________________________________________________________
mbox_priorbox (Concatenate) (6, 11658, 1, 8) 0 anchor_reshape_0[0][0]
anchor_reshape_1[0][0]
anchor_reshape_2[0][0]
anchor_reshape_3[0][0]
anchor_reshape_4[0][0]
anchor_reshape_5[0][0]
__________________________________________________________________________________________________
concatenate_2 (Concatenate) (6, 11658, 1, 14) 0 mbox_conf_softmax[0][0]
mbox_loc[0][0]
mbox_priorbox[0][0]
__________________________________________________________________________________________________
ssd_predictions (Reshape) (6, 11658, 14) 0 concatenate_2[0][0]
==================================================================================================
Total params: 30,747,096
Trainable params: 30,692,376
Non-trainable params: 54,720
__________________________________________________________________________________________________
2022-02-02 14:09:23,583 [INFO] __main__: Number of images in the training dataset: 2160
2022-02-02 14:09:23,583 [INFO] __main__: Number of images in the validation dataset: 239
Epoch 1/80000
2/360 [..............................] - ETA: 10:48:15 - loss: 80.5255/usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (2.858239). Check your callbacks.
% delta_t_median)
360/360 [==============================] - 461s 1s/step - loss: 22.4974
8dbd4e108ea4:47:75 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0>
8dbd4e108ea4:47:75 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
8dbd4e108ea4:47:75 [0] NCCL INFO NET/IB : No device found.
8dbd4e108ea4:47:75 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0>
8dbd4e108ea4:47:75 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda11.1
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 00/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 01/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 02/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 03/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 04/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 05/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 06/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 07/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 08/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 09/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 10/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 11/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 12/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 13/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 14/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 15/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 16/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 17/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 18/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 19/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 20/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 21/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 22/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 23/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 24/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 25/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 26/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 27/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 28/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 29/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 30/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 31/32 : 0
8dbd4e108ea4:47:75 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [1] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [2] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [3] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [4] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [5] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [6] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [7] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [8] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [9] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [10] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [11] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [12] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [13] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [14] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [15] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [16] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [17] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [18] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [19] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [20] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [21] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [22] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [23] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [24] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [25] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [26] -1/-1/-1->0->-1|-1->0->-1/-
8dbd4e108ea4:47:75 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
8dbd4e108ea4:47:75 [0] NCCL INFO comm 0x7f261438fbd0 rank 0 nranks 1 cudaDev 0 busId 1000 - Init COMPLETE
Epoch 00001: saving model to /workspace/tao-experiments/dssd_head_detection/unpruned2/weights/dssd_resnet34_epoch_001.tlt
Epoch 2/80000
360/360 [==============================] - 274s 761ms/step - loss: 19.9372
Epoch 00002: saving model to /workspace/tao-experiments/dssd_head_detection/unpruned2/weights/dssd_resnet34_epoch_002.tlt
Epoch 3/80000
360/360 [==============================] - 276s 766ms/step - loss: 19.6461
Epoch 00003: saving model to /workspace/tao-experiments/dssd_head_detection/unpruned2/weights/dssd_resnet34_epoch_003.tlt
Epoch 4/80000
53/360 [===>..........................] - ETA: 3:57 - loss: 19.4941^C
Traceback (most recent call last):
File "/usr/local/bin/dssd", line 8, in <module>
sys.exit(main())
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/entrypoint/ssd.py", line 12, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/entrypoint/entrypoint.py", line 337, in launch_job
File "/usr/lib/python3.6/subprocess.py", line 306, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python3.6/subprocess.py", line 289, in call
return p.wait(timeout=timeout)
File "/usr/lib/python3.6/subprocess.py", line 1477, in wait
(pid, sts) = self._try_wait(0)
File "/usr/lib/python3.6/subprocess.py", line 1424, in _try_wait
(pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 366, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 482, in return_func
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 362, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 274, in run_experiment
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1039, in fit
validation_steps=validation_steps)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_arrays.py", line 154, in fit_loop
outs = f(ins)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
return self._call(inputs)
File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in _call
fetched = self._callable_fn(*array_vals)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1472, in __call__
run_metadata_ptr)
KeyboardInterrupt
!tao dssd train --gpus 1 --gpu_index=$GPU_INDEX \
-e $SPECS_DIR/train1.txt \
-r $USER_EXPERIMENT_DIR/unpruned2 \
-k $KEY \
-m $USER_EXPERIMENT_DIR/unpruned2/weights/dssd_resnet34_epoch_003.tlt \
--initial_epoch 4
2022-03-06 00:10:41,896 [INFO] root: Registry: ['nvcr.io']
2022-03-06 00:10:41,976 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/dell/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
2022-03-05 21:10:49,257 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2022-03-05 21:10:49,257 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:64: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
2022-03-05 21:10:49,329 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:64: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:67: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2022-03-05 21:10:49,330 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:67: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2022-03-05 21:10:49,661 [INFO] iva.ssd.utils.spec_loader: Merging specification from /workspace/tao-experiments/dssd_head_detection/specs/train1.txt
2022-03-05 21:10:49,676 [INFO] __main__: Loading pretrained weights. This may take a while...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
2022-03-05 21:10:49,676 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
2022-03-05 21:10:49,677 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
2022-03-05 21:10:49,695 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.
2022-03-05 21:10:51,629 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.
Using DALI augmentation pipeline.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/box_coder/input_encoder.py:493: The name tf.log is deprecated. Please use tf.math.log instead.
2022-03-05 21:10:52,778 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/box_coder/input_encoder.py:493: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
2022-03-05 21:10:58,453 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
2022-03-05 21:10:58,454 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
2022-03-05 21:10:59,853 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
2022-03-05 21:11:01,908 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
2022-03-05 21:11:04,455 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
2022-03-05 21:11:05,208 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
Resume optimizer from pretrained model
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 366, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 494, in return_func
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 482, in return_func
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 362, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 148, in run_experiment
File "/usr/local/lib/python3.6/dist-packages/horovod/keras/__init__.py", line 61, in DistributedOptimizer
sparse_as_dense)
File "/usr/local/lib/python3.6/dist-packages/horovod/_keras/__init__.py", line 82, in create_distributed_optimizer
dict(_DistributedOptimizer.__dict__))
TypeError: type 'NoneType' is not an acceptable base type
2022-03-06 00:52:34,700 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.