DSSD resume error

Please provide the following information when requesting support.

dssd model does not resume

My logs:

 print("To run with multigpu, please change --gpus based on the number of available GPUs in your machine.")
> !tao dssd train --gpus 1 --gpu_index=$GPU_INDEX \
>                 -e $SPECS_DIR/train1.txt \
>                 -r $USER_EXPERIMENT_DIR/unpruned2 \
>                 -k $KEY \
>                 -m $USER_EXPERIMENT_DIR/pretrained_resnet34/pretrained_object_detection_vresnet34/resnet_34.hdf5
To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
2022-02-02 17:06:43,794 [INFO] root: Registry: ['nvcr.io']
.................................
Weights for those layers can not be loaded: ['activation_2', 'add_17', 'add_18', 'add_19', 'add_20', 'add_21', 'add_22', 'add_23', 'add_24', 'add_25', 'add_26', 'add_27', 'add_28', 'add_29', 'add_30', 'add_31', 'add_32']
STOP trainig now and check the pre-train model if this is not expected!
Initialize optimizer
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:121: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2022-02-02 14:09:22,847 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:121: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:122: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2022-02-02 14:09:22,847 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:122: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:123: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2022-02-02 14:09:22,848 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/utils/tensor_utils.py:123: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Input (InputLayer)              (6, 3, 300, 300)     0                                            
__________________________________________________________________________________________________
conv1 (Conv2D)                  (6, 64, 150, 150)    9408        Input[0][0]                      
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (6, 64, 150, 150)    256         conv1[0][0]                      
__________________________________________________________________________________________________
activation_2 (Activation)       (6, 64, 150, 150)    0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
block_1a_conv_1 (Conv2D)        (6, 64, 75, 75)      36864       activation_2[0][0]               
__________________________________________________________________________________________________
block_1a_bn_1 (BatchNormalizati (6, 64, 75, 75)      256         block_1a_conv_1[0][0]            
__________________________________________________________________________________________________
block_1a_relu_1 (Activation)    (6, 64, 75, 75)      0           block_1a_bn_1[0][0]              
__________________________________________________________________________________________________
block_1a_conv_2 (Conv2D)        (6, 64, 75, 75)      36864       block_1a_relu_1[0][0]            
__________________________________________________________________________________________________
block_1a_conv_shortcut (Conv2D) (6, 64, 75, 75)      4096        activation_2[0][0]               
__________________________________________________________________________________________________
block_1a_bn_2 (BatchNormalizati (6, 64, 75, 75)      256         block_1a_conv_2[0][0]            
__________________________________________________________________________________________________
block_1a_bn_shortcut (BatchNorm (6, 64, 75, 75)      256         block_1a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_17 (Add)                    (6, 64, 75, 75)      0           block_1a_bn_2[0][0]              
                                                                 block_1a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1a_relu (Activation)      (6, 64, 75, 75)      0           add_17[0][0]                     
__________________________________________________________________________________________________
block_1b_conv_1 (Conv2D)        (6, 64, 75, 75)      36864       block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_bn_1 (BatchNormalizati (6, 64, 75, 75)      256         block_1b_conv_1[0][0]            
__________________________________________________________________________________________________
block_1b_relu_1 (Activation)    (6, 64, 75, 75)      0           block_1b_bn_1[0][0]              
__________________________________________________________________________________________________
block_1b_conv_2 (Conv2D)        (6, 64, 75, 75)      36864       block_1b_relu_1[0][0]            
__________________________________________________________________________________________________
block_1b_conv_shortcut (Conv2D) (6, 64, 75, 75)      4096        block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_bn_2 (BatchNormalizati (6, 64, 75, 75)      256         block_1b_conv_2[0][0]            
__________________________________________________________________________________________________
block_1b_bn_shortcut (BatchNorm (6, 64, 75, 75)      256         block_1b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_18 (Add)                    (6, 64, 75, 75)      0           block_1b_bn_2[0][0]              
                                                                 block_1b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1b_relu (Activation)      (6, 64, 75, 75)      0           add_18[0][0]                     
__________________________________________________________________________________________________
block_1c_conv_1 (Conv2D)        (6, 64, 75, 75)      36864       block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_1c_bn_1 (BatchNormalizati (6, 64, 75, 75)      256         block_1c_conv_1[0][0]            
__________________________________________________________________________________________________
block_1c_relu_1 (Activation)    (6, 64, 75, 75)      0           block_1c_bn_1[0][0]              
__________________________________________________________________________________________________
block_1c_conv_2 (Conv2D)        (6, 64, 75, 75)      36864       block_1c_relu_1[0][0]            
__________________________________________________________________________________________________
block_1c_conv_shortcut (Conv2D) (6, 64, 75, 75)      4096        block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_1c_bn_2 (BatchNormalizati (6, 64, 75, 75)      256         block_1c_conv_2[0][0]            
__________________________________________________________________________________________________
block_1c_bn_shortcut (BatchNorm (6, 64, 75, 75)      256         block_1c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_19 (Add)                    (6, 64, 75, 75)      0           block_1c_bn_2[0][0]              
                                                                 block_1c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1c_relu (Activation)      (6, 64, 75, 75)      0           add_19[0][0]                     
__________________________________________________________________________________________________
block_2a_conv_1 (Conv2D)        (6, 128, 38, 38)     73728       block_1c_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_1 (BatchNormalizati (6, 128, 38, 38)     512         block_2a_conv_1[0][0]            
__________________________________________________________________________________________________
block_2a_relu_1 (Activation)    (6, 128, 38, 38)     0           block_2a_bn_1[0][0]              
__________________________________________________________________________________________________
block_2a_conv_2 (Conv2D)        (6, 128, 38, 38)     147456      block_2a_relu_1[0][0]            
__________________________________________________________________________________________________
block_2a_conv_shortcut (Conv2D) (6, 128, 38, 38)     8192        block_1c_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_2 (BatchNormalizati (6, 128, 38, 38)     512         block_2a_conv_2[0][0]            
__________________________________________________________________________________________________
block_2a_bn_shortcut (BatchNorm (6, 128, 38, 38)     512         block_2a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_20 (Add)                    (6, 128, 38, 38)     0           block_2a_bn_2[0][0]              
                                                                 block_2a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2a_relu (Activation)      (6, 128, 38, 38)     0           add_20[0][0]                     
__________________________________________________________________________________________________
block_2b_conv_1 (Conv2D)        (6, 128, 38, 38)     147456      block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_bn_1 (BatchNormalizati (6, 128, 38, 38)     512         block_2b_conv_1[0][0]            
__________________________________________________________________________________________________
block_2b_relu_1 (Activation)    (6, 128, 38, 38)     0           block_2b_bn_1[0][0]              
__________________________________________________________________________________________________
block_2b_conv_2 (Conv2D)        (6, 128, 38, 38)     147456      block_2b_relu_1[0][0]            
__________________________________________________________________________________________________
block_2b_conv_shortcut (Conv2D) (6, 128, 38, 38)     16384       block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_bn_2 (BatchNormalizati (6, 128, 38, 38)     512         block_2b_conv_2[0][0]            
__________________________________________________________________________________________________
block_2b_bn_shortcut (BatchNorm (6, 128, 38, 38)     512         block_2b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_21 (Add)                    (6, 128, 38, 38)     0           block_2b_bn_2[0][0]              
                                                                 block_2b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2b_relu (Activation)      (6, 128, 38, 38)     0           add_21[0][0]                     
__________________________________________________________________________________________________
block_2c_conv_1 (Conv2D)        (6, 128, 38, 38)     147456      block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_2c_bn_1 (BatchNormalizati (6, 128, 38, 38)     512         block_2c_conv_1[0][0]            
__________________________________________________________________________________________________
block_2c_relu_1 (Activation)    (6, 128, 38, 38)     0           block_2c_bn_1[0][0]              
__________________________________________________________________________________________________
block_2c_conv_2 (Conv2D)        (6, 128, 38, 38)     147456      block_2c_relu_1[0][0]            
__________________________________________________________________________________________________
block_2c_conv_shortcut (Conv2D) (6, 128, 38, 38)     16384       block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_2c_bn_2 (BatchNormalizati (6, 128, 38, 38)     512         block_2c_conv_2[0][0]            
__________________________________________________________________________________________________
block_2c_bn_shortcut (BatchNorm (6, 128, 38, 38)     512         block_2c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_22 (Add)                    (6, 128, 38, 38)     0           block_2c_bn_2[0][0]              
                                                                 block_2c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2c_relu (Activation)      (6, 128, 38, 38)     0           add_22[0][0]                     
__________________________________________________________________________________________________
block_2d_conv_1 (Conv2D)        (6, 128, 38, 38)     147456      block_2c_relu[0][0]              
__________________________________________________________________________________________________
block_2d_bn_1 (BatchNormalizati (6, 128, 38, 38)     512         block_2d_conv_1[0][0]            
__________________________________________________________________________________________________
block_2d_relu_1 (Activation)    (6, 128, 38, 38)     0           block_2d_bn_1[0][0]              
__________________________________________________________________________________________________
block_2d_conv_2 (Conv2D)        (6, 128, 38, 38)     147456      block_2d_relu_1[0][0]            
__________________________________________________________________________________________________
block_2d_conv_shortcut (Conv2D) (6, 128, 38, 38)     16384       block_2c_relu[0][0]              
__________________________________________________________________________________________________
block_2d_bn_2 (BatchNormalizati (6, 128, 38, 38)     512         block_2d_conv_2[0][0]            
__________________________________________________________________________________________________
block_2d_bn_shortcut (BatchNorm (6, 128, 38, 38)     512         block_2d_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_23 (Add)                    (6, 128, 38, 38)     0           block_2d_bn_2[0][0]              
                                                                 block_2d_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2d_relu (Activation)      (6, 128, 38, 38)     0           add_23[0][0]                     
__________________________________________________________________________________________________
block_3a_conv_1 (Conv2D)        (6, 256, 19, 19)     294912      block_2d_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_1 (BatchNormalizati (6, 256, 19, 19)     1024        block_3a_conv_1[0][0]            
__________________________________________________________________________________________________
block_3a_relu_1 (Activation)    (6, 256, 19, 19)     0           block_3a_bn_1[0][0]              
__________________________________________________________________________________________________
block_3a_conv_2 (Conv2D)        (6, 256, 19, 19)     589824      block_3a_relu_1[0][0]            
__________________________________________________________________________________________________
block_3a_conv_shortcut (Conv2D) (6, 256, 19, 19)     32768       block_2d_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_2 (BatchNormalizati (6, 256, 19, 19)     1024        block_3a_conv_2[0][0]            
__________________________________________________________________________________________________
block_3a_bn_shortcut (BatchNorm (6, 256, 19, 19)     1024        block_3a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_24 (Add)                    (6, 256, 19, 19)     0           block_3a_bn_2[0][0]              
                                                                 block_3a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3a_relu (Activation)      (6, 256, 19, 19)     0           add_24[0][0]                     
__________________________________________________________________________________________________
block_3b_conv_1 (Conv2D)        (6, 256, 19, 19)     589824      block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_bn_1 (BatchNormalizati (6, 256, 19, 19)     1024        block_3b_conv_1[0][0]            
__________________________________________________________________________________________________
block_3b_relu_1 (Activation)    (6, 256, 19, 19)     0           block_3b_bn_1[0][0]              
__________________________________________________________________________________________________
block_3b_conv_2 (Conv2D)        (6, 256, 19, 19)     589824      block_3b_relu_1[0][0]            
__________________________________________________________________________________________________
block_3b_conv_shortcut (Conv2D) (6, 256, 19, 19)     65536       block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_bn_2 (BatchNormalizati (6, 256, 19, 19)     1024        block_3b_conv_2[0][0]            
__________________________________________________________________________________________________
block_3b_bn_shortcut (BatchNorm (6, 256, 19, 19)     1024        block_3b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_25 (Add)                    (6, 256, 19, 19)     0           block_3b_bn_2[0][0]              
                                                                 block_3b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3b_relu (Activation)      (6, 256, 19, 19)     0           add_25[0][0]                     
__________________________________________________________________________________________________
block_3c_conv_1 (Conv2D)        (6, 256, 19, 19)     589824      block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_3c_bn_1 (BatchNormalizati (6, 256, 19, 19)     1024        block_3c_conv_1[0][0]            
__________________________________________________________________________________________________
block_3c_relu_1 (Activation)    (6, 256, 19, 19)     0           block_3c_bn_1[0][0]              
__________________________________________________________________________________________________
block_3c_conv_2 (Conv2D)        (6, 256, 19, 19)     589824      block_3c_relu_1[0][0]            
__________________________________________________________________________________________________
block_3c_conv_shortcut (Conv2D) (6, 256, 19, 19)     65536       block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_3c_bn_2 (BatchNormalizati (6, 256, 19, 19)     1024        block_3c_conv_2[0][0]            
__________________________________________________________________________________________________
block_3c_bn_shortcut (BatchNorm (6, 256, 19, 19)     1024        block_3c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_26 (Add)                    (6, 256, 19, 19)     0           block_3c_bn_2[0][0]              
                                                                 block_3c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3c_relu (Activation)      (6, 256, 19, 19)     0           add_26[0][0]                     
__________________________________________________________________________________________________
block_3d_conv_1 (Conv2D)        (6, 256, 19, 19)     589824      block_3c_relu[0][0]              
__________________________________________________________________________________________________
block_3d_bn_1 (BatchNormalizati (6, 256, 19, 19)     1024        block_3d_conv_1[0][0]            
__________________________________________________________________________________________________
block_3d_relu_1 (Activation)    (6, 256, 19, 19)     0           block_3d_bn_1[0][0]              
__________________________________________________________________________________________________
block_3d_conv_2 (Conv2D)        (6, 256, 19, 19)     589824      block_3d_relu_1[0][0]            
__________________________________________________________________________________________________
block_3d_conv_shortcut (Conv2D) (6, 256, 19, 19)     65536       block_3c_relu[0][0]              
__________________________________________________________________________________________________
block_3d_bn_2 (BatchNormalizati (6, 256, 19, 19)     1024        block_3d_conv_2[0][0]            
__________________________________________________________________________________________________
block_3d_bn_shortcut (BatchNorm (6, 256, 19, 19)     1024        block_3d_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_27 (Add)                    (6, 256, 19, 19)     0           block_3d_bn_2[0][0]              
                                                                 block_3d_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3d_relu (Activation)      (6, 256, 19, 19)     0           add_27[0][0]                     
__________________________________________________________________________________________________
block_3e_conv_1 (Conv2D)        (6, 256, 19, 19)     589824      block_3d_relu[0][0]              
__________________________________________________________________________________________________
block_3e_bn_1 (BatchNormalizati (6, 256, 19, 19)     1024        block_3e_conv_1[0][0]            
__________________________________________________________________________________________________
block_3e_relu_1 (Activation)    (6, 256, 19, 19)     0           block_3e_bn_1[0][0]              
__________________________________________________________________________________________________
block_3e_conv_2 (Conv2D)        (6, 256, 19, 19)     589824      block_3e_relu_1[0][0]            
__________________________________________________________________________________________________
block_3e_conv_shortcut (Conv2D) (6, 256, 19, 19)     65536       block_3d_relu[0][0]              
__________________________________________________________________________________________________
block_3e_bn_2 (BatchNormalizati (6, 256, 19, 19)     1024        block_3e_conv_2[0][0]            
__________________________________________________________________________________________________
block_3e_bn_shortcut (BatchNorm (6, 256, 19, 19)     1024        block_3e_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_28 (Add)                    (6, 256, 19, 19)     0           block_3e_bn_2[0][0]              
                                                                 block_3e_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3e_relu (Activation)      (6, 256, 19, 19)     0           add_28[0][0]                     
__________________________________________________________________________________________________
block_3f_conv_1 (Conv2D)        (6, 256, 19, 19)     589824      block_3e_relu[0][0]              
__________________________________________________________________________________________________
block_3f_bn_1 (BatchNormalizati (6, 256, 19, 19)     1024        block_3f_conv_1[0][0]            
__________________________________________________________________________________________________
block_3f_relu_1 (Activation)    (6, 256, 19, 19)     0           block_3f_bn_1[0][0]              
__________________________________________________________________________________________________
block_3f_conv_2 (Conv2D)        (6, 256, 19, 19)     589824      block_3f_relu_1[0][0]            
__________________________________________________________________________________________________
block_3f_conv_shortcut (Conv2D) (6, 256, 19, 19)     65536       block_3e_relu[0][0]              
__________________________________________________________________________________________________
block_3f_bn_2 (BatchNormalizati (6, 256, 19, 19)     1024        block_3f_conv_2[0][0]            
__________________________________________________________________________________________________
block_3f_bn_shortcut (BatchNorm (6, 256, 19, 19)     1024        block_3f_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_29 (Add)                    (6, 256, 19, 19)     0           block_3f_bn_2[0][0]              
                                                                 block_3f_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3f_relu (Activation)      (6, 256, 19, 19)     0           add_29[0][0]                     
__________________________________________________________________________________________________
block_4a_conv_1 (Conv2D)        (6, 512, 19, 19)     1179648     block_3f_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_1 (BatchNormalizati (6, 512, 19, 19)     2048        block_4a_conv_1[0][0]            
__________________________________________________________________________________________________
block_4a_relu_1 (Activation)    (6, 512, 19, 19)     0           block_4a_bn_1[0][0]              
__________________________________________________________________________________________________
block_4a_conv_2 (Conv2D)        (6, 512, 19, 19)     2359296     block_4a_relu_1[0][0]            
__________________________________________________________________________________________________
block_4a_conv_shortcut (Conv2D) (6, 512, 19, 19)     131072      block_3f_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_2 (BatchNormalizati (6, 512, 19, 19)     2048        block_4a_conv_2[0][0]            
__________________________________________________________________________________________________
block_4a_bn_shortcut (BatchNorm (6, 512, 19, 19)     2048        block_4a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_30 (Add)                    (6, 512, 19, 19)     0           block_4a_bn_2[0][0]              
                                                                 block_4a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4a_relu (Activation)      (6, 512, 19, 19)     0           add_30[0][0]                     
__________________________________________________________________________________________________
block_4b_conv_1 (Conv2D)        (6, 512, 19, 19)     2359296     block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_bn_1 (BatchNormalizati (6, 512, 19, 19)     2048        block_4b_conv_1[0][0]            
__________________________________________________________________________________________________
block_4b_relu_1 (Activation)    (6, 512, 19, 19)     0           block_4b_bn_1[0][0]              
__________________________________________________________________________________________________
block_4b_conv_2 (Conv2D)        (6, 512, 19, 19)     2359296     block_4b_relu_1[0][0]            
__________________________________________________________________________________________________
block_4b_conv_shortcut (Conv2D) (6, 512, 19, 19)     262144      block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_bn_2 (BatchNormalizati (6, 512, 19, 19)     2048        block_4b_conv_2[0][0]            
__________________________________________________________________________________________________
block_4b_bn_shortcut (BatchNorm (6, 512, 19, 19)     2048        block_4b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_31 (Add)                    (6, 512, 19, 19)     0           block_4b_bn_2[0][0]              
                                                                 block_4b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4b_relu (Activation)      (6, 512, 19, 19)     0           add_31[0][0]                     
__________________________________________________________________________________________________
block_4c_conv_1 (Conv2D)        (6, 512, 19, 19)     2359296     block_4b_relu[0][0]              
__________________________________________________________________________________________________
block_4c_bn_1 (BatchNormalizati (6, 512, 19, 19)     2048        block_4c_conv_1[0][0]            
__________________________________________________________________________________________________
block_4c_relu_1 (Activation)    (6, 512, 19, 19)     0           block_4c_bn_1[0][0]              
__________________________________________________________________________________________________
block_4c_conv_2 (Conv2D)        (6, 512, 19, 19)     2359296     block_4c_relu_1[0][0]            
__________________________________________________________________________________________________
block_4c_conv_shortcut (Conv2D) (6, 512, 19, 19)     262144      block_4b_relu[0][0]              
__________________________________________________________________________________________________
block_4c_bn_2 (BatchNormalizati (6, 512, 19, 19)     2048        block_4c_conv_2[0][0]            
__________________________________________________________________________________________________
block_4c_bn_shortcut (BatchNorm (6, 512, 19, 19)     2048        block_4c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_32 (Add)                    (6, 512, 19, 19)     0           block_4c_bn_2[0][0]              
                                                                 block_4c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4c_relu (Activation)      (6, 512, 19, 19)     0           add_32[0][0]                     
__________________________________________________________________________________________________
ssd_expand_block_0_conv_0 (Conv (6, 256, 19, 19)     131328      block_4c_relu[0][0]              
__________________________________________________________________________________________________
ssd_expand_block_0_relu_0 (ReLU (6, 256, 19, 19)     0           ssd_expand_block_0_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_0_conv_1 (Conv (6, 256, 19, 19)     589824      ssd_expand_block_0_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_0_bn_1 (BatchN (6, 256, 19, 19)     1024        ssd_expand_block_0_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_0_relu_1 (ReLU (6, 256, 19, 19)     0           ssd_expand_block_0_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_1_conv_0 (Conv (6, 128, 19, 19)     32896       ssd_expand_block_0_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_relu_0 (ReLU (6, 128, 19, 19)     0           ssd_expand_block_1_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_conv_1 (Conv (6, 256, 10, 10)     294912      ssd_expand_block_1_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_bn_1 (BatchN (6, 256, 10, 10)     1024        ssd_expand_block_1_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_relu_1 (ReLU (6, 256, 10, 10)     0           ssd_expand_block_1_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_2_conv_0 (Conv (6, 64, 10, 10)      16448       ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_relu_0 (ReLU (6, 64, 10, 10)      0           ssd_expand_block_2_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_conv_1 (Conv (6, 128, 5, 5)       73728       ssd_expand_block_2_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_bn_1 (BatchN (6, 128, 5, 5)       512         ssd_expand_block_2_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_relu_1 (ReLU (6, 128, 5, 5)       0           ssd_expand_block_2_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_3_conv_0 (Conv (6, 64, 5, 5)        8256        ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_relu_0 (ReLU (6, 64, 5, 5)        0           ssd_expand_block_3_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_conv_1 (Conv (6, 128, 3, 3)       73728       ssd_expand_block_3_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_bn_1 (BatchN (6, 128, 3, 3)       512         ssd_expand_block_3_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_relu_1 (ReLU (6, 128, 3, 3)       0           ssd_expand_block_3_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_4_conv_0 (Conv (6, 64, 3, 3)        8256        ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_relu_0 (ReLU (6, 64, 3, 3)        0           ssd_expand_block_4_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_conv_1 (Conv (6, 128, 2, 2)       73728       ssd_expand_block_4_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_bn_1 (BatchN (6, 128, 2, 2)       512         ssd_expand_block_4_conv_1[0][0]  
__________________________________________________________________________________________________
dssd_large_conv_0_block_0 (Conv (6, 128, 3, 3)       147456      ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_relu_1 (ReLU (6, 128, 2, 2)       0           ssd_expand_block_4_bn_1[0][0]    
__________________________________________________________________________________________________
dssd_large_bn_0_block_0 (BatchN (6, 128, 3, 3)       512         dssd_large_conv_0_block_0[0][0]  
__________________________________________________________________________________________________
dssd_small_deconv_block_0 (Conv (6, 128, 4, 4)       65536       ssd_expand_block_4_relu_1[0][0]  
__________________________________________________________________________________________________
dssd_large_relu_0_block_0 (ReLU (6, 128, 3, 3)       0           dssd_large_bn_0_block_0[0][0]    
__________________________________________________________________________________________________
dssd_pad_0 (ZeroPadding2D)      (6, 128, 4, 4)       0           dssd_small_deconv_block_0[0][0]  
__________________________________________________________________________________________________
dssd_large_conv_1_block_0 (Conv (6, 128, 3, 3)       147456      dssd_large_relu_0_block_0[0][0]  
__________________________________________________________________________________________________
dssd_small_conv_block_0 (Conv2D (6, 128, 3, 3)       65536       dssd_pad_0[0][0]                 
__________________________________________________________________________________________________
dssd_large_bn_1_block_0 (BatchN (6, 128, 3, 3)       512         dssd_large_conv_1_block_0[0][0]  
__________________________________________________________________________________________________
dssd_small_bn_block_0 (BatchNor (6, 128, 3, 3)       512         dssd_small_conv_block_0[0][0]    
__________________________________________________________________________________________________
dssd_mul_block_0 (Multiply)     (6, 128, 3, 3)       0           dssd_large_bn_1_block_0[0][0]    
                                                                 dssd_small_bn_block_0[0][0]      
__________________________________________________________________________________________________
dssd_large_conv_0_block_1 (Conv (6, 128, 5, 5)       147456      ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
dssd_relu_block_0 (ReLU)        (6, 128, 3, 3)       0           dssd_mul_block_0[0][0]           
__________________________________________________________________________________________________
dssd_large_bn_0_block_1 (BatchN (6, 128, 5, 5)       512         dssd_large_conv_0_block_1[0][0]  
__________________________________________________________________________________________________
dssd_small_deconv_block_1 (Conv (6, 128, 6, 6)       65536       dssd_relu_block_0[0][0]          
__________________________________________________________________________________________________
dssd_large_relu_0_block_1 (ReLU (6, 128, 5, 5)       0           dssd_large_bn_0_block_1[0][0]    
__________________________________________________________________________________________________
dssd_pad_1 (ZeroPadding2D)      (6, 128, 6, 6)       0           dssd_small_deconv_block_1[0][0]  
__________________________________________________________________________________________________
dssd_large_conv_1_block_1 (Conv (6, 128, 5, 5)       147456      dssd_large_relu_0_block_1[0][0]  
__________________________________________________________________________________________________
dssd_small_conv_block_1 (Conv2D (6, 128, 5, 5)       65536       dssd_pad_1[0][0]                 
__________________________________________________________________________________________________
dssd_large_bn_1_block_1 (BatchN (6, 128, 5, 5)       512         dssd_large_conv_1_block_1[0][0]  
__________________________________________________________________________________________________
dssd_small_bn_block_1 (BatchNor (6, 128, 5, 5)       512         dssd_small_conv_block_1[0][0]    
__________________________________________________________________________________________________
dssd_mul_block_1 (Multiply)     (6, 128, 5, 5)       0           dssd_large_bn_1_block_1[0][0]    
                                                                 dssd_small_bn_block_1[0][0]      
__________________________________________________________________________________________________
dssd_large_conv_0_block_2 (Conv (6, 256, 10, 10)     589824      ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
dssd_relu_block_1 (ReLU)        (6, 128, 5, 5)       0           dssd_mul_block_1[0][0]           
__________________________________________________________________________________________________
dssd_large_bn_0_block_2 (BatchN (6, 256, 10, 10)     1024        dssd_large_conv_0_block_2[0][0]  
__________________________________________________________________________________________________
dssd_small_deconv_block_2 (Conv (6, 256, 10, 10)     131072      dssd_relu_block_1[0][0]          
__________________________________________________________________________________________________
dssd_large_relu_0_block_2 (ReLU (6, 256, 10, 10)     0           dssd_large_bn_0_block_2[0][0]    
__________________________________________________________________________________________________
dssd_pad_2 (ZeroPadding2D)      (6, 256, 12, 12)     0           dssd_small_deconv_block_2[0][0]  
__________________________________________________________________________________________________
dssd_large_conv_1_block_2 (Conv (6, 256, 10, 10)     589824      dssd_large_relu_0_block_2[0][0]  
__________________________________________________________________________________________________
dssd_small_conv_block_2 (Conv2D (6, 256, 10, 10)     589824      dssd_pad_2[0][0]                 
__________________________________________________________________________________________________
dssd_large_bn_1_block_2 (BatchN (6, 256, 10, 10)     1024        dssd_large_conv_1_block_2[0][0]  
__________________________________________________________________________________________________
dssd_small_bn_block_2 (BatchNor (6, 256, 10, 10)     1024        dssd_small_conv_block_2[0][0]    
__________________________________________________________________________________________________
dssd_mul_block_2 (Multiply)     (6, 256, 10, 10)     0           dssd_large_bn_1_block_2[0][0]    
                                                                 dssd_small_bn_block_2[0][0]      
__________________________________________________________________________________________________
dssd_large_conv_0_block_3 (Conv (6, 256, 19, 19)     589824      ssd_expand_block_0_relu_1[0][0]  
__________________________________________________________________________________________________
dssd_relu_block_2 (ReLU)        (6, 256, 10, 10)     0           dssd_mul_block_2[0][0]           
__________________________________________________________________________________________________
dssd_large_bn_0_block_3 (BatchN (6, 256, 19, 19)     1024        dssd_large_conv_0_block_3[0][0]  
__________________________________________________________________________________________________
dssd_small_deconv_block_3 (Conv (6, 256, 20, 20)     262144      dssd_relu_block_2[0][0]          
__________________________________________________________________________________________________
dssd_large_relu_0_block_3 (ReLU (6, 256, 19, 19)     0           dssd_large_bn_0_block_3[0][0]    
__________________________________________________________________________________________________
dssd_pad_3 (ZeroPadding2D)      (6, 256, 20, 20)     0           dssd_small_deconv_block_3[0][0]  
__________________________________________________________________________________________________
dssd_large_conv_1_block_3 (Conv (6, 256, 19, 19)     589824      dssd_large_relu_0_block_3[0][0]  
__________________________________________________________________________________________________
dssd_small_conv_block_3 (Conv2D (6, 256, 19, 19)     262144      dssd_pad_3[0][0]                 
__________________________________________________________________________________________________
dssd_large_bn_1_block_3 (BatchN (6, 256, 19, 19)     1024        dssd_large_conv_1_block_3[0][0]  
__________________________________________________________________________________________________
dssd_small_bn_block_3 (BatchNor (6, 256, 19, 19)     1024        dssd_small_conv_block_3[0][0]    
__________________________________________________________________________________________________
dssd_mul_block_3 (Multiply)     (6, 256, 19, 19)     0           dssd_large_bn_1_block_3[0][0]    
                                                                 dssd_small_bn_block_3[0][0]      
__________________________________________________________________________________________________
dssd_large_conv_0_block_4 (Conv (6, 128, 38, 38)     147456      block_2d_relu[0][0]              
__________________________________________________________________________________________________
dssd_relu_block_3 (ReLU)        (6, 256, 19, 19)     0           dssd_mul_block_3[0][0]           
__________________________________________________________________________________________________
dssd_large_bn_0_block_4 (BatchN (6, 128, 38, 38)     512         dssd_large_conv_0_block_4[0][0]  
__________________________________________________________________________________________________
dssd_small_deconv_block_4 (Conv (6, 128, 38, 38)     131072      dssd_relu_block_3[0][0]          
__________________________________________________________________________________________________
dssd_large_relu_0_block_4 (ReLU (6, 128, 38, 38)     0           dssd_large_bn_0_block_4[0][0]    
__________________________________________________________________________________________________
dssd_pad_4 (ZeroPadding2D)      (6, 128, 40, 40)     0           dssd_small_deconv_block_4[0][0]  
__________________________________________________________________________________________________
dssd_large_conv_1_block_4 (Conv (6, 128, 38, 38)     147456      dssd_large_relu_0_block_4[0][0]  
__________________________________________________________________________________________________
dssd_small_conv_block_4 (Conv2D (6, 128, 38, 38)     147456      dssd_pad_4[0][0]                 
__________________________________________________________________________________________________
dssd_large_bn_1_block_4 (BatchN (6, 128, 38, 38)     512         dssd_large_conv_1_block_4[0][0]  
__________________________________________________________________________________________________
dssd_small_bn_block_4 (BatchNor (6, 128, 38, 38)     512         dssd_small_conv_block_4[0][0]    
__________________________________________________________________________________________________
dssd_mul_block_4 (Multiply)     (6, 128, 38, 38)     0           dssd_large_bn_1_block_4[0][0]    
                                                                 dssd_small_bn_block_4[0][0]      
__________________________________________________________________________________________________
dssd_relu_block_4 (ReLU)        (6, 128, 38, 38)     0           dssd_mul_block_4[0][0]           
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_0 (Conv2 (6, 128, 38, 38)     16384       dssd_relu_block_4[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_1 (Conv2 (6, 128, 19, 19)     32768       dssd_relu_block_3[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_2 (Conv2 (6, 128, 10, 10)     32768       dssd_relu_block_2[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_3 (Conv2 (6, 128, 5, 5)       16384       dssd_relu_block_1[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_4 (Conv2 (6, 128, 3, 3)       16384       dssd_relu_block_0[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_0_block_5 (Conv2 (6, 128, 2, 2)       16384       ssd_expand_block_4_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_0 (BatchNo (6, 128, 38, 38)     512         ssd_mpred_conv_0_block_0[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_1 (BatchNo (6, 128, 19, 19)     512         ssd_mpred_conv_0_block_1[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_2 (BatchNo (6, 128, 10, 10)     512         ssd_mpred_conv_0_block_2[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_3 (BatchNo (6, 128, 5, 5)       512         ssd_mpred_conv_0_block_3[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_4 (BatchNo (6, 128, 3, 3)       512         ssd_mpred_conv_0_block_4[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_0_block_5 (BatchNo (6, 128, 2, 2)       512         ssd_mpred_conv_0_block_5[0][0]   
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_0 (ReLU) (6, 128, 38, 38)     0           ssd_mpred_bn_0_block_0[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_1 (ReLU) (6, 128, 19, 19)     0           ssd_mpred_bn_0_block_1[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_2 (ReLU) (6, 128, 10, 10)     0           ssd_mpred_bn_0_block_2[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_3 (ReLU) (6, 128, 5, 5)       0           ssd_mpred_bn_0_block_3[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_4 (ReLU) (6, 128, 3, 3)       0           ssd_mpred_bn_0_block_4[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_0_block_5 (ReLU) (6, 128, 2, 2)       0           ssd_mpred_bn_0_block_5[0][0]     
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_0 (Conv2 (6, 128, 38, 38)     16384       ssd_mpred_relu_0_block_0[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_1 (Conv2 (6, 128, 19, 19)     16384       ssd_mpred_relu_0_block_1[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_2 (Conv2 (6, 128, 10, 10)     16384       ssd_mpred_relu_0_block_2[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_3 (Conv2 (6, 128, 5, 5)       16384       ssd_mpred_relu_0_block_3[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_4 (Conv2 (6, 128, 3, 3)       16384       ssd_mpred_relu_0_block_4[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_1_block_5 (Conv2 (6, 128, 2, 2)       16384       ssd_mpred_relu_0_block_5[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_0 (BatchNo (6, 128, 38, 38)     512         ssd_mpred_conv_1_block_0[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_1 (BatchNo (6, 128, 19, 19)     512         ssd_mpred_conv_1_block_1[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_2 (BatchNo (6, 128, 10, 10)     512         ssd_mpred_conv_1_block_2[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_3 (BatchNo (6, 128, 5, 5)       512         ssd_mpred_conv_1_block_3[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_4 (BatchNo (6, 128, 3, 3)       512         ssd_mpred_conv_1_block_4[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_1_block_5 (BatchNo (6, 128, 2, 2)       512         ssd_mpred_conv_1_block_5[0][0]   
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_0 (ReLU) (6, 128, 38, 38)     0           ssd_mpred_bn_1_block_0[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_1 (ReLU) (6, 128, 19, 19)     0           ssd_mpred_bn_1_block_1[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_2 (ReLU) (6, 128, 10, 10)     0           ssd_mpred_bn_1_block_2[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_3 (ReLU) (6, 128, 5, 5)       0           ssd_mpred_bn_1_block_3[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_4 (ReLU) (6, 128, 3, 3)       0           ssd_mpred_bn_1_block_4[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_1_block_5 (ReLU) (6, 128, 2, 2)       0           ssd_mpred_bn_1_block_5[0][0]     
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_0 (Conv2 (6, 512, 38, 38)     65536       ssd_mpred_relu_1_block_0[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_0 (Conv2 (6, 512, 38, 38)     65536       dssd_relu_block_4[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_1 (Conv2 (6, 512, 19, 19)     65536       ssd_mpred_relu_1_block_1[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_1 (Conv2 (6, 512, 19, 19)     131072      dssd_relu_block_3[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_2 (Conv2 (6, 512, 10, 10)     65536       ssd_mpred_relu_1_block_2[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_2 (Conv2 (6, 512, 10, 10)     131072      dssd_relu_block_2[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_3 (Conv2 (6, 512, 5, 5)       65536       ssd_mpred_relu_1_block_3[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_3 (Conv2 (6, 512, 5, 5)       65536       dssd_relu_block_1[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_4 (Conv2 (6, 512, 3, 3)       65536       ssd_mpred_relu_1_block_4[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_4 (Conv2 (6, 512, 3, 3)       65536       dssd_relu_block_0[0][0]          
__________________________________________________________________________________________________
ssd_mpred_conv_2_block_5 (Conv2 (6, 512, 2, 2)       65536       ssd_mpred_relu_1_block_5[0][0]   
__________________________________________________________________________________________________
ssd_mpred_conv_3_block_5 (Conv2 (6, 512, 2, 2)       65536       ssd_expand_block_4_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_0 (BatchNo (6, 512, 38, 38)     2048        ssd_mpred_conv_2_block_0[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_0 (BatchNo (6, 512, 38, 38)     2048        ssd_mpred_conv_3_block_0[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_1 (BatchNo (6, 512, 19, 19)     2048        ssd_mpred_conv_2_block_1[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_1 (BatchNo (6, 512, 19, 19)     2048        ssd_mpred_conv_3_block_1[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_2 (BatchNo (6, 512, 10, 10)     2048        ssd_mpred_conv_2_block_2[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_2 (BatchNo (6, 512, 10, 10)     2048        ssd_mpred_conv_3_block_2[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_3 (BatchNo (6, 512, 5, 5)       2048        ssd_mpred_conv_2_block_3[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_3 (BatchNo (6, 512, 5, 5)       2048        ssd_mpred_conv_3_block_3[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_4 (BatchNo (6, 512, 3, 3)       2048        ssd_mpred_conv_2_block_4[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_4 (BatchNo (6, 512, 3, 3)       2048        ssd_mpred_conv_3_block_4[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_2_block_5 (BatchNo (6, 512, 2, 2)       2048        ssd_mpred_conv_2_block_5[0][0]   
__________________________________________________________________________________________________
ssd_mpred_bn_3_block_5 (BatchNo (6, 512, 2, 2)       2048        ssd_mpred_conv_3_block_5[0][0]   
__________________________________________________________________________________________________
ssd_mpred_add_block_0 (Add)     (6, 512, 38, 38)     0           ssd_mpred_bn_2_block_0[0][0]     
                                                                 ssd_mpred_bn_3_block_0[0][0]     
__________________________________________________________________________________________________
ssd_mpred_add_block_1 (Add)     (6, 512, 19, 19)     0           ssd_mpred_bn_2_block_1[0][0]     
                                                                 ssd_mpred_bn_3_block_1[0][0]     
__________________________________________________________________________________________________
ssd_mpred_add_block_2 (Add)     (6, 512, 10, 10)     0           ssd_mpred_bn_2_block_2[0][0]     
                                                                 ssd_mpred_bn_3_block_2[0][0]     
__________________________________________________________________________________________________
ssd_mpred_add_block_3 (Add)     (6, 512, 5, 5)       0           ssd_mpred_bn_2_block_3[0][0]     
                                                                 ssd_mpred_bn_3_block_3[0][0]     
__________________________________________________________________________________________________
ssd_mpred_add_block_4 (Add)     (6, 512, 3, 3)       0           ssd_mpred_bn_2_block_4[0][0]     
                                                                 ssd_mpred_bn_3_block_4[0][0]     
__________________________________________________________________________________________________
ssd_mpred_add_block_5 (Add)     (6, 512, 2, 2)       0           ssd_mpred_bn_2_block_5[0][0]     
                                                                 ssd_mpred_bn_3_block_5[0][0]     
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_0 (ReLU) (6, 512, 38, 38)     0           ssd_mpred_add_block_0[0][0]      
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_1 (ReLU) (6, 512, 19, 19)     0           ssd_mpred_add_block_1[0][0]      
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_2 (ReLU) (6, 512, 10, 10)     0           ssd_mpred_add_block_2[0][0]      
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_3 (ReLU) (6, 512, 5, 5)       0           ssd_mpred_add_block_3[0][0]      
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_4 (ReLU) (6, 512, 3, 3)       0           ssd_mpred_add_block_4[0][0]      
__________________________________________________________________________________________________
ssd_mpred_relu_3_block_5 (ReLU) (6, 512, 2, 2)       0           ssd_mpred_add_block_5[0][0]      
__________________________________________________________________________________________________
ssd_conf_0 (Conv2D)             (6, 12, 38, 38)      55308       ssd_mpred_relu_3_block_0[0][0]   
__________________________________________________________________________________________________
ssd_conf_1 (Conv2D)             (6, 12, 19, 19)      55308       ssd_mpred_relu_3_block_1[0][0]   
__________________________________________________________________________________________________
ssd_conf_2 (Conv2D)             (6, 12, 10, 10)      55308       ssd_mpred_relu_3_block_2[0][0]   
__________________________________________________________________________________________________
ssd_conf_3 (Conv2D)             (6, 12, 5, 5)        55308       ssd_mpred_relu_3_block_3[0][0]   
__________________________________________________________________________________________________
ssd_conf_4 (Conv2D)             (6, 12, 3, 3)        55308       ssd_mpred_relu_3_block_4[0][0]   
__________________________________________________________________________________________________
ssd_conf_5 (Conv2D)             (6, 12, 2, 2)        55308       ssd_mpred_relu_3_block_5[0][0]   
__________________________________________________________________________________________________
permute_13 (Permute)            (6, 38, 38, 12)      0           ssd_conf_0[0][0]                 
__________________________________________________________________________________________________
permute_14 (Permute)            (6, 19, 19, 12)      0           ssd_conf_1[0][0]                 
__________________________________________________________________________________________________
permute_15 (Permute)            (6, 10, 10, 12)      0           ssd_conf_2[0][0]                 
__________________________________________________________________________________________________
permute_16 (Permute)            (6, 5, 5, 12)        0           ssd_conf_3[0][0]                 
__________________________________________________________________________________________________
permute_17 (Permute)            (6, 3, 3, 12)        0           ssd_conf_4[0][0]                 
__________________________________________________________________________________________________
permute_18 (Permute)            (6, 2, 2, 12)        0           ssd_conf_5[0][0]                 
__________________________________________________________________________________________________
conf_reshape_0 (Reshape)        (6, 8664, 1, 2)      0           permute_13[0][0]                 
__________________________________________________________________________________________________
conf_reshape_1 (Reshape)        (6, 2166, 1, 2)      0           permute_14[0][0]                 
__________________________________________________________________________________________________
conf_reshape_2 (Reshape)        (6, 600, 1, 2)       0           permute_15[0][0]                 
__________________________________________________________________________________________________
conf_reshape_3 (Reshape)        (6, 150, 1, 2)       0           permute_16[0][0]                 
__________________________________________________________________________________________________
conf_reshape_4 (Reshape)        (6, 54, 1, 2)        0           permute_17[0][0]                 
__________________________________________________________________________________________________
conf_reshape_5 (Reshape)        (6, 24, 1, 2)        0           permute_18[0][0]                 
__________________________________________________________________________________________________
mbox_conf (Concatenate)         (6, 11658, 1, 2)     0           conf_reshape_0[0][0]             
                                                                 conf_reshape_1[0][0]             
                                                                 conf_reshape_2[0][0]             
                                                                 conf_reshape_3[0][0]             
                                                                 conf_reshape_4[0][0]             
                                                                 conf_reshape_5[0][0]             
__________________________________________________________________________________________________
ssd_loc_0 (Conv2D)              (6, 24, 38, 38)      110616      ssd_mpred_relu_3_block_0[0][0]   
__________________________________________________________________________________________________
ssd_loc_1 (Conv2D)              (6, 24, 19, 19)      110616      ssd_mpred_relu_3_block_1[0][0]   
__________________________________________________________________________________________________
ssd_loc_2 (Conv2D)              (6, 24, 10, 10)      110616      ssd_mpred_relu_3_block_2[0][0]   
__________________________________________________________________________________________________
ssd_loc_3 (Conv2D)              (6, 24, 5, 5)        110616      ssd_mpred_relu_3_block_3[0][0]   
__________________________________________________________________________________________________
ssd_loc_4 (Conv2D)              (6, 24, 3, 3)        110616      ssd_mpred_relu_3_block_4[0][0]   
__________________________________________________________________________________________________
ssd_loc_5 (Conv2D)              (6, 24, 2, 2)        110616      ssd_mpred_relu_3_block_5[0][0]   
__________________________________________________________________________________________________
before_softmax_permute (Permute (6, 2, 1, 11658)     0           mbox_conf[0][0]                  
__________________________________________________________________________________________________
permute_19 (Permute)            (6, 38, 38, 24)      0           ssd_loc_0[0][0]                  
__________________________________________________________________________________________________
permute_20 (Permute)            (6, 19, 19, 24)      0           ssd_loc_1[0][0]                  
__________________________________________________________________________________________________
permute_21 (Permute)            (6, 10, 10, 24)      0           ssd_loc_2[0][0]                  
__________________________________________________________________________________________________
permute_22 (Permute)            (6, 5, 5, 24)        0           ssd_loc_3[0][0]                  
__________________________________________________________________________________________________
permute_23 (Permute)            (6, 3, 3, 24)        0           ssd_loc_4[0][0]                  
__________________________________________________________________________________________________
permute_24 (Permute)            (6, 2, 2, 24)        0           ssd_loc_5[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_0 (AnchorBoxes)      (6, 1444, 6, 8)      0           ssd_loc_0[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_1 (AnchorBoxes)      (6, 361, 6, 8)       0           ssd_loc_1[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_2 (AnchorBoxes)      (6, 100, 6, 8)       0           ssd_loc_2[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_3 (AnchorBoxes)      (6, 25, 6, 8)        0           ssd_loc_3[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_4 (AnchorBoxes)      (6, 9, 6, 8)         0           ssd_loc_4[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_5 (AnchorBoxes)      (6, 4, 6, 8)         0           ssd_loc_5[0][0]                  
__________________________________________________________________________________________________
mbox_conf_softmax_ (Softmax)    (6, 2, 1, 11658)     0           before_softmax_permute[0][0]     
__________________________________________________________________________________________________
loc_reshape_0 (Reshape)         (6, 8664, 1, 4)      0           permute_19[0][0]                 
__________________________________________________________________________________________________
loc_reshape_1 (Reshape)         (6, 2166, 1, 4)      0           permute_20[0][0]                 
__________________________________________________________________________________________________
loc_reshape_2 (Reshape)         (6, 600, 1, 4)       0           permute_21[0][0]                 
__________________________________________________________________________________________________
loc_reshape_3 (Reshape)         (6, 150, 1, 4)       0           permute_22[0][0]                 
__________________________________________________________________________________________________
loc_reshape_4 (Reshape)         (6, 54, 1, 4)        0           permute_23[0][0]                 
__________________________________________________________________________________________________
loc_reshape_5 (Reshape)         (6, 24, 1, 4)        0           permute_24[0][0]                 
__________________________________________________________________________________________________
anchor_reshape_0 (Reshape)      (6, 8664, 1, 8)      0           ssd_anchor_0[0][0]               
__________________________________________________________________________________________________
anchor_reshape_1 (Reshape)      (6, 2166, 1, 8)      0           ssd_anchor_1[0][0]               
__________________________________________________________________________________________________
anchor_reshape_2 (Reshape)      (6, 600, 1, 8)       0           ssd_anchor_2[0][0]               
__________________________________________________________________________________________________
anchor_reshape_3 (Reshape)      (6, 150, 1, 8)       0           ssd_anchor_3[0][0]               
__________________________________________________________________________________________________
anchor_reshape_4 (Reshape)      (6, 54, 1, 8)        0           ssd_anchor_4[0][0]               
__________________________________________________________________________________________________
anchor_reshape_5 (Reshape)      (6, 24, 1, 8)        0           ssd_anchor_5[0][0]               
__________________________________________________________________________________________________
mbox_conf_softmax (Permute)     (6, 11658, 1, 2)     0           mbox_conf_softmax_[0][0]         
__________________________________________________________________________________________________
mbox_loc (Concatenate)          (6, 11658, 1, 4)     0           loc_reshape_0[0][0]              
                                                                 loc_reshape_1[0][0]              
                                                                 loc_reshape_2[0][0]              
                                                                 loc_reshape_3[0][0]              
                                                                 loc_reshape_4[0][0]              
                                                                 loc_reshape_5[0][0]              
__________________________________________________________________________________________________
mbox_priorbox (Concatenate)     (6, 11658, 1, 8)     0           anchor_reshape_0[0][0]           
                                                                 anchor_reshape_1[0][0]           
                                                                 anchor_reshape_2[0][0]           
                                                                 anchor_reshape_3[0][0]           
                                                                 anchor_reshape_4[0][0]           
                                                                 anchor_reshape_5[0][0]           
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (6, 11658, 1, 14)    0           mbox_conf_softmax[0][0]          
                                                                 mbox_loc[0][0]                   
                                                                 mbox_priorbox[0][0]              
__________________________________________________________________________________________________
ssd_predictions (Reshape)       (6, 11658, 14)       0           concatenate_2[0][0]              
==================================================================================================
Total params: 30,747,096
Trainable params: 30,692,376
Non-trainable params: 54,720
__________________________________________________________________________________________________
2022-02-02 14:09:23,583 [INFO] __main__: Number of images in the training dataset:	  2160
2022-02-02 14:09:23,583 [INFO] __main__: Number of images in the validation dataset:	   239
Epoch 1/80000
  2/360 [..............................] - ETA: 10:48:15 - loss: 80.5255/usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (2.858239). Check your callbacks.
  % delta_t_median)
360/360 [==============================] - 461s 1s/step - loss: 22.4974
8dbd4e108ea4:47:75 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0>
8dbd4e108ea4:47:75 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
8dbd4e108ea4:47:75 [0] NCCL INFO NET/IB : No device found.
8dbd4e108ea4:47:75 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.2<0>
8dbd4e108ea4:47:75 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda11.1
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 00/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 01/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 02/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 03/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 04/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 05/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 06/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 07/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 08/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 09/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 10/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 11/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 12/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 13/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 14/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 15/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 16/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 17/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 18/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 19/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 20/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 21/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 22/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 23/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 24/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 25/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 26/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 27/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 28/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 29/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 30/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Channel 31/32 :    0
8dbd4e108ea4:47:75 [0] NCCL INFO Trees [0] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [1] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [2] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [3] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [4] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [5] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [6] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [7] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [8] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [9] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [10] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [11] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [12] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [13] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [14] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [15] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [16] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [17] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [18] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [19] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [20] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [21] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [22] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [23] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [24] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [25] -1/-1/-1->0->-1|-1->0->-1/-1/-1 [26] -1/-1/-1->0->-1|-1->0->-1/-
8dbd4e108ea4:47:75 [0] NCCL INFO 32 coll channels, 32 p2p channels, 32 p2p channels per peer
8dbd4e108ea4:47:75 [0] NCCL INFO comm 0x7f261438fbd0 rank 0 nranks 1 cudaDev 0 busId 1000 - Init COMPLETE

Epoch 00001: saving model to /workspace/tao-experiments/dssd_head_detection/unpruned2/weights/dssd_resnet34_epoch_001.tlt
Epoch 2/80000
360/360 [==============================] - 274s 761ms/step - loss: 19.9372

Epoch 00002: saving model to /workspace/tao-experiments/dssd_head_detection/unpruned2/weights/dssd_resnet34_epoch_002.tlt
Epoch 3/80000
360/360 [==============================] - 276s 766ms/step - loss: 19.6461

Epoch 00003: saving model to /workspace/tao-experiments/dssd_head_detection/unpruned2/weights/dssd_resnet34_epoch_003.tlt
Epoch 4/80000
 53/360 [===>..........................] - ETA: 3:57 - loss: 19.4941^C
Traceback (most recent call last):
  File "/usr/local/bin/dssd", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/entrypoint/ssd.py", line 12, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/entrypoint/entrypoint.py", line 337, in launch_job
  File "/usr/lib/python3.6/subprocess.py", line 306, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 289, in call
    return p.wait(timeout=timeout)
  File "/usr/lib/python3.6/subprocess.py", line 1477, in wait
    (pid, sts) = self._try_wait(0)
  File "/usr/lib/python3.6/subprocess.py", line 1424, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 366, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 482, in return_func
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 362, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 274, in run_experiment
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1039, in fit
    validation_steps=validation_steps)
  File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_arrays.py", line 154, in fit_loop
    outs = f(ins)
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1472, in __call__
    run_metadata_ptr)
KeyboardInterrupt
!tao dssd train --gpus 1 --gpu_index=$GPU_INDEX \
                -e $SPECS_DIR/train1.txt \
                -r $USER_EXPERIMENT_DIR/unpruned2 \
                -k $KEY \
                -m $USER_EXPERIMENT_DIR/unpruned2/weights/dssd_resnet34_epoch_003.tlt \
                --initial_epoch 4
2022-03-06 00:10:41,896 [INFO] root: Registry: ['nvcr.io']
2022-03-06 00:10:41,976 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/dell/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

2022-03-05 21:10:49,257 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2022-03-05 21:10:49,257 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:64: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2022-03-05 21:10:49,329 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:64: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:67: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2022-03-05 21:10:49,330 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py:67: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2022-03-05 21:10:49,661 [INFO] iva.ssd.utils.spec_loader: Merging specification from /workspace/tao-experiments/dssd_head_detection/specs/train1.txt
2022-03-05 21:10:49,676 [INFO] __main__: Loading pretrained weights. This may take a while...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

2022-03-05 21:10:49,676 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

2022-03-05 21:10:49,677 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

2022-03-05 21:10:49,695 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

2022-03-05 21:10:51,629 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

Using DALI augmentation pipeline.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/box_coder/input_encoder.py:493: The name tf.log is deprecated. Please use tf.math.log instead.

2022-03-05 21:10:52,778 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/box_coder/input_encoder.py:493: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2022-03-05 21:10:58,453 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

2022-03-05 21:10:58,454 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

2022-03-05 21:10:59,853 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

2022-03-05 21:11:01,908 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

2022-03-05 21:11:04,455 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2022-03-05 21:11:05,208 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

Resume optimizer from pretrained model
Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 366, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 494, in return_func
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 482, in return_func
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 362, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/ssd/scripts/train.py", line 148, in run_experiment
  File "/usr/local/lib/python3.6/dist-packages/horovod/keras/__init__.py", line 61, in DistributedOptimizer
    sparse_as_dense)
  File "/usr/local/lib/python3.6/dist-packages/horovod/_keras/__init__.py", line 82, in create_distributed_optimizer
    dict(_DistributedOptimizer.__dict__))
TypeError: type 'NoneType' is not an acceptable base type
2022-03-06 00:52:34,700 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Could you please the training spec file?

train1.txt (1.4 KB)
is here

I cannot reproduce your error.
I train with KITTI dataset for several epochs without any pretrained model.
Then I press “ctrl+c” and resume training with “–initial_epoch 4” and 3rd tlt model. It works.

Could you double check? More, to narrow down, could you try to train from scratch as I did?

I checked with Kitti dataset but I still get the same error.
dssd.ipynb (141.0 KB)

To narrow down, could you try to train from scratch?

!tao dssd train --gpus 1 --gpu_index=$GPU_INDEX
-e $SPECS_DIR/dssd_train_resnet18_kitti.txt
-r $USER_EXPERIMENT_DIR/experiment_dir_unpruned
-k $KEY

Training did not resume, restarted

!tao dssd train --gpus 1 --gpu_index=$GPU_INDEX \
                -e $SPECS_DIR/dssd_train_resnet18_kitti.txt \
                -r $USER_EXPERIMENT_DIR/experiment_dir_unpruned \
                -k $KEY 
2022-03-07 11:12:12,087 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2022-03-07 11:12:12,566 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

Epoch 1/80
   2/1684 [..............................] - ETA: 23:43:57 - loss: 62.7133/usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (1.626442). Check your callbacks.
  % delta_t_median)
 301/1684 [====>.........................] - ETA: 14:01 - loss: 24.1156^C

I solved the problem,I deleted the checkpoint. I think there is an error in the sample kitti notebook.

!tao dssd train --gpus 1 --gpu_index=$GPU_INDEX \
                -e $SPECS_DIR/dssd_train_resnet18_kitti.txt \
                -r $USER_EXPERIMENT_DIR/experiment_dir_unpruned \
                -k $KEY  \
                --initial_epoch 2

Glad to know the issue is gone. I am afraid that you set different result folder when run training and resume training, right?

No, i deleted -m (the model path)

Hi,
As mentioned earlier, I run with below step
I train with KITTI dataset for several epochs without any pretrained model.
Then I press “ctrl+c” and resume training with “–initial_epoch 4” and 3rd tlt model. It works.

So, when I resume training, I use “-m 3rd tlt model”.

So, your error may not come from the “-m”.

Please make sure the result folder keeps the same when run training and resume training.

i tried without any pre-trained model but i get error
dssd.ipynb (128.8 KB)

To narrow down, can you run below steps in the terminal instead of notebook?

  1. login docker
    $ tao dssd run /bin/bash
  2. train for several epochs

# dssd train -e path_to_dssd_train_resnet18_kitti.txt
-r path_to_experiment_dir_unpruned
-k key

  1. press ctrl + c
  2. resume training
 # dssd train  -e path_to_dssd_train_resnet18_kitti.txt 
            -r path_to_experiment_dir_unpruned 
            -k key 
             -m path_to_experiment_dir_unpruned/weights/dssd_resnet18_epoch_003.tlt
              --initial_epoch 4

I couldn’t try because cuda error occurred even though I set the batch size 2

Do you mean you run into above new error even run training instead of resuming training?
Could you share how did you trigger the docker?
And what is the result of “$nvidia-smi” ?

yes, I get an error when using terminal. But batch size gpu 4 in ipynotebook

Train config on notebook

Could you run $nvidia-smi before running "tao dssd run /bin/bash "?

Seems that you meet error during dssd training. And the error log “CUDA allocation failed” or “dalishareoutput” results from lacking gpu memory.
Could you try batch size 1 ?

Or, try to use lower input size
output_width: 128
output_height: 128

I set batch size 1,and output_width 128,output_height 128,but another error occurred