Very slow initialization of training and first epoch

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (RTX 3070)
• Network Type (Yolo_v4)
• TLT Version

Configuration of the TLT Instance

dockers:
        nvidia/tlt-streamanalytics:
                docker_registry: nvcr.io
                docker_tag: v3.0-py3
                tasks:
                        1. augment
                        2. bpnet
                        3. classification
                        4. detectnet_v2
                        5. dssd
                        6. emotionnet
                        7. faster_rcnn
                        8. fpenet
                        9. gazenet
                        10. gesturenet
                        11. heartratenet
                        12. lprnet
                        13. mask_rcnn
                        14. multitask_classification
                        15. retinanet
                        16. ssd
                        17. unet
                        18. yolo_v3
                        19. yolo_v4
                        20. tlt-converter
        nvidia/tlt-pytorch:
                docker_registry: nvcr.io
                docker_tag: v3.0-py3
                tasks:
                        1. speech_to_text
                        2. speech_to_text_citrinet
                        3. text_classification
                        4. question_answering
                        5. token_classification
                        6. intent_slot_classification
                        7. punctuation_and_capitalization
format_version: 1.0
tlt_version: 3.0
published_date: 04/16/2021

• Training spec file(default spec file from Yolo_v4)
• How to reproduce the issue?

Run tlt yolo_v4 train with parameters from the sample notebook.

I had to wait 10+ minutes for training just to start. How can I try to debug it?

2

Where did you need to wait for 10+ minutes? Did you have logs?
There is a known limitation for slow training speed in each epoch. But it will not take a long time before you see “Epoch 1” after initialization.

3

So I start a tlt train command at
2021-07-02 12:20:11,719
at
2021-07-02 12:20:27,750 - loading a model (3 minutes)
2021-07-02 12:23:20,990 - message about dataset (I have 10 images in dataset)
…
I wait here for several minutes
+
10 minutes for 1 epoch
…
Epoch 1/1
2/10 [=====>…] - ETA: 34:42 - loss: 96.4671 /usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (3.052041). Check your callbacks.
% delta_t_median)
10/10 [==============================] - 522s 52s/step - loss: 92.5794

Whole logs

To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
2021-07-02 14:20:11,719 [INFO] root: Registry: ['nvcr.io']
2021-07-02 14:20:11,998 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the ~/.tlt_mounts.json file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

2021-07-02 12:20:17,634 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2021-07-02 12:20:17,634 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:52: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2021-07-02 12:20:17,760 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:52: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:55: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2021-07-02 12:20:17,761 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:55: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

2021-07-02 12:20:18,079 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

2021-07-02 12:20:18,080 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

2021-07-02 12:20:18,097 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

2021-07-02 12:20:19,463 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

2021-07-02 12:20:19,488 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

2021-07-02 12:20:22,664 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2021-07-02 12:20:23,062 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

2021-07-02 12:20:23,063 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

2021-07-02 12:20:24,383 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

2021-07-02 12:20:26,412 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.

2021-07-02 12:20:26,416 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

2021-07-02 12:20:27,418 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2021-07-02 12:20:27,750 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Input (InputLayer)              (None, 3, 384, 384)  0                                            
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 64, 192, 192) 9408        Input[0][0]                      
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 64, 192, 192) 256         conv1[0][0]                      
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 64, 192, 192) 0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
block_1a_conv_1 (Conv2D)        (None, 64, 96, 96)   4096        activation_2[0][0]               
__________________________________________________________________________________________________
block_1a_bn_1 (BatchNormalizati (None, 64, 96, 96)   256         block_1a_conv_1[0][0]            
__________________________________________________________________________________________________
block_1a_relu_1 (Activation)    (None, 64, 96, 96)   0           block_1a_bn_1[0][0]              
__________________________________________________________________________________________________
block_1a_conv_2 (Conv2D)        (None, 64, 96, 96)   36864       block_1a_relu_1[0][0]            
__________________________________________________________________________________________________
block_1a_bn_2 (BatchNormalizati (None, 64, 96, 96)   256         block_1a_conv_2[0][0]            
__________________________________________________________________________________________________
block_1a_relu_2 (Activation)    (None, 64, 96, 96)   0           block_1a_bn_2[0][0]              
__________________________________________________________________________________________________
block_1a_conv_3 (Conv2D)        (None, 256, 96, 96)  16384       block_1a_relu_2[0][0]            
__________________________________________________________________________________________________
block_1a_conv_shortcut (Conv2D) (None, 256, 96, 96)  16384       activation_2[0][0]               
__________________________________________________________________________________________________
block_1a_bn_3 (BatchNormalizati (None, 256, 96, 96)  1024        block_1a_conv_3[0][0]            
__________________________________________________________________________________________________
block_1a_bn_shortcut (BatchNorm (None, 256, 96, 96)  1024        block_1a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_17 (Add)                    (None, 256, 96, 96)  0           block_1a_bn_3[0][0]              
                                                                 block_1a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1a_relu (Activation)      (None, 256, 96, 96)  0           add_17[0][0]                     
__________________________________________________________________________________________________
block_1b_conv_1 (Conv2D)        (None, 64, 96, 96)   16384       block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_bn_1 (BatchNormalizati (None, 64, 96, 96)   256         block_1b_conv_1[0][0]            
__________________________________________________________________________________________________
block_1b_relu_1 (Activation)    (None, 64, 96, 96)   0           block_1b_bn_1[0][0]              
__________________________________________________________________________________________________
block_1b_conv_2 (Conv2D)        (None, 64, 96, 96)   36864       block_1b_relu_1[0][0]            
__________________________________________________________________________________________________
block_1b_bn_2 (BatchNormalizati (None, 64, 96, 96)   256         block_1b_conv_2[0][0]            
__________________________________________________________________________________________________
block_1b_relu_2 (Activation)    (None, 64, 96, 96)   0           block_1b_bn_2[0][0]              
__________________________________________________________________________________________________
block_1b_conv_3 (Conv2D)        (None, 256, 96, 96)  16384       block_1b_relu_2[0][0]            
__________________________________________________________________________________________________
block_1b_conv_shortcut (Conv2D) (None, 256, 96, 96)  65536       block_1a_relu[0][0]              
__________________________________________________________________________________________________
block_1b_bn_3 (BatchNormalizati (None, 256, 96, 96)  1024        block_1b_conv_3[0][0]            
__________________________________________________________________________________________________
block_1b_bn_shortcut (BatchNorm (None, 256, 96, 96)  1024        block_1b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_18 (Add)                    (None, 256, 96, 96)  0           block_1b_bn_3[0][0]              
                                                                 block_1b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1b_relu (Activation)      (None, 256, 96, 96)  0           add_18[0][0]                     
__________________________________________________________________________________________________
block_1c_conv_1 (Conv2D)        (None, 64, 96, 96)   16384       block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_1c_bn_1 (BatchNormalizati (None, 64, 96, 96)   256         block_1c_conv_1[0][0]            
__________________________________________________________________________________________________
block_1c_relu_1 (Activation)    (None, 64, 96, 96)   0           block_1c_bn_1[0][0]              
__________________________________________________________________________________________________
block_1c_conv_2 (Conv2D)        (None, 64, 96, 96)   36864       block_1c_relu_1[0][0]            
__________________________________________________________________________________________________
block_1c_bn_2 (BatchNormalizati (None, 64, 96, 96)   256         block_1c_conv_2[0][0]            
__________________________________________________________________________________________________
block_1c_relu_2 (Activation)    (None, 64, 96, 96)   0           block_1c_bn_2[0][0]              
__________________________________________________________________________________________________
block_1c_conv_3 (Conv2D)        (None, 256, 96, 96)  16384       block_1c_relu_2[0][0]            
__________________________________________________________________________________________________
block_1c_conv_shortcut (Conv2D) (None, 256, 96, 96)  65536       block_1b_relu[0][0]              
__________________________________________________________________________________________________
block_1c_bn_3 (BatchNormalizati (None, 256, 96, 96)  1024        block_1c_conv_3[0][0]            
__________________________________________________________________________________________________
block_1c_bn_shortcut (BatchNorm (None, 256, 96, 96)  1024        block_1c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_19 (Add)                    (None, 256, 96, 96)  0           block_1c_bn_3[0][0]              
                                                                 block_1c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_1c_relu (Activation)      (None, 256, 96, 96)  0           add_19[0][0]                     
__________________________________________________________________________________________________
block_2a_conv_1 (Conv2D)        (None, 128, 48, 48)  32768       block_1c_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_1 (BatchNormalizati (None, 128, 48, 48)  512         block_2a_conv_1[0][0]            
__________________________________________________________________________________________________
block_2a_relu_1 (Activation)    (None, 128, 48, 48)  0           block_2a_bn_1[0][0]              
__________________________________________________________________________________________________
block_2a_conv_2 (Conv2D)        (None, 128, 48, 48)  147456      block_2a_relu_1[0][0]            
__________________________________________________________________________________________________
block_2a_bn_2 (BatchNormalizati (None, 128, 48, 48)  512         block_2a_conv_2[0][0]            
__________________________________________________________________________________________________
block_2a_relu_2 (Activation)    (None, 128, 48, 48)  0           block_2a_bn_2[0][0]              
__________________________________________________________________________________________________
block_2a_conv_3 (Conv2D)        (None, 512, 48, 48)  65536       block_2a_relu_2[0][0]            
__________________________________________________________________________________________________
block_2a_conv_shortcut (Conv2D) (None, 512, 48, 48)  131072      block_1c_relu[0][0]              
__________________________________________________________________________________________________
block_2a_bn_3 (BatchNormalizati (None, 512, 48, 48)  2048        block_2a_conv_3[0][0]            
__________________________________________________________________________________________________
block_2a_bn_shortcut (BatchNorm (None, 512, 48, 48)  2048        block_2a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_20 (Add)                    (None, 512, 48, 48)  0           block_2a_bn_3[0][0]              
                                                                 block_2a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2a_relu (Activation)      (None, 512, 48, 48)  0           add_20[0][0]                     
__________________________________________________________________________________________________
block_2b_conv_1 (Conv2D)        (None, 128, 48, 48)  65536       block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_bn_1 (BatchNormalizati (None, 128, 48, 48)  512         block_2b_conv_1[0][0]            
__________________________________________________________________________________________________
block_2b_relu_1 (Activation)    (None, 128, 48, 48)  0           block_2b_bn_1[0][0]              
__________________________________________________________________________________________________
block_2b_conv_2 (Conv2D)        (None, 128, 48, 48)  147456      block_2b_relu_1[0][0]            
__________________________________________________________________________________________________
block_2b_bn_2 (BatchNormalizati (None, 128, 48, 48)  512         block_2b_conv_2[0][0]            
__________________________________________________________________________________________________
block_2b_relu_2 (Activation)    (None, 128, 48, 48)  0           block_2b_bn_2[0][0]              
__________________________________________________________________________________________________
block_2b_conv_3 (Conv2D)        (None, 512, 48, 48)  65536       block_2b_relu_2[0][0]            
__________________________________________________________________________________________________
block_2b_conv_shortcut (Conv2D) (None, 512, 48, 48)  262144      block_2a_relu[0][0]              
__________________________________________________________________________________________________
block_2b_bn_3 (BatchNormalizati (None, 512, 48, 48)  2048        block_2b_conv_3[0][0]            
__________________________________________________________________________________________________
block_2b_bn_shortcut (BatchNorm (None, 512, 48, 48)  2048        block_2b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_21 (Add)                    (None, 512, 48, 48)  0           block_2b_bn_3[0][0]              
                                                                 block_2b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2b_relu (Activation)      (None, 512, 48, 48)  0           add_21[0][0]                     
__________________________________________________________________________________________________
block_2c_conv_1 (Conv2D)        (None, 128, 48, 48)  65536       block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_2c_bn_1 (BatchNormalizati (None, 128, 48, 48)  512         block_2c_conv_1[0][0]            
__________________________________________________________________________________________________
block_2c_relu_1 (Activation)    (None, 128, 48, 48)  0           block_2c_bn_1[0][0]              
__________________________________________________________________________________________________
block_2c_conv_2 (Conv2D)        (None, 128, 48, 48)  147456      block_2c_relu_1[0][0]            
__________________________________________________________________________________________________
block_2c_bn_2 (BatchNormalizati (None, 128, 48, 48)  512         block_2c_conv_2[0][0]            
__________________________________________________________________________________________________
block_2c_relu_2 (Activation)    (None, 128, 48, 48)  0           block_2c_bn_2[0][0]              
__________________________________________________________________________________________________
block_2c_conv_3 (Conv2D)        (None, 512, 48, 48)  65536       block_2c_relu_2[0][0]            
__________________________________________________________________________________________________
block_2c_conv_shortcut (Conv2D) (None, 512, 48, 48)  262144      block_2b_relu[0][0]              
__________________________________________________________________________________________________
block_2c_bn_3 (BatchNormalizati (None, 512, 48, 48)  2048        block_2c_conv_3[0][0]            
__________________________________________________________________________________________________
block_2c_bn_shortcut (BatchNorm (None, 512, 48, 48)  2048        block_2c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_22 (Add)                    (None, 512, 48, 48)  0           block_2c_bn_3[0][0]              
                                                                 block_2c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2c_relu (Activation)      (None, 512, 48, 48)  0           add_22[0][0]                     
__________________________________________________________________________________________________
block_2d_conv_1 (Conv2D)        (None, 128, 48, 48)  65536       block_2c_relu[0][0]              
__________________________________________________________________________________________________
block_2d_bn_1 (BatchNormalizati (None, 128, 48, 48)  512         block_2d_conv_1[0][0]            
__________________________________________________________________________________________________
block_2d_relu_1 (Activation)    (None, 128, 48, 48)  0           block_2d_bn_1[0][0]              
__________________________________________________________________________________________________
block_2d_conv_2 (Conv2D)        (None, 128, 48, 48)  147456      block_2d_relu_1[0][0]            
__________________________________________________________________________________________________
block_2d_bn_2 (BatchNormalizati (None, 128, 48, 48)  512         block_2d_conv_2[0][0]            
__________________________________________________________________________________________________
block_2d_relu_2 (Activation)    (None, 128, 48, 48)  0           block_2d_bn_2[0][0]              
__________________________________________________________________________________________________
block_2d_conv_3 (Conv2D)        (None, 512, 48, 48)  65536       block_2d_relu_2[0][0]            
__________________________________________________________________________________________________
block_2d_conv_shortcut (Conv2D) (None, 512, 48, 48)  262144      block_2c_relu[0][0]              
__________________________________________________________________________________________________
block_2d_bn_3 (BatchNormalizati (None, 512, 48, 48)  2048        block_2d_conv_3[0][0]            
__________________________________________________________________________________________________
block_2d_bn_shortcut (BatchNorm (None, 512, 48, 48)  2048        block_2d_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_23 (Add)                    (None, 512, 48, 48)  0           block_2d_bn_3[0][0]              
                                                                 block_2d_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_2d_relu (Activation)      (None, 512, 48, 48)  0           add_23[0][0]                     
__________________________________________________________________________________________________
block_3a_conv_1 (Conv2D)        (None, 256, 24, 24)  131072      block_2d_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_1 (BatchNormalizati (None, 256, 24, 24)  1024        block_3a_conv_1[0][0]            
__________________________________________________________________________________________________
block_3a_relu_1 (Activation)    (None, 256, 24, 24)  0           block_3a_bn_1[0][0]              
__________________________________________________________________________________________________
block_3a_conv_2 (Conv2D)        (None, 256, 24, 24)  589824      block_3a_relu_1[0][0]            
__________________________________________________________________________________________________
block_3a_bn_2 (BatchNormalizati (None, 256, 24, 24)  1024        block_3a_conv_2[0][0]            
__________________________________________________________________________________________________
block_3a_relu_2 (Activation)    (None, 256, 24, 24)  0           block_3a_bn_2[0][0]              
__________________________________________________________________________________________________
block_3a_conv_3 (Conv2D)        (None, 1024, 24, 24) 262144      block_3a_relu_2[0][0]            
__________________________________________________________________________________________________
block_3a_conv_shortcut (Conv2D) (None, 1024, 24, 24) 524288      block_2d_relu[0][0]              
__________________________________________________________________________________________________
block_3a_bn_3 (BatchNormalizati (None, 1024, 24, 24) 4096        block_3a_conv_3[0][0]            
__________________________________________________________________________________________________
block_3a_bn_shortcut (BatchNorm (None, 1024, 24, 24) 4096        block_3a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_24 (Add)                    (None, 1024, 24, 24) 0           block_3a_bn_3[0][0]              
                                                                 block_3a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3a_relu (Activation)      (None, 1024, 24, 24) 0           add_24[0][0]                     
__________________________________________________________________________________________________
block_3b_conv_1 (Conv2D)        (None, 256, 24, 24)  262144      block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_bn_1 (BatchNormalizati (None, 256, 24, 24)  1024        block_3b_conv_1[0][0]            
__________________________________________________________________________________________________
block_3b_relu_1 (Activation)    (None, 256, 24, 24)  0           block_3b_bn_1[0][0]              
__________________________________________________________________________________________________
block_3b_conv_2 (Conv2D)        (None, 256, 24, 24)  589824      block_3b_relu_1[0][0]            
__________________________________________________________________________________________________
block_3b_bn_2 (BatchNormalizati (None, 256, 24, 24)  1024        block_3b_conv_2[0][0]            
__________________________________________________________________________________________________
block_3b_relu_2 (Activation)    (None, 256, 24, 24)  0           block_3b_bn_2[0][0]              
__________________________________________________________________________________________________
block_3b_conv_3 (Conv2D)        (None, 1024, 24, 24) 262144      block_3b_relu_2[0][0]            
__________________________________________________________________________________________________
block_3b_conv_shortcut (Conv2D) (None, 1024, 24, 24) 1048576     block_3a_relu[0][0]              
__________________________________________________________________________________________________
block_3b_bn_3 (BatchNormalizati (None, 1024, 24, 24) 4096        block_3b_conv_3[0][0]            
__________________________________________________________________________________________________
block_3b_bn_shortcut (BatchNorm (None, 1024, 24, 24) 4096        block_3b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_25 (Add)                    (None, 1024, 24, 24) 0           block_3b_bn_3[0][0]              
                                                                 block_3b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3b_relu (Activation)      (None, 1024, 24, 24) 0           add_25[0][0]                     
__________________________________________________________________________________________________
block_3c_conv_1 (Conv2D)        (None, 256, 24, 24)  262144      block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_3c_bn_1 (BatchNormalizati (None, 256, 24, 24)  1024        block_3c_conv_1[0][0]            
__________________________________________________________________________________________________
block_3c_relu_1 (Activation)    (None, 256, 24, 24)  0           block_3c_bn_1[0][0]              
__________________________________________________________________________________________________
block_3c_conv_2 (Conv2D)        (None, 256, 24, 24)  589824      block_3c_relu_1[0][0]            
__________________________________________________________________________________________________
block_3c_bn_2 (BatchNormalizati (None, 256, 24, 24)  1024        block_3c_conv_2[0][0]            
__________________________________________________________________________________________________
block_3c_relu_2 (Activation)    (None, 256, 24, 24)  0           block_3c_bn_2[0][0]              
__________________________________________________________________________________________________
block_3c_conv_3 (Conv2D)        (None, 1024, 24, 24) 262144      block_3c_relu_2[0][0]            
__________________________________________________________________________________________________
block_3c_conv_shortcut (Conv2D) (None, 1024, 24, 24) 1048576     block_3b_relu[0][0]              
__________________________________________________________________________________________________
block_3c_bn_3 (BatchNormalizati (None, 1024, 24, 24) 4096        block_3c_conv_3[0][0]            
__________________________________________________________________________________________________
block_3c_bn_shortcut (BatchNorm (None, 1024, 24, 24) 4096        block_3c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_26 (Add)                    (None, 1024, 24, 24) 0           block_3c_bn_3[0][0]              
                                                                 block_3c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3c_relu (Activation)      (None, 1024, 24, 24) 0           add_26[0][0]                     
__________________________________________________________________________________________________
block_3d_conv_1 (Conv2D)        (None, 256, 24, 24)  262144      block_3c_relu[0][0]              
__________________________________________________________________________________________________
block_3d_bn_1 (BatchNormalizati (None, 256, 24, 24)  1024        block_3d_conv_1[0][0]            
__________________________________________________________________________________________________
block_3d_relu_1 (Activation)    (None, 256, 24, 24)  0           block_3d_bn_1[0][0]              
__________________________________________________________________________________________________
block_3d_conv_2 (Conv2D)        (None, 256, 24, 24)  589824      block_3d_relu_1[0][0]            
__________________________________________________________________________________________________
block_3d_bn_2 (BatchNormalizati (None, 256, 24, 24)  1024        block_3d_conv_2[0][0]            
__________________________________________________________________________________________________
block_3d_relu_2 (Activation)    (None, 256, 24, 24)  0           block_3d_bn_2[0][0]              
__________________________________________________________________________________________________
block_3d_conv_3 (Conv2D)        (None, 1024, 24, 24) 262144      block_3d_relu_2[0][0]            
__________________________________________________________________________________________________
block_3d_conv_shortcut (Conv2D) (None, 1024, 24, 24) 1048576     block_3c_relu[0][0]              
__________________________________________________________________________________________________
block_3d_bn_3 (BatchNormalizati (None, 1024, 24, 24) 4096        block_3d_conv_3[0][0]            
__________________________________________________________________________________________________
block_3d_bn_shortcut (BatchNorm (None, 1024, 24, 24) 4096        block_3d_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_27 (Add)                    (None, 1024, 24, 24) 0           block_3d_bn_3[0][0]              
                                                                 block_3d_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3d_relu (Activation)      (None, 1024, 24, 24) 0           add_27[0][0]                     
__________________________________________________________________________________________________
block_3e_conv_1 (Conv2D)        (None, 256, 24, 24)  262144      block_3d_relu[0][0]              
__________________________________________________________________________________________________
block_3e_bn_1 (BatchNormalizati (None, 256, 24, 24)  1024        block_3e_conv_1[0][0]            
__________________________________________________________________________________________________
block_3e_relu_1 (Activation)    (None, 256, 24, 24)  0           block_3e_bn_1[0][0]              
__________________________________________________________________________________________________
block_3e_conv_2 (Conv2D)        (None, 256, 24, 24)  589824      block_3e_relu_1[0][0]            
__________________________________________________________________________________________________
block_3e_bn_2 (BatchNormalizati (None, 256, 24, 24)  1024        block_3e_conv_2[0][0]            
__________________________________________________________________________________________________
block_3e_relu_2 (Activation)    (None, 256, 24, 24)  0           block_3e_bn_2[0][0]              
__________________________________________________________________________________________________
block_3e_conv_3 (Conv2D)        (None, 1024, 24, 24) 262144      block_3e_relu_2[0][0]            
__________________________________________________________________________________________________
block_3e_conv_shortcut (Conv2D) (None, 1024, 24, 24) 1048576     block_3d_relu[0][0]              
__________________________________________________________________________________________________
block_3e_bn_3 (BatchNormalizati (None, 1024, 24, 24) 4096        block_3e_conv_3[0][0]            
__________________________________________________________________________________________________
block_3e_bn_shortcut (BatchNorm (None, 1024, 24, 24) 4096        block_3e_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_28 (Add)                    (None, 1024, 24, 24) 0           block_3e_bn_3[0][0]              
                                                                 block_3e_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3e_relu (Activation)      (None, 1024, 24, 24) 0           add_28[0][0]                     
__________________________________________________________________________________________________
block_3f_conv_1 (Conv2D)        (None, 256, 24, 24)  262144      block_3e_relu[0][0]              
__________________________________________________________________________________________________
block_3f_bn_1 (BatchNormalizati (None, 256, 24, 24)  1024        block_3f_conv_1[0][0]            
__________________________________________________________________________________________________
block_3f_relu_1 (Activation)    (None, 256, 24, 24)  0           block_3f_bn_1[0][0]              
__________________________________________________________________________________________________
block_3f_conv_2 (Conv2D)        (None, 256, 24, 24)  589824      block_3f_relu_1[0][0]            
__________________________________________________________________________________________________
block_3f_bn_2 (BatchNormalizati (None, 256, 24, 24)  1024        block_3f_conv_2[0][0]            
__________________________________________________________________________________________________
block_3f_relu_2 (Activation)    (None, 256, 24, 24)  0           block_3f_bn_2[0][0]              
__________________________________________________________________________________________________
block_3f_conv_3 (Conv2D)        (None, 1024, 24, 24) 262144      block_3f_relu_2[0][0]            
__________________________________________________________________________________________________
block_3f_conv_shortcut (Conv2D) (None, 1024, 24, 24) 1048576     block_3e_relu[0][0]              
__________________________________________________________________________________________________
block_3f_bn_3 (BatchNormalizati (None, 1024, 24, 24) 4096        block_3f_conv_3[0][0]            
__________________________________________________________________________________________________
block_3f_bn_shortcut (BatchNorm (None, 1024, 24, 24) 4096        block_3f_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_29 (Add)                    (None, 1024, 24, 24) 0           block_3f_bn_3[0][0]              
                                                                 block_3f_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_3f_relu (Activation)      (None, 1024, 24, 24) 0           add_29[0][0]                     
__________________________________________________________________________________________________
block_4a_conv_1 (Conv2D)        (None, 512, 24, 24)  524288      block_3f_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_1 (BatchNormalizati (None, 512, 24, 24)  2048        block_4a_conv_1[0][0]            
__________________________________________________________________________________________________
block_4a_relu_1 (Activation)    (None, 512, 24, 24)  0           block_4a_bn_1[0][0]              
__________________________________________________________________________________________________
block_4a_conv_2 (Conv2D)        (None, 512, 24, 24)  2359296     block_4a_relu_1[0][0]            
__________________________________________________________________________________________________
block_4a_bn_2 (BatchNormalizati (None, 512, 24, 24)  2048        block_4a_conv_2[0][0]            
__________________________________________________________________________________________________
block_4a_relu_2 (Activation)    (None, 512, 24, 24)  0           block_4a_bn_2[0][0]              
__________________________________________________________________________________________________
block_4a_conv_3 (Conv2D)        (None, 2048, 24, 24) 1048576     block_4a_relu_2[0][0]            
__________________________________________________________________________________________________
block_4a_conv_shortcut (Conv2D) (None, 2048, 24, 24) 2097152     block_3f_relu[0][0]              
__________________________________________________________________________________________________
block_4a_bn_3 (BatchNormalizati (None, 2048, 24, 24) 8192        block_4a_conv_3[0][0]            
__________________________________________________________________________________________________
block_4a_bn_shortcut (BatchNorm (None, 2048, 24, 24) 8192        block_4a_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_30 (Add)                    (None, 2048, 24, 24) 0           block_4a_bn_3[0][0]              
                                                                 block_4a_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4a_relu (Activation)      (None, 2048, 24, 24) 0           add_30[0][0]                     
__________________________________________________________________________________________________
block_4b_conv_1 (Conv2D)        (None, 512, 24, 24)  1048576     block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_bn_1 (BatchNormalizati (None, 512, 24, 24)  2048        block_4b_conv_1[0][0]            
__________________________________________________________________________________________________
block_4b_relu_1 (Activation)    (None, 512, 24, 24)  0           block_4b_bn_1[0][0]              
__________________________________________________________________________________________________
block_4b_conv_2 (Conv2D)        (None, 512, 24, 24)  2359296     block_4b_relu_1[0][0]            
__________________________________________________________________________________________________
block_4b_bn_2 (BatchNormalizati (None, 512, 24, 24)  2048        block_4b_conv_2[0][0]            
__________________________________________________________________________________________________
block_4b_relu_2 (Activation)    (None, 512, 24, 24)  0           block_4b_bn_2[0][0]              
__________________________________________________________________________________________________
block_4b_conv_3 (Conv2D)        (None, 2048, 24, 24) 1048576     block_4b_relu_2[0][0]            
__________________________________________________________________________________________________
block_4b_conv_shortcut (Conv2D) (None, 2048, 24, 24) 4194304     block_4a_relu[0][0]              
__________________________________________________________________________________________________
block_4b_bn_3 (BatchNormalizati (None, 2048, 24, 24) 8192        block_4b_conv_3[0][0]            
__________________________________________________________________________________________________
block_4b_bn_shortcut (BatchNorm (None, 2048, 24, 24) 8192        block_4b_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_31 (Add)                    (None, 2048, 24, 24) 0           block_4b_bn_3[0][0]              
                                                                 block_4b_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4b_relu (Activation)      (None, 2048, 24, 24) 0           add_31[0][0]                     
__________________________________________________________________________________________________
block_4c_conv_1 (Conv2D)        (None, 512, 24, 24)  1048576     block_4b_relu[0][0]              
__________________________________________________________________________________________________
block_4c_bn_1 (BatchNormalizati (None, 512, 24, 24)  2048        block_4c_conv_1[0][0]            
__________________________________________________________________________________________________
block_4c_relu_1 (Activation)    (None, 512, 24, 24)  0           block_4c_bn_1[0][0]              
__________________________________________________________________________________________________
block_4c_conv_2 (Conv2D)        (None, 512, 24, 24)  2359296     block_4c_relu_1[0][0]            
__________________________________________________________________________________________________
block_4c_bn_2 (BatchNormalizati (None, 512, 24, 24)  2048        block_4c_conv_2[0][0]            
__________________________________________________________________________________________________
block_4c_relu_2 (Activation)    (None, 512, 24, 24)  0           block_4c_bn_2[0][0]              
__________________________________________________________________________________________________
block_4c_conv_3 (Conv2D)        (None, 2048, 24, 24) 1048576     block_4c_relu_2[0][0]            
__________________________________________________________________________________________________
block_4c_conv_shortcut (Conv2D) (None, 2048, 24, 24) 4194304     block_4b_relu[0][0]              
__________________________________________________________________________________________________
block_4c_bn_3 (BatchNormalizati (None, 2048, 24, 24) 8192        block_4c_conv_3[0][0]            
__________________________________________________________________________________________________
block_4c_bn_shortcut (BatchNorm (None, 2048, 24, 24) 8192        block_4c_conv_shortcut[0][0]     
__________________________________________________________________________________________________
add_32 (Add)                    (None, 2048, 24, 24) 0           block_4c_bn_3[0][0]              
                                                                 block_4c_bn_shortcut[0][0]       
__________________________________________________________________________________________________
block_4c_relu (Activation)      (None, 2048, 24, 24) 0           add_32[0][0]                     
__________________________________________________________________________________________________
yolo_spp_pool_1 (MaxPooling2D)  (None, 2048, 24, 24) 0           block_4c_relu[0][0]              
__________________________________________________________________________________________________
yolo_spp_pool_2 (MaxPooling2D)  (None, 2048, 24, 24) 0           block_4c_relu[0][0]              
__________________________________________________________________________________________________
yolo_spp_pool_3 (MaxPooling2D)  (None, 2048, 24, 24) 0           block_4c_relu[0][0]              
__________________________________________________________________________________________________
yolo_spp_concat (Concatenate)   (None, 8192, 24, 24) 0           yolo_spp_pool_1[0][0]            
                                                                 yolo_spp_pool_2[0][0]            
                                                                 yolo_spp_pool_3[0][0]            
                                                                 block_4c_relu[0][0]              
__________________________________________________________________________________________________
yolo_spp_conv (Conv2D)          (None, 512, 24, 24)  4194304     yolo_spp_concat[0][0]            
__________________________________________________________________________________________________
yolo_spp_conv_bn (BatchNormaliz (None, 512, 24, 24)  2048        yolo_spp_conv[0][0]              
__________________________________________________________________________________________________
yolo_spp_conv_lrelu (LeakyReLU) (None, 512, 24, 24)  0           yolo_spp_conv_bn[0][0]           
__________________________________________________________________________________________________
yolo_expand_conv1 (Conv2D)      (None, 1024, 12, 12) 4718592     yolo_spp_conv_lrelu[0][0]        
__________________________________________________________________________________________________
yolo_expand_conv1_bn (BatchNorm (None, 1024, 12, 12) 4096        yolo_expand_conv1[0][0]          
__________________________________________________________________________________________________
yolo_expand_conv1_lrelu (LeakyR (None, 1024, 12, 12) 0           yolo_expand_conv1_bn[0][0]       
__________________________________________________________________________________________________
yolo_conv1_1 (Conv2D)           (None, 512, 12, 12)  524288      yolo_expand_conv1_lrelu[0][0]    
__________________________________________________________________________________________________
yolo_conv1_1_bn (BatchNormaliza (None, 512, 12, 12)  2048        yolo_conv1_1[0][0]               
__________________________________________________________________________________________________
yolo_conv1_1_lrelu (LeakyReLU)  (None, 512, 12, 12)  0           yolo_conv1_1_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv1_2 (Conv2D)           (None, 1024, 12, 12) 4718592     yolo_conv1_1_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv1_2_bn (BatchNormaliza (None, 1024, 12, 12) 4096        yolo_conv1_2[0][0]               
__________________________________________________________________________________________________
yolo_conv1_2_lrelu (LeakyReLU)  (None, 1024, 12, 12) 0           yolo_conv1_2_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv1_3 (Conv2D)           (None, 512, 12, 12)  524288      yolo_conv1_2_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv1_3_bn (BatchNormaliza (None, 512, 12, 12)  2048        yolo_conv1_3[0][0]               
__________________________________________________________________________________________________
yolo_conv1_3_lrelu (LeakyReLU)  (None, 512, 12, 12)  0           yolo_conv1_3_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv1_4 (Conv2D)           (None, 1024, 12, 12) 4718592     yolo_conv1_3_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv1_4_bn (BatchNormaliza (None, 1024, 12, 12) 4096        yolo_conv1_4[0][0]               
__________________________________________________________________________________________________
yolo_conv1_4_lrelu (LeakyReLU)  (None, 1024, 12, 12) 0           yolo_conv1_4_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv1_5 (Conv2D)           (None, 512, 12, 12)  524288      yolo_conv1_4_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv1_5_bn (BatchNormaliza (None, 512, 12, 12)  2048        yolo_conv1_5[0][0]               
__________________________________________________________________________________________________
yolo_conv1_5_lrelu (LeakyReLU)  (None, 512, 12, 12)  0           yolo_conv1_5_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv2 (Conv2D)             (None, 256, 12, 12)  131072      yolo_conv1_5_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv2_bn (BatchNormalizati (None, 256, 12, 12)  1024        yolo_conv2[0][0]                 
__________________________________________________________________________________________________
yolo_conv2_lrelu (LeakyReLU)    (None, 256, 12, 12)  0           yolo_conv2_bn[0][0]              
__________________________________________________________________________________________________
upsample0 (UpSampling2D)        (None, 256, 24, 24)  0           yolo_conv2_lrelu[0][0]           
__________________________________________________________________________________________________
concatenate_3 (Concatenate)     (None, 768, 24, 24)  0           upsample0[0][0]                  
                                                                 block_4c_relu_2[0][0]            
__________________________________________________________________________________________________
yolo_conv3_1 (Conv2D)           (None, 256, 24, 24)  196608      concatenate_3[0][0]              
__________________________________________________________________________________________________
yolo_conv3_1_bn (BatchNormaliza (None, 256, 24, 24)  1024        yolo_conv3_1[0][0]               
__________________________________________________________________________________________________
yolo_conv3_1_lrelu (LeakyReLU)  (None, 256, 24, 24)  0           yolo_conv3_1_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv3_2 (Conv2D)           (None, 512, 24, 24)  1179648     yolo_conv3_1_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv3_2_bn (BatchNormaliza (None, 512, 24, 24)  2048        yolo_conv3_2[0][0]               
__________________________________________________________________________________________________
yolo_conv3_2_lrelu (LeakyReLU)  (None, 512, 24, 24)  0           yolo_conv3_2_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv3_3 (Conv2D)           (None, 256, 24, 24)  131072      yolo_conv3_2_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv3_3_bn (BatchNormaliza (None, 256, 24, 24)  1024        yolo_conv3_3[0][0]               
__________________________________________________________________________________________________
yolo_conv3_3_lrelu (LeakyReLU)  (None, 256, 24, 24)  0           yolo_conv3_3_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv3_4 (Conv2D)           (None, 512, 24, 24)  1179648     yolo_conv3_3_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv3_4_bn (BatchNormaliza (None, 512, 24, 24)  2048        yolo_conv3_4[0][0]               
__________________________________________________________________________________________________
yolo_conv3_4_lrelu (LeakyReLU)  (None, 512, 24, 24)  0           yolo_conv3_4_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv3_5 (Conv2D)           (None, 256, 24, 24)  131072      yolo_conv3_4_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv3_5_bn (BatchNormaliza (None, 256, 24, 24)  1024        yolo_conv3_5[0][0]               
__________________________________________________________________________________________________
yolo_conv3_5_lrelu (LeakyReLU)  (None, 256, 24, 24)  0           yolo_conv3_5_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv4 (Conv2D)             (None, 128, 24, 24)  32768       yolo_conv3_5_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv4_bn (BatchNormalizati (None, 128, 24, 24)  512         yolo_conv4[0][0]                 
__________________________________________________________________________________________________
yolo_conv4_lrelu (LeakyReLU)    (None, 128, 24, 24)  0           yolo_conv4_bn[0][0]              
__________________________________________________________________________________________________
upsample1 (UpSampling2D)        (None, 128, 48, 48)  0           yolo_conv4_lrelu[0][0]           
__________________________________________________________________________________________________
expand_upsample (UpSampling2D)  (None, 256, 48, 48)  0           block_3f_relu_2[0][0]            
__________________________________________________________________________________________________
concatenate_4 (Concatenate)     (None, 384, 48, 48)  0           upsample1[0][0]                  
                                                                 expand_upsample[0][0]            
__________________________________________________________________________________________________
yolo_conv5_1 (Conv2D)           (None, 128, 48, 48)  49152       concatenate_4[0][0]              
__________________________________________________________________________________________________
yolo_conv5_1_bn (BatchNormaliza (None, 128, 48, 48)  512         yolo_conv5_1[0][0]               
__________________________________________________________________________________________________
yolo_conv5_1_lrelu (LeakyReLU)  (None, 128, 48, 48)  0           yolo_conv5_1_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv5_2 (Conv2D)           (None, 256, 48, 48)  294912      yolo_conv5_1_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv5_2_bn (BatchNormaliza (None, 256, 48, 48)  1024        yolo_conv5_2[0][0]               
__________________________________________________________________________________________________
yolo_conv5_2_lrelu (LeakyReLU)  (None, 256, 48, 48)  0           yolo_conv5_2_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv5_3 (Conv2D)           (None, 128, 48, 48)  32768       yolo_conv5_2_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv5_3_bn (BatchNormaliza (None, 128, 48, 48)  512         yolo_conv5_3[0][0]               
__________________________________________________________________________________________________
yolo_conv5_3_lrelu (LeakyReLU)  (None, 128, 48, 48)  0           yolo_conv5_3_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv5_4 (Conv2D)           (None, 256, 48, 48)  294912      yolo_conv5_3_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv5_4_bn (BatchNormaliza (None, 256, 48, 48)  1024        yolo_conv5_4[0][0]               
__________________________________________________________________________________________________
yolo_conv5_4_lrelu (LeakyReLU)  (None, 256, 48, 48)  0           yolo_conv5_4_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv5_5 (Conv2D)           (None, 128, 48, 48)  32768       yolo_conv5_4_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv5_5_bn (BatchNormaliza (None, 128, 48, 48)  512         yolo_conv5_5[0][0]               
__________________________________________________________________________________________________
yolo_conv5_5_lrelu (LeakyReLU)  (None, 128, 48, 48)  0           yolo_conv5_5_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv1_6 (Conv2D)           (None, 1024, 12, 12) 4718592     yolo_conv1_5_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv3_6 (Conv2D)           (None, 512, 24, 24)  1179648     yolo_conv3_5_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv5_6 (Conv2D)           (None, 256, 48, 48)  294912      yolo_conv5_5_lrelu[0][0]         
__________________________________________________________________________________________________
yolo_conv1_6_bn (BatchNormaliza (None, 1024, 12, 12) 4096        yolo_conv1_6[0][0]               
__________________________________________________________________________________________________
yolo_conv3_6_bn (BatchNormaliza (None, 512, 24, 24)  2048        yolo_conv3_6[0][0]               
__________________________________________________________________________________________________
yolo_conv5_6_bn (BatchNormaliza (None, 256, 48, 48)  1024        yolo_conv5_6[0][0]               
__________________________________________________________________________________________________
yolo_conv1_6_lrelu (LeakyReLU)  (None, 1024, 12, 12) 0           yolo_conv1_6_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv3_6_lrelu (LeakyReLU)  (None, 512, 24, 24)  0           yolo_conv3_6_bn[0][0]            
__________________________________________________________________________________________________
yolo_conv5_6_lrelu (LeakyReLU)  (None, 256, 48, 48)  0           yolo_conv5_6_bn[0][0]            
__________________________________________________________________________________________________
conv_big_object (Conv2D)        (None, 24, 12, 12)   24600       yolo_conv1_6_lrelu[0][0]         
__________________________________________________________________________________________________
conv_mid_object (Conv2D)        (None, 24, 24, 24)   12312       yolo_conv3_6_lrelu[0][0]         
__________________________________________________________________________________________________
conv_sm_object (Conv2D)         (None, 24, 48, 48)   6168        yolo_conv5_6_lrelu[0][0]         
__________________________________________________________________________________________________
bg_permute (Permute)            (None, 12, 12, 24)   0           conv_big_object[0][0]            
__________________________________________________________________________________________________
md_permute (Permute)            (None, 24, 24, 24)   0           conv_mid_object[0][0]            
__________________________________________________________________________________________________
sm_permute (Permute)            (None, 48, 48, 24)   0           conv_sm_object[0][0]             
__________________________________________________________________________________________________
bg_reshape (Reshape)            (None, 432, 8)       0           bg_permute[0][0]                 
__________________________________________________________________________________________________
md_reshape (Reshape)            (None, 1728, 8)      0           md_permute[0][0]                 
__________________________________________________________________________________________________
sm_reshape (Reshape)            (None, 6912, 8)      0           sm_permute[0][0]                 
__________________________________________________________________________________________________
bg_anchor (YOLOAnchorBox)       (None, 432, 6)       0           conv_big_object[0][0]            
__________________________________________________________________________________________________
bg_bbox_processor (BBoxPostProc (None, 432, 8)       0           bg_reshape[0][0]                 
__________________________________________________________________________________________________
md_anchor (YOLOAnchorBox)       (None, 1728, 6)      0           conv_mid_object[0][0]            
__________________________________________________________________________________________________
md_bbox_processor (BBoxPostProc (None, 1728, 8)      0           md_reshape[0][0]                 
__________________________________________________________________________________________________
sm_anchor (YOLOAnchorBox)       (None, 6912, 6)      0           conv_sm_object[0][0]             
__________________________________________________________________________________________________
sm_bbox_processor (BBoxPostProc (None, 6912, 8)      0           sm_reshape[0][0]                 
__________________________________________________________________________________________________
encoded_bg (Concatenate)        (None, 432, 14)      0           bg_anchor[0][0]                  
                                                                 bg_bbox_processor[0][0]          
__________________________________________________________________________________________________
encoded_md (Concatenate)        (None, 1728, 14)     0           md_anchor[0][0]                  
                                                                 md_bbox_processor[0][0]          
__________________________________________________________________________________________________
encoded_sm (Concatenate)        (None, 6912, 14)     0           sm_anchor[0][0]                  
                                                                 sm_bbox_processor[0][0]          
__________________________________________________________________________________________________
encoded_detections (Concatenate (None, 9072, 14)     0           encoded_bg[0][0]                 
                                                                 encoded_md[0][0]                 
                                                                 encoded_sm[0][0]                 
==================================================================================================
Total params: 68,040,712
Trainable params: 67,945,096
Non-trainable params: 95,616
__________________________________________________________________________________________________
2021-07-02 12:23:20,990 [INFO] __main__: Number of images in the training dataset:	    10
Epoch 1/1
 2/10 [=====>........................] - ETA: 34:42 - loss: 96.4671  /usr/local/lib/python3.6/dist-packages/keras/callbacks.py:122: UserWarning: Method on_batch_end() is slow compared to the batch update (3.052041). Check your callbacks.
  % delta_t_median)
10/10 [==============================] - 522s 52s/step - loss: 92.5794

Epoch 00001: saving model to /workspace/tlt-experiments/yolo_v4/experiment_dir_unpruned/weights/yolov4_resnet50_epoch_001.tlt
4

Do you mean you wait for 10+ minutes between
2021-07-02 12:23:20,990 [INFO] __main__: Number of images in the training dataset: 10
and
Epoch 1/1

5

No, those two appear at the same time and then I wait 10+ minutes before I see the first info about the progress of the 1 epochs

6

OK. Will check. At that point, it will indeed take time to move on. But 10+ minutes is a bit longer.
Can you share your training spec? You mention that it is the same as default spec file from Yolo_v4. Appreciate that you can share here.
More, if possible, could you share the 10 images and labels?

7

For Resnet18 it is 1-2 minutes model loading + 11-37s/step first epoch (when I run multiple times for the test, and there is no other processes using gpu

For Resnet50 it is 3 minutes model loading + 4-5 minutes to start 1st epoch + 16s/step first epoch
For Resnet50 (image size 384x384)
it is 3 minutes model loading + + 4-5 minutes to start 1st epoch + 55s/step first epoch

I use batch size = 1 because I have OOM with 8GB videoram for image size 1248x348

small_train_dataset.zip (7.6 MB)
yolo_v4_train_resnet18_kitti.txt (2.3 KB)
yolo_v4_train_resnet50_kitti.txt (2.3 KB)
yolo_v4_train_resnet50_kitti_384x384.txt (2.3 KB)

8

Thanks for the info. I will check if I can reproduce.

9

I ran resnet18 with your shared spec file and small_train_dataset. But I cannot reproduce.
After “Epoch 1/1”, it cost about 10s to start the progress of 1 epoch.
I ran on Geforce 1080 Ti gpu card.
Could you double check? Or check NVIDIA TAO Documentation ?

10

I have the same issue here, whether with a T4 or an A100 GPU. Take 20 minutes to get started for the first epoch.

11

It seems it depends on the backbone used. YoloV4 cspdarknet53 takes the longest (20 minutes) while YoloV4 resnet18 takes about 5 minutes to load. YoloV3 with similar backbones takes less time to load.

1 Like
12

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.