when I run the command:
!tlt yolo_v4 train -e $SPECS_DIR/yolo_v4_train_resnet18_kitti.txt
-r $USER_EXPERIMENT_DIR/experiment_dir_unpruned
-k $KEY
–gpus 1
(from nvidia’s jupyter notebook)
I get this error:
To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
2021-04-19 20:38:07,133 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the ~/.tlt_mounts.json file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/init.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
2021-04-19 20:38:14,646 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/init.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/init.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2021-04-19 20:38:14,646 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/init.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:49: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
2021-04-19 20:38:14,731 [WARNING] tensorflow: From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:49: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:52: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-04-19 20:38:14,731 [WARNING] tensorflow: From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:52: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
2021-04-19 20:38:15,642 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
2021-04-19 20:38:15,645 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
2021-04-19 20:38:15,677 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
2021-04-19 20:38:16,343 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
2021-04-19 20:38:16,663 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.
2021-04-19 20:38:19,327 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
2021-04-19 20:38:19,576 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
2021-04-19 20:38:19,576 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
2021-04-19 20:38:20,661 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
2021-04-19 20:38:22,007 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.
2021-04-19 20:38:22,010 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
2021-04-19 20:38:22,747 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
2021-04-19 20:38:22,931 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.
Layer (type) Output Shape Param # Connected to
Input (InputLayer) (None, 3, 384, 1248) 0
conv1 (Conv2D) (None, 64, 192, 624) 9408 Input[0][0]
bn_conv1 (BatchNormalization) (None, 64, 192, 624) 256 conv1[0][0]
activation_2 (Activation) (None, 64, 192, 624) 0 bn_conv1[0][0]
block_1a_conv_1 (Conv2D) (None, 64, 96, 312) 36864 activation_2[0][0]
block_1a_bn_1 (BatchNormalizati (None, 64, 96, 312) 256 block_1a_conv_1[0][0]
block_1a_relu_1 (Activation) (None, 64, 96, 312) 0 block_1a_bn_1[0][0]
block_1a_conv_2 (Conv2D) (None, 64, 96, 312) 36864 block_1a_relu_1[0][0]
block_1a_conv_shortcut (Conv2D) (None, 64, 96, 312) 4096 activation_2[0][0]
block_1a_bn_2 (BatchNormalizati (None, 64, 96, 312) 256 block_1a_conv_2[0][0]
block_1a_bn_shortcut (BatchNorm (None, 64, 96, 312) 256 block_1a_conv_shortcut[0][0]
add_9 (Add) (None, 64, 96, 312) 0 block_1a_bn_2[0][0]
block_1a_bn_shortcut[0][0]
block_1a_relu (Activation) (None, 64, 96, 312) 0 add_9[0][0]
block_1b_conv_1 (Conv2D) (None, 64, 96, 312) 36864 block_1a_relu[0][0]
block_1b_bn_1 (BatchNormalizati (None, 64, 96, 312) 256 block_1b_conv_1[0][0]
block_1b_relu_1 (Activation) (None, 64, 96, 312) 0 block_1b_bn_1[0][0]
block_1b_conv_2 (Conv2D) (None, 64, 96, 312) 36864 block_1b_relu_1[0][0]
block_1b_conv_shortcut (Conv2D) (None, 64, 96, 312) 4096 block_1a_relu[0][0]
block_1b_bn_2 (BatchNormalizati (None, 64, 96, 312) 256 block_1b_conv_2[0][0]
block_1b_bn_shortcut (BatchNorm (None, 64, 96, 312) 256 block_1b_conv_shortcut[0][0]
add_10 (Add) (None, 64, 96, 312) 0 block_1b_bn_2[0][0]
block_1b_bn_shortcut[0][0]
block_1b_relu (Activation) (None, 64, 96, 312) 0 add_10[0][0]
block_2a_conv_1 (Conv2D) (None, 128, 48, 156) 73728 block_1b_relu[0][0]
block_2a_bn_1 (BatchNormalizati (None, 128, 48, 156) 512 block_2a_conv_1[0][0]
block_2a_relu_1 (Activation) (None, 128, 48, 156) 0 block_2a_bn_1[0][0]
block_2a_conv_2 (Conv2D) (None, 128, 48, 156) 147456 block_2a_relu_1[0][0]
block_2a_conv_shortcut (Conv2D) (None, 128, 48, 156) 8192 block_1b_relu[0][0]
block_2a_bn_2 (BatchNormalizati (None, 128, 48, 156) 512 block_2a_conv_2[0][0]
block_2a_bn_shortcut (BatchNorm (None, 128, 48, 156) 512 block_2a_conv_shortcut[0][0]
add_11 (Add) (None, 128, 48, 156) 0 block_2a_bn_2[0][0]
block_2a_bn_shortcut[0][0]
block_2a_relu (Activation) (None, 128, 48, 156) 0 add_11[0][0]
block_2b_conv_1 (Conv2D) (None, 128, 48, 156) 147456 block_2a_relu[0][0]
block_2b_bn_1 (BatchNormalizati (None, 128, 48, 156) 512 block_2b_conv_1[0][0]
block_2b_relu_1 (Activation) (None, 128, 48, 156) 0 block_2b_bn_1[0][0]
block_2b_conv_2 (Conv2D) (None, 128, 48, 156) 147456 block_2b_relu_1[0][0]
block_2b_conv_shortcut (Conv2D) (None, 128, 48, 156) 16384 block_2a_relu[0][0]
block_2b_bn_2 (BatchNormalizati (None, 128, 48, 156) 512 block_2b_conv_2[0][0]
block_2b_bn_shortcut (BatchNorm (None, 128, 48, 156) 512 block_2b_conv_shortcut[0][0]
add_12 (Add) (None, 128, 48, 156) 0 block_2b_bn_2[0][0]
block_2b_bn_shortcut[0][0]
block_2b_relu (Activation) (None, 128, 48, 156) 0 add_12[0][0]
block_3a_conv_1 (Conv2D) (None, 256, 24, 78) 294912 block_2b_relu[0][0]
block_3a_bn_1 (BatchNormalizati (None, 256, 24, 78) 1024 block_3a_conv_1[0][0]
block_3a_relu_1 (Activation) (None, 256, 24, 78) 0 block_3a_bn_1[0][0]
block_3a_conv_2 (Conv2D) (None, 256, 24, 78) 589824 block_3a_relu_1[0][0]
block_3a_conv_shortcut (Conv2D) (None, 256, 24, 78) 32768 block_2b_relu[0][0]
block_3a_bn_2 (BatchNormalizati (None, 256, 24, 78) 1024 block_3a_conv_2[0][0]
block_3a_bn_shortcut (BatchNorm (None, 256, 24, 78) 1024 block_3a_conv_shortcut[0][0]
add_13 (Add) (None, 256, 24, 78) 0 block_3a_bn_2[0][0]
block_3a_bn_shortcut[0][0]
block_3a_relu (Activation) (None, 256, 24, 78) 0 add_13[0][0]
block_3b_conv_1 (Conv2D) (None, 256, 24, 78) 589824 block_3a_relu[0][0]
block_3b_bn_1 (BatchNormalizati (None, 256, 24, 78) 1024 block_3b_conv_1[0][0]
block_3b_relu_1 (Activation) (None, 256, 24, 78) 0 block_3b_bn_1[0][0]
block_3b_conv_2 (Conv2D) (None, 256, 24, 78) 589824 block_3b_relu_1[0][0]
block_3b_conv_shortcut (Conv2D) (None, 256, 24, 78) 65536 block_3a_relu[0][0]
block_3b_bn_2 (BatchNormalizati (None, 256, 24, 78) 1024 block_3b_conv_2[0][0]
block_3b_bn_shortcut (BatchNorm (None, 256, 24, 78) 1024 block_3b_conv_shortcut[0][0]
add_14 (Add) (None, 256, 24, 78) 0 block_3b_bn_2[0][0]
block_3b_bn_shortcut[0][0]
block_3b_relu (Activation) (None, 256, 24, 78) 0 add_14[0][0]
block_4a_conv_1 (Conv2D) (None, 512, 24, 78) 1179648 block_3b_relu[0][0]
block_4a_bn_1 (BatchNormalizati (None, 512, 24, 78) 2048 block_4a_conv_1[0][0]
block_4a_relu_1 (Activation) (None, 512, 24, 78) 0 block_4a_bn_1[0][0]
block_4a_conv_2 (Conv2D) (None, 512, 24, 78) 2359296 block_4a_relu_1[0][0]
block_4a_conv_shortcut (Conv2D) (None, 512, 24, 78) 131072 block_3b_relu[0][0]
block_4a_bn_2 (BatchNormalizati (None, 512, 24, 78) 2048 block_4a_conv_2[0][0]
block_4a_bn_shortcut (BatchNorm (None, 512, 24, 78) 2048 block_4a_conv_shortcut[0][0]
add_15 (Add) (None, 512, 24, 78) 0 block_4a_bn_2[0][0]
block_4a_bn_shortcut[0][0]
block_4a_relu (Activation) (None, 512, 24, 78) 0 add_15[0][0]
block_4b_conv_1 (Conv2D) (None, 512, 24, 78) 2359296 block_4a_relu[0][0]
block_4b_bn_1 (BatchNormalizati (None, 512, 24, 78) 2048 block_4b_conv_1[0][0]
block_4b_relu_1 (Activation) (None, 512, 24, 78) 0 block_4b_bn_1[0][0]
block_4b_conv_2 (Conv2D) (None, 512, 24, 78) 2359296 block_4b_relu_1[0][0]
block_4b_conv_shortcut (Conv2D) (None, 512, 24, 78) 262144 block_4a_relu[0][0]
block_4b_bn_2 (BatchNormalizati (None, 512, 24, 78) 2048 block_4b_conv_2[0][0]
block_4b_bn_shortcut (BatchNorm (None, 512, 24, 78) 2048 block_4b_conv_shortcut[0][0]
add_16 (Add) (None, 512, 24, 78) 0 block_4b_bn_2[0][0]
block_4b_bn_shortcut[0][0]
block_4b_relu (Activation) (None, 512, 24, 78) 0 add_16[0][0]
yolo_spp_pool_1 (MaxPooling2D) (None, 512, 24, 78) 0 block_4b_relu[0][0]
yolo_spp_pool_2 (MaxPooling2D) (None, 512, 24, 78) 0 block_4b_relu[0][0]
yolo_spp_pool_3 (MaxPooling2D) (None, 512, 24, 78) 0 block_4b_relu[0][0]
yolo_spp_concat (Concatenate) (None, 2048, 24, 78) 0 yolo_spp_pool_1[0][0]
yolo_spp_pool_2[0][0]
yolo_spp_pool_3[0][0]
block_4b_relu[0][0]
yolo_spp_conv (Conv2D) (None, 512, 24, 78) 1048576 yolo_spp_concat[0][0]
yolo_spp_conv_bn (BatchNormaliz (None, 512, 24, 78) 2048 yolo_spp_conv[0][0]
yolo_spp_conv_lrelu (LeakyReLU) (None, 512, 24, 78) 0 yolo_spp_conv_bn[0][0]
yolo_expand_conv1 (Conv2D) (None, 512, 12, 39) 2359296 yolo_spp_conv_lrelu[0][0]
yolo_expand_conv1_bn (BatchNorm (None, 512, 12, 39) 2048 yolo_expand_conv1[0][0]
yolo_expand_conv1_lrelu (LeakyR (None, 512, 12, 39) 0 yolo_expand_conv1_bn[0][0]
yolo_conv1_1 (Conv2D) (None, 256, 12, 39) 131072 yolo_expand_conv1_lrelu[0][0]
yolo_conv1_1_bn (BatchNormaliza (None, 256, 12, 39) 1024 yolo_conv1_1[0][0]
yolo_conv1_1_lrelu (LeakyReLU) (None, 256, 12, 39) 0 yolo_conv1_1_bn[0][0]
yolo_conv1_2 (Conv2D) (None, 512, 12, 39) 1179648 yolo_conv1_1_lrelu[0][0]
yolo_conv1_2_bn (BatchNormaliza (None, 512, 12, 39) 2048 yolo_conv1_2[0][0]
yolo_conv1_2_lrelu (LeakyReLU) (None, 512, 12, 39) 0 yolo_conv1_2_bn[0][0]
yolo_conv1_3 (Conv2D) (None, 256, 12, 39) 131072 yolo_conv1_2_lrelu[0][0]
yolo_conv1_3_bn (BatchNormaliza (None, 256, 12, 39) 1024 yolo_conv1_3[0][0]
yolo_conv1_3_lrelu (LeakyReLU) (None, 256, 12, 39) 0 yolo_conv1_3_bn[0][0]
yolo_conv1_4 (Conv2D) (None, 512, 12, 39) 1179648 yolo_conv1_3_lrelu[0][0]
yolo_conv1_4_bn (BatchNormaliza (None, 512, 12, 39) 2048 yolo_conv1_4[0][0]
yolo_conv1_4_lrelu (LeakyReLU) (None, 512, 12, 39) 0 yolo_conv1_4_bn[0][0]
yolo_conv1_5 (Conv2D) (None, 256, 12, 39) 131072 yolo_conv1_4_lrelu[0][0]
yolo_conv1_5_bn (BatchNormaliza (None, 256, 12, 39) 1024 yolo_conv1_5[0][0]
yolo_conv1_5_lrelu (LeakyReLU) (None, 256, 12, 39) 0 yolo_conv1_5_bn[0][0]
yolo_conv2 (Conv2D) (None, 128, 12, 39) 32768 yolo_conv1_5_lrelu[0][0]
yolo_conv2_bn (BatchNormalizati (None, 128, 12, 39) 512 yolo_conv2[0][0]
yolo_conv2_lrelu (LeakyReLU) (None, 128, 12, 39) 0 yolo_conv2_bn[0][0]
upsample0 (UpSampling2D) (None, 128, 24, 78) 0 yolo_conv2_lrelu[0][0]
concatenate_3 (Concatenate) (None, 384, 24, 78) 0 upsample0[0][0]
block_3b_relu[0][0]
yolo_conv3_1 (Conv2D) (None, 128, 24, 78) 49152 concatenate_3[0][0]
yolo_conv3_1_bn (BatchNormaliza (None, 128, 24, 78) 512 yolo_conv3_1[0][0]
yolo_conv3_1_lrelu (LeakyReLU) (None, 128, 24, 78) 0 yolo_conv3_1_bn[0][0]
yolo_conv3_2 (Conv2D) (None, 256, 24, 78) 294912 yolo_conv3_1_lrelu[0][0]
yolo_conv3_2_bn (BatchNormaliza (None, 256, 24, 78) 1024 yolo_conv3_2[0][0]
yolo_conv3_2_lrelu (LeakyReLU) (None, 256, 24, 78) 0 yolo_conv3_2_bn[0][0]
yolo_conv3_3 (Conv2D) (None, 128, 24, 78) 32768 yolo_conv3_2_lrelu[0][0]
yolo_conv3_3_bn (BatchNormaliza (None, 128, 24, 78) 512 yolo_conv3_3[0][0]
yolo_conv3_3_lrelu (LeakyReLU) (None, 128, 24, 78) 0 yolo_conv3_3_bn[0][0]
yolo_conv3_4 (Conv2D) (None, 256, 24, 78) 294912 yolo_conv3_3_lrelu[0][0]
yolo_conv3_4_bn (BatchNormaliza (None, 256, 24, 78) 1024 yolo_conv3_4[0][0]
yolo_conv3_4_lrelu (LeakyReLU) (None, 256, 24, 78) 0 yolo_conv3_4_bn[0][0]
yolo_conv3_5 (Conv2D) (None, 128, 24, 78) 32768 yolo_conv3_4_lrelu[0][0]
yolo_conv3_5_bn (BatchNormaliza (None, 128, 24, 78) 512 yolo_conv3_5[0][0]
yolo_conv3_5_lrelu (LeakyReLU) (None, 128, 24, 78) 0 yolo_conv3_5_bn[0][0]
yolo_conv4 (Conv2D) (None, 64, 24, 78) 8192 yolo_conv3_5_lrelu[0][0]
yolo_conv4_bn (BatchNormalizati (None, 64, 24, 78) 256 yolo_conv4[0][0]
yolo_conv4_lrelu (LeakyReLU) (None, 64, 24, 78) 0 yolo_conv4_bn[0][0]
upsample1 (UpSampling2D) (None, 64, 48, 156) 0 yolo_conv4_lrelu[0][0]
concatenate_4 (Concatenate) (None, 192, 48, 156) 0 upsample1[0][0]
block_2b_relu[0][0]
yolo_conv5_1 (Conv2D) (None, 64, 48, 156) 12288 concatenate_4[0][0]
yolo_conv5_1_bn (BatchNormaliza (None, 64, 48, 156) 256 yolo_conv5_1[0][0]
yolo_conv5_1_lrelu (LeakyReLU) (None, 64, 48, 156) 0 yolo_conv5_1_bn[0][0]
yolo_conv5_2 (Conv2D) (None, 128, 48, 156) 73728 yolo_conv5_1_lrelu[0][0]
yolo_conv5_2_bn (BatchNormaliza (None, 128, 48, 156) 512 yolo_conv5_2[0][0]
yolo_conv5_2_lrelu (LeakyReLU) (None, 128, 48, 156) 0 yolo_conv5_2_bn[0][0]
yolo_conv5_3 (Conv2D) (None, 64, 48, 156) 8192 yolo_conv5_2_lrelu[0][0]
yolo_conv5_3_bn (BatchNormaliza (None, 64, 48, 156) 256 yolo_conv5_3[0][0]
yolo_conv5_3_lrelu (LeakyReLU) (None, 64, 48, 156) 0 yolo_conv5_3_bn[0][0]
yolo_conv5_4 (Conv2D) (None, 128, 48, 156) 73728 yolo_conv5_3_lrelu[0][0]
yolo_conv5_4_bn (BatchNormaliza (None, 128, 48, 156) 512 yolo_conv5_4[0][0]
yolo_conv5_4_lrelu (LeakyReLU) (None, 128, 48, 156) 0 yolo_conv5_4_bn[0][0]
yolo_conv5_5 (Conv2D) (None, 64, 48, 156) 8192 yolo_conv5_4_lrelu[0][0]
yolo_conv5_5_bn (BatchNormaliza (None, 64, 48, 156) 256 yolo_conv5_5[0][0]
yolo_conv5_5_lrelu (LeakyReLU) (None, 64, 48, 156) 0 yolo_conv5_5_bn[0][0]
yolo_conv1_6 (Conv2D) (None, 512, 12, 39) 1179648 yolo_conv1_5_lrelu[0][0]
yolo_conv3_6 (Conv2D) (None, 256, 24, 78) 294912 yolo_conv3_5_lrelu[0][0]
yolo_conv5_6 (Conv2D) (None, 128, 48, 156) 73728 yolo_conv5_5_lrelu[0][0]
yolo_conv1_6_bn (BatchNormaliza (None, 512, 12, 39) 2048 yolo_conv1_6[0][0]
yolo_conv3_6_bn (BatchNormaliza (None, 256, 24, 78) 1024 yolo_conv3_6[0][0]
yolo_conv5_6_bn (BatchNormaliza (None, 128, 48, 156) 512 yolo_conv5_6[0][0]
yolo_conv1_6_lrelu (LeakyReLU) (None, 512, 12, 39) 0 yolo_conv1_6_bn[0][0]
yolo_conv3_6_lrelu (LeakyReLU) (None, 256, 24, 78) 0 yolo_conv3_6_bn[0][0]
yolo_conv5_6_lrelu (LeakyReLU) (None, 128, 48, 156) 0 yolo_conv5_6_bn[0][0]
conv_big_object (Conv2D) (None, 24, 12, 39) 12312 yolo_conv1_6_lrelu[0][0]
conv_mid_object (Conv2D) (None, 24, 24, 78) 6168 yolo_conv3_6_lrelu[0][0]
conv_sm_object (Conv2D) (None, 24, 48, 156) 3096 yolo_conv5_6_lrelu[0][0]
bg_permute (Permute) (None, 12, 39, 24) 0 conv_big_object[0][0]
md_permute (Permute) (None, 24, 78, 24) 0 conv_mid_object[0][0]
sm_permute (Permute) (None, 48, 156, 24) 0 conv_sm_object[0][0]
bg_reshape (Reshape) (None, 1404, 8) 0 bg_permute[0][0]
md_reshape (Reshape) (None, 5616, 8) 0 md_permute[0][0]
sm_reshape (Reshape) (None, 22464, 8) 0 sm_permute[0][0]
bg_anchor (YOLOAnchorBox) (None, 1404, 6) 0 conv_big_object[0][0]
bg_bbox_processor (BBoxPostProc (None, 1404, 8) 0 bg_reshape[0][0]
md_anchor (YOLOAnchorBox) (None, 5616, 6) 0 conv_mid_object[0][0]
md_bbox_processor (BBoxPostProc (None, 5616, 8) 0 md_reshape[0][0]
sm_anchor (YOLOAnchorBox) (None, 22464, 6) 0 conv_sm_object[0][0]
sm_bbox_processor (BBoxPostProc (None, 22464, 8) 0 sm_reshape[0][0]
encoded_bg (Concatenate) (None, 1404, 14) 0 bg_anchor[0][0]
bg_bbox_processor[0][0]
encoded_md (Concatenate) (None, 5616, 14) 0 md_anchor[0][0]
md_bbox_processor[0][0]
encoded_sm (Concatenate) (None, 22464, 14) 0 sm_anchor[0][0]
sm_bbox_processor[0][0]
encoded_detections (Concatenate (None, 29484, 14) 0 encoded_bg[0][0]
encoded_md[0][0]
encoded_sm[0][0]
Total params: 20,215,304
Trainable params: 20,193,160
Non-trainable params: 22,144
2021-04-19 20:39:05,392 [INFO] main: Number of images in the training dataset: 7481
Epoch 1/80
Traceback (most recent call last):
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py”, line 209, in
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py”, line 205, in main
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py”, line 162, in run_experiment
File “/usr/local/lib/python3.6/dist-packages/keras/legacy/interfaces.py”, line 91, in wrapper
return func(*args, **kwargs)
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 1418, in fit_generator
initial_epoch=initial_epoch)
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training_generator.py”, line 217, in fit_generator
class_weight=class_weight)
File “/usr/local/lib/python3.6/dist-packages/keras/engine/training.py”, line 1217, in train_on_batch
outputs = self.train_function(ins)
File “/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py”, line 2715, in call
return self._call(inputs)
File “/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py”, line 2675, in _call
fetched = self._callable_fn(*array_vals)
File “/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py”, line 1472, in call
run_metadata_ptr)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: OOM when allocating tensor with shape[8,128,48,156] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node yolo_conv5_6_lrelu_1/LeakyRelu}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[loss_1/add_49/_4291]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
(1) Resource exhausted: OOM when allocating tensor with shape[8,128,48,156] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node yolo_conv5_6_lrelu_1/LeakyRelu}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
0 successful operations.
0 derived errors ignored.
Traceback (most recent call last):
File “/usr/local/bin/yolo_v4”, line 8, in
sys.exit(main())
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/entrypoint/yolo_v4.py”, line 12, in main
File “/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/entrypoint/entrypoint.py”, line 296, in launch_job
AssertionError: Process run failed.
2021-04-19 20:42:07,249 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.