I attached the training spec in my previous post…did you mean my retrain spec? They are actually the same, just that in my “Training Spec” I am using a previously trained, unpruned model as the pretrained_model_file and in the “Retrain spec” I am using the resulting pruned model as the pretrained_model_file.
Here are the situations I’ve tried:
- Train with training spec where I use the pretrained Mobilenet V2 model from the NGC model registry (
.hdf5filetype). The first 50 epochs all have 0% mAP, and eventually I was able to achieve 94% mAP after several hours of training. - Prune the model
- Retrain the pruned model. The first 50 epochs are all back at 0% mAP and it takes several hours again to get back up to 94% mAP
I’ve also tried:
- Train with training spec where I use my own previously trained model, this time it is a
.tltfile. Again, the first 50 or so epochs are back to 0% mAP, even though I can confirm that thistltmodel used as thepretrained_model_filehas an mAP of around 94% when I evaluate it separately. I was eventually able to achieve around 94% mAP after several hours of training. - Prune the model
- Retrain the pruned model. The first 50 epochs are, again, all back at 0% mAP and it takes several hours again to get back up to 94% mAP.
My theory is that the weights aren’t actually being initialized, but I cannot confirm this by looking at the logs during training.
Here is the training log, up to the first validation checkpoint, for the situation where I am trying to train an unpruned model using my previously trained model .tlt file that has a 94% mAP:
Using TensorFlow backend.
2020-03-20 18:15:14.574814: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-03-20 18:15:14.668492: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-03-20 18:15:14.668915: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x663eeb0 executing computations on platform CUDA. Devices:
2020-03-20 18:15:14.668937: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce GTX 1070 Ti, Compute Capability 6.1
2020-03-20 18:15:14.670438: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2020-03-20 18:15:14.670735: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x66a9060 executing computations on platform Host. Devices:
2020-03-20 18:15:14.670753: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2020-03-20 18:15:14.670908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce GTX 1070 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.33GiB
2020-03-20 18:15:14.670926: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-03-20 18:15:14.671460: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-20 18:15:14.671473: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2020-03-20 18:15:14.671483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2020-03-20 18:15:14.671613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7129 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-03-20 18:15:14,672 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /workspace/examples/detectnet_v2/specs/detectnet_v2_train_mobilenet_v2_kitti.txt.
2020-03-20 18:15:14,672 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/examples/detectnet_v2/specs/detectnet_v2_train_mobilenet_v2_kitti.txt
WARNING:tensorflow:From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
2020-03-20 18:15:14,678 [WARNING] tensorflow: From ./detectnet_v2/dataloader/utilities.py:114: tf_record_iterator (from tensorflow.python.lib.io.tf_record) is deprecated and will be removed in a future version.
Instructions for updating:
Use eager execution and:
`tf.data.TFRecordDataset(path)`
2020-03-20 18:15:14,712 [INFO] iva.detectnet_v2.scripts.train: Cannot iterate over exactly 394 samples with a batch size of 4; each epoch will therefore take one extra step.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2020-03-20 18:15:14,716 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/__init__.py:91: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
2020-03-20 18:15:14,725 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/horovod/tensorflow/__init__.py:91: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Deprecated in favor of operator or tf.math.divide.
/usr/local/lib/python2.7/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
warnings.warn('No training configuration found in save file: '
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, 3, 720, 1280) 0
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D) (None, 3, 722, 1282) 0 input_1[0][0]
__________________________________________________________________________________________________
conv1 (Conv2D) (None, 32, 360, 640) 864 conv1_pad[0][0]
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization) (None, 32, 360, 640) 128 conv1[0][0]
__________________________________________________________________________________________________
re_lu_1 (ReLU) (None, 32, 360, 640) 0 bn_conv1[0][0]
__________________________________________________________________________________________________
expanded_conv_depthwise_pad (Ze (None, 32, 362, 642) 0 re_lu_1[0][0]
__________________________________________________________________________________________________
expanded_conv_depthwise (Depthw (None, 32, 360, 640) 288 expanded_conv_depthwise_pad[0][0]
__________________________________________________________________________________________________
expanded_conv_depthwise_bn (Bat (None, 32, 360, 640) 128 expanded_conv_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_2 (ReLU) (None, 32, 360, 640) 0 expanded_conv_depthwise_bn[0][0]
__________________________________________________________________________________________________
expanded_conv_project (Conv2D) (None, 16, 360, 640) 512 re_lu_2[0][0]
__________________________________________________________________________________________________
expanded_conv_project_bn (Batch (None, 16, 360, 640) 64 expanded_conv_project[0][0]
__________________________________________________________________________________________________
block_1_expand (Conv2D) (None, 96, 360, 640) 1536 expanded_conv_project_bn[0][0]
__________________________________________________________________________________________________
block_1_expand_bn (BatchNormali (None, 96, 360, 640) 384 block_1_expand[0][0]
__________________________________________________________________________________________________
re_lu_3 (ReLU) (None, 96, 360, 640) 0 block_1_expand_bn[0][0]
__________________________________________________________________________________________________
block_1_depthwise_pad (ZeroPadd (None, 96, 362, 642) 0 re_lu_3[0][0]
__________________________________________________________________________________________________
block_1_depthwise (DepthwiseCon (None, 96, 180, 320) 864 block_1_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_1_depthwise_bn (BatchNorm (None, 96, 180, 320) 384 block_1_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_4 (ReLU) (None, 96, 180, 320) 0 block_1_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_1_project (Conv2D) (None, 24, 180, 320) 2304 re_lu_4[0][0]
__________________________________________________________________________________________________
block_1_project_bn (BatchNormal (None, 24, 180, 320) 96 block_1_project[0][0]
__________________________________________________________________________________________________
block_2_expand (Conv2D) (None, 144, 180, 320 3456 block_1_project_bn[0][0]
__________________________________________________________________________________________________
block_2_expand_bn (BatchNormali (None, 144, 180, 320 576 block_2_expand[0][0]
__________________________________________________________________________________________________
re_lu_5 (ReLU) (None, 144, 180, 320 0 block_2_expand_bn[0][0]
__________________________________________________________________________________________________
block_2_depthwise_pad (ZeroPadd (None, 144, 182, 322 0 re_lu_5[0][0]
__________________________________________________________________________________________________
block_2_depthwise (DepthwiseCon (None, 144, 180, 320 1296 block_2_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_2_depthwise_bn (BatchNorm (None, 144, 180, 320 576 block_2_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_6 (ReLU) (None, 144, 180, 320 0 block_2_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_2_project (Conv2D) (None, 24, 180, 320) 3456 re_lu_6[0][0]
__________________________________________________________________________________________________
block_2_project_bn (BatchNormal (None, 24, 180, 320) 96 block_2_project[0][0]
__________________________________________________________________________________________________
block_2_add (Add) (None, 24, 180, 320) 0 block_1_project_bn[0][0]
block_2_project_bn[0][0]
__________________________________________________________________________________________________
block_3_expand (Conv2D) (None, 144, 180, 320 3456 block_2_add[0][0]
__________________________________________________________________________________________________
block_3_expand_bn (BatchNormali (None, 144, 180, 320 576 block_3_expand[0][0]
__________________________________________________________________________________________________
re_lu_7 (ReLU) (None, 144, 180, 320 0 block_3_expand_bn[0][0]
__________________________________________________________________________________________________
block_3_depthwise_pad (ZeroPadd (None, 144, 182, 322 0 re_lu_7[0][0]
__________________________________________________________________________________________________
block_3_depthwise (DepthwiseCon (None, 144, 90, 160) 1296 block_3_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_3_depthwise_bn (BatchNorm (None, 144, 90, 160) 576 block_3_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_8 (ReLU) (None, 144, 90, 160) 0 block_3_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_3_project (Conv2D) (None, 32, 90, 160) 4608 re_lu_8[0][0]
__________________________________________________________________________________________________
block_3_project_bn (BatchNormal (None, 32, 90, 160) 128 block_3_project[0][0]
__________________________________________________________________________________________________
block_4_expand (Conv2D) (None, 192, 90, 160) 6144 block_3_project_bn[0][0]
__________________________________________________________________________________________________
block_4_expand_bn (BatchNormali (None, 192, 90, 160) 768 block_4_expand[0][0]
__________________________________________________________________________________________________
re_lu_9 (ReLU) (None, 192, 90, 160) 0 block_4_expand_bn[0][0]
__________________________________________________________________________________________________
block_4_depthwise_pad (ZeroPadd (None, 192, 92, 162) 0 re_lu_9[0][0]
__________________________________________________________________________________________________
block_4_depthwise (DepthwiseCon (None, 192, 90, 160) 1728 block_4_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_4_depthwise_bn (BatchNorm (None, 192, 90, 160) 768 block_4_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_10 (ReLU) (None, 192, 90, 160) 0 block_4_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_4_project (Conv2D) (None, 32, 90, 160) 6144 re_lu_10[0][0]
__________________________________________________________________________________________________
block_4_project_bn (BatchNormal (None, 32, 90, 160) 128 block_4_project[0][0]
__________________________________________________________________________________________________
block_4_add (Add) (None, 32, 90, 160) 0 block_3_project_bn[0][0]
block_4_project_bn[0][0]
__________________________________________________________________________________________________
block_5_expand (Conv2D) (None, 192, 90, 160) 6144 block_4_add[0][0]
__________________________________________________________________________________________________
block_5_expand_bn (BatchNormali (None, 192, 90, 160) 768 block_5_expand[0][0]
__________________________________________________________________________________________________
re_lu_11 (ReLU) (None, 192, 90, 160) 0 block_5_expand_bn[0][0]
__________________________________________________________________________________________________
block_5_depthwise_pad (ZeroPadd (None, 192, 92, 162) 0 re_lu_11[0][0]
__________________________________________________________________________________________________
block_5_depthwise (DepthwiseCon (None, 192, 90, 160) 1728 block_5_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_5_depthwise_bn (BatchNorm (None, 192, 90, 160) 768 block_5_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_12 (ReLU) (None, 192, 90, 160) 0 block_5_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_5_project (Conv2D) (None, 32, 90, 160) 6144 re_lu_12[0][0]
__________________________________________________________________________________________________
block_5_project_bn (BatchNormal (None, 32, 90, 160) 128 block_5_project[0][0]
__________________________________________________________________________________________________
block_5_add (Add) (None, 32, 90, 160) 0 block_4_add[0][0]
block_5_project_bn[0][0]
__________________________________________________________________________________________________
block_6_expand (Conv2D) (None, 192, 90, 160) 6144 block_5_add[0][0]
__________________________________________________________________________________________________
block_6_expand_bn (BatchNormali (None, 192, 90, 160) 768 block_6_expand[0][0]
__________________________________________________________________________________________________
re_lu_13 (ReLU) (None, 192, 90, 160) 0 block_6_expand_bn[0][0]
__________________________________________________________________________________________________
block_6_depthwise_pad (ZeroPadd (None, 192, 92, 162) 0 re_lu_13[0][0]
__________________________________________________________________________________________________
block_6_depthwise (DepthwiseCon (None, 192, 45, 80) 1728 block_6_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_6_depthwise_bn (BatchNorm (None, 192, 45, 80) 768 block_6_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_14 (ReLU) (None, 192, 45, 80) 0 block_6_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_6_project (Conv2D) (None, 64, 45, 80) 12288 re_lu_14[0][0]
__________________________________________________________________________________________________
block_6_project_bn (BatchNormal (None, 64, 45, 80) 256 block_6_project[0][0]
__________________________________________________________________________________________________
block_7_expand (Conv2D) (None, 384, 45, 80) 24576 block_6_project_bn[0][0]
__________________________________________________________________________________________________
block_7_expand_bn (BatchNormali (None, 384, 45, 80) 1536 block_7_expand[0][0]
__________________________________________________________________________________________________
re_lu_15 (ReLU) (None, 384, 45, 80) 0 block_7_expand_bn[0][0]
__________________________________________________________________________________________________
block_7_depthwise_pad (ZeroPadd (None, 384, 47, 82) 0 re_lu_15[0][0]
__________________________________________________________________________________________________
block_7_depthwise (DepthwiseCon (None, 384, 45, 80) 3456 block_7_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_7_depthwise_bn (BatchNorm (None, 384, 45, 80) 1536 block_7_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_16 (ReLU) (None, 384, 45, 80) 0 block_7_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_7_project (Conv2D) (None, 64, 45, 80) 24576 re_lu_16[0][0]
__________________________________________________________________________________________________
block_7_project_bn (BatchNormal (None, 64, 45, 80) 256 block_7_project[0][0]
__________________________________________________________________________________________________
block_7_add (Add) (None, 64, 45, 80) 0 block_6_project_bn[0][0]
block_7_project_bn[0][0]
__________________________________________________________________________________________________
block_8_expand (Conv2D) (None, 384, 45, 80) 24576 block_7_add[0][0]
__________________________________________________________________________________________________
block_8_expand_bn (BatchNormali (None, 384, 45, 80) 1536 block_8_expand[0][0]
__________________________________________________________________________________________________
re_lu_17 (ReLU) (None, 384, 45, 80) 0 block_8_expand_bn[0][0]
__________________________________________________________________________________________________
block_8_depthwise_pad (ZeroPadd (None, 384, 47, 82) 0 re_lu_17[0][0]
__________________________________________________________________________________________________
block_8_depthwise (DepthwiseCon (None, 384, 45, 80) 3456 block_8_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_8_depthwise_bn (BatchNorm (None, 384, 45, 80) 1536 block_8_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_18 (ReLU) (None, 384, 45, 80) 0 block_8_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_8_project (Conv2D) (None, 64, 45, 80) 24576 re_lu_18[0][0]
__________________________________________________________________________________________________
block_8_project_bn (BatchNormal (None, 64, 45, 80) 256 block_8_project[0][0]
__________________________________________________________________________________________________
block_8_add (Add) (None, 64, 45, 80) 0 block_7_add[0][0]
block_8_project_bn[0][0]
__________________________________________________________________________________________________
block_9_expand (Conv2D) (None, 384, 45, 80) 24576 block_8_add[0][0]
__________________________________________________________________________________________________
block_9_expand_bn (BatchNormali (None, 384, 45, 80) 1536 block_9_expand[0][0]
__________________________________________________________________________________________________
re_lu_19 (ReLU) (None, 384, 45, 80) 0 block_9_expand_bn[0][0]
__________________________________________________________________________________________________
block_9_depthwise_pad (ZeroPadd (None, 384, 47, 82) 0 re_lu_19[0][0]
__________________________________________________________________________________________________
block_9_depthwise (DepthwiseCon (None, 384, 45, 80) 3456 block_9_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_9_depthwise_bn (BatchNorm (None, 384, 45, 80) 1536 block_9_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_20 (ReLU) (None, 384, 45, 80) 0 block_9_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_9_project (Conv2D) (None, 64, 45, 80) 24576 re_lu_20[0][0]
__________________________________________________________________________________________________
block_9_project_bn (BatchNormal (None, 64, 45, 80) 256 block_9_project[0][0]
__________________________________________________________________________________________________
block_9_add (Add) (None, 64, 45, 80) 0 block_8_add[0][0]
block_9_project_bn[0][0]
__________________________________________________________________________________________________
block_10_expand (Conv2D) (None, 384, 45, 80) 24576 block_9_add[0][0]
__________________________________________________________________________________________________
block_10_expand_bn (BatchNormal (None, 384, 45, 80) 1536 block_10_expand[0][0]
__________________________________________________________________________________________________
re_lu_21 (ReLU) (None, 384, 45, 80) 0 block_10_expand_bn[0][0]
__________________________________________________________________________________________________
block_10_depthwise_pad (ZeroPad (None, 384, 47, 82) 0 re_lu_21[0][0]
__________________________________________________________________________________________________
block_10_depthwise (DepthwiseCo (None, 384, 45, 80) 3456 block_10_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_10_depthwise_bn (BatchNor (None, 384, 45, 80) 1536 block_10_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_22 (ReLU) (None, 384, 45, 80) 0 block_10_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_10_project (Conv2D) (None, 96, 45, 80) 36864 re_lu_22[0][0]
__________________________________________________________________________________________________
block_10_project_bn (BatchNorma (None, 96, 45, 80) 384 block_10_project[0][0]
__________________________________________________________________________________________________
block_11_expand (Conv2D) (None, 576, 45, 80) 55296 block_10_project_bn[0][0]
__________________________________________________________________________________________________
block_11_expand_bn (BatchNormal (None, 576, 45, 80) 2304 block_11_expand[0][0]
__________________________________________________________________________________________________
re_lu_23 (ReLU) (None, 576, 45, 80) 0 block_11_expand_bn[0][0]
__________________________________________________________________________________________________
block_11_depthwise_pad (ZeroPad (None, 576, 47, 82) 0 re_lu_23[0][0]
__________________________________________________________________________________________________
block_11_depthwise (DepthwiseCo (None, 576, 45, 80) 5184 block_11_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_11_depthwise_bn (BatchNor (None, 576, 45, 80) 2304 block_11_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_24 (ReLU) (None, 576, 45, 80) 0 block_11_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_11_project (Conv2D) (None, 96, 45, 80) 55296 re_lu_24[0][0]
__________________________________________________________________________________________________
block_11_project_bn (BatchNorma (None, 96, 45, 80) 384 block_11_project[0][0]
__________________________________________________________________________________________________
block_11_add (Add) (None, 96, 45, 80) 0 block_10_project_bn[0][0]
block_11_project_bn[0][0]
__________________________________________________________________________________________________
block_12_expand (Conv2D) (None, 576, 45, 80) 55296 block_11_add[0][0]
__________________________________________________________________________________________________
block_12_expand_bn (BatchNormal (None, 576, 45, 80) 2304 block_12_expand[0][0]
__________________________________________________________________________________________________
re_lu_25 (ReLU) (None, 576, 45, 80) 0 block_12_expand_bn[0][0]
__________________________________________________________________________________________________
block_12_depthwise_pad (ZeroPad (None, 576, 47, 82) 0 re_lu_25[0][0]
__________________________________________________________________________________________________
block_12_depthwise (DepthwiseCo (None, 576, 45, 80) 5184 block_12_depthwise_pad[0][0]
__________________________________________________________________________________________________
block_12_depthwise_bn (BatchNor (None, 576, 45, 80) 2304 block_12_depthwise[0][0]
__________________________________________________________________________________________________
re_lu_26 (ReLU) (None, 576, 45, 80) 0 block_12_depthwise_bn[0][0]
__________________________________________________________________________________________________
block_12_project (Conv2D) (None, 96, 45, 80) 55296 re_lu_26[0][0]
__________________________________________________________________________________________________
block_12_project_bn (BatchNorma (None, 96, 45, 80) 384 block_12_project[0][0]
__________________________________________________________________________________________________
block_12_add (Add) (None, 96, 45, 80) 0 block_11_add[0][0]
block_12_project_bn[0][0]
__________________________________________________________________________________________________
output_bbox (Conv2D) (None, 20, 45, 80) 1940 block_12_add[0][0]
__________________________________________________________________________________________________
output_cov (Conv2D) (None, 5, 45, 80) 485 block_12_add[0][0]
==================================================================================================
Total params: 561,081
Trainable params: 544,953
Non-trainable params: 16,128
__________________________________________________________________________________________________
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2020-03-20 18:15:18,274 [INFO] iva.detectnet_v2.scripts.train: Found 394 samples in training set
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
target/truncation is not updated to match the crop areaif the dataset contains target/truncation.
2020-03-20 18:15:23,069 [INFO] iva.detectnet_v2.scripts.train: Found 63 samples in validation set
INFO:tensorflow:Create CheckpointSaverHook.
2020-03-20 18:15:26,773 [INFO] tensorflow: Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.
2020-03-20 18:15:27,678 [INFO] tensorflow: Graph was finalized.
2020-03-20 18:15:27.678562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2020-03-20 18:15:27.678589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-03-20 18:15:27.678596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2020-03-20 18:15:27.678603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2020-03-20 18:15:27.678741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 7129 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1070 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Running local_init_op.
2020-03-20 18:15:28,990 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2020-03-20 18:15:29,211 [INFO] tensorflow: Done running local_init_op.
INFO:tensorflow:Saving checkpoints for step-0.
2020-03-20 18:15:36,307 [INFO] tensorflow: Saving checkpoints for step-0.
2020-03-20 18:16:01.212227: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2020-03-20 18:16:01.318690: I tensorflow/core/kernels/cuda_solvers.cc:159] Creating CudaSolver handles for stream 0x66e4a60
2020-03-20 18:16:02.231869: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.09GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.238245: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.280204: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.08GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.283842: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.24GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.385723: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.07GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.386907: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.24GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.389052: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.05GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.390794: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.08GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.481579: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.06GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2020-03-20 18:16:02.484193: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.40GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
INFO:tensorflow:epoch = 0.0, loss = 0.54143083, step = 0
2020-03-20 18:16:03,019 [INFO] tensorflow: epoch = 0.0, loss = 0.54143083, step = 0
2020-03-20 18:16:03,019 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 0/500: loss: 0.54143 Time taken: 0:00:00 ETA: 0:00:00
INFO:tensorflow:epoch = 0.07070707070707072, loss = 0.547261, step = 7 (7.983 sec)
2020-03-20 18:16:11,001 [INFO] tensorflow: epoch = 0.07070707070707072, loss = 0.547261, step = 7 (7.983 sec)
INFO:tensorflow:global_step/sec: 0.804866
2020-03-20 18:16:14,201 [INFO] tensorflow: global_step/sec: 0.804866
INFO:tensorflow:epoch = 0.14141414141414144, loss = 0.5353711, step = 14 (5.714 sec)
2020-03-20 18:16:16,715 [INFO] tensorflow: epoch = 0.14141414141414144, loss = 0.5353711, step = 14 (5.714 sec)
INFO:tensorflow:global_step/sec: 1.99343
2020-03-20 18:16:18,716 [INFO] tensorflow: global_step/sec: 1.99343
2020-03-20 18:16:21,823 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 4.383
INFO:tensorflow:epoch = 0.25252525252525254, loss = 0.5337452, step = 25 (5.615 sec)
2020-03-20 18:16:22,330 [INFO] tensorflow: epoch = 0.25252525252525254, loss = 0.5337452, step = 25 (5.615 sec)
INFO:tensorflow:global_step/sec: 1.95584
2020-03-20 18:16:23,317 [INFO] tensorflow: global_step/sec: 1.95584
INFO:tensorflow:epoch = 0.36363636363636365, loss = 0.52291155, step = 36 (5.520 sec)
2020-03-20 18:16:27,850 [INFO] tensorflow: epoch = 0.36363636363636365, loss = 0.52291155, step = 36 (5.520 sec)
INFO:tensorflow:global_step/sec: 1.98533
2020-03-20 18:16:27,851 [INFO] tensorflow: global_step/sec: 1.98533
INFO:tensorflow:global_step/sec: 1.96531
2020-03-20 18:16:32,430 [INFO] tensorflow: global_step/sec: 1.96531
INFO:tensorflow:epoch = 0.4747474747474748, loss = 0.5189539, step = 47 (5.587 sec)
2020-03-20 18:16:33,437 [INFO] tensorflow: epoch = 0.4747474747474748, loss = 0.5189539, step = 47 (5.587 sec)
2020-03-20 18:16:34,474 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.905
INFO:tensorflow:global_step/sec: 1.97808
2020-03-20 18:16:36,980 [INFO] tensorflow: global_step/sec: 1.97808
INFO:tensorflow:epoch = 0.5858585858585859, loss = 0.51284915, step = 58 (5.570 sec)
2020-03-20 18:16:39,007 [INFO] tensorflow: epoch = 0.5858585858585859, loss = 0.51284915, step = 58 (5.570 sec)
INFO:tensorflow:global_step/sec: 1.92838
2020-03-20 18:16:41,647 [INFO] tensorflow: global_step/sec: 1.92838
INFO:tensorflow:epoch = 0.696969696969697, loss = 0.51052964, step = 69 (5.674 sec)
2020-03-20 18:16:44,680 [INFO] tensorflow: epoch = 0.696969696969697, loss = 0.51052964, step = 69 (5.674 sec)
INFO:tensorflow:global_step/sec: 1.98563
2020-03-20 18:16:46,180 [INFO] tensorflow: global_step/sec: 1.98563
2020-03-20 18:16:47,239 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.835
INFO:tensorflow:epoch = 0.8080808080808082, loss = 0.508822, step = 80 (5.693 sec)
2020-03-20 18:16:50,374 [INFO] tensorflow: epoch = 0.8080808080808082, loss = 0.508822, step = 80 (5.693 sec)
INFO:tensorflow:global_step/sec: 1.91123
2020-03-20 18:16:50,889 [INFO] tensorflow: global_step/sec: 1.91123
INFO:tensorflow:global_step/sec: 1.92121
2020-03-20 18:16:55,573 [INFO] tensorflow: global_step/sec: 1.92121
INFO:tensorflow:epoch = 0.9191919191919192, loss = 0.50488627, step = 91 (5.703 sec)
2020-03-20 18:16:56,077 [INFO] tensorflow: epoch = 0.9191919191919192, loss = 0.50488627, step = 91 (5.703 sec)
INFO:tensorflow:global_step/sec: 1.85098
2020-03-20 18:17:00,436 [INFO] tensorflow: global_step/sec: 1.85098
2020-03-20 18:17:00,436 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 1/500: loss: 0.00096 Time taken: 0:01:00.898557 ETA: 8:26:28.379943
2020-03-20 18:17:00,436 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.577
INFO:tensorflow:epoch = 1.0303030303030305, loss = 0.0009843046, step = 102 (5.908 sec)
2020-03-20 18:17:01,985 [INFO] tensorflow: epoch = 1.0303030303030305, loss = 0.0009843046, step = 102 (5.908 sec)
INFO:tensorflow:global_step/sec: 1.93703
2020-03-20 18:17:05,082 [INFO] tensorflow: global_step/sec: 1.93703
INFO:tensorflow:epoch = 1.1414141414141414, loss = 0.00095792516, step = 113 (5.737 sec)
2020-03-20 18:17:07,722 [INFO] tensorflow: epoch = 1.1414141414141414, loss = 0.00095792516, step = 113 (5.737 sec)
INFO:tensorflow:global_step/sec: 1.90025
2020-03-20 18:17:09,818 [INFO] tensorflow: global_step/sec: 1.90025
INFO:tensorflow:epoch = 1.2525252525252526, loss = 0.0009852392, step = 124 (5.682 sec)
2020-03-20 18:17:13,404 [INFO] tensorflow: epoch = 1.2525252525252526, loss = 0.0009852392, step = 124 (5.682 sec)
2020-03-20 18:17:13,404 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.711
INFO:tensorflow:global_step/sec: 1.9556
2020-03-20 18:17:14,420 [INFO] tensorflow: global_step/sec: 1.9556
INFO:tensorflow:epoch = 1.3636363636363638, loss = 0.00089812273, step = 135 (5.703 sec)
2020-03-20 18:17:19,107 [INFO] tensorflow: epoch = 1.3636363636363638, loss = 0.00089812273, step = 135 (5.703 sec)
INFO:tensorflow:global_step/sec: 1.92018
2020-03-20 18:17:19,107 [INFO] tensorflow: global_step/sec: 1.92018
INFO:tensorflow:global_step/sec: 1.91623
2020-03-20 18:17:23,804 [INFO] tensorflow: global_step/sec: 1.91623
INFO:tensorflow:epoch = 1.474747474747475, loss = 0.0009279676, step = 146 (5.738 sec)
2020-03-20 18:17:24,844 [INFO] tensorflow: epoch = 1.474747474747475, loss = 0.0009279676, step = 146 (5.738 sec)
2020-03-20 18:17:26,380 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.707
INFO:tensorflow:global_step/sec: 1.93415
2020-03-20 18:17:28,457 [INFO] tensorflow: global_step/sec: 1.93415
INFO:tensorflow:epoch = 1.585858585858586, loss = 0.0009106908, step = 157 (5.671 sec)
2020-03-20 18:17:30,516 [INFO] tensorflow: epoch = 1.585858585858586, loss = 0.0009106908, step = 157 (5.671 sec)
INFO:tensorflow:global_step/sec: 1.93502
2020-03-20 18:17:33,108 [INFO] tensorflow: global_step/sec: 1.93502
INFO:tensorflow:epoch = 1.696969696969697, loss = 0.0009452889, step = 168 (5.735 sec)
2020-03-20 18:17:36,251 [INFO] tensorflow: epoch = 1.696969696969697, loss = 0.0009452889, step = 168 (5.735 sec)
INFO:tensorflow:global_step/sec: 1.92834
2020-03-20 18:17:37,776 [INFO] tensorflow: global_step/sec: 1.92834
2020-03-20 18:17:39,344 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.714
INFO:tensorflow:epoch = 1.8080808080808082, loss = 0.0009398138, step = 179 (5.668 sec)
2020-03-20 18:17:41,919 [INFO] tensorflow: epoch = 1.8080808080808082, loss = 0.0009398138, step = 179 (5.668 sec)
INFO:tensorflow:global_step/sec: 1.92218
2020-03-20 18:17:42,458 [INFO] tensorflow: global_step/sec: 1.92218
INFO:tensorflow:global_step/sec: 1.9167
2020-03-20 18:17:47,153 [INFO] tensorflow: global_step/sec: 1.9167
INFO:tensorflow:epoch = 1.9191919191919193, loss = 0.0009455455, step = 190 (5.757 sec)
2020-03-20 18:17:47,676 [INFO] tensorflow: epoch = 1.9191919191919193, loss = 0.0009455455, step = 190 (5.757 sec)
INFO:tensorflow:global_step/sec: 1.91853
2020-03-20 18:17:51,844 [INFO] tensorflow: global_step/sec: 1.91853
2020-03-20 18:17:51,845 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 2/500: loss: 0.00092 Time taken: 0:00:51.417653 ETA: 7:06:45.991194
2020-03-20 18:17:52,362 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.682
INFO:tensorflow:epoch = 2.0303030303030303, loss = 0.0009290565, step = 201 (5.761 sec)
2020-03-20 18:17:53,438 [INFO] tensorflow: epoch = 2.0303030303030303, loss = 0.0009290565, step = 201 (5.761 sec)
INFO:tensorflow:global_step/sec: 1.8937
2020-03-20 18:17:56,597 [INFO] tensorflow: global_step/sec: 1.8937
INFO:tensorflow:epoch = 2.1414141414141414, loss = 0.0009679984, step = 212 (5.769 sec)
2020-03-20 18:17:59,207 [INFO] tensorflow: epoch = 2.1414141414141414, loss = 0.0009679984, step = 212 (5.769 sec)
INFO:tensorflow:global_step/sec: 1.91842
2020-03-20 18:18:01,288 [INFO] tensorflow: global_step/sec: 1.91842
INFO:tensorflow:epoch = 2.2525252525252526, loss = 0.0009079506, step = 223 (5.740 sec)
2020-03-20 18:18:04,946 [INFO] tensorflow: epoch = 2.2525252525252526, loss = 0.0009079506, step = 223 (5.740 sec)
2020-03-20 18:18:05,468 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.630
INFO:tensorflow:global_step/sec: 1.91774
2020-03-20 18:18:05,981 [INFO] tensorflow: global_step/sec: 1.91774
INFO:tensorflow:epoch = 2.3636363636363638, loss = 0.000969926, step = 234 (5.768 sec)
2020-03-20 18:18:10,714 [INFO] tensorflow: epoch = 2.3636363636363638, loss = 0.000969926, step = 234 (5.768 sec)
INFO:tensorflow:global_step/sec: 1.90139
2020-03-20 18:18:10,715 [INFO] tensorflow: global_step/sec: 1.90139
INFO:tensorflow:global_step/sec: 1.90066
2020-03-20 18:18:15,450 [INFO] tensorflow: global_step/sec: 1.90066
INFO:tensorflow:epoch = 2.474747474747475, loss = 0.00086104346, step = 245 (5.783 sec)
2020-03-20 18:18:16,497 [INFO] tensorflow: epoch = 2.474747474747475, loss = 0.00086104346, step = 245 (5.783 sec)
2020-03-20 18:18:18,603 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.613
INFO:tensorflow:global_step/sec: 1.91597
2020-03-20 18:18:20,147 [INFO] tensorflow: global_step/sec: 1.91597
INFO:tensorflow:epoch = 2.585858585858586, loss = 0.0009799077, step = 256 (5.711 sec)
2020-03-20 18:18:22,208 [INFO] tensorflow: epoch = 2.585858585858586, loss = 0.0009799077, step = 256 (5.711 sec)
INFO:tensorflow:global_step/sec: 1.91937
2020-03-20 18:18:24,836 [INFO] tensorflow: global_step/sec: 1.91937
INFO:tensorflow:epoch = 2.6969696969696972, loss = 0.00094006327, step = 267 (5.753 sec)
2020-03-20 18:18:27,961 [INFO] tensorflow: epoch = 2.6969696969696972, loss = 0.00094006327, step = 267 (5.753 sec)
INFO:tensorflow:global_step/sec: 1.92418
2020-03-20 18:18:29,514 [INFO] tensorflow: global_step/sec: 1.92418
2020-03-20 18:18:31,623 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.681
INFO:tensorflow:epoch = 2.8080808080808084, loss = 0.000921318, step = 278 (5.777 sec)
2020-03-20 18:18:33,738 [INFO] tensorflow: epoch = 2.8080808080808084, loss = 0.000921318, step = 278 (5.777 sec)
INFO:tensorflow:global_step/sec: 1.89646
2020-03-20 18:18:34,259 [INFO] tensorflow: global_step/sec: 1.89646
INFO:tensorflow:global_step/sec: 1.91425
2020-03-20 18:18:38,961 [INFO] tensorflow: global_step/sec: 1.91425
INFO:tensorflow:epoch = 2.9191919191919196, loss = 0.0009311149, step = 289 (5.749 sec)
2020-03-20 18:18:39,486 [INFO] tensorflow: epoch = 2.9191919191919196, loss = 0.0009311149, step = 289 (5.749 sec)
INFO:tensorflow:global_step/sec: 1.89139
2020-03-20 18:18:43,719 [INFO] tensorflow: global_step/sec: 1.89139
2020-03-20 18:18:43,720 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 3/500: loss: 0.00102 Time taken: 0:00:51.835348 ETA: 7:09:22.167956
2020-03-20 18:18:44,758 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.614
INFO:tensorflow:epoch = 3.0303030303030307, loss = 0.00094757596, step = 300 (5.801 sec)
2020-03-20 18:18:45,287 [INFO] tensorflow: epoch = 3.0303030303030307, loss = 0.00094757596, step = 300 (5.801 sec)
INFO:tensorflow:global_step/sec: 1.91187
2020-03-20 18:18:48,427 [INFO] tensorflow: global_step/sec: 1.91187
INFO:tensorflow:epoch = 3.141414141414142, loss = 0.0009914932, step = 311 (5.777 sec)
2020-03-20 18:18:51,064 [INFO] tensorflow: epoch = 3.141414141414142, loss = 0.0009914932, step = 311 (5.777 sec)
INFO:tensorflow:global_step/sec: 1.91593
2020-03-20 18:18:53,124 [INFO] tensorflow: global_step/sec: 1.91593
INFO:tensorflow:epoch = 3.2525252525252526, loss = 0.0009799146, step = 322 (5.715 sec)
2020-03-20 18:18:56,779 [INFO] tensorflow: epoch = 3.2525252525252526, loss = 0.0009799146, step = 322 (5.715 sec)
INFO:tensorflow:global_step/sec: 1.90648
2020-03-20 18:18:57,845 [INFO] tensorflow: global_step/sec: 1.90648
2020-03-20 18:18:57,845 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.641
INFO:tensorflow:epoch = 3.3636363636363638, loss = 0.00084329536, step = 333 (5.735 sec)
2020-03-20 18:19:02,513 [INFO] tensorflow: epoch = 3.3636363636363638, loss = 0.00084329536, step = 333 (5.735 sec)
INFO:tensorflow:global_step/sec: 1.92764
2020-03-20 18:19:02,514 [INFO] tensorflow: global_step/sec: 1.92764
INFO:tensorflow:global_step/sec: 1.91651
2020-03-20 18:19:07,210 [INFO] tensorflow: global_step/sec: 1.91651
INFO:tensorflow:epoch = 3.474747474747475, loss = 0.00087175716, step = 344 (5.763 sec)
2020-03-20 18:19:08,276 [INFO] tensorflow: epoch = 3.474747474747475, loss = 0.00087175716, step = 344 (5.763 sec)
2020-03-20 18:19:10,886 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.668
INFO:tensorflow:global_step/sec: 1.90723
2020-03-20 18:19:11,929 [INFO] tensorflow: global_step/sec: 1.90723
INFO:tensorflow:epoch = 3.585858585858586, loss = 0.00095952966, step = 355 (5.780 sec)
2020-03-20 18:19:14,056 [INFO] tensorflow: epoch = 3.585858585858586, loss = 0.00095952966, step = 355 (5.780 sec)
INFO:tensorflow:global_step/sec: 1.90483
2020-03-20 18:19:16,654 [INFO] tensorflow: global_step/sec: 1.90483
INFO:tensorflow:epoch = 3.6969696969696972, loss = 0.0009282512, step = 366 (5.716 sec)
2020-03-20 18:19:19,772 [INFO] tensorflow: epoch = 3.6969696969696972, loss = 0.0009282512, step = 366 (5.716 sec)
INFO:tensorflow:global_step/sec: 1.91711
2020-03-20 18:19:21,348 [INFO] tensorflow: global_step/sec: 1.91711
2020-03-20 18:19:23,980 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.637
INFO:tensorflow:epoch = 3.8080808080808084, loss = 0.0009802162, step = 377 (5.758 sec)
2020-03-20 18:19:25,530 [INFO] tensorflow: epoch = 3.8080808080808084, loss = 0.0009802162, step = 377 (5.758 sec)
INFO:tensorflow:global_step/sec: 1.92291
2020-03-20 18:19:26,029 [INFO] tensorflow: global_step/sec: 1.92291
INFO:tensorflow:global_step/sec: 1.91303
2020-03-20 18:19:30,733 [INFO] tensorflow: global_step/sec: 1.91303
INFO:tensorflow:epoch = 3.9191919191919196, loss = 0.0009512324, step = 388 (5.735 sec)
2020-03-20 18:19:31,265 [INFO] tensorflow: epoch = 3.9191919191919196, loss = 0.0009512324, step = 388 (5.735 sec)
INFO:tensorflow:global_step/sec: 1.9056
2020-03-20 18:19:35,456 [INFO] tensorflow: global_step/sec: 1.9056
2020-03-20 18:19:35,457 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 4/500: loss: 0.00093 Time taken: 0:00:51.785088 ETA: 7:08:05.403648
INFO:tensorflow:epoch = 4.03030303030303, loss = 0.0009777864, step = 399 (5.765 sec)
2020-03-20 18:19:37,030 [INFO] tensorflow: epoch = 4.03030303030303, loss = 0.0009777864, step = 399 (5.765 sec)
2020-03-20 18:19:37,030 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.663
INFO:tensorflow:global_step/sec: 1.91782
2020-03-20 18:19:40,149 [INFO] tensorflow: global_step/sec: 1.91782
INFO:tensorflow:epoch = 4.141414141414142, loss = 0.0009197957, step = 410 (5.788 sec)
2020-03-20 18:19:42,818 [INFO] tensorflow: epoch = 4.141414141414142, loss = 0.0009197957, step = 410 (5.788 sec)
INFO:tensorflow:global_step/sec: 1.8968
2020-03-20 18:19:44,894 [INFO] tensorflow: global_step/sec: 1.8968
INFO:tensorflow:epoch = 4.252525252525253, loss = 0.0009904406, step = 421 (5.740 sec)
2020-03-20 18:19:48,557 [INFO] tensorflow: epoch = 4.252525252525253, loss = 0.0009904406, step = 421 (5.740 sec)
INFO:tensorflow:global_step/sec: 1.91622
2020-03-20 18:19:49,591 [INFO] tensorflow: global_step/sec: 1.91622
2020-03-20 18:19:50,109 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.646
INFO:tensorflow:epoch = 4.363636363636364, loss = 0.0009531719, step = 432 (5.767 sec)
2020-03-20 18:19:54,324 [INFO] tensorflow: epoch = 4.363636363636364, loss = 0.0009531719, step = 432 (5.767 sec)
INFO:tensorflow:global_step/sec: 1.90127
2020-03-20 18:19:54,324 [INFO] tensorflow: global_step/sec: 1.90127
INFO:tensorflow:global_step/sec: 1.91397
2020-03-20 18:19:59,026 [INFO] tensorflow: global_step/sec: 1.91397
INFO:tensorflow:epoch = 4.474747474747475, loss = 0.0009472327, step = 443 (5.742 sec)
2020-03-20 18:20:00,065 [INFO] tensorflow: epoch = 4.474747474747475, loss = 0.0009472327, step = 443 (5.742 sec)
2020-03-20 18:20:03,241 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.615
INFO:tensorflow:global_step/sec: 1.89664
2020-03-20 18:20:03,772 [INFO] tensorflow: global_step/sec: 1.89664
INFO:tensorflow:epoch = 4.5858585858585865, loss = 0.00094064797, step = 454 (5.816 sec)
2020-03-20 18:20:05,882 [INFO] tensorflow: epoch = 4.5858585858585865, loss = 0.00094064797, step = 454 (5.816 sec)
INFO:tensorflow:global_step/sec: 1.90762
2020-03-20 18:20:08,490 [INFO] tensorflow: global_step/sec: 1.90762
INFO:tensorflow:epoch = 4.696969696969697, loss = 0.0009906166, step = 465 (5.740 sec)
2020-03-20 18:20:11,622 [INFO] tensorflow: epoch = 4.696969696969697, loss = 0.0009906166, step = 465 (5.740 sec)
INFO:tensorflow:global_step/sec: 1.91803
2020-03-20 18:20:13,182 [INFO] tensorflow: global_step/sec: 1.91803
2020-03-20 18:20:16,365 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 7.620
INFO:tensorflow:epoch = 4.808080808080809, loss = 0.00091935333, step = 476 (5.790 sec)
2020-03-20 18:20:17,411 [INFO] tensorflow: epoch = 4.808080808080809, loss = 0.00091935333, step = 476 (5.790 sec)
INFO:tensorflow:global_step/sec: 1.89198
2020-03-20 18:20:17,939 [INFO] tensorflow: global_step/sec: 1.89198
INFO:tensorflow:global_step/sec: 1.92182
2020-03-20 18:20:22,622 [INFO] tensorflow: global_step/sec: 1.92182
INFO:tensorflow:epoch = 4.91919191919192, loss = 0.00093365816, step = 487 (5.718 sec)
2020-03-20 18:20:23,129 [INFO] tensorflow: epoch = 4.91919191919192, loss = 0.00093365816, step = 487 (5.718 sec)
INFO:tensorflow:Saving checkpoints for step-495.
2020-03-20 18:20:26,800 [INFO] tensorflow: Saving checkpoints for step-495.
2020-03-20 18:20:26,955 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 15, 0.00s/step
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/evaluation/metadata.py:38: UserWarning: One or more metadata field(s) are missing from ground_truth batch_data, and will be replaced with defaults: ['frame/camera_location']
2020-03-20 18:21:32,403 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 15, 6.54s/step
Epoch 5/500
=========================
Validation cost: 0.000688
Mean average_precision (in %): 0.0000
class name average precision (in %)
----------------- --------------------------
hose_connected 0
hose_disconnected 0
truck 0
valve_closed 0
valve_open 0
Median Inference Time: 0.029676
I have also tried setting load_graph to false in the train spec, and the initial mAP is still reverting back to 0%.