sing TensorFlow backend. 2020-07-01 06:31:35,894 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /workspace/examples/detectnet_v2/specs/detectnet_v2_retrain_resnet18_kitti_lego.txt. 2020-07-01 06:31:35,894 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/examples/detectnet_v2/specs/detectnet_v2_retrain_resnet18_kitti_lego.txt 2020-07-01 06:31:35,917 [INFO] iva.detectnet_v2.scripts.train: Cannot iterate over exactly 61 samples with a batch size of 4; each epoch will therefore take one extra step. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 544, 1408) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 272, 704) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 272, 704) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 272, 704) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 136, 352) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 136, 352) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 136, 352) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 136, 352) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 136, 352) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 136, 352) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 136, 352) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 136, 352) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 136, 352) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 136, 352) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 136, 352) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 136, 352) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 136, 352) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 136, 352) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 136, 352) 0 block_1b_bn_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 136, 352) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 68, 176) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 68, 176) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 68, 176) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 68, 176) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 68, 176) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 68, 176) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 68, 176) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 68, 176) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 68, 176) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 68, 176) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 68, 176) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 68, 176) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 68, 176) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 68, 176) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 68, 176) 0 block_2b_bn_2[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 68, 176) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 34, 88) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 34, 88) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 34, 88) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 34, 88) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 34, 88) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 34, 88) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 34, 88) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 34, 88) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 34, 88) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 34, 88) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 34, 88) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 34, 88) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 34, 88) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 34, 88) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 34, 88) 0 block_3b_bn_2[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 34, 88) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 34, 88) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 34, 88) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 34, 88) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 34, 88) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 34, 88) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 34, 88) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 34, 88) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 34, 88) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 34, 88) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 34, 88) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 34, 88) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 34, 88) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 34, 88) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 34, 88) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 34, 88) 0 block_4b_bn_2[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 34, 88) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 12, 34, 88) 6156 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 3, 34, 88) 1539 block_4b_relu[0][0] ================================================================================================== Total params: 11,203,023 Trainable params: 11,193,295 Non-trainable params: 9,728 __________________________________________________________________________________________________ target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. 2020-07-01 06:31:41,719 [INFO] iva.detectnet_v2.scripts.train: Found 61 samples in training set target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. 2020-07-01 06:31:47,214 [INFO] iva.detectnet_v2.scripts.train: Found 9 samples in validation set 2020-07-01 06:32:26,673 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 0/120: loss: 0.00369 Time taken: 0:00:00 ETA: 0:00:00 2020-07-01 06:32:32,234 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 1/120: loss: 0.00063 Time taken: 0:00:10.566415 ETA: 0:20:57.403385 2020-07-01 06:32:33,743 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.155 2020-07-01 06:32:35,208 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 2/120: loss: 0.00063 Time taken: 0:00:02.978774 ETA: 0:05:51.495332 2020-07-01 06:32:38,077 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 3/120: loss: 0.00064 Time taken: 0:00:02.868126 ETA: 0:05:35.570742 2020-07-01 06:32:38,259 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.145 2020-07-01 06:32:40,969 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 4/120: loss: 0.00062 Time taken: 0:00:02.883355 ETA: 0:05:34.469180 2020-07-01 06:32:42,806 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.992 2020-07-01 06:32:43,918 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 5/120: loss: 0.00064 Time taken: 0:00:02.949524 ETA: 0:05:39.195260 2020-07-01 06:32:46,846 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 6/120: loss: 0.00063 Time taken: 0:00:02.929633 ETA: 0:05:33.978162 2020-07-01 06:32:47,401 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.764 2020-07-01 06:32:49,803 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 7/120: loss: 0.00061 Time taken: 0:00:02.944624 ETA: 0:05:32.742512 2020-07-01 06:32:51,956 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.957 2020-07-01 06:32:52,671 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 8/120: loss: 0.00062 Time taken: 0:00:02.886671 ETA: 0:05:23.307152 2020-07-01 06:32:55,554 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 9/120: loss: 0.00063 Time taken: 0:00:02.878302 ETA: 0:05:19.491522 2020-07-01 06:32:56,446 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.272 2020-07-01 06:33:02,159 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 10/120: loss: 0.00063 Time taken: 0:00:06.597845 ETA: 0:12:05.762950 2020-07-01 06:33:04,674 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 12.153 2020-07-01 06:33:05,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 11/120: loss: 0.00069 Time taken: 0:00:02.883359 ETA: 0:05:14.286131 2020-07-01 06:33:07,911 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 12/120: loss: 0.00067 Time taken: 0:00:02.878127 ETA: 0:05:10.837716 2020-07-01 06:33:09,207 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.063 2020-07-01 06:33:10,869 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 13/120: loss: 0.00068 Time taken: 0:00:02.949242 ETA: 0:05:15.568894 2020-07-01 06:33:13,845 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 14/120: loss: 0.00064 Time taken: 0:00:02.974066 ETA: 0:05:15.250996 2020-07-01 06:33:13,845 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.564 2020-07-01 06:33:16,809 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 15/120: loss: 0.00068 Time taken: 0:00:02.970223 ETA: 0:05:11.873415 2020-07-01 06:33:18,472 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.613 2020-07-01 06:33:19,748 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 16/120: loss: 0.00066 Time taken: 0:00:02.934829 ETA: 0:05:05.222216 2020-07-01 06:33:22,689 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.00065 Time taken: 0:00:02.944393 ETA: 0:05:03.272479 2020-07-01 06:33:23,056 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.817 2020-07-01 06:33:25,636 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.00064 Time taken: 0:00:02.939558 ETA: 0:04:59.834916 2020-07-01 06:33:27,673 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.659 2020-07-01 06:33:28,618 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 19/120: loss: 0.00061 Time taken: 0:00:02.980538 ETA: 0:05:01.034338 2020-07-01 06:33:35,504 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 20/120: loss: 0.00062 Time taken: 0:00:06.881680 ETA: 0:11:28.168000 2020-07-01 06:33:36,223 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.697 2020-07-01 06:33:38,437 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 21/120: loss: 0.00063 Time taken: 0:00:02.939897 ETA: 0:04:51.049803 2020-07-01 06:33:40,841 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.655 2020-07-01 06:33:41,399 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 22/120: loss: 0.00065 Time taken: 0:00:02.958555 ETA: 0:04:49.938390 2020-07-01 06:33:44,302 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 23/120: loss: 0.00067 Time taken: 0:00:02.908461 ETA: 0:04:42.120717 2020-07-01 06:33:45,380 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.031 2020-07-01 06:33:47,201 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 24/120: loss: 0.00063 Time taken: 0:00:02.893381 ETA: 0:04:37.764576 2020-07-01 06:33:49,961 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.828 2020-07-01 06:33:50,142 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 25/120: loss: 0.00069 Time taken: 0:00:02.943833 ETA: 0:04:39.664135 2020-07-01 06:33:53,103 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 26/120: loss: 0.00063 Time taken: 0:00:02.963838 ETA: 0:04:38.600772 2020-07-01 06:33:54,548 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.806 2020-07-01 06:33:56,012 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 27/120: loss: 0.00061 Time taken: 0:00:02.903282 ETA: 0:04:30.005226 2020-07-01 06:33:58,935 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 28/120: loss: 0.00062 Time taken: 0:00:02.918908 ETA: 0:04:28.539536 2020-07-01 06:33:59,120 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.875 2020-07-01 06:34:01,879 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 29/120: loss: 0.00067 Time taken: 0:00:02.940572 ETA: 0:04:27.592052 2020-07-01 06:34:03,712 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.779 2020-07-01 06:34:08,430 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 45/45 [00:00<00:00, 13 Matching predictions to ground truth, class 2/3.: 100%|█| 7/7 [00:00<00:00, 6909 Matching predictions to ground truth, class 3/3.: 100%|█| 130/130 [00:00<00:00, Epoch 30/120 ========================= Validation cost: 0.000139 Mean average_precision (in %): 62.1795 class name average precision (in %) ------------ -------------------------- bus 66.6667 followme 36.5385 police 83.3333 Median Inference Time: 0.168490 2020-07-01 06:34:10,264 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 30/120: loss: 0.00064 Time taken: 0:00:08.380891 ETA: 0:12:34.280190 2020-07-01 06:34:13,170 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 31/120: loss: 0.00057 Time taken: 0:00:02.904790 ETA: 0:04:18.526310 2020-07-01 06:34:13,709 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 10.003 2020-07-01 06:34:16,104 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 32/120: loss: 0.00057 Time taken: 0:00:02.944368 ETA: 0:04:19.104384 2020-07-01 06:34:18,276 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.901 2020-07-01 06:34:18,998 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 33/120: loss: 0.00058 Time taken: 0:00:02.893032 ETA: 0:04:11.693784 2020-07-01 06:34:21,888 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 34/120: loss: 0.00056 Time taken: 0:00:02.888747 ETA: 0:04:08.432242 2020-07-01 06:34:22,869 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.770 2020-07-01 06:34:24,875 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 35/120: loss: 0.00061 Time taken: 0:00:02.990039 ETA: 0:04:14.153315 2020-07-01 06:34:27,381 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.163 2020-07-01 06:34:27,750 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 36/120: loss: 0.00057 Time taken: 0:00:02.868002 ETA: 0:04:00.912168 2020-07-01 06:34:30,702 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 37/120: loss: 0.00059 Time taken: 0:00:02.955543 ETA: 0:04:05.310069 2020-07-01 06:34:31,976 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.766 2020-07-01 06:34:33,619 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 38/120: loss: 0.00057 Time taken: 0:00:02.913368 ETA: 0:03:58.896176 2020-07-01 06:34:36,552 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 39/120: loss: 0.00058 Time taken: 0:00:02.934525 ETA: 0:03:57.696525 2020-07-01 06:34:36,552 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.854 2020-07-01 06:34:43,459 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 34/34 [00:00<00:00, 11 Matching predictions to ground truth, class 2/3.: 100%|█| 11/11 [00:00<00:00, 11 Matching predictions to ground truth, class 3/3.: 100%|█| 100/100 [00:00<00:00, Epoch 40/120 ========================= Validation cost: 0.000123 Mean average_precision (in %): 65.1190 class name average precision (in %) ------------ -------------------------- bus 60 followme 55.3571 police 80 Median Inference Time: 0.019600 2020-07-01 06:34:44,013 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 40/120: loss: 0.00063 Time taken: 0:00:07.446212 ETA: 0:09:55.696960 2020-07-01 06:34:45,675 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 10.962 2020-07-01 06:34:46,959 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 41/120: loss: 0.00057 Time taken: 0:00:02.955692 ETA: 0:03:53.499668 2020-07-01 06:34:49,874 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 42/120: loss: 0.00055 Time taken: 0:00:02.916407 ETA: 0:03:47.479746 2020-07-01 06:34:50,249 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.865 2020-07-01 06:34:52,834 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 43/120: loss: 0.00056 Time taken: 0:00:02.959637 ETA: 0:03:47.892049 2020-07-01 06:34:54,964 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.209 2020-07-01 06:34:55,937 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 44/120: loss: 0.00056 Time taken: 0:00:03.095384 ETA: 0:03:55.249184 2020-07-01 06:34:58,975 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 45/120: loss: 0.00055 Time taken: 0:00:03.041031 ETA: 0:03:48.077325 2020-07-01 06:34:59,732 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 20.975 2020-07-01 06:35:01,948 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 46/120: loss: 0.00055 Time taken: 0:00:02.969951 ETA: 0:03:39.776374 2020-07-01 06:35:04,344 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.682 2020-07-01 06:35:04,894 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 47/120: loss: 0.00054 Time taken: 0:00:02.948944 ETA: 0:03:35.272912 2020-07-01 06:35:07,834 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 48/120: loss: 0.00057 Time taken: 0:00:02.938669 ETA: 0:03:31.584168 2020-07-01 06:35:08,926 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.826 2020-07-01 06:35:10,738 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 49/120: loss: 0.00058 Time taken: 0:00:02.908991 ETA: 0:03:26.538361 2020-07-01 06:35:17,244 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 38/38 [00:00<00:00, 12 Matching predictions to ground truth, class 2/3.: 100%|█| 6/6 [00:00<00:00, 6342 Matching predictions to ground truth, class 3/3.: 100%|█| 38/38 [00:00<00:00, 12 Epoch 50/120 ========================= Validation cost: 0.000089 Mean average_precision (in %): 64.9206 class name average precision (in %) ------------ -------------------------- bus 80 followme 31.4286 police 83.3333 Median Inference Time: 0.019753 2020-07-01 06:35:17,500 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.665 2020-07-01 06:35:17,684 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 50/120: loss: 0.00053 Time taken: 0:00:06.939224 ETA: 0:08:05.745680 2020-07-01 06:35:20,655 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 51/120: loss: 0.00054 Time taken: 0:00:02.973307 ETA: 0:03:25.158183 2020-07-01 06:35:22,110 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.695 2020-07-01 06:35:23,578 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 52/120: loss: 0.00052 Time taken: 0:00:02.918491 ETA: 0:03:18.457388 2020-07-01 06:35:26,518 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 53/120: loss: 0.00052 Time taken: 0:00:02.945413 ETA: 0:03:17.342671 2020-07-01 06:35:26,793 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.354 2020-07-01 06:35:29,523 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 54/120: loss: 0.00052 Time taken: 0:00:02.995671 ETA: 0:03:17.714286 2020-07-01 06:35:31,310 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.137 2020-07-01 06:35:32,388 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 55/120: loss: 0.00053 Time taken: 0:00:02.874682 ETA: 0:03:06.854330 2020-07-01 06:35:35,297 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 56/120: loss: 0.00053 Time taken: 0:00:02.908184 ETA: 0:03:06.123776 2020-07-01 06:35:35,844 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.059 2020-07-01 06:35:38,203 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 57/120: loss: 0.00052 Time taken: 0:00:02.903361 ETA: 0:03:02.911743 2020-07-01 06:35:40,390 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.999 2020-07-01 06:35:41,115 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 58/120: loss: 0.00053 Time taken: 0:00:02.908142 ETA: 0:03:00.304804 2020-07-01 06:35:44,096 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 59/120: loss: 0.00049 Time taken: 0:00:02.974783 ETA: 0:03:01.461763 2020-07-01 06:35:45,011 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.639 2020-07-01 06:35:50,979 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 26/26 [00:00<00:00, 74 Matching predictions to ground truth, class 2/3.: 100%|█| 6/6 [00:00<00:00, 6416 Matching predictions to ground truth, class 3/3.: 100%|█| 36/36 [00:00<00:00, 95 Epoch 60/120 ========================= Validation cost: 0.000112 Mean average_precision (in %): 65.8333 class name average precision (in %) ------------ -------------------------- bus 80 followme 37.5 police 80 Median Inference Time: 0.019354 2020-07-01 06:35:51,411 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 60/120: loss: 0.00051 Time taken: 0:00:07.323582 ETA: 0:07:19.414920 2020-07-01 06:35:53,983 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.146 2020-07-01 06:35:54,373 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 61/120: loss: 0.00050 Time taken: 0:00:02.942958 ETA: 0:02:53.634522 2020-07-01 06:35:57,402 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 62/120: loss: 0.00049 Time taken: 0:00:03.036504 ETA: 0:02:56.117232 2020-07-01 06:35:58,720 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.114 2020-07-01 06:36:00,442 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 63/120: loss: 0.00049 Time taken: 0:00:03.039894 ETA: 0:02:53.273958 2020-07-01 06:36:03,443 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 64/120: loss: 0.00050 Time taken: 0:00:03.009917 ETA: 0:02:48.555352 2020-07-01 06:36:03,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.172 2020-07-01 06:36:06,384 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 65/120: loss: 0.00049 Time taken: 0:00:02.934509 ETA: 0:02:41.397995 2020-07-01 06:36:08,047 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.725 2020-07-01 06:36:09,382 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 66/120: loss: 0.00051 Time taken: 0:00:02.999578 ETA: 0:02:41.977212 2020-07-01 06:36:12,282 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 67/120: loss: 0.00049 Time taken: 0:00:02.897928 ETA: 0:02:33.590184 2020-07-01 06:36:12,644 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.754 2020-07-01 06:36:15,212 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 68/120: loss: 0.00048 Time taken: 0:00:02.932923 ETA: 0:02:32.511996 2020-07-01 06:36:17,205 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.925 2020-07-01 06:36:18,106 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 69/120: loss: 0.00051 Time taken: 0:00:02.893233 ETA: 0:02:27.554883 2020-07-01 06:36:24,586 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 35/35 [00:00<00:00, 91 Matching predictions to ground truth, class 2/3.: 100%|█| 9/9 [00:00<00:00, 8989 Matching predictions to ground truth, class 3/3.: 100%|█| 37/37 [00:00<00:00, 11 Epoch 70/120 ========================= Validation cost: 0.000108 Mean average_precision (in %): 71.8803 class name average precision (in %) ------------ -------------------------- bus 83.3333 followme 52.3077 police 80 Median Inference Time: 0.019620 2020-07-01 06:36:25,016 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 70/120: loss: 0.00048 Time taken: 0:00:06.904037 ETA: 0:05:45.201850 2020-07-01 06:36:25,731 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.729 2020-07-01 06:36:27,874 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 71/120: loss: 0.00048 Time taken: 0:00:02.865251 ETA: 0:02:20.397299 2020-07-01 06:36:30,299 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.894 2020-07-01 06:36:30,849 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 72/120: loss: 0.00046 Time taken: 0:00:02.970179 ETA: 0:02:22.568592 2020-07-01 06:36:33,773 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 73/120: loss: 0.00048 Time taken: 0:00:02.908062 ETA: 0:02:16.678914 2020-07-01 06:36:34,984 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.347 2020-07-01 06:36:36,809 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 74/120: loss: 0.00047 Time taken: 0:00:03.050355 ETA: 0:02:20.316330 2020-07-01 06:36:39,547 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.918 2020-07-01 06:36:39,742 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 75/120: loss: 0.00045 Time taken: 0:00:02.917990 ETA: 0:02:11.309550 2020-07-01 06:36:42,725 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 76/120: loss: 0.00045 Time taken: 0:00:02.994801 ETA: 0:02:11.771244 2020-07-01 06:36:44,182 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.578 2020-07-01 06:36:45,640 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 77/120: loss: 0.00045 Time taken: 0:00:02.918137 ETA: 0:02:05.479891 2020-07-01 06:36:48,558 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 78/120: loss: 0.00044 Time taken: 0:00:02.913031 ETA: 0:02:02.347302 2020-07-01 06:36:48,823 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.548 2020-07-01 06:36:51,541 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 79/120: loss: 0.00045 Time taken: 0:00:02.984090 ETA: 0:02:02.347690 2020-07-01 06:36:53,338 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.147 2020-07-01 06:36:58,259 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 22/22 [00:00<00:00, 67 Matching predictions to ground truth, class 2/3.: 100%|█| 6/6 [00:00<00:00, 6335 Matching predictions to ground truth, class 3/3.: 100%|█| 24/24 [00:00<00:00, 17 Epoch 80/120 ========================= Validation cost: 0.000117 Mean average_precision (in %): 68.5470 class name average precision (in %) ------------ -------------------------- bus 80 followme 42.3077 police 83.3333 Median Inference Time: 0.020295 2020-07-01 06:36:58,678 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 80/120: loss: 0.00045 Time taken: 0:00:07.133015 ETA: 0:04:45.320600 2020-07-01 06:37:01,603 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 81/120: loss: 0.00054 Time taken: 0:00:02.932567 ETA: 0:01:54.370113 2020-07-01 06:37:02,149 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.350 2020-07-01 06:37:04,525 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 82/120: loss: 0.00044 Time taken: 0:00:02.918349 ETA: 0:01:50.897262 2020-07-01 06:37:06,716 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.902 2020-07-01 06:37:07,457 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 83/120: loss: 0.00044 Time taken: 0:00:02.923454 ETA: 0:01:48.167798 2020-07-01 06:37:10,387 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 84/120: loss: 0.00047 Time taken: 0:00:02.938330 ETA: 0:01:45.779880 2020-07-01 06:37:11,312 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.760 2020-07-01 06:37:13,337 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 85/120: loss: 0.00045 Time taken: 0:00:02.944570 ETA: 0:01:43.059950 2020-07-01 06:37:15,906 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.769 2020-07-01 06:37:16,282 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 86/120: loss: 0.00041 Time taken: 0:00:02.943270 ETA: 0:01:40.071180 2020-07-01 06:37:19,200 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 87/120: loss: 0.00041 Time taken: 0:00:02.923533 ETA: 0:01:36.476589 2020-07-01 06:37:20,475 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.888 2020-07-01 06:37:22,095 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 88/120: loss: 0.00042 Time taken: 0:00:02.892761 ETA: 0:01:32.568352 2020-07-01 06:37:25,053 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 89/120: loss: 0.00041 Time taken: 0:00:02.958986 ETA: 0:01:31.728566 2020-07-01 06:37:25,054 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.840 2020-07-01 06:37:31,640 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 20/20 [00:00<00:00, 65 Matching predictions to ground truth, class 2/3.: 100%|█| 7/7 [00:00<00:00, 4435 Matching predictions to ground truth, class 3/3.: 100%|█| 29/29 [00:00<00:00, 20 Epoch 90/120 ========================= Validation cost: 0.000094 Mean average_precision (in %): 72.5000 class name average precision (in %) ------------ -------------------------- bus 80 followme 57.5 police 80 Median Inference Time: 0.019303 2020-07-01 06:37:32,042 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 90/120: loss: 0.00040 Time taken: 0:00:06.984963 ETA: 0:03:29.548890 2020-07-01 06:37:33,656 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.625 2020-07-01 06:37:34,916 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 91/120: loss: 0.00040 Time taken: 0:00:02.873826 ETA: 0:01:23.340954 2020-07-01 06:37:37,803 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 92/120: loss: 0.00042 Time taken: 0:00:02.888705 ETA: 0:01:20.883740 2020-07-01 06:37:38,161 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.199 2020-07-01 06:37:40,686 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 93/120: loss: 0.00038 Time taken: 0:00:02.882261 ETA: 0:01:17.821047 2020-07-01 06:37:42,725 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.915 2020-07-01 06:37:43,625 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 94/120: loss: 0.00039 Time taken: 0:00:02.938809 ETA: 0:01:16.409034 2020-07-01 06:37:46,504 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 95/120: loss: 0.00041 Time taken: 0:00:02.873279 ETA: 0:01:11.831975 2020-07-01 06:37:47,219 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.252 2020-07-01 06:37:49,377 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 96/120: loss: 0.00039 Time taken: 0:00:02.878369 ETA: 0:01:09.080856 2020-07-01 06:37:51,721 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.214 2020-07-01 06:37:52,267 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 97/120: loss: 0.00039 Time taken: 0:00:02.888256 ETA: 0:01:06.429888 2020-07-01 06:37:55,141 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 98/120: loss: 0.00040 Time taken: 0:00:02.873421 ETA: 0:01:03.215262 2020-07-01 06:37:56,219 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.234 2020-07-01 06:37:58,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 99/120: loss: 0.00038 Time taken: 0:00:02.887514 ETA: 0:01:00.637794 2020-07-01 06:38:04,870 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 17/17 [00:00<00:00, 14 Matching predictions to ground truth, class 2/3.: 100%|█| 5/5 [00:00<00:00, 4897 Matching predictions to ground truth, class 3/3.: 100%|█| 23/23 [00:00<00:00, 17 Epoch 100/120 ========================= Validation cost: 0.000069 Mean average_precision (in %): 72.2727 class name average precision (in %) ------------ -------------------------- bus 80 followme 56.8182 police 80 Median Inference Time: 0.019314 2020-07-01 06:38:05,082 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.283 2020-07-01 06:38:05,265 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 100/120: loss: 0.00039 Time taken: 0:00:07.230052 ETA: 0:02:24.601040 2020-07-01 06:38:08,128 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 101/120: loss: 0.00041 Time taken: 0:00:02.867388 ETA: 0:00:54.480372 2020-07-01 06:38:09,658 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.857 2020-07-01 06:38:11,093 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 102/120: loss: 0.00039 Time taken: 0:00:02.959974 ETA: 0:00:53.279532 2020-07-01 06:38:13,988 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 103/120: loss: 0.00038 Time taken: 0:00:02.898308 ETA: 0:00:49.271236 2020-07-01 06:38:14,172 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.156 2020-07-01 06:38:16,881 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 104/120: loss: 0.00038 Time taken: 0:00:02.892951 ETA: 0:00:46.287216 2020-07-01 06:38:18,692 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.121 2020-07-01 06:38:19,789 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 105/120: loss: 0.00038 Time taken: 0:00:02.903108 ETA: 0:00:43.546620 2020-07-01 06:38:22,738 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 106/120: loss: 0.00039 Time taken: 0:00:02.949072 ETA: 0:00:41.287008 2020-07-01 06:38:23,282 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.790 2020-07-01 06:38:25,624 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 107/120: loss: 0.00039 Time taken: 0:00:02.888494 ETA: 0:00:37.550422 2020-07-01 06:38:27,777 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.246 2020-07-01 06:38:28,506 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 108/120: loss: 0.00039 Time taken: 0:00:02.877768 ETA: 0:00:34.533216 2020-07-01 06:38:31,395 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 109/120: loss: 0.00038 Time taken: 0:00:02.887645 ETA: 0:00:31.764095 2020-07-01 06:38:32,295 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.138 2020-07-01 06:38:38,144 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 18/18 [00:00<00:00, 11 Matching predictions to ground truth, class 2/3.: 100%|█| 5/5 [00:00<00:00, 4980 Matching predictions to ground truth, class 3/3.: 100%|█| 20/20 [00:00<00:00, 63 Epoch 110/120 ========================= Validation cost: 0.000079 Mean average_precision (in %): 72.2222 class name average precision (in %) ------------ -------------------------- bus 83.3333 followme 53.3333 police 80 Median Inference Time: 0.018995 2020-07-01 06:38:38,532 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 110/120: loss: 0.00038 Time taken: 0:00:07.141035 ETA: 0:01:11.410350 2020-07-01 06:38:41,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.444 2020-07-01 06:38:41,391 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 111/120: loss: 0.00038 Time taken: 0:00:02.859366 ETA: 0:00:25.734294 2020-07-01 06:38:44,265 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 112/120: loss: 0.00039 Time taken: 0:00:02.868013 ETA: 0:00:22.944104 2020-07-01 06:38:45,520 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.288 2020-07-01 06:38:47,134 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 113/120: loss: 0.00038 Time taken: 0:00:02.872940 ETA: 0:00:20.110580 2020-07-01 06:38:49,998 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 114/120: loss: 0.00038 Time taken: 0:00:02.862607 ETA: 0:00:17.175642 2020-07-01 06:38:49,999 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.331 2020-07-01 06:38:52,888 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 115/120: loss: 0.00038 Time taken: 0:00:02.888509 ETA: 0:00:14.442545 2020-07-01 06:38:54,504 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.194 2020-07-01 06:38:55,776 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 116/120: loss: 0.00038 Time taken: 0:00:02.888667 ETA: 0:00:11.554668 2020-07-01 06:38:58,645 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 117/120: loss: 0.00038 Time taken: 0:00:02.867858 ETA: 0:00:08.603574 2020-07-01 06:38:59,007 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.211 2020-07-01 06:39:01,539 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 118/120: loss: 0.00038 Time taken: 0:00:02.892958 ETA: 0:00:05.785916 2020-07-01 06:39:03,540 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.060 2020-07-01 06:39:04,444 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 119/120: loss: 0.00039 Time taken: 0:00:02.903297 ETA: 0:00:02.903297 2020-07-01 06:39:11,159 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 21/21 [00:00<00:00, 60 Matching predictions to ground truth, class 2/3.: 100%|█| 6/6 [00:00<00:00, 6115 Matching predictions to ground truth, class 3/3.: 100%|█| 23/23 [00:00<00:00, 70 Epoch 120/120 ========================= Validation cost: 0.000079 Mean average_precision (in %): 68.0556 class name average precision (in %) ------------ -------------------------- bus 83.3333 followme 37.5 police 83.3333 Median Inference Time: 0.020477 Time taken to run iva.detectnet_v2.scripts.train:main: 0:07:36.404767.