Using TensorFlow backend. 2020-07-01 05:07:26,046 [INFO] iva.detectnet_v2.scripts.train: Loading experiment spec at /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti_lego.txt. 2020-07-01 05:07:26,047 [INFO] iva.detectnet_v2.spec_handler.spec_loader: Merging specification from /workspace/examples/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti_lego.txt 2020-07-01 05:07:26,068 [INFO] iva.detectnet_v2.scripts.train: Cannot iterate over exactly 61 samples with a batch size of 4; each epoch will therefore take one extra step. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 544, 1408) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 272, 704) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 272, 704) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 272, 704) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 136, 352) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 136, 352) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 136, 352) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 136, 352) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 136, 352) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 136, 352) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 136, 352) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 136, 352) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 136, 352) 0 add_1[0][0] __________________________________________________________________________________________________ block_1b_conv_1 (Conv2D) (None, 64, 136, 352) 36928 block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_bn_1 (BatchNormalizati (None, 64, 136, 352) 256 block_1b_conv_1[0][0] __________________________________________________________________________________________________ block_1b_relu_1 (Activation) (None, 64, 136, 352) 0 block_1b_bn_1[0][0] __________________________________________________________________________________________________ block_1b_conv_2 (Conv2D) (None, 64, 136, 352) 36928 block_1b_relu_1[0][0] __________________________________________________________________________________________________ block_1b_bn_2 (BatchNormalizati (None, 64, 136, 352) 256 block_1b_conv_2[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 64, 136, 352) 0 block_1b_bn_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ block_1b_relu (Activation) (None, 64, 136, 352) 0 add_2[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 68, 176) 73856 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 68, 176) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 68, 176) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 68, 176) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 68, 176) 8320 block_1b_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 68, 176) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 68, 176) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 128, 68, 176) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 68, 176) 0 add_3[0][0] __________________________________________________________________________________________________ block_2b_conv_1 (Conv2D) (None, 128, 68, 176) 147584 block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_bn_1 (BatchNormalizati (None, 128, 68, 176) 512 block_2b_conv_1[0][0] __________________________________________________________________________________________________ block_2b_relu_1 (Activation) (None, 128, 68, 176) 0 block_2b_bn_1[0][0] __________________________________________________________________________________________________ block_2b_conv_2 (Conv2D) (None, 128, 68, 176) 147584 block_2b_relu_1[0][0] __________________________________________________________________________________________________ block_2b_bn_2 (BatchNormalizati (None, 128, 68, 176) 512 block_2b_conv_2[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 128, 68, 176) 0 block_2b_bn_2[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ block_2b_relu (Activation) (None, 128, 68, 176) 0 add_4[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 34, 88) 295168 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 34, 88) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 34, 88) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 34, 88) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 34, 88) 33024 block_2b_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 34, 88) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 34, 88) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_5 (Add) (None, 256, 34, 88) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 34, 88) 0 add_5[0][0] __________________________________________________________________________________________________ block_3b_conv_1 (Conv2D) (None, 256, 34, 88) 590080 block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_bn_1 (BatchNormalizati (None, 256, 34, 88) 1024 block_3b_conv_1[0][0] __________________________________________________________________________________________________ block_3b_relu_1 (Activation) (None, 256, 34, 88) 0 block_3b_bn_1[0][0] __________________________________________________________________________________________________ block_3b_conv_2 (Conv2D) (None, 256, 34, 88) 590080 block_3b_relu_1[0][0] __________________________________________________________________________________________________ block_3b_bn_2 (BatchNormalizati (None, 256, 34, 88) 1024 block_3b_conv_2[0][0] __________________________________________________________________________________________________ add_6 (Add) (None, 256, 34, 88) 0 block_3b_bn_2[0][0] block_3a_relu[0][0] __________________________________________________________________________________________________ block_3b_relu (Activation) (None, 256, 34, 88) 0 add_6[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 34, 88) 1180160 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 34, 88) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 34, 88) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 34, 88) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 34, 88) 131584 block_3b_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 34, 88) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 34, 88) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_7 (Add) (None, 512, 34, 88) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 34, 88) 0 add_7[0][0] __________________________________________________________________________________________________ block_4b_conv_1 (Conv2D) (None, 512, 34, 88) 2359808 block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_bn_1 (BatchNormalizati (None, 512, 34, 88) 2048 block_4b_conv_1[0][0] __________________________________________________________________________________________________ block_4b_relu_1 (Activation) (None, 512, 34, 88) 0 block_4b_bn_1[0][0] __________________________________________________________________________________________________ block_4b_conv_2 (Conv2D) (None, 512, 34, 88) 2359808 block_4b_relu_1[0][0] __________________________________________________________________________________________________ block_4b_bn_2 (BatchNormalizati (None, 512, 34, 88) 2048 block_4b_conv_2[0][0] __________________________________________________________________________________________________ add_8 (Add) (None, 512, 34, 88) 0 block_4b_bn_2[0][0] block_4a_relu[0][0] __________________________________________________________________________________________________ block_4b_relu (Activation) (None, 512, 34, 88) 0 add_8[0][0] __________________________________________________________________________________________________ output_bbox (Conv2D) (None, 12, 34, 88) 6156 block_4b_relu[0][0] __________________________________________________________________________________________________ output_cov (Conv2D) (None, 3, 34, 88) 1539 block_4b_relu[0][0] ================================================================================================== Total params: 11,203,023 Trainable params: 11,193,295 Non-trainable params: 9,728 __________________________________________________________________________________________________ target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. 2020-07-01 05:07:36,172 [INFO] iva.detectnet_v2.scripts.train: Found 61 samples in training set target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. target/truncation is not updated to match the crop areaif the dataset contains target/truncation. 2020-07-01 05:07:41,828 [INFO] iva.detectnet_v2.scripts.train: Found 9 samples in validation set 2020-07-01 05:08:20,108 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 0/120: loss: 0.08885 Time taken: 0:00:00 ETA: 0:00:00 2020-07-01 05:08:25,581 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 1/120: loss: 0.00137 Time taken: 0:00:10.314594 ETA: 0:20:27.436686 2020-07-01 05:08:27,052 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 8.357 2020-07-01 05:08:28,530 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 2/120: loss: 0.00131 Time taken: 0:00:02.949130 ETA: 0:05:47.997340 2020-07-01 05:08:31,431 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 3/120: loss: 0.00108 Time taken: 0:00:02.903438 ETA: 0:05:39.702246 2020-07-01 05:08:31,620 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.893 2020-07-01 05:08:34,303 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 4/120: loss: 0.00129 Time taken: 0:00:02.868533 ETA: 0:05:32.749828 2020-07-01 05:08:36,088 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.382 2020-07-01 05:08:37,179 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 5/120: loss: 0.00113 Time taken: 0:00:02.877233 ETA: 0:05:30.881795 2020-07-01 05:08:40,052 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 6/120: loss: 0.00123 Time taken: 0:00:02.873048 ETA: 0:05:27.527472 2020-07-01 05:08:40,597 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.180 2020-07-01 05:08:42,935 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 7/120: loss: 0.00104 Time taken: 0:00:02.882584 ETA: 0:05:25.731992 2020-07-01 05:08:45,084 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.286 2020-07-01 05:08:45,820 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 8/120: loss: 0.00101 Time taken: 0:00:02.883155 ETA: 0:05:22.913360 2020-07-01 05:08:48,707 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 9/120: loss: 0.00087 Time taken: 0:00:02.883075 ETA: 0:05:20.021325 2020-07-01 05:08:49,663 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.842 2020-07-01 05:08:55,625 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 10/120: loss: 0.00095 Time taken: 0:00:06.910633 ETA: 0:12:40.169630 2020-07-01 05:08:58,120 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.825 2020-07-01 05:08:58,489 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 11/120: loss: 0.00086 Time taken: 0:00:02.868469 ETA: 0:05:12.663121 2020-07-01 05:09:01,366 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 12/120: loss: 0.00090 Time taken: 0:00:02.882666 ETA: 0:05:11.327928 2020-07-01 05:09:02,630 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.174 2020-07-01 05:09:04,245 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 13/120: loss: 0.00086 Time taken: 0:00:02.878881 ETA: 0:05:08.040267 2020-07-01 05:09:07,169 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 14/120: loss: 0.00091 Time taken: 0:00:02.918012 ETA: 0:05:09.309272 2020-07-01 05:09:07,169 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.031 2020-07-01 05:09:10,036 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 15/120: loss: 0.00077 Time taken: 0:00:02.868172 ETA: 0:05:01.158060 2020-07-01 05:09:11,650 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.318 2020-07-01 05:09:12,911 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 16/120: loss: 0.00083 Time taken: 0:00:02.872896 ETA: 0:04:58.781184 2020-07-01 05:09:15,787 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.00089 Time taken: 0:00:02.878411 ETA: 0:04:56.476333 2020-07-01 05:09:16,213 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.917 2020-07-01 05:09:18,750 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.00080 Time taken: 0:00:02.960132 ETA: 0:05:01.933464 2020-07-01 05:09:20,717 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.202 2020-07-01 05:09:21,625 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 19/120: loss: 0.00084 Time taken: 0:00:02.872602 ETA: 0:04:50.132802 2020-07-01 05:09:28,473 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 20/120: loss: 0.00076 Time taken: 0:00:06.826958 ETA: 0:11:22.695800 2020-07-01 05:09:29,247 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 11.723 2020-07-01 05:09:31,534 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 21/120: loss: 0.00081 Time taken: 0:00:03.070580 ETA: 0:05:03.987420 2020-07-01 05:09:33,890 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.540 2020-07-01 05:09:34,438 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 22/120: loss: 0.00078 Time taken: 0:00:02.918191 ETA: 0:04:45.982718 2020-07-01 05:09:37,321 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 23/120: loss: 0.00076 Time taken: 0:00:02.883137 ETA: 0:04:39.664289 2020-07-01 05:09:38,404 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.158 2020-07-01 05:09:40,205 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 24/120: loss: 0.00076 Time taken: 0:00:02.877884 ETA: 0:04:36.276864 2020-07-01 05:09:42,907 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.206 2020-07-01 05:09:43,094 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 25/120: loss: 0.00078 Time taken: 0:00:02.883056 ETA: 0:04:33.890320 2020-07-01 05:09:45,984 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 26/120: loss: 0.00077 Time taken: 0:00:02.897749 ETA: 0:04:32.388406 2020-07-01 05:09:47,429 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.118 2020-07-01 05:09:48,863 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 27/120: loss: 0.00078 Time taken: 0:00:02.878069 ETA: 0:04:27.660417 2020-07-01 05:09:51,749 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 28/120: loss: 0.00073 Time taken: 0:00:02.883166 ETA: 0:04:25.251272 2020-07-01 05:09:52,015 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.808 2020-07-01 05:09:54,708 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 29/120: loss: 0.00072 Time taken: 0:00:02.959863 ETA: 0:04:29.347533 2020-07-01 05:09:56,499 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.301 2020-07-01 05:10:01,264 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 6005/6005 [00:00<00:00 Matching predictions to ground truth, class 2/3.: 100%|█| 4942/4942 [00:00<00:00 Matching predictions to ground truth, class 3/3.: 100%|█| 5103/5103 [00:00<00:00 Epoch 30/120 ========================= Validation cost: 0.000187 Mean average_precision (in %): 25.7630 class name average precision (in %) ------------ -------------------------- bus 47.7778 followme 6.52174 police 22.9896 Median Inference Time: 0.156410 2020-07-01 05:10:12,738 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 30/120: loss: 0.00069 Time taken: 0:00:18.023172 ETA: 0:27:02.085480 2020-07-01 05:10:15,599 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 31/120: loss: 0.00069 Time taken: 0:00:02.866165 ETA: 0:04:15.088685 2020-07-01 05:10:16,138 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 5.092 2020-07-01 05:10:18,473 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 32/120: loss: 0.00079 Time taken: 0:00:02.877740 ETA: 0:04:13.241120 2020-07-01 05:10:20,616 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.334 2020-07-01 05:10:21,343 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 33/120: loss: 0.00072 Time taken: 0:00:02.863170 ETA: 0:04:09.095790 2020-07-01 05:10:24,369 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 34/120: loss: 0.00071 Time taken: 0:00:03.014434 ETA: 0:04:19.241324 2020-07-01 05:10:25,284 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.424 2020-07-01 05:10:27,323 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 35/120: loss: 0.00069 Time taken: 0:00:02.964418 ETA: 0:04:11.975530 2020-07-01 05:10:29,849 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.909 2020-07-01 05:10:30,218 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 36/120: loss: 0.00069 Time taken: 0:00:02.898591 ETA: 0:04:03.481644 2020-07-01 05:10:33,111 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 37/120: loss: 0.00071 Time taken: 0:00:02.888084 ETA: 0:03:59.710972 2020-07-01 05:10:34,377 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.086 2020-07-01 05:10:36,015 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 38/120: loss: 0.00072 Time taken: 0:00:02.903198 ETA: 0:03:58.062236 2020-07-01 05:10:38,900 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 39/120: loss: 0.00068 Time taken: 0:00:02.887851 ETA: 0:03:53.915931 2020-07-01 05:10:38,900 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.110 2020-07-01 05:10:45,702 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 3625/3625 [00:00<00:00 Matching predictions to ground truth, class 2/3.: 100%|█| 2049/2049 [00:00<00:00 Matching predictions to ground truth, class 3/3.: 100%|█| 3163/3163 [00:00<00:00 Epoch 40/120 ========================= Validation cost: 0.000145 Mean average_precision (in %): 32.7151 class name average precision (in %) ------------ -------------------------- bus 30 followme 27.9412 police 40.2041 Median Inference Time: 0.020632 2020-07-01 05:10:54,675 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 40/120: loss: 0.00071 Time taken: 0:00:15.757471 ETA: 0:21:00.597680 2020-07-01 05:10:56,354 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 5.729 2020-07-01 05:10:57,601 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 41/120: loss: 0.00071 Time taken: 0:00:02.939922 ETA: 0:03:52.253838 2020-07-01 05:11:00,461 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 42/120: loss: 0.00071 Time taken: 0:00:02.857513 ETA: 0:03:42.886014 2020-07-01 05:11:00,824 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.376 2020-07-01 05:11:03,330 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 43/120: loss: 0.00071 Time taken: 0:00:02.867886 ETA: 0:03:40.827222 2020-07-01 05:11:05,310 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.291 2020-07-01 05:11:06,220 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 44/120: loss: 0.00069 Time taken: 0:00:02.894095 ETA: 0:03:39.951220 2020-07-01 05:11:09,099 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 45/120: loss: 0.00068 Time taken: 0:00:02.877408 ETA: 0:03:35.805600 2020-07-01 05:11:09,828 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.136 2020-07-01 05:11:11,983 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 46/120: loss: 0.00073 Time taken: 0:00:02.883044 ETA: 0:03:33.345256 2020-07-01 05:11:14,327 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.229 2020-07-01 05:11:14,871 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 47/120: loss: 0.00068 Time taken: 0:00:02.887777 ETA: 0:03:30.807721 2020-07-01 05:11:17,764 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 48/120: loss: 0.00075 Time taken: 0:00:02.893345 ETA: 0:03:28.320840 2020-07-01 05:11:18,859 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.068 2020-07-01 05:11:20,656 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 49/120: loss: 0.00070 Time taken: 0:00:02.887556 ETA: 0:03:25.016476 2020-07-01 05:11:27,246 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 1377/1377 [00:00<00:00 Matching predictions to ground truth, class 2/3.: 100%|█| 905/905 [00:00<00:00, Matching predictions to ground truth, class 3/3.: 100%|█| 1973/1973 [00:00<00:00 Epoch 50/120 ========================= Validation cost: 0.000119 Mean average_precision (in %): 22.2153 class name average precision (in %) ------------ -------------------------- bus 28 followme 9.23077 police 29.415 Median Inference Time: 0.020414 2020-07-01 05:11:35,394 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 6.048 2020-07-01 05:11:35,587 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 50/120: loss: 0.00069 Time taken: 0:00:14.917894 ETA: 0:17:24.252580 2020-07-01 05:11:38,453 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 51/120: loss: 0.00066 Time taken: 0:00:02.877092 ETA: 0:03:18.519348 2020-07-01 05:11:39,878 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.305 2020-07-01 05:11:41,312 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 52/120: loss: 0.00070 Time taken: 0:00:02.862406 ETA: 0:03:14.643608 2020-07-01 05:11:44,184 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 53/120: loss: 0.00066 Time taken: 0:00:02.867774 ETA: 0:03:12.140858 2020-07-01 05:11:44,367 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.281 2020-07-01 05:11:47,070 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 54/120: loss: 0.00068 Time taken: 0:00:02.883484 ETA: 0:03:10.309944 2020-07-01 05:11:48,857 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.273 2020-07-01 05:11:49,944 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 55/120: loss: 0.00071 Time taken: 0:00:02.873168 ETA: 0:03:06.755920 2020-07-01 05:11:52,832 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 56/120: loss: 0.00067 Time taken: 0:00:02.888361 ETA: 0:03:04.855104 2020-07-01 05:11:53,375 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.137 2020-07-01 05:11:55,723 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 57/120: loss: 0.00069 Time taken: 0:00:02.893057 ETA: 0:03:02.262591 2020-07-01 05:11:57,869 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.250 2020-07-01 05:11:58,587 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 58/120: loss: 0.00071 Time taken: 0:00:02.862855 ETA: 0:02:57.497010 2020-07-01 05:12:01,474 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 59/120: loss: 0.00070 Time taken: 0:00:02.883348 ETA: 0:02:55.884228 2020-07-01 05:12:02,374 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.202 2020-07-01 05:12:08,150 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 563/563 [00:00<00:00, Matching predictions to ground truth, class 2/3.: 100%|█| 396/396 [00:00<00:00, Matching predictions to ground truth, class 3/3.: 100%|█| 946/946 [00:00<00:00, Epoch 60/120 ========================= Validation cost: 0.000122 Mean average_precision (in %): 25.8906 class name average precision (in %) ------------ -------------------------- bus 38.7879 followme 5 police 33.8838 Median Inference Time: 0.020297 2020-07-01 05:12:15,844 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 60/120: loss: 0.00067 Time taken: 0:00:14.359244 ETA: 0:14:21.554640 2020-07-01 05:12:18,336 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 6.265 2020-07-01 05:12:18,701 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 61/120: loss: 0.00069 Time taken: 0:00:02.869307 ETA: 0:02:49.289113 2020-07-01 05:12:21,627 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 62/120: loss: 0.00070 Time taken: 0:00:02.924329 ETA: 0:02:49.611082 2020-07-01 05:12:22,875 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.032 2020-07-01 05:12:24,490 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 63/120: loss: 0.00065 Time taken: 0:00:02.862107 ETA: 0:02:43.140099 2020-07-01 05:12:27,365 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 64/120: loss: 0.00064 Time taken: 0:00:02.873221 ETA: 0:02:40.900376 2020-07-01 05:12:27,365 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.271 2020-07-01 05:12:30,246 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 65/120: loss: 0.00073 Time taken: 0:00:02.882676 ETA: 0:02:38.547180 2020-07-01 05:12:31,860 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.249 2020-07-01 05:12:33,126 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 66/120: loss: 0.00068 Time taken: 0:00:02.877097 ETA: 0:02:35.363238 2020-07-01 05:12:36,002 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 67/120: loss: 0.00068 Time taken: 0:00:02.878802 ETA: 0:02:32.576506 2020-07-01 05:12:36,366 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.192 2020-07-01 05:12:38,883 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 68/120: loss: 0.00067 Time taken: 0:00:02.877340 ETA: 0:02:29.621680 2020-07-01 05:12:40,874 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.186 2020-07-01 05:12:41,794 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 69/120: loss: 0.00069 Time taken: 0:00:02.899972 ETA: 0:02:27.898572 2020-07-01 05:12:48,509 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 308/308 [00:00<00:00, Matching predictions to ground truth, class 2/3.: 100%|█| 226/226 [00:00<00:00, Matching predictions to ground truth, class 3/3.: 100%|█| 435/435 [00:00<00:00, Epoch 70/120 ========================= Validation cost: 0.000116 Mean average_precision (in %): 18.2778 class name average precision (in %) ------------ -------------------------- bus 23.3333 followme 10 police 21.5 Median Inference Time: 0.020104 2020-07-01 05:12:56,084 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 70/120: loss: 0.00070 Time taken: 0:00:14.286557 ETA: 0:11:54.327850 2020-07-01 05:12:56,798 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 6.280 2020-07-01 05:12:58,942 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 71/120: loss: 0.00067 Time taken: 0:00:02.872361 ETA: 0:02:20.745689 2020-07-01 05:13:01,269 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.368 2020-07-01 05:13:01,806 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 72/120: loss: 0.00064 Time taken: 0:00:02.863438 ETA: 0:02:17.445024 2020-07-01 05:13:04,661 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 73/120: loss: 0.00068 Time taken: 0:00:02.852289 ETA: 0:02:14.057583 2020-07-01 05:13:05,788 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.131 2020-07-01 05:13:07,577 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 74/120: loss: 0.00067 Time taken: 0:00:02.913692 ETA: 0:02:14.029832 2020-07-01 05:13:10,285 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.238 2020-07-01 05:13:10,463 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 75/120: loss: 0.00067 Time taken: 0:00:02.888421 ETA: 0:02:09.978945 2020-07-01 05:13:13,348 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 76/120: loss: 0.00066 Time taken: 0:00:02.874927 ETA: 0:02:06.496788 2020-07-01 05:13:14,780 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.250 2020-07-01 05:13:16,237 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 77/120: loss: 0.00064 Time taken: 0:00:02.890089 ETA: 0:02:04.273827 2020-07-01 05:13:19,127 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 78/120: loss: 0.00065 Time taken: 0:00:02.898171 ETA: 0:02:01.723182 2020-07-01 05:13:19,310 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.077 2020-07-01 05:13:21,996 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 79/120: loss: 0.00063 Time taken: 0:00:02.867892 ETA: 0:01:57.583572 2020-07-01 05:13:23,794 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.303 2020-07-01 05:13:28,525 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 191/191 [00:00<00:00, Matching predictions to ground truth, class 2/3.: 100%|█| 158/158 [00:00<00:00, Matching predictions to ground truth, class 3/3.: 100%|█| 322/322 [00:00<00:00, Epoch 80/120 ========================= Validation cost: 0.000111 Mean average_precision (in %): 35.2685 class name average precision (in %) ------------ -------------------------- bus 60 followme 17.3333 police 28.4722 Median Inference Time: 0.019987 2020-07-01 05:13:35,407 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 80/120: loss: 0.00063 Time taken: 0:00:13.402729 ETA: 0:08:56.109160 2020-07-01 05:13:38,260 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 81/120: loss: 0.00068 Time taken: 0:00:02.858708 ETA: 0:01:51.489612 2020-07-01 05:13:38,800 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 6.664 2020-07-01 05:13:41,128 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 82/120: loss: 0.00062 Time taken: 0:00:02.868018 ETA: 0:01:48.984684 2020-07-01 05:13:43,277 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.340 2020-07-01 05:13:43,999 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 83/120: loss: 0.00062 Time taken: 0:00:02.867408 ETA: 0:01:46.094096 2020-07-01 05:13:46,863 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 84/120: loss: 0.00064 Time taken: 0:00:02.862898 ETA: 0:01:43.064328 2020-07-01 05:13:47,818 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.024 2020-07-01 05:13:49,793 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 85/120: loss: 0.00068 Time taken: 0:00:02.928618 ETA: 0:01:42.501630 2020-07-01 05:13:52,296 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.329 2020-07-01 05:13:52,665 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 86/120: loss: 0.00064 Time taken: 0:00:02.869990 ETA: 0:01:37.579660 2020-07-01 05:13:55,605 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 87/120: loss: 0.00065 Time taken: 0:00:02.940917 ETA: 0:01:37.050261 2020-07-01 05:13:56,864 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.895 2020-07-01 05:13:58,501 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 88/120: loss: 0.00063 Time taken: 0:00:02.893724 ETA: 0:01:32.599168 2020-07-01 05:14:01,384 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 89/120: loss: 0.00065 Time taken: 0:00:02.882940 ETA: 0:01:29.371140 2020-07-01 05:14:01,432 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 21.890 2020-07-01 05:14:08,020 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 173/173 [00:00<00:00, Matching predictions to ground truth, class 2/3.: 100%|█| 79/79 [00:00<00:00, 11 Matching predictions to ground truth, class 3/3.: 100%|█| 220/220 [00:00<00:00, Epoch 90/120 ========================= Validation cost: 0.000107 Mean average_precision (in %): 39.6296 class name average precision (in %) ------------ -------------------------- bus 55 followme 30.5556 police 33.3333 Median Inference Time: 0.019671 2020-07-01 05:14:14,739 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 90/120: loss: 0.00064 Time taken: 0:00:13.346373 ETA: 0:06:40.391190 2020-07-01 05:14:16,340 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 6.708 2020-07-01 05:14:17,610 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 91/120: loss: 0.00063 Time taken: 0:00:02.876412 ETA: 0:01:23.415948 2020-07-01 05:14:20,474 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 92/120: loss: 0.00062 Time taken: 0:00:02.865019 ETA: 0:01:20.220532 2020-07-01 05:14:20,834 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.255 2020-07-01 05:14:23,339 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 93/120: loss: 0.00064 Time taken: 0:00:02.867771 ETA: 0:01:17.429817 2020-07-01 05:14:25,310 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.342 2020-07-01 05:14:26,214 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 94/120: loss: 0.00062 Time taken: 0:00:02.873258 ETA: 0:01:14.704708 2020-07-01 05:14:29,086 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 95/120: loss: 0.00062 Time taken: 0:00:02.867850 ETA: 0:01:11.696250 2020-07-01 05:14:29,812 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.212 2020-07-01 05:14:31,966 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 96/120: loss: 0.00062 Time taken: 0:00:02.882985 ETA: 0:01:09.191640 2020-07-01 05:14:34,297 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.298 2020-07-01 05:14:34,839 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 97/120: loss: 0.00064 Time taken: 0:00:02.872632 ETA: 0:01:06.070536 2020-07-01 05:14:37,749 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 98/120: loss: 0.00064 Time taken: 0:00:02.908649 ETA: 0:01:03.990278 2020-07-01 05:14:38,827 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.080 2020-07-01 05:14:40,647 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 99/120: loss: 0.00064 Time taken: 0:00:02.885750 ETA: 0:01:00.600750 2020-07-01 05:14:47,192 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 146/146 [00:00<00:00, Matching predictions to ground truth, class 2/3.: 100%|█| 76/76 [00:00<00:00, 10 Matching predictions to ground truth, class 3/3.: 100%|█| 159/159 [00:00<00:00, Epoch 100/120 ========================= Validation cost: 0.000092 Mean average_precision (in %): 35.6879 class name average precision (in %) ------------ -------------------------- bus 60 followme 26.0638 police 21 Median Inference Time: 0.020146 2020-07-01 05:14:53,498 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 6.816 2020-07-01 05:14:53,686 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 100/120: loss: 0.00065 Time taken: 0:00:13.039179 ETA: 0:04:20.783580 2020-07-01 05:14:56,548 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 101/120: loss: 0.00064 Time taken: 0:00:02.866234 ETA: 0:00:54.458446 2020-07-01 05:14:57,983 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.304 2020-07-01 05:14:59,423 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 102/120: loss: 0.00064 Time taken: 0:00:02.877731 ETA: 0:00:51.799158 2020-07-01 05:15:02,292 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 103/120: loss: 0.00063 Time taken: 0:00:02.867092 ETA: 0:00:48.740564 2020-07-01 05:15:02,474 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.269 2020-07-01 05:15:05,173 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 104/120: loss: 0.00063 Time taken: 0:00:02.883771 ETA: 0:00:46.140336 2020-07-01 05:15:06,954 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.319 2020-07-01 05:15:08,035 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 105/120: loss: 0.00065 Time taken: 0:00:02.857277 ETA: 0:00:42.859155 2020-07-01 05:15:10,915 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 106/120: loss: 0.00063 Time taken: 0:00:02.883168 ETA: 0:00:40.364352 2020-07-01 05:15:11,460 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.197 2020-07-01 05:15:13,794 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 107/120: loss: 0.00064 Time taken: 0:00:02.877675 ETA: 0:00:37.409775 2020-07-01 05:15:15,942 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.312 2020-07-01 05:15:16,676 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 108/120: loss: 0.00061 Time taken: 0:00:02.877854 ETA: 0:00:34.534248 2020-07-01 05:15:19,565 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 109/120: loss: 0.00061 Time taken: 0:00:02.887729 ETA: 0:00:31.765019 2020-07-01 05:15:20,463 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.119 2020-07-01 05:15:26,195 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 163/163 [00:00<00:00, Matching predictions to ground truth, class 2/3.: 100%|█| 63/63 [00:00<00:00, 82 Matching predictions to ground truth, class 3/3.: 100%|█| 156/156 [00:00<00:00, Epoch 110/120 ========================= Validation cost: 0.000103 Mean average_precision (in %): 41.7107 class name average precision (in %) ------------ -------------------------- bus 66.6667 followme 25.1321 police 33.3333 Median Inference Time: 0.020074 2020-07-01 05:15:32,731 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 110/120: loss: 0.00062 Time taken: 0:00:13.157517 ETA: 0:02:11.575170 2020-07-01 05:15:35,230 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 6.772 2020-07-01 05:15:35,595 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 111/120: loss: 0.00063 Time taken: 0:00:02.872178 ETA: 0:00:25.849602 2020-07-01 05:15:38,445 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 112/120: loss: 0.00062 Time taken: 0:00:02.852995 ETA: 0:00:22.823960 2020-07-01 05:15:39,702 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.365 2020-07-01 05:15:41,318 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 113/120: loss: 0.00064 Time taken: 0:00:02.872751 ETA: 0:00:20.109257 2020-07-01 05:15:44,191 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 114/120: loss: 0.00063 Time taken: 0:00:02.868006 ETA: 0:00:17.208036 2020-07-01 05:15:44,191 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.276 2020-07-01 05:15:47,066 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 115/120: loss: 0.00063 Time taken: 0:00:02.878372 ETA: 0:00:14.391860 2020-07-01 05:15:48,682 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.265 2020-07-01 05:15:49,939 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 116/120: loss: 0.00061 Time taken: 0:00:02.872742 ETA: 0:00:11.490968 2020-07-01 05:15:52,820 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 117/120: loss: 0.00064 Time taken: 0:00:02.878651 ETA: 0:00:08.635953 2020-07-01 05:15:53,181 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.233 2020-07-01 05:15:55,702 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 118/120: loss: 0.00062 Time taken: 0:00:02.882417 ETA: 0:00:05.764834 2020-07-01 05:15:57,683 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 22.213 2020-07-01 05:15:58,592 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 119/120: loss: 0.00066 Time taken: 0:00:02.888434 ETA: 0:00:02.888434 2020-07-01 05:16:05,021 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 2, 0.00s/step Matching predictions to ground truth, class 1/3.: 100%|█| 146/146 [00:00<00:00, Matching predictions to ground truth, class 2/3.: 100%|█| 59/59 [00:00<00:00, 81 Matching predictions to ground truth, class 3/3.: 100%|█| 134/134 [00:00<00:00, Epoch 120/120 ========================= Validation cost: 0.000105 Mean average_precision (in %): 32.8116 class name average precision (in %) ------------ -------------------------- bus 63.3333 followme 6.04167 police 29.0598 Median Inference Time: 0.019658 Time taken to run iva.detectnet_v2.scripts.train:main: 0:08:47.516181. print("For multi-GPU, please uncomment and run this instead. Change --gpus based on your machine.") # Please check the outp