INFO:tensorflow:epoch = 0.0, loss = 0.08217385, step = 0
2020-01-13 09:03:11,264 [INFO] tensorflow: epoch = 0.0, loss = 0.08217385, step = 0
2020-01-13 09:03:11,266 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 0/120: loss: 0.08217 Time taken: 0:00:00 ETA: 0:00:00
INFO:tensorflow:global_step/sec: 0.13889
2020-01-13 09:03:18,465 [INFO] tensorflow: global_step/sec: 0.13889
INFO:tensorflow:epoch = 0.25, loss = 0.081661165, step = 2 (8.003 sec)
2020-01-13 09:03:19,266 [INFO] tensorflow: epoch = 0.25, loss = 0.081661165, step = 2 (8.003 sec)
INFO:tensorflow:global_step/sec: 1.24539
2020-01-13 09:03:19,268 [INFO] tensorflow: global_step/sec: 1.24539
INFO:tensorflow:global_step/sec: 1.23939
2020-01-13 09:03:20,075 [INFO] tensorflow: global_step/sec: 1.23939
INFO:tensorflow:global_step/sec: 1.25462
2020-01-13 09:03:20,872 [INFO] tensorflow: global_step/sec: 1.25462
INFO:tensorflow:global_step/sec: 1.24212
2020-01-13 09:03:21,677 [INFO] tensorflow: global_step/sec: 1.24212
INFO:tensorflow:global_step/sec: 1.25098
2020-01-13 09:03:22,476 [INFO] tensorflow: global_step/sec: 1.25098
INFO:tensorflow:global_step/sec: 1.23253
2020-01-13 09:03:23,288 [INFO] tensorflow: global_step/sec: 1.23253
INFO:tensorflow:global_step/sec: 0.671053
2020-01-13 09:03:24,778 [INFO] tensorflow: global_step/sec: 0.671053
2020-01-13 09:03:24,779 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 1/120: loss: 0.88263 Time taken: 0:00:28.790975 ETA: 0:57:06.126025
INFO:tensorflow:epoch = 1.125, loss = 0.89238954, step = 9 (6.317 sec)
2020-01-13 09:03:25,583 [INFO] tensorflow: epoch = 1.125, loss = 0.89238954, step = 9 (6.317 sec)
INFO:tensorflow:global_step/sec: 1.23907
2020-01-13 09:03:25,585 [INFO] tensorflow: global_step/sec: 1.23907
INFO:tensorflow:global_step/sec: 1.25482
2020-01-13 09:03:26,382 [INFO] tensorflow: global_step/sec: 1.25482
INFO:tensorflow:global_step/sec: 1.23969
2020-01-13 09:03:27,188 [INFO] tensorflow: global_step/sec: 1.23969
INFO:tensorflow:global_step/sec: 1.25345
2020-01-13 09:03:27,986 [INFO] tensorflow: global_step/sec: 1.25345
INFO:tensorflow:global_step/sec: 1.24189
2020-01-13 09:03:28,791 [INFO] tensorflow: global_step/sec: 1.24189
INFO:tensorflow:global_step/sec: 1.23142
2020-01-13 09:03:29,604 [INFO] tensorflow: global_step/sec: 1.23142
INFO:tensorflow:global_step/sec: 1.25021
2020-01-13 09:03:30,403 [INFO] tensorflow: global_step/sec: 1.25021
INFO:tensorflow:global_step/sec: 1.23834
2020-01-13 09:03:31,211 [INFO] tensorflow: global_step/sec: 1.23834
2020-01-13 09:03:31,212 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 2/120: loss: 0.88019 Time taken: 0:00:06.424891 ETA: 0:12:38.137138
INFO:tensorflow:epoch = 2.125, loss = 0.88394403, step = 17 (6.437 sec)
2020-01-13 09:03:32,020 [INFO] tensorflow: epoch = 2.125, loss = 0.88394403, step = 17 (6.437 sec)
INFO:tensorflow:global_step/sec: 1.23332
2020-01-13 09:03:32,022 [INFO] tensorflow: global_step/sec: 1.23332
INFO:tensorflow:global_step/sec: 1.23491
2020-01-13 09:03:32,832 [INFO] tensorflow: global_step/sec: 1.23491
INFO:tensorflow:global_step/sec: 1.23355
2020-01-13 09:03:33,642 [INFO] tensorflow: global_step/sec: 1.23355
INFO:tensorflow:global_step/sec: 1.23715
2020-01-13 09:03:34,450 [INFO] tensorflow: global_step/sec: 1.23715
INFO:tensorflow:global_step/sec: 1.23514
2020-01-13 09:03:35,260 [INFO] tensorflow: global_step/sec: 1.23514
INFO:tensorflow:global_step/sec: 1.23509
2020-01-13 09:03:36,070 [INFO] tensorflow: global_step/sec: 1.23509
INFO:tensorflow:global_step/sec: 1.23888
2020-01-13 09:03:36,877 [INFO] tensorflow: global_step/sec: 1.23888
INFO:tensorflow:global_step/sec: 1.2291
2020-01-13 09:03:37,691 [INFO] tensorflow: global_step/sec: 1.2291
2020-01-13 09:03:37,692 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 3/120: loss: 0.84273 Time taken: 0:00:06.469167 ETA: 0:12:36.892539
2020-01-13 09:03:37,693 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.412
INFO:tensorflow:epoch = 3.125, loss = 0.84840333, step = 25 (6.482 sec)
2020-01-13 09:03:38,502 [INFO] tensorflow: epoch = 3.125, loss = 0.84840333, step = 25 (6.482 sec)
INFO:tensorflow:global_step/sec: 1.22973
2020-01-13 09:03:38,504 [INFO] tensorflow: global_step/sec: 1.22973
INFO:tensorflow:global_step/sec: 1.23322
2020-01-13 09:03:39,315 [INFO] tensorflow: global_step/sec: 1.23322
INFO:tensorflow:global_step/sec: 1.23214
2020-01-13 09:03:40,126 [INFO] tensorflow: global_step/sec: 1.23214
INFO:tensorflow:global_step/sec: 1.23505
2020-01-13 09:03:40,936 [INFO] tensorflow: global_step/sec: 1.23505
INFO:tensorflow:global_step/sec: 1.23297
2020-01-13 09:03:41,747 [INFO] tensorflow: global_step/sec: 1.23297
INFO:tensorflow:global_step/sec: 1.23499
2020-01-13 09:03:42,557 [INFO] tensorflow: global_step/sec: 1.23499
INFO:tensorflow:global_step/sec: 1.23409
2020-01-13 09:03:43,367 [INFO] tensorflow: global_step/sec: 1.23409
INFO:tensorflow:global_step/sec: 1.22928
2020-01-13 09:03:44,180 [INFO] tensorflow: global_step/sec: 1.22928
2020-01-13 09:03:44,182 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 4/120: loss: 0.82718 Time taken: 0:00:06.487857 ETA: 0:12:32.591412
INFO:tensorflow:epoch = 4.125, loss = 0.8112773, step = 33 (6.487 sec)
2020-01-13 09:03:44,989 [INFO] tensorflow: epoch = 4.125, loss = 0.8112773, step = 33 (6.487 sec)
INFO:tensorflow:global_step/sec: 1.23456
2020-01-13 09:03:44,990 [INFO] tensorflow: global_step/sec: 1.23456
INFO:tensorflow:global_step/sec: 1.23234
2020-01-13 09:03:45,802 [INFO] tensorflow: global_step/sec: 1.23234
INFO:tensorflow:global_step/sec: 1.23289
2020-01-13 09:03:46,613 [INFO] tensorflow: global_step/sec: 1.23289
INFO:tensorflow:global_step/sec: 1.23381
2020-01-13 09:03:47,424 [INFO] tensorflow: global_step/sec: 1.23381
INFO:tensorflow:global_step/sec: 1.24023
2020-01-13 09:03:48,230 [INFO] tensorflow: global_step/sec: 1.24023
INFO:tensorflow:global_step/sec: 1.23209
2020-01-13 09:03:49,041 [INFO] tensorflow: global_step/sec: 1.23209
INFO:tensorflow:global_step/sec: 1.22952
2020-01-13 09:03:49,855 [INFO] tensorflow: global_step/sec: 1.22952
INFO:tensorflow:Saving checkpoints for step-40.
2020-01-13 09:03:49,856 [INFO] tensorflow: Saving checkpoints for step-40.
INFO:tensorflow:epoch = 5.0, loss = 0.793715, step = 40 (6.136 sec)
2020-01-13 09:03:51,125 [INFO] tensorflow: epoch = 5.0, loss = 0.793715, step = 40 (6.136 sec)
INFO:tensorflow:global_step/sec: 0.785983
2020-01-13 09:03:51,127 [INFO] tensorflow: global_step/sec: 0.785983
2020-01-13 09:03:51,128 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 5/120: loss: 0.79372 Time taken: 0:00:06.949062 ETA: 0:13:19.142130
INFO:tensorflow:global_step/sec: 1.23249
2020-01-13 09:03:51,939 [INFO] tensorflow: global_step/sec: 1.23249
INFO:tensorflow:global_step/sec: 1.22749
2020-01-13 09:03:52,753 [INFO] tensorflow: global_step/sec: 1.22749
INFO:tensorflow:global_step/sec: 1.22863
2020-01-13 09:03:53,567 [INFO] tensorflow: global_step/sec: 1.22863
INFO:tensorflow:global_step/sec: 1.22833
2020-01-13 09:03:54,381 [INFO] tensorflow: global_step/sec: 1.22833
INFO:tensorflow:global_step/sec: 1.23041
2020-01-13 09:03:55,194 [INFO] tensorflow: global_step/sec: 1.23041
INFO:tensorflow:global_step/sec: 1.2312
2020-01-13 09:03:56,006 [INFO] tensorflow: global_step/sec: 1.2312
INFO:tensorflow:global_step/sec: 1.22334
2020-01-13 09:03:56,824 [INFO] tensorflow: global_step/sec: 1.22334
INFO:tensorflow:epoch = 6.0, loss = 0.7258079, step = 48 (6.520 sec)
2020-01-13 09:03:57,645 [INFO] tensorflow: epoch = 6.0, loss = 0.7258079, step = 48 (6.520 sec)
INFO:tensorflow:global_step/sec: 1.2156
2020-01-13 09:03:57,646 [INFO] tensorflow: global_step/sec: 1.2156
2020-01-13 09:03:57,647 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 6/120: loss: 0.72581 Time taken: 0:00:06.508513 ETA: 0:12:21.970482
INFO:tensorflow:global_step/sec: 1.22714
2020-01-13 09:03:58,461 [INFO] tensorflow: global_step/sec: 1.22714
2020-01-13 09:03:58,462 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 19.259
INFO:tensorflow:global_step/sec: 1.22683
2020-01-13 09:03:59,276 [INFO] tensorflow: global_step/sec: 1.22683
INFO:tensorflow:global_step/sec: 1.2248
2020-01-13 09:04:00,093 [INFO] tensorflow: global_step/sec: 1.2248
INFO:tensorflow:global_step/sec: 1.225
2020-01-13 09:04:00,909 [INFO] tensorflow: global_step/sec: 1.225
INFO:tensorflow:global_step/sec: 1.22329
2020-01-13 09:04:01,726 [INFO] tensorflow: global_step/sec: 1.22329
INFO:tensorflow:global_step/sec: 1.22106
2020-01-13 09:04:02,545 [INFO] tensorflow: global_step/sec: 1.22106
INFO:tensorflow:global_step/sec: 1.22076
2020-01-13 09:04:03,365 [INFO] tensorflow: global_step/sec: 1.22076
INFO:tensorflow:epoch = 7.0, loss = 0.65784985, step = 56 (6.545 sec)
2020-01-13 09:04:04,190 [INFO] tensorflow: epoch = 7.0, loss = 0.65784985, step = 56 (6.545 sec)
INFO:tensorflow:global_step/sec: 1.20875
2020-01-13 09:04:04,192 [INFO] tensorflow: global_step/sec: 1.20875
2020-01-13 09:04:04,193 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 7/120: loss: 0.65785 Time taken: 0:00:06.541966 ETA: 0:12:19.242158
INFO:tensorflow:global_step/sec: 1.22432
2020-01-13 09:04:05,009 [INFO] tensorflow: global_step/sec: 1.22432
INFO:tensorflow:global_step/sec: 1.21864
2020-01-13 09:04:05,829 [INFO] tensorflow: global_step/sec: 1.21864
INFO:tensorflow:global_step/sec: 1.22613
2020-01-13 09:04:06,645 [INFO] tensorflow: global_step/sec: 1.22613
INFO:tensorflow:global_step/sec: 1.22405
2020-01-13 09:04:07,462 [INFO] tensorflow: global_step/sec: 1.22405
INFO:tensorflow:global_step/sec: 1.21197
2020-01-13 09:04:08,287 [INFO] tensorflow: global_step/sec: 1.21197
INFO:tensorflow:global_step/sec: 1.21861
2020-01-13 09:04:09,108 [INFO] tensorflow: global_step/sec: 1.21861
INFO:tensorflow:global_step/sec: 1.21626
2020-01-13 09:04:09,930 [INFO] tensorflow: global_step/sec: 1.21626
INFO:tensorflow:epoch = 8.0, loss = 0.5598427, step = 64 (6.569 sec)
2020-01-13 09:04:10,760 [INFO] tensorflow: epoch = 8.0, loss = 0.5598427, step = 64 (6.569 sec)
INFO:tensorflow:global_step/sec: 1.20245
2020-01-13 09:04:10,761 [INFO] tensorflow: global_step/sec: 1.20245
2020-01-13 09:04:10,763 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 8/120: loss: 0.55984 Time taken: 0:00:06.559767 ETA: 0:12:14.693904
INFO:tensorflow:global_step/sec: 1.22415
2020-01-13 09:04:11,578 [INFO] tensorflow: global_step/sec: 1.22415
INFO:tensorflow:global_step/sec: 1.21284
2020-01-13 09:04:12,403 [INFO] tensorflow: global_step/sec: 1.21284
INFO:tensorflow:global_step/sec: 1.21669
2020-01-13 09:04:13,225 [INFO] tensorflow: global_step/sec: 1.21669
INFO:tensorflow:global_step/sec: 1.22166
2020-01-13 09:04:14,043 [INFO] tensorflow: global_step/sec: 1.22166
INFO:tensorflow:global_step/sec: 1.21026
2020-01-13 09:04:14,869 [INFO] tensorflow: global_step/sec: 1.21026
INFO:tensorflow:global_step/sec: 1.21357
2020-01-13 09:04:15,694 [INFO] tensorflow: global_step/sec: 1.21357
INFO:tensorflow:global_step/sec: 1.21719
2020-01-13 09:04:16,515 [INFO] tensorflow: global_step/sec: 1.21719
INFO:tensorflow:epoch = 9.0, loss = 0.4610829, step = 72 (6.586 sec)
2020-01-13 09:04:17,345 [INFO] tensorflow: epoch = 9.0, loss = 0.4610829, step = 72 (6.586 sec)
INFO:tensorflow:global_step/sec: 1.2021
2020-01-13 09:04:17,347 [INFO] tensorflow: global_step/sec: 1.2021
2020-01-13 09:04:17,348 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 9/120: loss: 0.46108 Time taken: 0:00:06.585149 ETA: 0:12:10.951539
INFO:tensorflow:global_step/sec: 1.20817
2020-01-13 09:04:18,175 [INFO] tensorflow: global_step/sec: 1.20817
INFO:tensorflow:global_step/sec: 1.20627
2020-01-13 09:04:19,004 [INFO] tensorflow: global_step/sec: 1.20627
2020-01-13 09:04:19,005 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 19.472
INFO:tensorflow:global_step/sec: 1.21284
2020-01-13 09:04:19,828 [INFO] tensorflow: global_step/sec: 1.21284
INFO:tensorflow:global_step/sec: 1.21585
2020-01-13 09:04:20,651 [INFO] tensorflow: global_step/sec: 1.21585
INFO:tensorflow:global_step/sec: 1.20792
2020-01-13 09:04:21,478 [INFO] tensorflow: global_step/sec: 1.20792
INFO:tensorflow:global_step/sec: 1.20794
2020-01-13 09:04:22,306 [INFO] tensorflow: global_step/sec: 1.20794
INFO:tensorflow:global_step/sec: 1.20136
2020-01-13 09:04:23,139 [INFO] tensorflow: global_step/sec: 1.20136
INFO:tensorflow:Saving checkpoints for step-80.
2020-01-13 09:04:23,140 [INFO] tensorflow: Saving checkpoints for step-80.
INFO:tensorflow:epoch = 10.0, loss = 0.37151942, step = 80 (7.085 sec)
2020-01-13 09:04:24,430 [INFO] tensorflow: epoch = 10.0, loss = 0.37151942, step = 80 (7.085 sec)
INFO:tensorflow:global_step/sec: 0.773477
2020-01-13 09:04:24,432 [INFO] tensorflow: global_step/sec: 0.773477
2020-01-13 09:04:24,433 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 10/120: loss: 0.37152 Time taken: 0:00:07.089200 ETA: 0:12:59.812000
INFO:tensorflow:global_step/sec: 1.20121
2020-01-13 09:04:25,264 [INFO] tensorflow: global_step/sec: 1.20121
INFO:tensorflow:global_step/sec: 1.20272
2020-01-13 09:04:26,096 [INFO] tensorflow: global_step/sec: 1.20272
INFO:tensorflow:global_step/sec: 1.20062
2020-01-13 09:04:26,928 [INFO] tensorflow: global_step/sec: 1.20062
INFO:tensorflow:global_step/sec: 1.20317
2020-01-13 09:04:27,760 [INFO] tensorflow: global_step/sec: 1.20317
INFO:tensorflow:global_step/sec: 1.20706
2020-01-13 09:04:28,588 [INFO] tensorflow: global_step/sec: 1.20706
INFO:tensorflow:global_step/sec: 1.20326
2020-01-13 09:04:29,419 [INFO] tensorflow: global_step/sec: 1.20326
INFO:tensorflow:global_step/sec: 1.20178
2020-01-13 09:04:30,251 [INFO] tensorflow: global_step/sec: 1.20178
INFO:tensorflow:epoch = 11.0, loss = 0.24042998, step = 88 (6.651 sec)
2020-01-13 09:04:31,081 [INFO] tensorflow: epoch = 11.0, loss = 0.24042998, step = 88 (6.651 sec)
INFO:tensorflow:global_step/sec: 1.20248
2020-01-13 09:04:31,083 [INFO] tensorflow: global_step/sec: 1.20248
2020-01-13 09:04:31,084 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 11/120: loss: 0.24043 Time taken: 0:00:06.648068 ETA: 0:12:04.639412
INFO:tensorflow:global_step/sec: 1.20146
2020-01-13 09:04:31,915 [INFO] tensorflow: global_step/sec: 1.20146
INFO:tensorflow:global_step/sec: 1.20348
2020-01-13 09:04:32,746 [INFO] tensorflow: global_step/sec: 1.20348
INFO:tensorflow:global_step/sec: 1.19836
2020-01-13 09:04:33,581 [INFO] tensorflow: global_step/sec: 1.19836
INFO:tensorflow:global_step/sec: 1.20527
2020-01-13 09:04:34,410 [INFO] tensorflow: global_step/sec: 1.20527
INFO:tensorflow:global_step/sec: 1.20532
2020-01-13 09:04:35,240 [INFO] tensorflow: global_step/sec: 1.20532
INFO:tensorflow:global_step/sec: 1.20257
2020-01-13 09:04:36,071 [INFO] tensorflow: global_step/sec: 1.20257
INFO:tensorflow:global_step/sec: 1.20472
2020-01-13 09:04:36,901 [INFO] tensorflow: global_step/sec: 1.20472
INFO:tensorflow:epoch = 12.0, loss = 0.15279289, step = 96 (6.653 sec)
2020-01-13 09:04:37,735 [INFO] tensorflow: epoch = 12.0, loss = 0.15279289, step = 96 (6.653 sec)
INFO:tensorflow:global_step/sec: 1.19832
2020-01-13 09:04:37,736 [INFO] tensorflow: global_step/sec: 1.19832
2020-01-13 09:04:37,737 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 12/120: loss: 0.15279 Time taken: 0:00:06.645763 ETA: 0:11:57.742404
INFO:tensorflow:global_step/sec: 1.20362
2020-01-13 09:04:38,567 [INFO] tensorflow: global_step/sec: 1.20362
INFO:tensorflow:global_step/sec: 1.2017
2020-01-13 09:04:39,399 [INFO] tensorflow: global_step/sec: 1.2017
INFO:tensorflow:global_step/sec: 1.20585
2020-01-13 09:04:40,228 [INFO] tensorflow: global_step/sec: 1.20585
2020-01-13 09:04:40,230 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.847
INFO:tensorflow:global_step/sec: 1.20181
2020-01-13 09:04:41,060 [INFO] tensorflow: global_step/sec: 1.20181
INFO:tensorflow:global_step/sec: 1.20076
2020-01-13 09:04:41,893 [INFO] tensorflow: global_step/sec: 1.20076
INFO:tensorflow:global_step/sec: 1.20406
2020-01-13 09:04:42,724 [INFO] tensorflow: global_step/sec: 1.20406
INFO:tensorflow:global_step/sec: 1.20429
2020-01-13 09:04:43,554 [INFO] tensorflow: global_step/sec: 1.20429
INFO:tensorflow:epoch = 13.0, loss = 0.09585952, step = 104 (6.665 sec)
2020-01-13 09:04:44,399 [INFO] tensorflow: epoch = 13.0, loss = 0.09585952, step = 104 (6.665 sec)
INFO:tensorflow:global_step/sec: 1.18113
2020-01-13 09:04:44,401 [INFO] tensorflow: global_step/sec: 1.18113
2020-01-13 09:04:44,402 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 13/120: loss: 0.09586 Time taken: 0:00:06.654223 ETA: 0:11:52.001861
INFO:tensorflow:global_step/sec: 1.20214
2020-01-13 09:04:45,233 [INFO] tensorflow: global_step/sec: 1.20214
INFO:tensorflow:global_step/sec: 1.20131
2020-01-13 09:04:46,065 [INFO] tensorflow: global_step/sec: 1.20131
INFO:tensorflow:global_step/sec: 1.19416
2020-01-13 09:04:46,902 [INFO] tensorflow: global_step/sec: 1.19416
INFO:tensorflow:global_step/sec: 1.19085
2020-01-13 09:04:47,742 [INFO] tensorflow: global_step/sec: 1.19085
INFO:tensorflow:global_step/sec: 1.19763
2020-01-13 09:04:48,577 [INFO] tensorflow: global_step/sec: 1.19763
INFO:tensorflow:global_step/sec: 1.19465
2020-01-13 09:04:49,414 [INFO] tensorflow: global_step/sec: 1.19465
INFO:tensorflow:epoch = 13.875, loss = 0.07340296, step = 111 (5.854 sec)
2020-01-13 09:04:50,253 [INFO] tensorflow: epoch = 13.875, loss = 0.07340296, step = 111 (5.854 sec)
INFO:tensorflow:global_step/sec: 1.18977
2020-01-13 09:04:50,255 [INFO] tensorflow: global_step/sec: 1.18977
INFO:tensorflow:global_step/sec: 1.18541
2020-01-13 09:04:51,098 [INFO] tensorflow: global_step/sec: 1.18541
2020-01-13 09:04:51,100 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 14/120: loss: 0.06708 Time taken: 0:00:06.697458 ETA: 0:11:49.930548
INFO:tensorflow:global_step/sec: 1.19475
2020-01-13 09:04:51,935 [INFO] tensorflow: global_step/sec: 1.19475
INFO:tensorflow:global_step/sec: 1.19367
2020-01-13 09:04:52,773 [INFO] tensorflow: global_step/sec: 1.19367
INFO:tensorflow:global_step/sec: 1.18981
2020-01-13 09:04:53,613 [INFO] tensorflow: global_step/sec: 1.18981
INFO:tensorflow:global_step/sec: 1.1982
2020-01-13 09:04:54,448 [INFO] tensorflow: global_step/sec: 1.1982
INFO:tensorflow:global_step/sec: 1.18652
2020-01-13 09:04:55,291 [INFO] tensorflow: global_step/sec: 1.18652
INFO:tensorflow:epoch = 14.75, loss = 0.0560713, step = 118 (5.877 sec)
2020-01-13 09:04:56,130 [INFO] tensorflow: epoch = 14.75, loss = 0.0560713, step = 118 (5.877 sec)
INFO:tensorflow:global_step/sec: 1.1892
2020-01-13 09:04:56,132 [INFO] tensorflow: global_step/sec: 1.1892
INFO:tensorflow:global_step/sec: 1.18348
2020-01-13 09:04:56,977 [INFO] tensorflow: global_step/sec: 1.18348
INFO:tensorflow:Saving checkpoints for step-120.
2020-01-13 09:04:56,978 [INFO] tensorflow: Saving checkpoints for step-120.
INFO:tensorflow:global_step/sec: 0.770842
2020-01-13 09:04:58,274 [INFO] tensorflow: global_step/sec: 0.770842
2020-01-13 09:04:58,276 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 15/120: loss: 0.05127 Time taken: 0:00:07.181245 ETA: 0:12:34.030725
INFO:tensorflow:global_step/sec: 1.1828
2020-01-13 09:04:59,119 [INFO] tensorflow: global_step/sec: 1.1828
INFO:tensorflow:global_step/sec: 1.18271
2020-01-13 09:04:59,965 [INFO] tensorflow: global_step/sec: 1.18271
INFO:tensorflow:global_step/sec: 1.18786
2020-01-13 09:05:00,807 [INFO] tensorflow: global_step/sec: 1.18786
INFO:tensorflow:global_step/sec: 1.18073
2020-01-13 09:05:01,654 [INFO] tensorflow: global_step/sec: 1.18073
2020-01-13 09:05:01,655 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.670
INFO:tensorflow:epoch = 15.625, loss = 0.045743596, step = 125 (6.365 sec)
2020-01-13 09:05:02,495 [INFO] tensorflow: epoch = 15.625, loss = 0.045743596, step = 125 (6.365 sec)
INFO:tensorflow:global_step/sec: 1.18607
2020-01-13 09:05:02,497 [INFO] tensorflow: global_step/sec: 1.18607
INFO:tensorflow:global_step/sec: 1.18488
2020-01-13 09:05:03,341 [INFO] tensorflow: global_step/sec: 1.18488
INFO:tensorflow:global_step/sec: 1.18146
2020-01-13 09:05:04,187 [INFO] tensorflow: global_step/sec: 1.18146
INFO:tensorflow:global_step/sec: 1.18053
2020-01-13 09:05:05,034 [INFO] tensorflow: global_step/sec: 1.18053
2020-01-13 09:05:05,036 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 16/120: loss: 0.04340 Time taken: 0:00:06.751384 ETA: 0:11:42.143936
INFO:tensorflow:global_step/sec: 1.19055
2020-01-13 09:05:05,874 [INFO] tensorflow: global_step/sec: 1.19055
INFO:tensorflow:global_step/sec: 1.18252
2020-01-13 09:05:06,720 [INFO] tensorflow: global_step/sec: 1.18252
INFO:tensorflow:global_step/sec: 1.19334
2020-01-13 09:05:07,558 [INFO] tensorflow: global_step/sec: 1.19334
INFO:tensorflow:epoch = 16.5, loss = 0.042903394, step = 132 (5.907 sec)
2020-01-13 09:05:08,402 [INFO] tensorflow: epoch = 16.5, loss = 0.042903394, step = 132 (5.907 sec)
INFO:tensorflow:global_step/sec: 1.18161
2020-01-13 09:05:08,404 [INFO] tensorflow: global_step/sec: 1.18161
INFO:tensorflow:global_step/sec: 1.18271
2020-01-13 09:05:09,250 [INFO] tensorflow: global_step/sec: 1.18271
INFO:tensorflow:global_step/sec: 1.18744
2020-01-13 09:05:10,092 [INFO] tensorflow: global_step/sec: 1.18744
INFO:tensorflow:global_step/sec: 1.18766
2020-01-13 09:05:10,934 [INFO] tensorflow: global_step/sec: 1.18766
INFO:tensorflow:global_step/sec: 1.17512
2020-01-13 09:05:11,785 [INFO] tensorflow: global_step/sec: 1.17512
2020-01-13 09:05:11,786 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.03505 Time taken: 0:00:06.741886 ETA: 0:11:34.414258
INFO:tensorflow:global_step/sec: 1.18949
2020-01-13 09:05:12,626 [INFO] tensorflow: global_step/sec: 1.18949
INFO:tensorflow:global_step/sec: 1.18392
2020-01-13 09:05:13,470 [INFO] tensorflow: global_step/sec: 1.18392
INFO:tensorflow:epoch = 17.375, loss = 0.03255378, step = 139 (5.915 sec)
2020-01-13 09:05:14,317 [INFO] tensorflow: epoch = 17.375, loss = 0.03255378, step = 139 (5.915 sec)
INFO:tensorflow:global_step/sec: 1.17861
2020-01-13 09:05:14,319 [INFO] tensorflow: global_step/sec: 1.17861
INFO:tensorflow:global_step/sec: 1.17902
2020-01-13 09:05:15,167 [INFO] tensorflow: global_step/sec: 1.17902
INFO:tensorflow:global_step/sec: 1.17729
2020-01-13 09:05:16,016 [INFO] tensorflow: global_step/sec: 1.17729
INFO:tensorflow:global_step/sec: 1.18077
2020-01-13 09:05:16,863 [INFO] tensorflow: global_step/sec: 1.18077
INFO:tensorflow:global_step/sec: 1.17633
2020-01-13 09:05:17,713 [INFO] tensorflow: global_step/sec: 1.17633
INFO:tensorflow:global_step/sec: 1.17275
2020-01-13 09:05:18,566 [INFO] tensorflow: global_step/sec: 1.17275
2020-01-13 09:05:18,567 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.03164 Time taken: 0:00:06.781169 ETA: 0:11:31.679238
INFO:tensorflow:global_step/sec: 1.17412
2020-01-13 09:05:19,418 [INFO] tensorflow: global_step/sec: 1.17412
INFO:tensorflow:epoch = 18.25, loss = 0.028710622, step = 146 (5.953 sec)
2020-01-13 09:05:20,270 [INFO] tensorflow: epoch = 18.25, loss = 0.028710622, step = 146 (5.953 sec)
INFO:tensorflow:global_step/sec: 1.17143
2020-01-13 09:05:20,271 [INFO] tensorflow: global_step/sec: 1.17143
INFO:tensorflow:global_step/sec: 1.17769
2020-01-13 09:05:21,120 [INFO] tensorflow: global_step/sec: 1.17769
INFO:tensorflow:global_step/sec: 1.17181
2020-01-13 09:05:21,974 [INFO] tensorflow: global_step/sec: 1.17181
INFO:tensorflow:global_step/sec: 1.17752
2020-01-13 09:05:22,823 [INFO] tensorflow: global_step/sec: 1.17752
2020-01-13 09:05:22,824 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.896
INFO:tensorflow:global_step/sec: 1.17529
2020-01-13 09:05:23,674 [INFO] tensorflow: global_step/sec: 1.17529
INFO:tensorflow:global_step/sec: 1.17504
2020-01-13 09:05:24,525 [INFO] tensorflow: global_step/sec: 1.17504
INFO:tensorflow:global_step/sec: 1.16963
2020-01-13 09:05:25,380 [INFO] tensorflow: global_step/sec: 1.16963
2020-01-13 09:05:25,381 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 19/120: loss: 0.02699 Time taken: 0:00:06.809700 ETA: 0:11:27.779700
INFO:tensorflow:epoch = 19.125, loss = 0.026556345, step = 153 (5.958 sec)
2020-01-13 09:05:26,228 [INFO] tensorflow: epoch = 19.125, loss = 0.026556345, step = 153 (5.958 sec)
INFO:tensorflow:global_step/sec: 1.17718
2020-01-13 09:05:26,229 [INFO] tensorflow: global_step/sec: 1.17718
INFO:tensorflow:global_step/sec: 1.1786
2020-01-13 09:05:27,078 [INFO] tensorflow: global_step/sec: 1.1786
INFO:tensorflow:global_step/sec: 1.17575
2020-01-13 09:05:27,928 [INFO] tensorflow: global_step/sec: 1.17575
INFO:tensorflow:global_step/sec: 1.1763
2020-01-13 09:05:28,778 [INFO] tensorflow: global_step/sec: 1.1763
INFO:tensorflow:global_step/sec: 1.17554
2020-01-13 09:05:29,629 [INFO] tensorflow: global_step/sec: 1.17554
INFO:tensorflow:global_step/sec: 1.17479
2020-01-13 09:05:30,480 [INFO] tensorflow: global_step/sec: 1.17479
INFO:tensorflow:global_step/sec: 1.17323
2020-01-13 09:05:31,333 [INFO] tensorflow: global_step/sec: 1.17323
INFO:tensorflow:Saving checkpoints for step-160.
2020-01-13 09:05:31,334 [INFO] tensorflow: Saving checkpoints for step-160.
INFO:tensorflow:epoch = 20.0, loss = 0.026538212, step = 160 (6.412 sec)
2020-01-13 09:05:32,640 [INFO] tensorflow: epoch = 20.0, loss = 0.026538212, step = 160 (6.412 sec)
INFO:tensorflow:global_step/sec: 0.763986
2020-01-13 09:05:32,642 [INFO] tensorflow: global_step/sec: 0.763986
2020-01-13 09:05:32,643 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 20/120: loss: 0.02654 Time taken: 0:00:07.265122 ETA: 0:12:06.512200
INFO:tensorflow:global_step/sec: 1.17021
2020-01-13 09:05:33,496 [INFO] tensorflow: global_step/sec: 1.17021
INFO:tensorflow:global_step/sec: 1.1748
2020-01-13 09:05:34,347 [INFO] tensorflow: global_step/sec: 1.1748
INFO:tensorflow:global_step/sec: 1.16869
2020-01-13 09:05:35,203 [INFO] tensorflow: global_step/sec: 1.16869
INFO:tensorflow:global_step/sec: 1.17663
2020-01-13 09:05:36,053 [INFO] tensorflow: global_step/sec: 1.17663
INFO:tensorflow:global_step/sec: 1.17693
2020-01-13 09:05:36,903 [INFO] tensorflow: global_step/sec: 1.17693
INFO:tensorflow:global_step/sec: 1.1789
2020-01-13 09:05:37,751 [INFO] tensorflow: global_step/sec: 1.1789
INFO:tensorflow:epoch = 20.875, loss = 0.022777596, step = 167 (5.965 sec)
2020-01-13 09:05:38,605 [INFO] tensorflow: epoch = 20.875, loss = 0.022777596, step = 167 (5.965 sec)
INFO:tensorflow:global_step/sec: 1.16868
2020-01-13 09:05:38,606 [INFO] tensorflow: global_step/sec: 1.16868
INFO:tensorflow:global_step/sec: 1.15941
2020-01-13 09:05:39,469 [INFO] tensorflow: global_step/sec: 1.15941
2020-01-13 09:05:39,471 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 21/120: loss: 0.02057 Time taken: 0:00:06.811818 ETA: 0:11:14.369982
INFO:tensorflow:global_step/sec: 1.16772
2020-01-13 09:05:40,325 [INFO] tensorflow: global_step/sec: 1.16772
INFO:tensorflow:global_step/sec: 1.17334
2020-01-13 09:05:41,178 [INFO] tensorflow: global_step/sec: 1.17334
INFO:tensorflow:global_step/sec: 1.16903
2020-01-13 09:05:42,033 [INFO] tensorflow: global_step/sec: 1.16903
INFO:tensorflow:global_step/sec: 1.1758
2020-01-13 09:05:42,884 [INFO] tensorflow: global_step/sec: 1.1758
INFO:tensorflow:global_step/sec: 1.17498
2020-01-13 09:05:43,735 [INFO] tensorflow: global_step/sec: 1.17498
INFO:tensorflow:epoch = 21.75, loss = 0.021212738, step = 174 (5.979 sec)
2020-01-13 09:05:44,584 [INFO] tensorflow: epoch = 21.75, loss = 0.021212738, step = 174 (5.979 sec)
INFO:tensorflow:global_step/sec: 1.17516
2020-01-13 09:05:44,586 [INFO] tensorflow: global_step/sec: 1.17516
2020-01-13 09:05:44,587 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.380
INFO:tensorflow:global_step/sec: 1.1727
2020-01-13 09:05:45,438 [INFO] tensorflow: global_step/sec: 1.1727
INFO:tensorflow:global_step/sec: 1.15644
2020-01-13 09:05:46,303 [INFO] tensorflow: global_step/sec: 1.15644
2020-01-13 09:05:46,305 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 22/120: loss: 0.02000 Time taken: 0:00:06.834812 ETA: 0:11:09.811576
INFO:tensorflow:global_step/sec: 1.16753
2020-01-13 09:05:47,160 [INFO] tensorflow: global_step/sec: 1.16753
INFO:tensorflow:global_step/sec: 1.16631
2020-01-13 09:05:48,017 [INFO] tensorflow: global_step/sec: 1.16631
INFO:tensorflow:global_step/sec: 1.16058
2020-01-13 09:05:48,879 [INFO] tensorflow: global_step/sec: 1.16058
INFO:tensorflow:global_step/sec: 1.17225
2020-01-13 09:05:49,732 [INFO] tensorflow: global_step/sec: 1.17225
INFO:tensorflow:epoch = 22.625, loss = 0.020029472, step = 181 (6.013 sec)
2020-01-13 09:05:50,597 [INFO] tensorflow: epoch = 22.625, loss = 0.020029472, step = 181 (6.013 sec)
INFO:tensorflow:global_step/sec: 1.15332
2020-01-13 09:05:50,599 [INFO] tensorflow: global_step/sec: 1.15332
INFO:tensorflow:global_step/sec: 1.16956
2020-01-13 09:05:51,454 [INFO] tensorflow: global_step/sec: 1.16956
INFO:tensorflow:global_step/sec: 1.17473
2020-01-13 09:05:52,305 [INFO] tensorflow: global_step/sec: 1.17473
INFO:tensorflow:global_step/sec: 1.15433
2020-01-13 09:05:53,171 [INFO] tensorflow: global_step/sec: 1.15433
2020-01-13 09:05:53,173 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 23/120: loss: 0.01813 Time taken: 0:00:06.865815 ETA: 0:11:05.984055
INFO:tensorflow:global_step/sec: 1.16736
2020-01-13 09:05:54,028 [INFO] tensorflow: global_step/sec: 1.16736
INFO:tensorflow:global_step/sec: 1.16585
2020-01-13 09:05:54,886 [INFO] tensorflow: global_step/sec: 1.16585
INFO:tensorflow:global_step/sec: 1.16298
2020-01-13 09:05:55,746 [INFO] tensorflow: global_step/sec: 1.16298
INFO:tensorflow:epoch = 23.5, loss = 0.018176988, step = 188 (6.004 sec)
2020-01-13 09:05:56,602 [INFO] tensorflow: epoch = 23.5, loss = 0.018176988, step = 188 (6.004 sec)
INFO:tensorflow:global_step/sec: 1.16599
2020-01-13 09:05:56,603 [INFO] tensorflow: global_step/sec: 1.16599
INFO:tensorflow:global_step/sec: 1.15859
2020-01-13 09:05:57,466 [INFO] tensorflow: global_step/sec: 1.15859
INFO:tensorflow:global_step/sec: 1.17219
2020-01-13 09:05:58,319 [INFO] tensorflow: global_step/sec: 1.17219
INFO:tensorflow:global_step/sec: 1.16631
2020-01-13 09:05:59,177 [INFO] tensorflow: global_step/sec: 1.16631
INFO:tensorflow:global_step/sec: 1.15975
2020-01-13 09:06:00,039 [INFO] tensorflow: global_step/sec: 1.15975
2020-01-13 09:06:00,041 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 24/120: loss: 0.01762 Time taken: 0:00:06.867600 ETA: 0:10:59.289600
INFO:tensorflow:global_step/sec: 1.15947
2020-01-13 09:06:00,902 [INFO] tensorflow: global_step/sec: 1.15947
INFO:tensorflow:global_step/sec: 1.16405
2020-01-13 09:06:01,761 [INFO] tensorflow: global_step/sec: 1.16405
INFO:tensorflow:epoch = 24.375, loss = 0.016502984, step = 195 (6.019 sec)
2020-01-13 09:06:02,620 [INFO] tensorflow: epoch = 24.375, loss = 0.016502984, step = 195 (6.019 sec)
INFO:tensorflow:global_step/sec: 1.16104
2020-01-13 09:06:02,622 [INFO] tensorflow: global_step/sec: 1.16104
INFO:tensorflow:global_step/sec: 1.16114
2020-01-13 09:06:03,483 [INFO] tensorflow: global_step/sec: 1.16114
INFO:tensorflow:global_step/sec: 1.15141
2020-01-13 09:06:04,352 [INFO] tensorflow: global_step/sec: 1.15141
INFO:tensorflow:global_step/sec: 1.16767
2020-01-13 09:06:05,208 [INFO] tensorflow: global_step/sec: 1.16767
INFO:tensorflow:global_step/sec: 1.16934
2020-01-13 09:06:06,063 [INFO] tensorflow: global_step/sec: 1.16934
INFO:tensorflow:Saving checkpoints for step-200.
2020-01-13 09:06:06,065 [INFO] tensorflow: Saving checkpoints for step-200.
2020-01-13 09:06:06,523 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.235
INFO:tensorflow:global_step/sec: 0.764018
2020-01-13 09:06:07,372 [INFO] tensorflow: global_step/sec: 0.764018
2020-01-13 09:06:07,374 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 25/120: loss: 0.01665 Time taken: 0:00:07.341518 ETA: 0:11:37.444210
INFO:tensorflow:global_step/sec: 1.16167
2020-01-13 09:06:08,233 [INFO] tensorflow: global_step/sec: 1.16167
INFO:tensorflow:epoch = 25.25, loss = 0.016083246, step = 202 (6.474 sec)
2020-01-13 09:06:09,094 [INFO] tensorflow: epoch = 25.25, loss = 0.016083246, step = 202 (6.474 sec)
INFO:tensorflow:global_step/sec: 1.15926
2020-01-13 09:06:09,095 [INFO] tensorflow: global_step/sec: 1.15926
INFO:tensorflow:global_step/sec: 1.15353
2020-01-13 09:06:09,962 [INFO] tensorflow: global_step/sec: 1.15353
INFO:tensorflow:global_step/sec: 1.15666
2020-01-13 09:06:10,827 [INFO] tensorflow: global_step/sec: 1.15666
INFO:tensorflow:global_step/sec: 1.16251
2020-01-13 09:06:11,687 [INFO] tensorflow: global_step/sec: 1.16251
INFO:tensorflow:global_step/sec: 1.15612
2020-01-13 09:06:12,552 [INFO] tensorflow: global_step/sec: 1.15612
INFO:tensorflow:global_step/sec: 1.1619
2020-01-13 09:06:13,413 [INFO] tensorflow: global_step/sec: 1.1619
INFO:tensorflow:global_step/sec: 1.15654
2020-01-13 09:06:14,277 [INFO] tensorflow: global_step/sec: 1.15654
2020-01-13 09:06:14,279 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 26/120: loss: 0.01415 Time taken: 0:00:06.892762 ETA: 0:10:47.919628
INFO:tensorflow:epoch = 26.125, loss = 0.014135221, step = 209 (6.050 sec)
2020-01-13 09:06:15,144 [INFO] tensorflow: epoch = 26.125, loss = 0.014135221, step = 209 (6.050 sec)
INFO:tensorflow:global_step/sec: 1.15156
2020-01-13 09:06:15,146 [INFO] tensorflow: global_step/sec: 1.15156
INFO:tensorflow:global_step/sec: 1.14705
2020-01-13 09:06:16,018 [INFO] tensorflow: global_step/sec: 1.14705
INFO:tensorflow:global_step/sec: 1.15132
2020-01-13 09:06:16,886 [INFO] tensorflow: global_step/sec: 1.15132
INFO:tensorflow:global_step/sec: 1.16414
2020-01-13 09:06:17,745 [INFO] tensorflow: global_step/sec: 1.16414
INFO:tensorflow:global_step/sec: 1.15915
2020-01-13 09:06:18,608 [INFO] tensorflow: global_step/sec: 1.15915
INFO:tensorflow:global_step/sec: 1.14999
2020-01-13 09:06:19,477 [INFO] tensorflow: global_step/sec: 1.14999
INFO:tensorflow:global_step/sec: 1.16229
2020-01-13 09:06:20,338 [INFO] tensorflow: global_step/sec: 1.16229
INFO:tensorflow:epoch = 27.0, loss = 0.013429198, step = 216 (6.059 sec)
2020-01-13 09:06:21,203 [INFO] tensorflow: epoch = 27.0, loss = 0.013429198, step = 216 (6.059 sec)
INFO:tensorflow:global_step/sec: 1.15327
2020-01-13 09:06:21,205 [INFO] tensorflow: global_step/sec: 1.15327
2020-01-13 09:06:21,206 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 27/120: loss: 0.01343 Time taken: 0:00:06.925481 ETA: 0:10:44.069733
INFO:tensorflow:global_step/sec: 1.15305
2020-01-13 09:06:22,072 [INFO] tensorflow: global_step/sec: 1.15305
INFO:tensorflow:global_step/sec: 1.15239
2020-01-13 09:06:22,940 [INFO] tensorflow: global_step/sec: 1.15239
INFO:tensorflow:global_step/sec: 1.1472
2020-01-13 09:06:23,812 [INFO] tensorflow: global_step/sec: 1.1472
INFO:tensorflow:global_step/sec: 1.14875
2020-01-13 09:06:24,682 [INFO] tensorflow: global_step/sec: 1.14875
INFO:tensorflow:global_step/sec: 1.1546
2020-01-13 09:06:25,548 [INFO] tensorflow: global_step/sec: 1.1546
INFO:tensorflow:global_step/sec: 1.16169
2020-01-13 09:06:26,409 [INFO] tensorflow: global_step/sec: 1.16169
INFO:tensorflow:epoch = 27.875, loss = 0.012523004, step = 223 (6.074 sec)
2020-01-13 09:06:27,277 [INFO] tensorflow: epoch = 27.875, loss = 0.012523004, step = 223 (6.074 sec)
INFO:tensorflow:global_step/sec: 1.14991
2020-01-13 09:06:27,279 [INFO] tensorflow: global_step/sec: 1.14991
INFO:tensorflow:global_step/sec: 1.1492
2020-01-13 09:06:28,149 [INFO] tensorflow: global_step/sec: 1.1492
2020-01-13 09:06:28,151 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 28/120: loss: 0.01252 Time taken: 0:00:06.938529 ETA: 0:10:38.344668
2020-01-13 09:06:28,152 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.495
INFO:tensorflow:global_step/sec: 1.14847
2020-01-13 09:06:29,020 [INFO] tensorflow: global_step/sec: 1.14847
INFO:tensorflow:global_step/sec: 1.14988
2020-01-13 09:06:29,889 [INFO] tensorflow: global_step/sec: 1.14988
INFO:tensorflow:global_step/sec: 1.1568
2020-01-13 09:06:30,754 [INFO] tensorflow: global_step/sec: 1.1568
INFO:tensorflow:global_step/sec: 1.15836
2020-01-13 09:06:31,617 [INFO] tensorflow: global_step/sec: 1.15836
INFO:tensorflow:global_step/sec: 1.15347
2020-01-13 09:06:32,484 [INFO] tensorflow: global_step/sec: 1.15347
INFO:tensorflow:epoch = 28.75, loss = 0.013229851, step = 230 (6.076 sec)
2020-01-13 09:06:33,353 [INFO] tensorflow: epoch = 28.75, loss = 0.013229851, step = 230 (6.076 sec)
INFO:tensorflow:global_step/sec: 1.14902
2020-01-13 09:06:33,354 [INFO] tensorflow: global_step/sec: 1.14902
INFO:tensorflow:global_step/sec: 1.15301
2020-01-13 09:06:34,222 [INFO] tensorflow: global_step/sec: 1.15301
INFO:tensorflow:global_step/sec: 1.14673
2020-01-13 09:06:35,094 [INFO] tensorflow: global_step/sec: 1.14673
2020-01-13 09:06:35,095 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 29/120: loss: 0.01129 Time taken: 0:00:06.941255 ETA: 0:10:31.654205
INFO:tensorflow:global_step/sec: 1.14455
2020-01-13 09:06:35,967 [INFO] tensorflow: global_step/sec: 1.14455
INFO:tensorflow:global_step/sec: 1.14991
2020-01-13 09:06:36,837 [INFO] tensorflow: global_step/sec: 1.14991
INFO:tensorflow:global_step/sec: 1.14701
2020-01-13 09:06:37,709 [INFO] tensorflow: global_step/sec: 1.14701
INFO:tensorflow:global_step/sec: 1.14811
2020-01-13 09:06:38,580 [INFO] tensorflow: global_step/sec: 1.14811
INFO:tensorflow:epoch = 29.625, loss = 0.011274138, step = 237 (6.097 sec)
2020-01-13 09:06:39,450 [INFO] tensorflow: epoch = 29.625, loss = 0.011274138, step = 237 (6.097 sec)
INFO:tensorflow:global_step/sec: 1.14733
2020-01-13 09:06:39,451 [INFO] tensorflow: global_step/sec: 1.14733
INFO:tensorflow:global_step/sec: 1.14823
2020-01-13 09:06:40,322 [INFO] tensorflow: global_step/sec: 1.14823
INFO:tensorflow:global_step/sec: 1.14756
2020-01-13 09:06:41,194 [INFO] tensorflow: global_step/sec: 1.14756
INFO:tensorflow:Saving checkpoints for step-240.
2020-01-13 09:06:41,195 [INFO] tensorflow: Saving checkpoints for step-240.
INFO:tensorflow:global_step/sec: 0.742871
2020-01-13 09:06:42,540 [INFO] tensorflow: global_step/sec: 0.742871
2020-01-13 09:06:42,541 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 30/120: loss: 0.01019 Time taken: 0:00:07.457480 ETA: 0:11:11.173200
INFO:tensorflow:global_step/sec: 1.14526
2020-01-13 09:06:43,413 [INFO] tensorflow: global_step/sec: 1.14526
INFO:tensorflow:global_step/sec: 1.14625
2020-01-13 09:06:44,285 [INFO] tensorflow: global_step/sec: 1.14625
INFO:tensorflow:global_step/sec: 1.14388
2020-01-13 09:06:45,160 [INFO] tensorflow: global_step/sec: 1.14388
INFO:tensorflow:epoch = 30.5, loss = 0.011310074, step = 244 (6.583 sec)
2020-01-13 09:06:46,033 [INFO] tensorflow: epoch = 30.5, loss = 0.011310074, step = 244 (6.583 sec)
INFO:tensorflow:global_step/sec: 1.14277
2020-01-13 09:06:46,035 [INFO] tensorflow: global_step/sec: 1.14277
INFO:tensorflow:global_step/sec: 1.14348
2020-01-13 09:06:46,909 [INFO] tensorflow: global_step/sec: 1.14348
INFO:tensorflow:global_step/sec: 1.15073
2020-01-13 09:06:47,778 [INFO] tensorflow: global_step/sec: 1.15073
INFO:tensorflow:global_step/sec: 1.14954
2020-01-13 09:06:48,648 [INFO] tensorflow: global_step/sec: 1.14954
INFO:tensorflow:global_step/sec: 1.14587
2020-01-13 09:06:49,521 [INFO] tensorflow: global_step/sec: 1.14587
2020-01-13 09:06:49,522 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 31/120: loss: 0.01118 Time taken: 0:00:06.971661 ETA: 0:10:20.477829
INFO:tensorflow:global_step/sec: 1.14874
2020-01-13 09:06:50,391 [INFO] tensorflow: global_step/sec: 1.14874
2020-01-13 09:06:50,393 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 17.985
INFO:tensorflow:global_step/sec: 1.14336
2020-01-13 09:06:51,266 [INFO] tensorflow: global_step/sec: 1.14336
INFO:tensorflow:epoch = 31.375, loss = 0.010314425, step = 251 (6.106 sec)
2020-01-13 09:06:52,139 [INFO] tensorflow: epoch = 31.375, loss = 0.010314425, step = 251 (6.106 sec)
INFO:tensorflow:global_step/sec: 1.14384
2020-01-13 09:06:52,140 [INFO] tensorflow: global_step/sec: 1.14384
INFO:tensorflow:global_step/sec: 1.14179
2020-01-13 09:06:53,016 [INFO] tensorflow: global_step/sec: 1.14179
INFO:tensorflow:global_step/sec: 1.15897
2020-01-13 09:06:53,879 [INFO] tensorflow: global_step/sec: 1.15897
INFO:tensorflow:global_step/sec: 1.15259
2020-01-13 09:06:54,747 [INFO] tensorflow: global_step/sec: 1.15259
INFO:tensorflow:global_step/sec: 1.14054
2020-01-13 09:06:55,623 [INFO] tensorflow: global_step/sec: 1.14054
INFO:tensorflow:global_step/sec: 1.13539
2020-01-13 09:06:56,504 [INFO] tensorflow: global_step/sec: 1.13539
2020-01-13 09:06:56,506 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 32/120: loss: 0.00953 Time taken: 0:00:06.970922 ETA: 0:10:13.441136
INFO:tensorflow:global_step/sec: 1.14327
2020-01-13 09:06:57,379 [INFO] tensorflow: global_step/sec: 1.14327
INFO:tensorflow:epoch = 32.25, loss = 0.010005676, step = 258 (6.109 sec)
2020-01-13 09:06:58,248 [INFO] tensorflow: epoch = 32.25, loss = 0.010005676, step = 258 (6.109 sec)
INFO:tensorflow:global_step/sec: 1.14908
2020-01-13 09:06:58,249 [INFO] tensorflow: global_step/sec: 1.14908
INFO:tensorflow:global_step/sec: 1.1475
2020-01-13 09:06:59,120 [INFO] tensorflow: global_step/sec: 1.1475
INFO:tensorflow:global_step/sec: 1.14703
2020-01-13 09:06:59,992 [INFO] tensorflow: global_step/sec: 1.14703
INFO:tensorflow:global_step/sec: 1.14211
2020-01-13 09:07:00,868 [INFO] tensorflow: global_step/sec: 1.14211
INFO:tensorflow:global_step/sec: 1.14662
2020-01-13 09:07:01,740 [INFO] tensorflow: global_step/sec: 1.14662
INFO:tensorflow:global_step/sec: 1.14617
2020-01-13 09:07:02,612 [INFO] tensorflow: global_step/sec: 1.14617
INFO:tensorflow:global_step/sec: 1.13784
2020-01-13 09:07:03,491 [INFO] tensorflow: global_step/sec: 1.13784
2020-01-13 09:07:03,493 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 33/120: loss: 0.00896 Time taken: 0:00:06.990394 ETA: 0:10:08.164278
INFO:tensorflow:epoch = 33.125, loss = 0.012269406, step = 265 (6.115 sec)
2020-01-13 09:07:04,363 [INFO] tensorflow: epoch = 33.125, loss = 0.012269406, step = 265 (6.115 sec)
INFO:tensorflow:global_step/sec: 1.14561
2020-01-13 09:07:04,364 [INFO] tensorflow: global_step/sec: 1.14561
INFO:tensorflow:global_step/sec: 1.14682
2020-01-13 09:07:05,236 [INFO] tensorflow: global_step/sec: 1.14682
INFO:tensorflow:global_step/sec: 1.14104
2020-01-13 09:07:06,113 [INFO] tensorflow: global_step/sec: 1.14104
INFO:tensorflow:global_step/sec: 1.14664
2020-01-13 09:07:06,985 [INFO] tensorflow: global_step/sec: 1.14664
INFO:tensorflow:global_step/sec: 1.14809
2020-01-13 09:07:07,856 [INFO] tensorflow: global_step/sec: 1.14809
INFO:tensorflow:global_step/sec: 1.14492
2020-01-13 09:07:08,729 [INFO] tensorflow: global_step/sec: 1.14492
INFO:tensorflow:global_step/sec: 1.15261
2020-01-13 09:07:09,597 [INFO] tensorflow: global_step/sec: 1.15261
INFO:tensorflow:epoch = 34.0, loss = 0.010143902, step = 272 (6.104 sec)
2020-01-13 09:07:10,466 [INFO] tensorflow: epoch = 34.0, loss = 0.010143902, step = 272 (6.104 sec)
INFO:tensorflow:global_step/sec: 1.14786
2020-01-13 09:07:10,468 [INFO] tensorflow: global_step/sec: 1.14786
2020-01-13 09:07:10,469 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 34/120: loss: 0.01014 Time taken: 0:00:06.979787 ETA: 0:10:00.261682
INFO:tensorflow:global_step/sec: 1.1439
2020-01-13 09:07:11,342 [INFO] tensorflow: global_step/sec: 1.1439
INFO:tensorflow:global_step/sec: 1.1445
2020-01-13 09:07:12,216 [INFO] tensorflow: global_step/sec: 1.1445
2020-01-13 09:07:12,217 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.328
INFO:tensorflow:global_step/sec: 1.14509
2020-01-13 09:07:13,089 [INFO] tensorflow: global_step/sec: 1.14509
INFO:tensorflow:global_step/sec: 1.14465
2020-01-13 09:07:13,963 [INFO] tensorflow: global_step/sec: 1.14465
INFO:tensorflow:global_step/sec: 1.14451
2020-01-13 09:07:14,836 [INFO] tensorflow: global_step/sec: 1.14451
INFO:tensorflow:global_step/sec: 1.14367
2020-01-13 09:07:15,711 [INFO] tensorflow: global_step/sec: 1.14367
INFO:tensorflow:epoch = 34.875, loss = 0.008242929, step = 279 (6.120 sec)
2020-01-13 09:07:16,587 [INFO] tensorflow: epoch = 34.875, loss = 0.008242929, step = 279 (6.120 sec)
INFO:tensorflow:global_step/sec: 1.13967
2020-01-13 09:07:16,588 [INFO] tensorflow: global_step/sec: 1.13967
INFO:tensorflow:Saving checkpoints for step-280.
2020-01-13 09:07:16,590 [INFO] tensorflow: Saving checkpoints for step-280.
INFO:tensorflow:global_step/sec: 0.757119
2020-01-13 09:07:17,909 [INFO] tensorflow: global_step/sec: 0.757119
2020-01-13 09:07:17,911 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 35/120: loss: 0.01004 Time taken: 0:00:07.446958 ETA: 0:10:32.991430
INFO:tensorflow:global_step/sec: 1.15672
2020-01-13 09:07:18,774 [INFO] tensorflow: global_step/sec: 1.15672
INFO:tensorflow:global_step/sec: 1.13438
2020-01-13 09:07:19,655 [INFO] tensorflow: global_step/sec: 1.13438
INFO:tensorflow:global_step/sec: 1.1385
2020-01-13 09:07:20,533 [INFO] tensorflow: global_step/sec: 1.1385
INFO:tensorflow:global_step/sec: 1.14118
2020-01-13 09:07:21,410 [INFO] tensorflow: global_step/sec: 1.14118
INFO:tensorflow:global_step/sec: 1.14438
2020-01-13 09:07:22,284 [INFO] tensorflow: global_step/sec: 1.14438
INFO:tensorflow:epoch = 35.75, loss = 0.008048964, step = 286 (6.572 sec)
2020-01-13 09:07:23,159 [INFO] tensorflow: epoch = 35.75, loss = 0.008048964, step = 286 (6.572 sec)
INFO:tensorflow:global_step/sec: 1.14069
2020-01-13 09:07:23,160 [INFO] tensorflow: global_step/sec: 1.14069
INFO:tensorflow:global_step/sec: 1.14209
2020-01-13 09:07:24,036 [INFO] tensorflow: global_step/sec: 1.14209
INFO:tensorflow:global_step/sec: 1.13879
2020-01-13 09:07:24,914 [INFO] tensorflow: global_step/sec: 1.13879
2020-01-13 09:07:24,916 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 36/120: loss: 0.00855 Time taken: 0:00:06.992730 ETA: 0:09:47.389320
INFO:tensorflow:global_step/sec: 1.14148
2020-01-13 09:07:25,790 [INFO] tensorflow: global_step/sec: 1.14148
INFO:tensorflow:global_step/sec: 1.14297
2020-01-13 09:07:26,665 [INFO] tensorflow: global_step/sec: 1.14297
INFO:tensorflow:global_step/sec: 1.13741
2020-01-13 09:07:27,544 [INFO] tensorflow: global_step/sec: 1.13741
INFO:tensorflow:global_step/sec: 1.14353
2020-01-13 09:07:28,419 [INFO] tensorflow: global_step/sec: 1.14353
INFO:tensorflow:epoch = 36.625, loss = 0.008221224, step = 293 (6.136 sec)
2020-01-13 09:07:29,295 [INFO] tensorflow: epoch = 36.625, loss = 0.008221224, step = 293 (6.136 sec)
INFO:tensorflow:global_step/sec: 1.13985
2020-01-13 09:07:29,296 [INFO] tensorflow: global_step/sec: 1.13985
INFO:tensorflow:global_step/sec: 1.14709
2020-01-13 09:07:30,168 [INFO] tensorflow: global_step/sec: 1.14709
INFO:tensorflow:global_step/sec: 1.14577
2020-01-13 09:07:31,041 [INFO] tensorflow: global_step/sec: 1.14577
INFO:tensorflow:global_step/sec: 1.13715
2020-01-13 09:07:31,920 [INFO] tensorflow: global_step/sec: 1.13715
2020-01-13 09:07:31,921 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 37/120: loss: 0.00899 Time taken: 0:00:07.001909 ETA: 0:09:41.158447
INFO:tensorflow:global_step/sec: 1.15095
2020-01-13 09:07:32,789 [INFO] tensorflow: global_step/sec: 1.15095
INFO:tensorflow:global_step/sec: 1.14333
2020-01-13 09:07:33,663 [INFO] tensorflow: global_step/sec: 1.14333
INFO:tensorflow:global_step/sec: 1.1424
2020-01-13 09:07:34,539 [INFO] tensorflow: global_step/sec: 1.1424
2020-01-13 09:07:34,540 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 17.919
INFO:tensorflow:epoch = 37.5, loss = 0.008026822, step = 300 (6.118 sec)
2020-01-13 09:07:35,413 [INFO] tensorflow: epoch = 37.5, loss = 0.008026822, step = 300 (6.118 sec)
INFO:tensorflow:global_step/sec: 1.1421
2020-01-13 09:07:35,414 [INFO] tensorflow: global_step/sec: 1.1421
INFO:tensorflow:global_step/sec: 1.13526
2020-01-13 09:07:36,295 [INFO] tensorflow: global_step/sec: 1.13526
INFO:tensorflow:global_step/sec: 1.14348
2020-01-13 09:07:37,170 [INFO] tensorflow: global_step/sec: 1.14348
INFO:tensorflow:global_step/sec: 1.14093
2020-01-13 09:07:38,046 [INFO] tensorflow: global_step/sec: 1.14093
INFO:tensorflow:global_step/sec: 1.12846
2020-01-13 09:07:38,932 [INFO] tensorflow: global_step/sec: 1.12846
2020-01-13 09:07:38,934 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 38/120: loss: 0.00701 Time taken: 0:00:07.007744 ETA: 0:09:34.635008
INFO:tensorflow:global_step/sec: 1.14285
2020-01-13 09:07:39,807 [INFO] tensorflow: global_step/sec: 1.14285
INFO:tensorflow:global_step/sec: 1.14732
2020-01-13 09:07:40,679 [INFO] tensorflow: global_step/sec: 1.14732
INFO:tensorflow:epoch = 38.375, loss = 0.0071663177, step = 307 (6.144 sec)
2020-01-13 09:07:41,557 [INFO] tensorflow: epoch = 38.375, loss = 0.0071663177, step = 307 (6.144 sec)
INFO:tensorflow:global_step/sec: 1.13685
2020-01-13 09:07:41,558 [INFO] tensorflow: global_step/sec: 1.13685
INFO:tensorflow:global_step/sec: 1.14295
2020-01-13 09:07:42,433 [INFO] tensorflow: global_step/sec: 1.14295
INFO:tensorflow:global_step/sec: 1.14265
2020-01-13 09:07:43,309 [INFO] tensorflow: global_step/sec: 1.14265
INFO:tensorflow:global_step/sec: 1.14257
2020-01-13 09:07:44,184 [INFO] tensorflow: global_step/sec: 1.14257
INFO:tensorflow:global_step/sec: 1.13315
2020-01-13 09:07:45,066 [INFO] tensorflow: global_step/sec: 1.13315
INFO:tensorflow:global_step/sec: 1.13501
2020-01-13 09:07:45,947 [INFO] tensorflow: global_step/sec: 1.13501
2020-01-13 09:07:45,949 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 39/120: loss: 0.00708 Time taken: 0:00:07.016607 ETA: 0:09:28.345167
INFO:tensorflow:global_step/sec: 1.13619
2020-01-13 09:07:46,828 [INFO] tensorflow: global_step/sec: 1.13619
INFO:tensorflow:epoch = 39.25, loss = 0.007587464, step = 314 (6.147 sec)
2020-01-13 09:07:47,704 [INFO] tensorflow: epoch = 39.25, loss = 0.007587464, step = 314 (6.147 sec)
INFO:tensorflow:global_step/sec: 1.13821
2020-01-13 09:07:47,706 [INFO] tensorflow: global_step/sec: 1.13821
INFO:tensorflow:global_step/sec: 1.13992
2020-01-13 09:07:48,583 [INFO] tensorflow: global_step/sec: 1.13992
INFO:tensorflow:global_step/sec: 1.13548
2020-01-13 09:07:49,464 [INFO] tensorflow: global_step/sec: 1.13548
INFO:tensorflow:global_step/sec: 1.14765
2020-01-13 09:07:50,335 [INFO] tensorflow: global_step/sec: 1.14765
INFO:tensorflow:global_step/sec: 1.1434
2020-01-13 09:07:51,210 [INFO] tensorflow: global_step/sec: 1.1434
INFO:tensorflow:global_step/sec: 1.14241
2020-01-13 09:07:52,085 [INFO] tensorflow: global_step/sec: 1.14241
INFO:tensorflow:Saving checkpoints for step-320.
2020-01-13 09:07:52,087 [INFO] tensorflow: Saving checkpoints for step-320.
INFO:tensorflow:global_step/sec: 0.734195
2020-01-13 09:07:53,447 [INFO] tensorflow: global_step/sec: 0.734195
2020-01-13 09:07:53,448 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 40/120: loss: 0.00644 Time taken: 0:00:07.517500 ETA: 0:10:01.400000
INFO:tensorflow:epoch = 40.125, loss = 0.006629837, step = 321 (6.609 sec)
2020-01-13 09:07:54,313 [INFO] tensorflow: epoch = 40.125, loss = 0.006629837, step = 321 (6.609 sec)
INFO:tensorflow:global_step/sec: 1.15255
2020-01-13 09:07:54,315 [INFO] tensorflow: global_step/sec: 1.15255
INFO:tensorflow:global_step/sec: 1.14249
2020-01-13 09:07:55,190 [INFO] tensorflow: global_step/sec: 1.14249
INFO:tensorflow:global_step/sec: 1.13538
2020-01-13 09:07:56,071 [INFO] tensorflow: global_step/sec: 1.13538
INFO:tensorflow:global_step/sec: 1.13793
2020-01-13 09:07:56,950 [INFO] tensorflow: global_step/sec: 1.13793
2020-01-13 09:07:56,951 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 17.849
INFO:tensorflow:global_step/sec: 1.14276
2020-01-13 09:07:57,825 [INFO] tensorflow: global_step/sec: 1.14276
INFO:tensorflow:global_step/sec: 1.13823
2020-01-13 09:07:58,703 [INFO] tensorflow: global_step/sec: 1.13823
INFO:tensorflow:global_step/sec: 1.14011
2020-01-13 09:07:59,581 [INFO] tensorflow: global_step/sec: 1.14011
INFO:tensorflow:epoch = 41.0, loss = 0.007186246, step = 328 (6.146 sec)
2020-01-13 09:08:00,459 [INFO] tensorflow: epoch = 41.0, loss = 0.007186246, step = 328 (6.146 sec)
INFO:tensorflow:global_step/sec: 1.13612
2020-01-13 09:08:00,461 [INFO] tensorflow: global_step/sec: 1.13612
2020-01-13 09:08:00,462 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 41/120: loss: 0.00719 Time taken: 0:00:06.997418 ETA: 0:09:12.796022
INFO:tensorflow:global_step/sec: 1.14459
2020-01-13 09:08:01,334 [INFO] tensorflow: global_step/sec: 1.14459
INFO:tensorflow:global_step/sec: 1.14661
2020-01-13 09:08:02,207 [INFO] tensorflow: global_step/sec: 1.14661
INFO:tensorflow:global_step/sec: 1.14291
2020-01-13 09:08:03,082 [INFO] tensorflow: global_step/sec: 1.14291
INFO:tensorflow:global_step/sec: 1.13793
2020-01-13 09:08:03,960 [INFO] tensorflow: global_step/sec: 1.13793
INFO:tensorflow:global_step/sec: 1.14532
2020-01-13 09:08:04,833 [INFO] tensorflow: global_step/sec: 1.14532
INFO:tensorflow:global_step/sec: 1.13839
2020-01-13 09:08:05,712 [INFO] tensorflow: global_step/sec: 1.13839
INFO:tensorflow:epoch = 41.875, loss = 0.005918531, step = 335 (6.132 sec)
2020-01-13 09:08:06,591 [INFO] tensorflow: epoch = 41.875, loss = 0.005918531, step = 335 (6.132 sec)
INFO:tensorflow:global_step/sec: 1.13543
2020-01-13 09:08:06,593 [INFO] tensorflow: global_step/sec: 1.13543
INFO:tensorflow:global_step/sec: 1.14001
2020-01-13 09:08:07,470 [INFO] tensorflow: global_step/sec: 1.14001
2020-01-13 09:08:07,471 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 42/120: loss: 0.00602 Time taken: 0:00:07.005010 ETA: 0:09:06.390780
INFO:tensorflow:global_step/sec: 1.14369
2020-01-13 09:08:08,344 [INFO] tensorflow: global_step/sec: 1.14369
INFO:tensorflow:global_step/sec: 1.13675
2020-01-13 09:08:09,224 [INFO] tensorflow: global_step/sec: 1.13675
INFO:tensorflow:global_step/sec: 1.13864
2020-01-13 09:08:10,102 [INFO] tensorflow: global_step/sec: 1.13864
INFO:tensorflow:global_step/sec: 1.14223
2020-01-13 09:08:10,978 [INFO] tensorflow: global_step/sec: 1.14223
INFO:tensorflow:global_step/sec: 1.13427
2020-01-13 09:08:11,859 [INFO] tensorflow: global_step/sec: 1.13427
INFO:tensorflow:epoch = 42.75, loss = 0.0057198703, step = 342 (6.149 sec)
2020-01-13 09:08:12,740 [INFO] tensorflow: epoch = 42.75, loss = 0.0057198703, step = 342 (6.149 sec)
INFO:tensorflow:global_step/sec: 1.13338
2020-01-13 09:08:12,742 [INFO] tensorflow: global_step/sec: 1.13338
INFO:tensorflow:global_step/sec: 1.14358
2020-01-13 09:08:13,616 [INFO] tensorflow: global_step/sec: 1.14358
INFO:tensorflow:global_step/sec: 1.13482
2020-01-13 09:08:14,497 [INFO] tensorflow: global_step/sec: 1.13482
2020-01-13 09:08:14,499 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 43/120: loss: 0.00610 Time taken: 0:00:07.023872 ETA: 0:09:00.838144
INFO:tensorflow:global_step/sec: 1.14127
2020-01-13 09:08:15,373 [INFO] tensorflow: global_step/sec: 1.14127
INFO:tensorflow:global_step/sec: 1.13718
2020-01-13 09:08:16,253 [INFO] tensorflow: global_step/sec: 1.13718
INFO:tensorflow:global_step/sec: 1.14298
2020-01-13 09:08:17,128 [INFO] tensorflow: global_step/sec: 1.14298
INFO:tensorflow:global_step/sec: 1.14212
2020-01-13 09:08:18,003 [INFO] tensorflow: global_step/sec: 1.14212
INFO:tensorflow:epoch = 43.625, loss = 0.005818277, step = 349 (6.144 sec)
2020-01-13 09:08:18,884 [INFO] tensorflow: epoch = 43.625, loss = 0.005818277, step = 349 (6.144 sec)
INFO:tensorflow:global_step/sec: 1.13321
2020-01-13 09:08:18,886 [INFO] tensorflow: global_step/sec: 1.13321
2020-01-13 09:08:18,887 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.235
INFO:tensorflow:global_step/sec: 1.14762
2020-01-13 09:08:19,757 [INFO] tensorflow: global_step/sec: 1.14762
INFO:tensorflow:global_step/sec: 1.14771
2020-01-13 09:08:20,628 [INFO] tensorflow: global_step/sec: 1.14771
INFO:tensorflow:global_step/sec: 1.13819
2020-01-13 09:08:21,507 [INFO] tensorflow: global_step/sec: 1.13819
2020-01-13 09:08:21,508 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 44/120: loss: 0.00621 Time taken: 0:00:07.013538 ETA: 0:08:53.028888
INFO:tensorflow:global_step/sec: 1.14067
2020-01-13 09:08:22,384 [INFO] tensorflow: global_step/sec: 1.14067
INFO:tensorflow:global_step/sec: 1.14117
2020-01-13 09:08:23,260 [INFO] tensorflow: global_step/sec: 1.14117
INFO:tensorflow:global_step/sec: 1.14585
2020-01-13 09:08:24,133 [INFO] tensorflow: global_step/sec: 1.14585
INFO:tensorflow:epoch = 44.5, loss = 0.0053089373, step = 356 (6.126 sec)
2020-01-13 09:08:25,011 [INFO] tensorflow: epoch = 44.5, loss = 0.0053089373, step = 356 (6.126 sec)
INFO:tensorflow:global_step/sec: 1.13703
2020-01-13 09:08:25,012 [INFO] tensorflow: global_step/sec: 1.13703
INFO:tensorflow:global_step/sec: 1.14844
2020-01-13 09:08:25,883 [INFO] tensorflow: global_step/sec: 1.14844
INFO:tensorflow:global_step/sec: 1.14762
2020-01-13 09:08:26,754 [INFO] tensorflow: global_step/sec: 1.14762
INFO:tensorflow:global_step/sec: 1.14598
2020-01-13 09:08:27,627 [INFO] tensorflow: global_step/sec: 1.14598
INFO:tensorflow:Saving checkpoints for step-360.
2020-01-13 09:08:27,628 [INFO] tensorflow: Saving checkpoints for step-360.
2020-01-13 09:08:28,091 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 1, 0.00s/step
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/evaluation/metadata.py:38: UserWarning: One or more metadata field(s) are missing from ground_truth batch_data, and will be replaced with defaults: [‘frame/camera_location’]
Matching predictions to ground truth, class 1/2.: 100%|##################################| 1/1 [00:00<00:00, 464.69it/s]
Matching predictions to ground truth, class 2/2.: 100%|##############################| 78/78 [00:00<00:00, 76029.68it/s]
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/evaluation/compute_metrics.py:717: RuntimeWarning: invalid value encountered in true_divide
Epoch 45/120
Validation cost: 0.018751
Mean average_precision (in %): nan
class name average precision (in %)
person nan
vehicle 0
Median Inference Time: 0.374451
INFO:tensorflow:epoch = 45.0, loss = 0.0052545117, step = 360 (22.807 sec)
2020-01-13 09:08:47,817 [INFO] tensorflow: epoch = 45.0, loss = 0.0052545117, step = 360 (22.807 sec)
INFO:tensorflow:global_step/sec: 0.0495251
2020-01-13 09:08:47,819 [INFO] tensorflow: global_step/sec: 0.0495251
2020-01-13 09:08:47,820 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 45/120: loss: 0.00525 Time taken: 0:00:26.299269 ETA: 0:32:52.445175
INFO:tensorflow:global_step/sec: 1.17189
2020-01-13 09:08:48,672 [INFO] tensorflow: global_step/sec: 1.17189
INFO:tensorflow:global_step/sec: 1.16685
2020-01-13 09:08:49,529 [INFO] tensorflow: global_step/sec: 1.16685
INFO:tensorflow:global_step/sec: 1.17651
2020-01-13 09:08:50,379 [INFO] tensorflow: global_step/sec: 1.17651
INFO:tensorflow:global_step/sec: 1.16843
2020-01-13 09:08:51,235 [INFO] tensorflow: global_step/sec: 1.16843
INFO:tensorflow:global_step/sec: 1.16548
2020-01-13 09:08:52,093 [INFO] tensorflow: global_step/sec: 1.16548
INFO:tensorflow:global_step/sec: 1.17084
2020-01-13 09:08:52,947 [INFO] tensorflow: global_step/sec: 1.17084
INFO:tensorflow:epoch = 45.875, loss = 0.005457866, step = 367 (5.983 sec)
2020-01-13 09:08:53,801 [INFO] tensorflow: epoch = 45.875, loss = 0.005457866, step = 367 (5.983 sec)
INFO:tensorflow:global_step/sec: 1.16914
2020-01-13 09:08:53,802 [INFO] tensorflow: global_step/sec: 1.16914
INFO:tensorflow:global_step/sec: 1.15881
2020-01-13 09:08:54,665 [INFO] tensorflow: global_step/sec: 1.15881
2020-01-13 09:08:54,667 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 46/120: loss: 0.00542 Time taken: 0:00:06.869998 ETA: 0:08:28.379852
INFO:tensorflow:global_step/sec: 1.16422
2020-01-13 09:08:55,524 [INFO] tensorflow: global_step/sec: 1.16422
INFO:tensorflow:global_step/sec: 1.17172
2020-01-13 09:08:56,378 [INFO] tensorflow: global_step/sec: 1.17172
INFO:tensorflow:global_step/sec: 1.158
2020-01-13 09:08:57,241 [INFO] tensorflow: global_step/sec: 1.158
INFO:tensorflow:global_step/sec: 1.16647
2020-01-13 09:08:58,098 [INFO] tensorflow: global_step/sec: 1.16647
INFO:tensorflow:global_step/sec: 1.17048
2020-01-13 09:08:58,953 [INFO] tensorflow: global_step/sec: 1.17048
INFO:tensorflow:epoch = 46.75, loss = 0.0049010357, step = 374 (6.015 sec)
2020-01-13 09:08:59,815 [INFO] tensorflow: epoch = 46.75, loss = 0.0049010357, step = 374 (6.015 sec)
INFO:tensorflow:global_step/sec: 1.15818
2020-01-13 09:08:59,816 [INFO] tensorflow: global_step/sec: 1.15818
2020-01-13 09:08:59,817 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.773
INFO:tensorflow:global_step/sec: 1.16719
2020-01-13 09:09:00,673 [INFO] tensorflow: global_step/sec: 1.16719
INFO:tensorflow:global_step/sec: 1.15324
2020-01-13 09:09:01,540 [INFO] tensorflow: global_step/sec: 1.15324
2020-01-13 09:09:01,542 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 47/120: loss: 0.00551 Time taken: 0:00:06.870875 ETA: 0:08:21.573875
INFO:tensorflow:global_step/sec: 1.16139
2020-01-13 09:09:02,401 [INFO] tensorflow: global_step/sec: 1.16139
INFO:tensorflow:global_step/sec: 1.16233
2020-01-13 09:09:03,261 [INFO] tensorflow: global_step/sec: 1.16233
INFO:tensorflow:global_step/sec: 1.16739
2020-01-13 09:09:04,118 [INFO] tensorflow: global_step/sec: 1.16739
INFO:tensorflow:glob