Always get 0 or nan precision during training detectnet

INFO:tensorflow:epoch = 0.0, loss = 0.08217385, step = 0
2020-01-13 09:03:11,264 [INFO] tensorflow: epoch = 0.0, loss = 0.08217385, step = 0
2020-01-13 09:03:11,266 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 0/120: loss: 0.08217 Time taken: 0:00:00 ETA: 0:00:00
INFO:tensorflow:global_step/sec: 0.13889
2020-01-13 09:03:18,465 [INFO] tensorflow: global_step/sec: 0.13889
INFO:tensorflow:epoch = 0.25, loss = 0.081661165, step = 2 (8.003 sec)
2020-01-13 09:03:19,266 [INFO] tensorflow: epoch = 0.25, loss = 0.081661165, step = 2 (8.003 sec)
INFO:tensorflow:global_step/sec: 1.24539
2020-01-13 09:03:19,268 [INFO] tensorflow: global_step/sec: 1.24539
INFO:tensorflow:global_step/sec: 1.23939
2020-01-13 09:03:20,075 [INFO] tensorflow: global_step/sec: 1.23939
INFO:tensorflow:global_step/sec: 1.25462
2020-01-13 09:03:20,872 [INFO] tensorflow: global_step/sec: 1.25462
INFO:tensorflow:global_step/sec: 1.24212
2020-01-13 09:03:21,677 [INFO] tensorflow: global_step/sec: 1.24212
INFO:tensorflow:global_step/sec: 1.25098
2020-01-13 09:03:22,476 [INFO] tensorflow: global_step/sec: 1.25098
INFO:tensorflow:global_step/sec: 1.23253
2020-01-13 09:03:23,288 [INFO] tensorflow: global_step/sec: 1.23253
INFO:tensorflow:global_step/sec: 0.671053
2020-01-13 09:03:24,778 [INFO] tensorflow: global_step/sec: 0.671053
2020-01-13 09:03:24,779 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 1/120: loss: 0.88263 Time taken: 0:00:28.790975 ETA: 0:57:06.126025
INFO:tensorflow:epoch = 1.125, loss = 0.89238954, step = 9 (6.317 sec)
2020-01-13 09:03:25,583 [INFO] tensorflow: epoch = 1.125, loss = 0.89238954, step = 9 (6.317 sec)
INFO:tensorflow:global_step/sec: 1.23907
2020-01-13 09:03:25,585 [INFO] tensorflow: global_step/sec: 1.23907
INFO:tensorflow:global_step/sec: 1.25482
2020-01-13 09:03:26,382 [INFO] tensorflow: global_step/sec: 1.25482
INFO:tensorflow:global_step/sec: 1.23969
2020-01-13 09:03:27,188 [INFO] tensorflow: global_step/sec: 1.23969
INFO:tensorflow:global_step/sec: 1.25345
2020-01-13 09:03:27,986 [INFO] tensorflow: global_step/sec: 1.25345
INFO:tensorflow:global_step/sec: 1.24189
2020-01-13 09:03:28,791 [INFO] tensorflow: global_step/sec: 1.24189
INFO:tensorflow:global_step/sec: 1.23142
2020-01-13 09:03:29,604 [INFO] tensorflow: global_step/sec: 1.23142
INFO:tensorflow:global_step/sec: 1.25021
2020-01-13 09:03:30,403 [INFO] tensorflow: global_step/sec: 1.25021
INFO:tensorflow:global_step/sec: 1.23834
2020-01-13 09:03:31,211 [INFO] tensorflow: global_step/sec: 1.23834
2020-01-13 09:03:31,212 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 2/120: loss: 0.88019 Time taken: 0:00:06.424891 ETA: 0:12:38.137138
INFO:tensorflow:epoch = 2.125, loss = 0.88394403, step = 17 (6.437 sec)
2020-01-13 09:03:32,020 [INFO] tensorflow: epoch = 2.125, loss = 0.88394403, step = 17 (6.437 sec)
INFO:tensorflow:global_step/sec: 1.23332
2020-01-13 09:03:32,022 [INFO] tensorflow: global_step/sec: 1.23332
INFO:tensorflow:global_step/sec: 1.23491
2020-01-13 09:03:32,832 [INFO] tensorflow: global_step/sec: 1.23491
INFO:tensorflow:global_step/sec: 1.23355
2020-01-13 09:03:33,642 [INFO] tensorflow: global_step/sec: 1.23355
INFO:tensorflow:global_step/sec: 1.23715
2020-01-13 09:03:34,450 [INFO] tensorflow: global_step/sec: 1.23715
INFO:tensorflow:global_step/sec: 1.23514
2020-01-13 09:03:35,260 [INFO] tensorflow: global_step/sec: 1.23514
INFO:tensorflow:global_step/sec: 1.23509
2020-01-13 09:03:36,070 [INFO] tensorflow: global_step/sec: 1.23509
INFO:tensorflow:global_step/sec: 1.23888
2020-01-13 09:03:36,877 [INFO] tensorflow: global_step/sec: 1.23888
INFO:tensorflow:global_step/sec: 1.2291
2020-01-13 09:03:37,691 [INFO] tensorflow: global_step/sec: 1.2291
2020-01-13 09:03:37,692 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 3/120: loss: 0.84273 Time taken: 0:00:06.469167 ETA: 0:12:36.892539
2020-01-13 09:03:37,693 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.412
INFO:tensorflow:epoch = 3.125, loss = 0.84840333, step = 25 (6.482 sec)
2020-01-13 09:03:38,502 [INFO] tensorflow: epoch = 3.125, loss = 0.84840333, step = 25 (6.482 sec)
INFO:tensorflow:global_step/sec: 1.22973
2020-01-13 09:03:38,504 [INFO] tensorflow: global_step/sec: 1.22973
INFO:tensorflow:global_step/sec: 1.23322
2020-01-13 09:03:39,315 [INFO] tensorflow: global_step/sec: 1.23322
INFO:tensorflow:global_step/sec: 1.23214
2020-01-13 09:03:40,126 [INFO] tensorflow: global_step/sec: 1.23214
INFO:tensorflow:global_step/sec: 1.23505
2020-01-13 09:03:40,936 [INFO] tensorflow: global_step/sec: 1.23505
INFO:tensorflow:global_step/sec: 1.23297
2020-01-13 09:03:41,747 [INFO] tensorflow: global_step/sec: 1.23297
INFO:tensorflow:global_step/sec: 1.23499
2020-01-13 09:03:42,557 [INFO] tensorflow: global_step/sec: 1.23499
INFO:tensorflow:global_step/sec: 1.23409
2020-01-13 09:03:43,367 [INFO] tensorflow: global_step/sec: 1.23409
INFO:tensorflow:global_step/sec: 1.22928
2020-01-13 09:03:44,180 [INFO] tensorflow: global_step/sec: 1.22928
2020-01-13 09:03:44,182 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 4/120: loss: 0.82718 Time taken: 0:00:06.487857 ETA: 0:12:32.591412
INFO:tensorflow:epoch = 4.125, loss = 0.8112773, step = 33 (6.487 sec)
2020-01-13 09:03:44,989 [INFO] tensorflow: epoch = 4.125, loss = 0.8112773, step = 33 (6.487 sec)
INFO:tensorflow:global_step/sec: 1.23456
2020-01-13 09:03:44,990 [INFO] tensorflow: global_step/sec: 1.23456
INFO:tensorflow:global_step/sec: 1.23234
2020-01-13 09:03:45,802 [INFO] tensorflow: global_step/sec: 1.23234
INFO:tensorflow:global_step/sec: 1.23289
2020-01-13 09:03:46,613 [INFO] tensorflow: global_step/sec: 1.23289
INFO:tensorflow:global_step/sec: 1.23381
2020-01-13 09:03:47,424 [INFO] tensorflow: global_step/sec: 1.23381
INFO:tensorflow:global_step/sec: 1.24023
2020-01-13 09:03:48,230 [INFO] tensorflow: global_step/sec: 1.24023
INFO:tensorflow:global_step/sec: 1.23209
2020-01-13 09:03:49,041 [INFO] tensorflow: global_step/sec: 1.23209
INFO:tensorflow:global_step/sec: 1.22952
2020-01-13 09:03:49,855 [INFO] tensorflow: global_step/sec: 1.22952
INFO:tensorflow:Saving checkpoints for step-40.
2020-01-13 09:03:49,856 [INFO] tensorflow: Saving checkpoints for step-40.
INFO:tensorflow:epoch = 5.0, loss = 0.793715, step = 40 (6.136 sec)
2020-01-13 09:03:51,125 [INFO] tensorflow: epoch = 5.0, loss = 0.793715, step = 40 (6.136 sec)
INFO:tensorflow:global_step/sec: 0.785983
2020-01-13 09:03:51,127 [INFO] tensorflow: global_step/sec: 0.785983
2020-01-13 09:03:51,128 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 5/120: loss: 0.79372 Time taken: 0:00:06.949062 ETA: 0:13:19.142130
INFO:tensorflow:global_step/sec: 1.23249
2020-01-13 09:03:51,939 [INFO] tensorflow: global_step/sec: 1.23249
INFO:tensorflow:global_step/sec: 1.22749
2020-01-13 09:03:52,753 [INFO] tensorflow: global_step/sec: 1.22749
INFO:tensorflow:global_step/sec: 1.22863
2020-01-13 09:03:53,567 [INFO] tensorflow: global_step/sec: 1.22863
INFO:tensorflow:global_step/sec: 1.22833
2020-01-13 09:03:54,381 [INFO] tensorflow: global_step/sec: 1.22833
INFO:tensorflow:global_step/sec: 1.23041
2020-01-13 09:03:55,194 [INFO] tensorflow: global_step/sec: 1.23041
INFO:tensorflow:global_step/sec: 1.2312
2020-01-13 09:03:56,006 [INFO] tensorflow: global_step/sec: 1.2312
INFO:tensorflow:global_step/sec: 1.22334
2020-01-13 09:03:56,824 [INFO] tensorflow: global_step/sec: 1.22334
INFO:tensorflow:epoch = 6.0, loss = 0.7258079, step = 48 (6.520 sec)
2020-01-13 09:03:57,645 [INFO] tensorflow: epoch = 6.0, loss = 0.7258079, step = 48 (6.520 sec)
INFO:tensorflow:global_step/sec: 1.2156
2020-01-13 09:03:57,646 [INFO] tensorflow: global_step/sec: 1.2156
2020-01-13 09:03:57,647 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 6/120: loss: 0.72581 Time taken: 0:00:06.508513 ETA: 0:12:21.970482
INFO:tensorflow:global_step/sec: 1.22714
2020-01-13 09:03:58,461 [INFO] tensorflow: global_step/sec: 1.22714
2020-01-13 09:03:58,462 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 19.259
INFO:tensorflow:global_step/sec: 1.22683
2020-01-13 09:03:59,276 [INFO] tensorflow: global_step/sec: 1.22683
INFO:tensorflow:global_step/sec: 1.2248
2020-01-13 09:04:00,093 [INFO] tensorflow: global_step/sec: 1.2248
INFO:tensorflow:global_step/sec: 1.225
2020-01-13 09:04:00,909 [INFO] tensorflow: global_step/sec: 1.225
INFO:tensorflow:global_step/sec: 1.22329
2020-01-13 09:04:01,726 [INFO] tensorflow: global_step/sec: 1.22329
INFO:tensorflow:global_step/sec: 1.22106
2020-01-13 09:04:02,545 [INFO] tensorflow: global_step/sec: 1.22106
INFO:tensorflow:global_step/sec: 1.22076
2020-01-13 09:04:03,365 [INFO] tensorflow: global_step/sec: 1.22076
INFO:tensorflow:epoch = 7.0, loss = 0.65784985, step = 56 (6.545 sec)
2020-01-13 09:04:04,190 [INFO] tensorflow: epoch = 7.0, loss = 0.65784985, step = 56 (6.545 sec)
INFO:tensorflow:global_step/sec: 1.20875
2020-01-13 09:04:04,192 [INFO] tensorflow: global_step/sec: 1.20875
2020-01-13 09:04:04,193 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 7/120: loss: 0.65785 Time taken: 0:00:06.541966 ETA: 0:12:19.242158
INFO:tensorflow:global_step/sec: 1.22432
2020-01-13 09:04:05,009 [INFO] tensorflow: global_step/sec: 1.22432
INFO:tensorflow:global_step/sec: 1.21864
2020-01-13 09:04:05,829 [INFO] tensorflow: global_step/sec: 1.21864
INFO:tensorflow:global_step/sec: 1.22613
2020-01-13 09:04:06,645 [INFO] tensorflow: global_step/sec: 1.22613
INFO:tensorflow:global_step/sec: 1.22405
2020-01-13 09:04:07,462 [INFO] tensorflow: global_step/sec: 1.22405
INFO:tensorflow:global_step/sec: 1.21197
2020-01-13 09:04:08,287 [INFO] tensorflow: global_step/sec: 1.21197
INFO:tensorflow:global_step/sec: 1.21861
2020-01-13 09:04:09,108 [INFO] tensorflow: global_step/sec: 1.21861
INFO:tensorflow:global_step/sec: 1.21626
2020-01-13 09:04:09,930 [INFO] tensorflow: global_step/sec: 1.21626
INFO:tensorflow:epoch = 8.0, loss = 0.5598427, step = 64 (6.569 sec)
2020-01-13 09:04:10,760 [INFO] tensorflow: epoch = 8.0, loss = 0.5598427, step = 64 (6.569 sec)
INFO:tensorflow:global_step/sec: 1.20245
2020-01-13 09:04:10,761 [INFO] tensorflow: global_step/sec: 1.20245
2020-01-13 09:04:10,763 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 8/120: loss: 0.55984 Time taken: 0:00:06.559767 ETA: 0:12:14.693904
INFO:tensorflow:global_step/sec: 1.22415
2020-01-13 09:04:11,578 [INFO] tensorflow: global_step/sec: 1.22415
INFO:tensorflow:global_step/sec: 1.21284
2020-01-13 09:04:12,403 [INFO] tensorflow: global_step/sec: 1.21284
INFO:tensorflow:global_step/sec: 1.21669
2020-01-13 09:04:13,225 [INFO] tensorflow: global_step/sec: 1.21669
INFO:tensorflow:global_step/sec: 1.22166
2020-01-13 09:04:14,043 [INFO] tensorflow: global_step/sec: 1.22166
INFO:tensorflow:global_step/sec: 1.21026
2020-01-13 09:04:14,869 [INFO] tensorflow: global_step/sec: 1.21026
INFO:tensorflow:global_step/sec: 1.21357
2020-01-13 09:04:15,694 [INFO] tensorflow: global_step/sec: 1.21357
INFO:tensorflow:global_step/sec: 1.21719
2020-01-13 09:04:16,515 [INFO] tensorflow: global_step/sec: 1.21719
INFO:tensorflow:epoch = 9.0, loss = 0.4610829, step = 72 (6.586 sec)
2020-01-13 09:04:17,345 [INFO] tensorflow: epoch = 9.0, loss = 0.4610829, step = 72 (6.586 sec)
INFO:tensorflow:global_step/sec: 1.2021
2020-01-13 09:04:17,347 [INFO] tensorflow: global_step/sec: 1.2021
2020-01-13 09:04:17,348 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 9/120: loss: 0.46108 Time taken: 0:00:06.585149 ETA: 0:12:10.951539
INFO:tensorflow:global_step/sec: 1.20817
2020-01-13 09:04:18,175 [INFO] tensorflow: global_step/sec: 1.20817
INFO:tensorflow:global_step/sec: 1.20627
2020-01-13 09:04:19,004 [INFO] tensorflow: global_step/sec: 1.20627
2020-01-13 09:04:19,005 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 19.472
INFO:tensorflow:global_step/sec: 1.21284
2020-01-13 09:04:19,828 [INFO] tensorflow: global_step/sec: 1.21284
INFO:tensorflow:global_step/sec: 1.21585
2020-01-13 09:04:20,651 [INFO] tensorflow: global_step/sec: 1.21585
INFO:tensorflow:global_step/sec: 1.20792
2020-01-13 09:04:21,478 [INFO] tensorflow: global_step/sec: 1.20792
INFO:tensorflow:global_step/sec: 1.20794
2020-01-13 09:04:22,306 [INFO] tensorflow: global_step/sec: 1.20794
INFO:tensorflow:global_step/sec: 1.20136
2020-01-13 09:04:23,139 [INFO] tensorflow: global_step/sec: 1.20136
INFO:tensorflow:Saving checkpoints for step-80.
2020-01-13 09:04:23,140 [INFO] tensorflow: Saving checkpoints for step-80.
INFO:tensorflow:epoch = 10.0, loss = 0.37151942, step = 80 (7.085 sec)
2020-01-13 09:04:24,430 [INFO] tensorflow: epoch = 10.0, loss = 0.37151942, step = 80 (7.085 sec)
INFO:tensorflow:global_step/sec: 0.773477
2020-01-13 09:04:24,432 [INFO] tensorflow: global_step/sec: 0.773477
2020-01-13 09:04:24,433 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 10/120: loss: 0.37152 Time taken: 0:00:07.089200 ETA: 0:12:59.812000
INFO:tensorflow:global_step/sec: 1.20121
2020-01-13 09:04:25,264 [INFO] tensorflow: global_step/sec: 1.20121
INFO:tensorflow:global_step/sec: 1.20272
2020-01-13 09:04:26,096 [INFO] tensorflow: global_step/sec: 1.20272
INFO:tensorflow:global_step/sec: 1.20062
2020-01-13 09:04:26,928 [INFO] tensorflow: global_step/sec: 1.20062
INFO:tensorflow:global_step/sec: 1.20317
2020-01-13 09:04:27,760 [INFO] tensorflow: global_step/sec: 1.20317
INFO:tensorflow:global_step/sec: 1.20706
2020-01-13 09:04:28,588 [INFO] tensorflow: global_step/sec: 1.20706
INFO:tensorflow:global_step/sec: 1.20326
2020-01-13 09:04:29,419 [INFO] tensorflow: global_step/sec: 1.20326
INFO:tensorflow:global_step/sec: 1.20178
2020-01-13 09:04:30,251 [INFO] tensorflow: global_step/sec: 1.20178
INFO:tensorflow:epoch = 11.0, loss = 0.24042998, step = 88 (6.651 sec)
2020-01-13 09:04:31,081 [INFO] tensorflow: epoch = 11.0, loss = 0.24042998, step = 88 (6.651 sec)
INFO:tensorflow:global_step/sec: 1.20248
2020-01-13 09:04:31,083 [INFO] tensorflow: global_step/sec: 1.20248
2020-01-13 09:04:31,084 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 11/120: loss: 0.24043 Time taken: 0:00:06.648068 ETA: 0:12:04.639412
INFO:tensorflow:global_step/sec: 1.20146
2020-01-13 09:04:31,915 [INFO] tensorflow: global_step/sec: 1.20146
INFO:tensorflow:global_step/sec: 1.20348
2020-01-13 09:04:32,746 [INFO] tensorflow: global_step/sec: 1.20348
INFO:tensorflow:global_step/sec: 1.19836
2020-01-13 09:04:33,581 [INFO] tensorflow: global_step/sec: 1.19836
INFO:tensorflow:global_step/sec: 1.20527
2020-01-13 09:04:34,410 [INFO] tensorflow: global_step/sec: 1.20527
INFO:tensorflow:global_step/sec: 1.20532
2020-01-13 09:04:35,240 [INFO] tensorflow: global_step/sec: 1.20532
INFO:tensorflow:global_step/sec: 1.20257
2020-01-13 09:04:36,071 [INFO] tensorflow: global_step/sec: 1.20257
INFO:tensorflow:global_step/sec: 1.20472
2020-01-13 09:04:36,901 [INFO] tensorflow: global_step/sec: 1.20472
INFO:tensorflow:epoch = 12.0, loss = 0.15279289, step = 96 (6.653 sec)
2020-01-13 09:04:37,735 [INFO] tensorflow: epoch = 12.0, loss = 0.15279289, step = 96 (6.653 sec)
INFO:tensorflow:global_step/sec: 1.19832
2020-01-13 09:04:37,736 [INFO] tensorflow: global_step/sec: 1.19832
2020-01-13 09:04:37,737 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 12/120: loss: 0.15279 Time taken: 0:00:06.645763 ETA: 0:11:57.742404
INFO:tensorflow:global_step/sec: 1.20362
2020-01-13 09:04:38,567 [INFO] tensorflow: global_step/sec: 1.20362
INFO:tensorflow:global_step/sec: 1.2017
2020-01-13 09:04:39,399 [INFO] tensorflow: global_step/sec: 1.2017
INFO:tensorflow:global_step/sec: 1.20585
2020-01-13 09:04:40,228 [INFO] tensorflow: global_step/sec: 1.20585
2020-01-13 09:04:40,230 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.847
INFO:tensorflow:global_step/sec: 1.20181
2020-01-13 09:04:41,060 [INFO] tensorflow: global_step/sec: 1.20181
INFO:tensorflow:global_step/sec: 1.20076
2020-01-13 09:04:41,893 [INFO] tensorflow: global_step/sec: 1.20076
INFO:tensorflow:global_step/sec: 1.20406
2020-01-13 09:04:42,724 [INFO] tensorflow: global_step/sec: 1.20406
INFO:tensorflow:global_step/sec: 1.20429
2020-01-13 09:04:43,554 [INFO] tensorflow: global_step/sec: 1.20429
INFO:tensorflow:epoch = 13.0, loss = 0.09585952, step = 104 (6.665 sec)
2020-01-13 09:04:44,399 [INFO] tensorflow: epoch = 13.0, loss = 0.09585952, step = 104 (6.665 sec)
INFO:tensorflow:global_step/sec: 1.18113
2020-01-13 09:04:44,401 [INFO] tensorflow: global_step/sec: 1.18113
2020-01-13 09:04:44,402 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 13/120: loss: 0.09586 Time taken: 0:00:06.654223 ETA: 0:11:52.001861
INFO:tensorflow:global_step/sec: 1.20214
2020-01-13 09:04:45,233 [INFO] tensorflow: global_step/sec: 1.20214
INFO:tensorflow:global_step/sec: 1.20131
2020-01-13 09:04:46,065 [INFO] tensorflow: global_step/sec: 1.20131
INFO:tensorflow:global_step/sec: 1.19416
2020-01-13 09:04:46,902 [INFO] tensorflow: global_step/sec: 1.19416
INFO:tensorflow:global_step/sec: 1.19085
2020-01-13 09:04:47,742 [INFO] tensorflow: global_step/sec: 1.19085
INFO:tensorflow:global_step/sec: 1.19763
2020-01-13 09:04:48,577 [INFO] tensorflow: global_step/sec: 1.19763
INFO:tensorflow:global_step/sec: 1.19465
2020-01-13 09:04:49,414 [INFO] tensorflow: global_step/sec: 1.19465
INFO:tensorflow:epoch = 13.875, loss = 0.07340296, step = 111 (5.854 sec)
2020-01-13 09:04:50,253 [INFO] tensorflow: epoch = 13.875, loss = 0.07340296, step = 111 (5.854 sec)
INFO:tensorflow:global_step/sec: 1.18977
2020-01-13 09:04:50,255 [INFO] tensorflow: global_step/sec: 1.18977
INFO:tensorflow:global_step/sec: 1.18541
2020-01-13 09:04:51,098 [INFO] tensorflow: global_step/sec: 1.18541
2020-01-13 09:04:51,100 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 14/120: loss: 0.06708 Time taken: 0:00:06.697458 ETA: 0:11:49.930548
INFO:tensorflow:global_step/sec: 1.19475
2020-01-13 09:04:51,935 [INFO] tensorflow: global_step/sec: 1.19475
INFO:tensorflow:global_step/sec: 1.19367
2020-01-13 09:04:52,773 [INFO] tensorflow: global_step/sec: 1.19367
INFO:tensorflow:global_step/sec: 1.18981
2020-01-13 09:04:53,613 [INFO] tensorflow: global_step/sec: 1.18981
INFO:tensorflow:global_step/sec: 1.1982
2020-01-13 09:04:54,448 [INFO] tensorflow: global_step/sec: 1.1982
INFO:tensorflow:global_step/sec: 1.18652
2020-01-13 09:04:55,291 [INFO] tensorflow: global_step/sec: 1.18652
INFO:tensorflow:epoch = 14.75, loss = 0.0560713, step = 118 (5.877 sec)
2020-01-13 09:04:56,130 [INFO] tensorflow: epoch = 14.75, loss = 0.0560713, step = 118 (5.877 sec)
INFO:tensorflow:global_step/sec: 1.1892
2020-01-13 09:04:56,132 [INFO] tensorflow: global_step/sec: 1.1892
INFO:tensorflow:global_step/sec: 1.18348
2020-01-13 09:04:56,977 [INFO] tensorflow: global_step/sec: 1.18348
INFO:tensorflow:Saving checkpoints for step-120.
2020-01-13 09:04:56,978 [INFO] tensorflow: Saving checkpoints for step-120.
INFO:tensorflow:global_step/sec: 0.770842
2020-01-13 09:04:58,274 [INFO] tensorflow: global_step/sec: 0.770842
2020-01-13 09:04:58,276 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 15/120: loss: 0.05127 Time taken: 0:00:07.181245 ETA: 0:12:34.030725
INFO:tensorflow:global_step/sec: 1.1828
2020-01-13 09:04:59,119 [INFO] tensorflow: global_step/sec: 1.1828
INFO:tensorflow:global_step/sec: 1.18271
2020-01-13 09:04:59,965 [INFO] tensorflow: global_step/sec: 1.18271
INFO:tensorflow:global_step/sec: 1.18786
2020-01-13 09:05:00,807 [INFO] tensorflow: global_step/sec: 1.18786
INFO:tensorflow:global_step/sec: 1.18073
2020-01-13 09:05:01,654 [INFO] tensorflow: global_step/sec: 1.18073
2020-01-13 09:05:01,655 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.670
INFO:tensorflow:epoch = 15.625, loss = 0.045743596, step = 125 (6.365 sec)
2020-01-13 09:05:02,495 [INFO] tensorflow: epoch = 15.625, loss = 0.045743596, step = 125 (6.365 sec)
INFO:tensorflow:global_step/sec: 1.18607
2020-01-13 09:05:02,497 [INFO] tensorflow: global_step/sec: 1.18607
INFO:tensorflow:global_step/sec: 1.18488
2020-01-13 09:05:03,341 [INFO] tensorflow: global_step/sec: 1.18488
INFO:tensorflow:global_step/sec: 1.18146
2020-01-13 09:05:04,187 [INFO] tensorflow: global_step/sec: 1.18146
INFO:tensorflow:global_step/sec: 1.18053
2020-01-13 09:05:05,034 [INFO] tensorflow: global_step/sec: 1.18053
2020-01-13 09:05:05,036 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 16/120: loss: 0.04340 Time taken: 0:00:06.751384 ETA: 0:11:42.143936
INFO:tensorflow:global_step/sec: 1.19055
2020-01-13 09:05:05,874 [INFO] tensorflow: global_step/sec: 1.19055
INFO:tensorflow:global_step/sec: 1.18252
2020-01-13 09:05:06,720 [INFO] tensorflow: global_step/sec: 1.18252
INFO:tensorflow:global_step/sec: 1.19334
2020-01-13 09:05:07,558 [INFO] tensorflow: global_step/sec: 1.19334
INFO:tensorflow:epoch = 16.5, loss = 0.042903394, step = 132 (5.907 sec)
2020-01-13 09:05:08,402 [INFO] tensorflow: epoch = 16.5, loss = 0.042903394, step = 132 (5.907 sec)
INFO:tensorflow:global_step/sec: 1.18161
2020-01-13 09:05:08,404 [INFO] tensorflow: global_step/sec: 1.18161
INFO:tensorflow:global_step/sec: 1.18271
2020-01-13 09:05:09,250 [INFO] tensorflow: global_step/sec: 1.18271
INFO:tensorflow:global_step/sec: 1.18744
2020-01-13 09:05:10,092 [INFO] tensorflow: global_step/sec: 1.18744
INFO:tensorflow:global_step/sec: 1.18766
2020-01-13 09:05:10,934 [INFO] tensorflow: global_step/sec: 1.18766
INFO:tensorflow:global_step/sec: 1.17512
2020-01-13 09:05:11,785 [INFO] tensorflow: global_step/sec: 1.17512
2020-01-13 09:05:11,786 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 17/120: loss: 0.03505 Time taken: 0:00:06.741886 ETA: 0:11:34.414258
INFO:tensorflow:global_step/sec: 1.18949
2020-01-13 09:05:12,626 [INFO] tensorflow: global_step/sec: 1.18949
INFO:tensorflow:global_step/sec: 1.18392
2020-01-13 09:05:13,470 [INFO] tensorflow: global_step/sec: 1.18392
INFO:tensorflow:epoch = 17.375, loss = 0.03255378, step = 139 (5.915 sec)
2020-01-13 09:05:14,317 [INFO] tensorflow: epoch = 17.375, loss = 0.03255378, step = 139 (5.915 sec)
INFO:tensorflow:global_step/sec: 1.17861
2020-01-13 09:05:14,319 [INFO] tensorflow: global_step/sec: 1.17861
INFO:tensorflow:global_step/sec: 1.17902
2020-01-13 09:05:15,167 [INFO] tensorflow: global_step/sec: 1.17902
INFO:tensorflow:global_step/sec: 1.17729
2020-01-13 09:05:16,016 [INFO] tensorflow: global_step/sec: 1.17729
INFO:tensorflow:global_step/sec: 1.18077
2020-01-13 09:05:16,863 [INFO] tensorflow: global_step/sec: 1.18077
INFO:tensorflow:global_step/sec: 1.17633
2020-01-13 09:05:17,713 [INFO] tensorflow: global_step/sec: 1.17633
INFO:tensorflow:global_step/sec: 1.17275
2020-01-13 09:05:18,566 [INFO] tensorflow: global_step/sec: 1.17275
2020-01-13 09:05:18,567 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 18/120: loss: 0.03164 Time taken: 0:00:06.781169 ETA: 0:11:31.679238
INFO:tensorflow:global_step/sec: 1.17412
2020-01-13 09:05:19,418 [INFO] tensorflow: global_step/sec: 1.17412
INFO:tensorflow:epoch = 18.25, loss = 0.028710622, step = 146 (5.953 sec)
2020-01-13 09:05:20,270 [INFO] tensorflow: epoch = 18.25, loss = 0.028710622, step = 146 (5.953 sec)
INFO:tensorflow:global_step/sec: 1.17143
2020-01-13 09:05:20,271 [INFO] tensorflow: global_step/sec: 1.17143
INFO:tensorflow:global_step/sec: 1.17769
2020-01-13 09:05:21,120 [INFO] tensorflow: global_step/sec: 1.17769
INFO:tensorflow:global_step/sec: 1.17181
2020-01-13 09:05:21,974 [INFO] tensorflow: global_step/sec: 1.17181
INFO:tensorflow:global_step/sec: 1.17752
2020-01-13 09:05:22,823 [INFO] tensorflow: global_step/sec: 1.17752
2020-01-13 09:05:22,824 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.896
INFO:tensorflow:global_step/sec: 1.17529
2020-01-13 09:05:23,674 [INFO] tensorflow: global_step/sec: 1.17529
INFO:tensorflow:global_step/sec: 1.17504
2020-01-13 09:05:24,525 [INFO] tensorflow: global_step/sec: 1.17504
INFO:tensorflow:global_step/sec: 1.16963
2020-01-13 09:05:25,380 [INFO] tensorflow: global_step/sec: 1.16963
2020-01-13 09:05:25,381 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 19/120: loss: 0.02699 Time taken: 0:00:06.809700 ETA: 0:11:27.779700
INFO:tensorflow:epoch = 19.125, loss = 0.026556345, step = 153 (5.958 sec)
2020-01-13 09:05:26,228 [INFO] tensorflow: epoch = 19.125, loss = 0.026556345, step = 153 (5.958 sec)
INFO:tensorflow:global_step/sec: 1.17718
2020-01-13 09:05:26,229 [INFO] tensorflow: global_step/sec: 1.17718
INFO:tensorflow:global_step/sec: 1.1786
2020-01-13 09:05:27,078 [INFO] tensorflow: global_step/sec: 1.1786
INFO:tensorflow:global_step/sec: 1.17575
2020-01-13 09:05:27,928 [INFO] tensorflow: global_step/sec: 1.17575
INFO:tensorflow:global_step/sec: 1.1763
2020-01-13 09:05:28,778 [INFO] tensorflow: global_step/sec: 1.1763
INFO:tensorflow:global_step/sec: 1.17554
2020-01-13 09:05:29,629 [INFO] tensorflow: global_step/sec: 1.17554
INFO:tensorflow:global_step/sec: 1.17479
2020-01-13 09:05:30,480 [INFO] tensorflow: global_step/sec: 1.17479
INFO:tensorflow:global_step/sec: 1.17323
2020-01-13 09:05:31,333 [INFO] tensorflow: global_step/sec: 1.17323
INFO:tensorflow:Saving checkpoints for step-160.
2020-01-13 09:05:31,334 [INFO] tensorflow: Saving checkpoints for step-160.
INFO:tensorflow:epoch = 20.0, loss = 0.026538212, step = 160 (6.412 sec)
2020-01-13 09:05:32,640 [INFO] tensorflow: epoch = 20.0, loss = 0.026538212, step = 160 (6.412 sec)
INFO:tensorflow:global_step/sec: 0.763986
2020-01-13 09:05:32,642 [INFO] tensorflow: global_step/sec: 0.763986
2020-01-13 09:05:32,643 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 20/120: loss: 0.02654 Time taken: 0:00:07.265122 ETA: 0:12:06.512200
INFO:tensorflow:global_step/sec: 1.17021
2020-01-13 09:05:33,496 [INFO] tensorflow: global_step/sec: 1.17021
INFO:tensorflow:global_step/sec: 1.1748
2020-01-13 09:05:34,347 [INFO] tensorflow: global_step/sec: 1.1748
INFO:tensorflow:global_step/sec: 1.16869
2020-01-13 09:05:35,203 [INFO] tensorflow: global_step/sec: 1.16869
INFO:tensorflow:global_step/sec: 1.17663
2020-01-13 09:05:36,053 [INFO] tensorflow: global_step/sec: 1.17663
INFO:tensorflow:global_step/sec: 1.17693
2020-01-13 09:05:36,903 [INFO] tensorflow: global_step/sec: 1.17693
INFO:tensorflow:global_step/sec: 1.1789
2020-01-13 09:05:37,751 [INFO] tensorflow: global_step/sec: 1.1789
INFO:tensorflow:epoch = 20.875, loss = 0.022777596, step = 167 (5.965 sec)
2020-01-13 09:05:38,605 [INFO] tensorflow: epoch = 20.875, loss = 0.022777596, step = 167 (5.965 sec)
INFO:tensorflow:global_step/sec: 1.16868
2020-01-13 09:05:38,606 [INFO] tensorflow: global_step/sec: 1.16868
INFO:tensorflow:global_step/sec: 1.15941
2020-01-13 09:05:39,469 [INFO] tensorflow: global_step/sec: 1.15941
2020-01-13 09:05:39,471 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 21/120: loss: 0.02057 Time taken: 0:00:06.811818 ETA: 0:11:14.369982
INFO:tensorflow:global_step/sec: 1.16772
2020-01-13 09:05:40,325 [INFO] tensorflow: global_step/sec: 1.16772
INFO:tensorflow:global_step/sec: 1.17334
2020-01-13 09:05:41,178 [INFO] tensorflow: global_step/sec: 1.17334
INFO:tensorflow:global_step/sec: 1.16903
2020-01-13 09:05:42,033 [INFO] tensorflow: global_step/sec: 1.16903
INFO:tensorflow:global_step/sec: 1.1758
2020-01-13 09:05:42,884 [INFO] tensorflow: global_step/sec: 1.1758
INFO:tensorflow:global_step/sec: 1.17498
2020-01-13 09:05:43,735 [INFO] tensorflow: global_step/sec: 1.17498
INFO:tensorflow:epoch = 21.75, loss = 0.021212738, step = 174 (5.979 sec)
2020-01-13 09:05:44,584 [INFO] tensorflow: epoch = 21.75, loss = 0.021212738, step = 174 (5.979 sec)
INFO:tensorflow:global_step/sec: 1.17516
2020-01-13 09:05:44,586 [INFO] tensorflow: global_step/sec: 1.17516
2020-01-13 09:05:44,587 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.380
INFO:tensorflow:global_step/sec: 1.1727
2020-01-13 09:05:45,438 [INFO] tensorflow: global_step/sec: 1.1727
INFO:tensorflow:global_step/sec: 1.15644
2020-01-13 09:05:46,303 [INFO] tensorflow: global_step/sec: 1.15644
2020-01-13 09:05:46,305 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 22/120: loss: 0.02000 Time taken: 0:00:06.834812 ETA: 0:11:09.811576
INFO:tensorflow:global_step/sec: 1.16753
2020-01-13 09:05:47,160 [INFO] tensorflow: global_step/sec: 1.16753
INFO:tensorflow:global_step/sec: 1.16631
2020-01-13 09:05:48,017 [INFO] tensorflow: global_step/sec: 1.16631
INFO:tensorflow:global_step/sec: 1.16058
2020-01-13 09:05:48,879 [INFO] tensorflow: global_step/sec: 1.16058
INFO:tensorflow:global_step/sec: 1.17225
2020-01-13 09:05:49,732 [INFO] tensorflow: global_step/sec: 1.17225
INFO:tensorflow:epoch = 22.625, loss = 0.020029472, step = 181 (6.013 sec)
2020-01-13 09:05:50,597 [INFO] tensorflow: epoch = 22.625, loss = 0.020029472, step = 181 (6.013 sec)
INFO:tensorflow:global_step/sec: 1.15332
2020-01-13 09:05:50,599 [INFO] tensorflow: global_step/sec: 1.15332
INFO:tensorflow:global_step/sec: 1.16956
2020-01-13 09:05:51,454 [INFO] tensorflow: global_step/sec: 1.16956
INFO:tensorflow:global_step/sec: 1.17473
2020-01-13 09:05:52,305 [INFO] tensorflow: global_step/sec: 1.17473
INFO:tensorflow:global_step/sec: 1.15433
2020-01-13 09:05:53,171 [INFO] tensorflow: global_step/sec: 1.15433
2020-01-13 09:05:53,173 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 23/120: loss: 0.01813 Time taken: 0:00:06.865815 ETA: 0:11:05.984055
INFO:tensorflow:global_step/sec: 1.16736
2020-01-13 09:05:54,028 [INFO] tensorflow: global_step/sec: 1.16736
INFO:tensorflow:global_step/sec: 1.16585
2020-01-13 09:05:54,886 [INFO] tensorflow: global_step/sec: 1.16585
INFO:tensorflow:global_step/sec: 1.16298
2020-01-13 09:05:55,746 [INFO] tensorflow: global_step/sec: 1.16298
INFO:tensorflow:epoch = 23.5, loss = 0.018176988, step = 188 (6.004 sec)
2020-01-13 09:05:56,602 [INFO] tensorflow: epoch = 23.5, loss = 0.018176988, step = 188 (6.004 sec)
INFO:tensorflow:global_step/sec: 1.16599
2020-01-13 09:05:56,603 [INFO] tensorflow: global_step/sec: 1.16599
INFO:tensorflow:global_step/sec: 1.15859
2020-01-13 09:05:57,466 [INFO] tensorflow: global_step/sec: 1.15859
INFO:tensorflow:global_step/sec: 1.17219
2020-01-13 09:05:58,319 [INFO] tensorflow: global_step/sec: 1.17219
INFO:tensorflow:global_step/sec: 1.16631
2020-01-13 09:05:59,177 [INFO] tensorflow: global_step/sec: 1.16631
INFO:tensorflow:global_step/sec: 1.15975
2020-01-13 09:06:00,039 [INFO] tensorflow: global_step/sec: 1.15975
2020-01-13 09:06:00,041 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 24/120: loss: 0.01762 Time taken: 0:00:06.867600 ETA: 0:10:59.289600
INFO:tensorflow:global_step/sec: 1.15947
2020-01-13 09:06:00,902 [INFO] tensorflow: global_step/sec: 1.15947
INFO:tensorflow:global_step/sec: 1.16405
2020-01-13 09:06:01,761 [INFO] tensorflow: global_step/sec: 1.16405
INFO:tensorflow:epoch = 24.375, loss = 0.016502984, step = 195 (6.019 sec)
2020-01-13 09:06:02,620 [INFO] tensorflow: epoch = 24.375, loss = 0.016502984, step = 195 (6.019 sec)
INFO:tensorflow:global_step/sec: 1.16104
2020-01-13 09:06:02,622 [INFO] tensorflow: global_step/sec: 1.16104
INFO:tensorflow:global_step/sec: 1.16114
2020-01-13 09:06:03,483 [INFO] tensorflow: global_step/sec: 1.16114
INFO:tensorflow:global_step/sec: 1.15141
2020-01-13 09:06:04,352 [INFO] tensorflow: global_step/sec: 1.15141
INFO:tensorflow:global_step/sec: 1.16767
2020-01-13 09:06:05,208 [INFO] tensorflow: global_step/sec: 1.16767
INFO:tensorflow:global_step/sec: 1.16934
2020-01-13 09:06:06,063 [INFO] tensorflow: global_step/sec: 1.16934
INFO:tensorflow:Saving checkpoints for step-200.
2020-01-13 09:06:06,065 [INFO] tensorflow: Saving checkpoints for step-200.
2020-01-13 09:06:06,523 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.235
INFO:tensorflow:global_step/sec: 0.764018
2020-01-13 09:06:07,372 [INFO] tensorflow: global_step/sec: 0.764018
2020-01-13 09:06:07,374 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 25/120: loss: 0.01665 Time taken: 0:00:07.341518 ETA: 0:11:37.444210
INFO:tensorflow:global_step/sec: 1.16167
2020-01-13 09:06:08,233 [INFO] tensorflow: global_step/sec: 1.16167
INFO:tensorflow:epoch = 25.25, loss = 0.016083246, step = 202 (6.474 sec)
2020-01-13 09:06:09,094 [INFO] tensorflow: epoch = 25.25, loss = 0.016083246, step = 202 (6.474 sec)
INFO:tensorflow:global_step/sec: 1.15926
2020-01-13 09:06:09,095 [INFO] tensorflow: global_step/sec: 1.15926
INFO:tensorflow:global_step/sec: 1.15353
2020-01-13 09:06:09,962 [INFO] tensorflow: global_step/sec: 1.15353
INFO:tensorflow:global_step/sec: 1.15666
2020-01-13 09:06:10,827 [INFO] tensorflow: global_step/sec: 1.15666
INFO:tensorflow:global_step/sec: 1.16251
2020-01-13 09:06:11,687 [INFO] tensorflow: global_step/sec: 1.16251
INFO:tensorflow:global_step/sec: 1.15612
2020-01-13 09:06:12,552 [INFO] tensorflow: global_step/sec: 1.15612
INFO:tensorflow:global_step/sec: 1.1619
2020-01-13 09:06:13,413 [INFO] tensorflow: global_step/sec: 1.1619
INFO:tensorflow:global_step/sec: 1.15654
2020-01-13 09:06:14,277 [INFO] tensorflow: global_step/sec: 1.15654
2020-01-13 09:06:14,279 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 26/120: loss: 0.01415 Time taken: 0:00:06.892762 ETA: 0:10:47.919628
INFO:tensorflow:epoch = 26.125, loss = 0.014135221, step = 209 (6.050 sec)
2020-01-13 09:06:15,144 [INFO] tensorflow: epoch = 26.125, loss = 0.014135221, step = 209 (6.050 sec)
INFO:tensorflow:global_step/sec: 1.15156
2020-01-13 09:06:15,146 [INFO] tensorflow: global_step/sec: 1.15156
INFO:tensorflow:global_step/sec: 1.14705
2020-01-13 09:06:16,018 [INFO] tensorflow: global_step/sec: 1.14705
INFO:tensorflow:global_step/sec: 1.15132
2020-01-13 09:06:16,886 [INFO] tensorflow: global_step/sec: 1.15132
INFO:tensorflow:global_step/sec: 1.16414
2020-01-13 09:06:17,745 [INFO] tensorflow: global_step/sec: 1.16414
INFO:tensorflow:global_step/sec: 1.15915
2020-01-13 09:06:18,608 [INFO] tensorflow: global_step/sec: 1.15915
INFO:tensorflow:global_step/sec: 1.14999
2020-01-13 09:06:19,477 [INFO] tensorflow: global_step/sec: 1.14999
INFO:tensorflow:global_step/sec: 1.16229
2020-01-13 09:06:20,338 [INFO] tensorflow: global_step/sec: 1.16229
INFO:tensorflow:epoch = 27.0, loss = 0.013429198, step = 216 (6.059 sec)
2020-01-13 09:06:21,203 [INFO] tensorflow: epoch = 27.0, loss = 0.013429198, step = 216 (6.059 sec)
INFO:tensorflow:global_step/sec: 1.15327
2020-01-13 09:06:21,205 [INFO] tensorflow: global_step/sec: 1.15327
2020-01-13 09:06:21,206 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 27/120: loss: 0.01343 Time taken: 0:00:06.925481 ETA: 0:10:44.069733
INFO:tensorflow:global_step/sec: 1.15305
2020-01-13 09:06:22,072 [INFO] tensorflow: global_step/sec: 1.15305
INFO:tensorflow:global_step/sec: 1.15239
2020-01-13 09:06:22,940 [INFO] tensorflow: global_step/sec: 1.15239
INFO:tensorflow:global_step/sec: 1.1472
2020-01-13 09:06:23,812 [INFO] tensorflow: global_step/sec: 1.1472
INFO:tensorflow:global_step/sec: 1.14875
2020-01-13 09:06:24,682 [INFO] tensorflow: global_step/sec: 1.14875
INFO:tensorflow:global_step/sec: 1.1546
2020-01-13 09:06:25,548 [INFO] tensorflow: global_step/sec: 1.1546
INFO:tensorflow:global_step/sec: 1.16169
2020-01-13 09:06:26,409 [INFO] tensorflow: global_step/sec: 1.16169
INFO:tensorflow:epoch = 27.875, loss = 0.012523004, step = 223 (6.074 sec)
2020-01-13 09:06:27,277 [INFO] tensorflow: epoch = 27.875, loss = 0.012523004, step = 223 (6.074 sec)
INFO:tensorflow:global_step/sec: 1.14991
2020-01-13 09:06:27,279 [INFO] tensorflow: global_step/sec: 1.14991
INFO:tensorflow:global_step/sec: 1.1492
2020-01-13 09:06:28,149 [INFO] tensorflow: global_step/sec: 1.1492
2020-01-13 09:06:28,151 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 28/120: loss: 0.01252 Time taken: 0:00:06.938529 ETA: 0:10:38.344668
2020-01-13 09:06:28,152 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.495
INFO:tensorflow:global_step/sec: 1.14847
2020-01-13 09:06:29,020 [INFO] tensorflow: global_step/sec: 1.14847
INFO:tensorflow:global_step/sec: 1.14988
2020-01-13 09:06:29,889 [INFO] tensorflow: global_step/sec: 1.14988
INFO:tensorflow:global_step/sec: 1.1568
2020-01-13 09:06:30,754 [INFO] tensorflow: global_step/sec: 1.1568
INFO:tensorflow:global_step/sec: 1.15836
2020-01-13 09:06:31,617 [INFO] tensorflow: global_step/sec: 1.15836
INFO:tensorflow:global_step/sec: 1.15347
2020-01-13 09:06:32,484 [INFO] tensorflow: global_step/sec: 1.15347
INFO:tensorflow:epoch = 28.75, loss = 0.013229851, step = 230 (6.076 sec)
2020-01-13 09:06:33,353 [INFO] tensorflow: epoch = 28.75, loss = 0.013229851, step = 230 (6.076 sec)
INFO:tensorflow:global_step/sec: 1.14902
2020-01-13 09:06:33,354 [INFO] tensorflow: global_step/sec: 1.14902
INFO:tensorflow:global_step/sec: 1.15301
2020-01-13 09:06:34,222 [INFO] tensorflow: global_step/sec: 1.15301
INFO:tensorflow:global_step/sec: 1.14673
2020-01-13 09:06:35,094 [INFO] tensorflow: global_step/sec: 1.14673
2020-01-13 09:06:35,095 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 29/120: loss: 0.01129 Time taken: 0:00:06.941255 ETA: 0:10:31.654205
INFO:tensorflow:global_step/sec: 1.14455
2020-01-13 09:06:35,967 [INFO] tensorflow: global_step/sec: 1.14455
INFO:tensorflow:global_step/sec: 1.14991
2020-01-13 09:06:36,837 [INFO] tensorflow: global_step/sec: 1.14991
INFO:tensorflow:global_step/sec: 1.14701
2020-01-13 09:06:37,709 [INFO] tensorflow: global_step/sec: 1.14701
INFO:tensorflow:global_step/sec: 1.14811
2020-01-13 09:06:38,580 [INFO] tensorflow: global_step/sec: 1.14811
INFO:tensorflow:epoch = 29.625, loss = 0.011274138, step = 237 (6.097 sec)
2020-01-13 09:06:39,450 [INFO] tensorflow: epoch = 29.625, loss = 0.011274138, step = 237 (6.097 sec)
INFO:tensorflow:global_step/sec: 1.14733
2020-01-13 09:06:39,451 [INFO] tensorflow: global_step/sec: 1.14733
INFO:tensorflow:global_step/sec: 1.14823
2020-01-13 09:06:40,322 [INFO] tensorflow: global_step/sec: 1.14823
INFO:tensorflow:global_step/sec: 1.14756
2020-01-13 09:06:41,194 [INFO] tensorflow: global_step/sec: 1.14756
INFO:tensorflow:Saving checkpoints for step-240.
2020-01-13 09:06:41,195 [INFO] tensorflow: Saving checkpoints for step-240.
INFO:tensorflow:global_step/sec: 0.742871
2020-01-13 09:06:42,540 [INFO] tensorflow: global_step/sec: 0.742871
2020-01-13 09:06:42,541 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 30/120: loss: 0.01019 Time taken: 0:00:07.457480 ETA: 0:11:11.173200
INFO:tensorflow:global_step/sec: 1.14526
2020-01-13 09:06:43,413 [INFO] tensorflow: global_step/sec: 1.14526
INFO:tensorflow:global_step/sec: 1.14625
2020-01-13 09:06:44,285 [INFO] tensorflow: global_step/sec: 1.14625
INFO:tensorflow:global_step/sec: 1.14388
2020-01-13 09:06:45,160 [INFO] tensorflow: global_step/sec: 1.14388
INFO:tensorflow:epoch = 30.5, loss = 0.011310074, step = 244 (6.583 sec)
2020-01-13 09:06:46,033 [INFO] tensorflow: epoch = 30.5, loss = 0.011310074, step = 244 (6.583 sec)
INFO:tensorflow:global_step/sec: 1.14277
2020-01-13 09:06:46,035 [INFO] tensorflow: global_step/sec: 1.14277
INFO:tensorflow:global_step/sec: 1.14348
2020-01-13 09:06:46,909 [INFO] tensorflow: global_step/sec: 1.14348
INFO:tensorflow:global_step/sec: 1.15073
2020-01-13 09:06:47,778 [INFO] tensorflow: global_step/sec: 1.15073
INFO:tensorflow:global_step/sec: 1.14954
2020-01-13 09:06:48,648 [INFO] tensorflow: global_step/sec: 1.14954
INFO:tensorflow:global_step/sec: 1.14587
2020-01-13 09:06:49,521 [INFO] tensorflow: global_step/sec: 1.14587
2020-01-13 09:06:49,522 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 31/120: loss: 0.01118 Time taken: 0:00:06.971661 ETA: 0:10:20.477829
INFO:tensorflow:global_step/sec: 1.14874
2020-01-13 09:06:50,391 [INFO] tensorflow: global_step/sec: 1.14874
2020-01-13 09:06:50,393 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 17.985
INFO:tensorflow:global_step/sec: 1.14336
2020-01-13 09:06:51,266 [INFO] tensorflow: global_step/sec: 1.14336
INFO:tensorflow:epoch = 31.375, loss = 0.010314425, step = 251 (6.106 sec)
2020-01-13 09:06:52,139 [INFO] tensorflow: epoch = 31.375, loss = 0.010314425, step = 251 (6.106 sec)
INFO:tensorflow:global_step/sec: 1.14384
2020-01-13 09:06:52,140 [INFO] tensorflow: global_step/sec: 1.14384
INFO:tensorflow:global_step/sec: 1.14179
2020-01-13 09:06:53,016 [INFO] tensorflow: global_step/sec: 1.14179
INFO:tensorflow:global_step/sec: 1.15897
2020-01-13 09:06:53,879 [INFO] tensorflow: global_step/sec: 1.15897
INFO:tensorflow:global_step/sec: 1.15259
2020-01-13 09:06:54,747 [INFO] tensorflow: global_step/sec: 1.15259
INFO:tensorflow:global_step/sec: 1.14054
2020-01-13 09:06:55,623 [INFO] tensorflow: global_step/sec: 1.14054
INFO:tensorflow:global_step/sec: 1.13539
2020-01-13 09:06:56,504 [INFO] tensorflow: global_step/sec: 1.13539
2020-01-13 09:06:56,506 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 32/120: loss: 0.00953 Time taken: 0:00:06.970922 ETA: 0:10:13.441136
INFO:tensorflow:global_step/sec: 1.14327
2020-01-13 09:06:57,379 [INFO] tensorflow: global_step/sec: 1.14327
INFO:tensorflow:epoch = 32.25, loss = 0.010005676, step = 258 (6.109 sec)
2020-01-13 09:06:58,248 [INFO] tensorflow: epoch = 32.25, loss = 0.010005676, step = 258 (6.109 sec)
INFO:tensorflow:global_step/sec: 1.14908
2020-01-13 09:06:58,249 [INFO] tensorflow: global_step/sec: 1.14908
INFO:tensorflow:global_step/sec: 1.1475
2020-01-13 09:06:59,120 [INFO] tensorflow: global_step/sec: 1.1475
INFO:tensorflow:global_step/sec: 1.14703
2020-01-13 09:06:59,992 [INFO] tensorflow: global_step/sec: 1.14703
INFO:tensorflow:global_step/sec: 1.14211
2020-01-13 09:07:00,868 [INFO] tensorflow: global_step/sec: 1.14211
INFO:tensorflow:global_step/sec: 1.14662
2020-01-13 09:07:01,740 [INFO] tensorflow: global_step/sec: 1.14662
INFO:tensorflow:global_step/sec: 1.14617
2020-01-13 09:07:02,612 [INFO] tensorflow: global_step/sec: 1.14617
INFO:tensorflow:global_step/sec: 1.13784
2020-01-13 09:07:03,491 [INFO] tensorflow: global_step/sec: 1.13784
2020-01-13 09:07:03,493 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 33/120: loss: 0.00896 Time taken: 0:00:06.990394 ETA: 0:10:08.164278
INFO:tensorflow:epoch = 33.125, loss = 0.012269406, step = 265 (6.115 sec)
2020-01-13 09:07:04,363 [INFO] tensorflow: epoch = 33.125, loss = 0.012269406, step = 265 (6.115 sec)
INFO:tensorflow:global_step/sec: 1.14561
2020-01-13 09:07:04,364 [INFO] tensorflow: global_step/sec: 1.14561
INFO:tensorflow:global_step/sec: 1.14682
2020-01-13 09:07:05,236 [INFO] tensorflow: global_step/sec: 1.14682
INFO:tensorflow:global_step/sec: 1.14104
2020-01-13 09:07:06,113 [INFO] tensorflow: global_step/sec: 1.14104
INFO:tensorflow:global_step/sec: 1.14664
2020-01-13 09:07:06,985 [INFO] tensorflow: global_step/sec: 1.14664
INFO:tensorflow:global_step/sec: 1.14809
2020-01-13 09:07:07,856 [INFO] tensorflow: global_step/sec: 1.14809
INFO:tensorflow:global_step/sec: 1.14492
2020-01-13 09:07:08,729 [INFO] tensorflow: global_step/sec: 1.14492
INFO:tensorflow:global_step/sec: 1.15261
2020-01-13 09:07:09,597 [INFO] tensorflow: global_step/sec: 1.15261
INFO:tensorflow:epoch = 34.0, loss = 0.010143902, step = 272 (6.104 sec)
2020-01-13 09:07:10,466 [INFO] tensorflow: epoch = 34.0, loss = 0.010143902, step = 272 (6.104 sec)
INFO:tensorflow:global_step/sec: 1.14786
2020-01-13 09:07:10,468 [INFO] tensorflow: global_step/sec: 1.14786
2020-01-13 09:07:10,469 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 34/120: loss: 0.01014 Time taken: 0:00:06.979787 ETA: 0:10:00.261682
INFO:tensorflow:global_step/sec: 1.1439
2020-01-13 09:07:11,342 [INFO] tensorflow: global_step/sec: 1.1439
INFO:tensorflow:global_step/sec: 1.1445
2020-01-13 09:07:12,216 [INFO] tensorflow: global_step/sec: 1.1445
2020-01-13 09:07:12,217 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.328
INFO:tensorflow:global_step/sec: 1.14509
2020-01-13 09:07:13,089 [INFO] tensorflow: global_step/sec: 1.14509
INFO:tensorflow:global_step/sec: 1.14465
2020-01-13 09:07:13,963 [INFO] tensorflow: global_step/sec: 1.14465
INFO:tensorflow:global_step/sec: 1.14451
2020-01-13 09:07:14,836 [INFO] tensorflow: global_step/sec: 1.14451
INFO:tensorflow:global_step/sec: 1.14367
2020-01-13 09:07:15,711 [INFO] tensorflow: global_step/sec: 1.14367
INFO:tensorflow:epoch = 34.875, loss = 0.008242929, step = 279 (6.120 sec)
2020-01-13 09:07:16,587 [INFO] tensorflow: epoch = 34.875, loss = 0.008242929, step = 279 (6.120 sec)
INFO:tensorflow:global_step/sec: 1.13967
2020-01-13 09:07:16,588 [INFO] tensorflow: global_step/sec: 1.13967
INFO:tensorflow:Saving checkpoints for step-280.
2020-01-13 09:07:16,590 [INFO] tensorflow: Saving checkpoints for step-280.
INFO:tensorflow:global_step/sec: 0.757119
2020-01-13 09:07:17,909 [INFO] tensorflow: global_step/sec: 0.757119
2020-01-13 09:07:17,911 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 35/120: loss: 0.01004 Time taken: 0:00:07.446958 ETA: 0:10:32.991430
INFO:tensorflow:global_step/sec: 1.15672
2020-01-13 09:07:18,774 [INFO] tensorflow: global_step/sec: 1.15672
INFO:tensorflow:global_step/sec: 1.13438
2020-01-13 09:07:19,655 [INFO] tensorflow: global_step/sec: 1.13438
INFO:tensorflow:global_step/sec: 1.1385
2020-01-13 09:07:20,533 [INFO] tensorflow: global_step/sec: 1.1385
INFO:tensorflow:global_step/sec: 1.14118
2020-01-13 09:07:21,410 [INFO] tensorflow: global_step/sec: 1.14118
INFO:tensorflow:global_step/sec: 1.14438
2020-01-13 09:07:22,284 [INFO] tensorflow: global_step/sec: 1.14438
INFO:tensorflow:epoch = 35.75, loss = 0.008048964, step = 286 (6.572 sec)
2020-01-13 09:07:23,159 [INFO] tensorflow: epoch = 35.75, loss = 0.008048964, step = 286 (6.572 sec)
INFO:tensorflow:global_step/sec: 1.14069
2020-01-13 09:07:23,160 [INFO] tensorflow: global_step/sec: 1.14069
INFO:tensorflow:global_step/sec: 1.14209
2020-01-13 09:07:24,036 [INFO] tensorflow: global_step/sec: 1.14209
INFO:tensorflow:global_step/sec: 1.13879
2020-01-13 09:07:24,914 [INFO] tensorflow: global_step/sec: 1.13879
2020-01-13 09:07:24,916 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 36/120: loss: 0.00855 Time taken: 0:00:06.992730 ETA: 0:09:47.389320
INFO:tensorflow:global_step/sec: 1.14148
2020-01-13 09:07:25,790 [INFO] tensorflow: global_step/sec: 1.14148
INFO:tensorflow:global_step/sec: 1.14297
2020-01-13 09:07:26,665 [INFO] tensorflow: global_step/sec: 1.14297
INFO:tensorflow:global_step/sec: 1.13741
2020-01-13 09:07:27,544 [INFO] tensorflow: global_step/sec: 1.13741
INFO:tensorflow:global_step/sec: 1.14353
2020-01-13 09:07:28,419 [INFO] tensorflow: global_step/sec: 1.14353
INFO:tensorflow:epoch = 36.625, loss = 0.008221224, step = 293 (6.136 sec)
2020-01-13 09:07:29,295 [INFO] tensorflow: epoch = 36.625, loss = 0.008221224, step = 293 (6.136 sec)
INFO:tensorflow:global_step/sec: 1.13985
2020-01-13 09:07:29,296 [INFO] tensorflow: global_step/sec: 1.13985
INFO:tensorflow:global_step/sec: 1.14709
2020-01-13 09:07:30,168 [INFO] tensorflow: global_step/sec: 1.14709
INFO:tensorflow:global_step/sec: 1.14577
2020-01-13 09:07:31,041 [INFO] tensorflow: global_step/sec: 1.14577
INFO:tensorflow:global_step/sec: 1.13715
2020-01-13 09:07:31,920 [INFO] tensorflow: global_step/sec: 1.13715
2020-01-13 09:07:31,921 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 37/120: loss: 0.00899 Time taken: 0:00:07.001909 ETA: 0:09:41.158447
INFO:tensorflow:global_step/sec: 1.15095
2020-01-13 09:07:32,789 [INFO] tensorflow: global_step/sec: 1.15095
INFO:tensorflow:global_step/sec: 1.14333
2020-01-13 09:07:33,663 [INFO] tensorflow: global_step/sec: 1.14333
INFO:tensorflow:global_step/sec: 1.1424
2020-01-13 09:07:34,539 [INFO] tensorflow: global_step/sec: 1.1424
2020-01-13 09:07:34,540 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 17.919
INFO:tensorflow:epoch = 37.5, loss = 0.008026822, step = 300 (6.118 sec)
2020-01-13 09:07:35,413 [INFO] tensorflow: epoch = 37.5, loss = 0.008026822, step = 300 (6.118 sec)
INFO:tensorflow:global_step/sec: 1.1421
2020-01-13 09:07:35,414 [INFO] tensorflow: global_step/sec: 1.1421
INFO:tensorflow:global_step/sec: 1.13526
2020-01-13 09:07:36,295 [INFO] tensorflow: global_step/sec: 1.13526
INFO:tensorflow:global_step/sec: 1.14348
2020-01-13 09:07:37,170 [INFO] tensorflow: global_step/sec: 1.14348
INFO:tensorflow:global_step/sec: 1.14093
2020-01-13 09:07:38,046 [INFO] tensorflow: global_step/sec: 1.14093
INFO:tensorflow:global_step/sec: 1.12846
2020-01-13 09:07:38,932 [INFO] tensorflow: global_step/sec: 1.12846
2020-01-13 09:07:38,934 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 38/120: loss: 0.00701 Time taken: 0:00:07.007744 ETA: 0:09:34.635008
INFO:tensorflow:global_step/sec: 1.14285
2020-01-13 09:07:39,807 [INFO] tensorflow: global_step/sec: 1.14285
INFO:tensorflow:global_step/sec: 1.14732
2020-01-13 09:07:40,679 [INFO] tensorflow: global_step/sec: 1.14732
INFO:tensorflow:epoch = 38.375, loss = 0.0071663177, step = 307 (6.144 sec)
2020-01-13 09:07:41,557 [INFO] tensorflow: epoch = 38.375, loss = 0.0071663177, step = 307 (6.144 sec)
INFO:tensorflow:global_step/sec: 1.13685
2020-01-13 09:07:41,558 [INFO] tensorflow: global_step/sec: 1.13685
INFO:tensorflow:global_step/sec: 1.14295
2020-01-13 09:07:42,433 [INFO] tensorflow: global_step/sec: 1.14295
INFO:tensorflow:global_step/sec: 1.14265
2020-01-13 09:07:43,309 [INFO] tensorflow: global_step/sec: 1.14265
INFO:tensorflow:global_step/sec: 1.14257
2020-01-13 09:07:44,184 [INFO] tensorflow: global_step/sec: 1.14257
INFO:tensorflow:global_step/sec: 1.13315
2020-01-13 09:07:45,066 [INFO] tensorflow: global_step/sec: 1.13315
INFO:tensorflow:global_step/sec: 1.13501
2020-01-13 09:07:45,947 [INFO] tensorflow: global_step/sec: 1.13501
2020-01-13 09:07:45,949 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 39/120: loss: 0.00708 Time taken: 0:00:07.016607 ETA: 0:09:28.345167
INFO:tensorflow:global_step/sec: 1.13619
2020-01-13 09:07:46,828 [INFO] tensorflow: global_step/sec: 1.13619
INFO:tensorflow:epoch = 39.25, loss = 0.007587464, step = 314 (6.147 sec)
2020-01-13 09:07:47,704 [INFO] tensorflow: epoch = 39.25, loss = 0.007587464, step = 314 (6.147 sec)
INFO:tensorflow:global_step/sec: 1.13821
2020-01-13 09:07:47,706 [INFO] tensorflow: global_step/sec: 1.13821
INFO:tensorflow:global_step/sec: 1.13992
2020-01-13 09:07:48,583 [INFO] tensorflow: global_step/sec: 1.13992
INFO:tensorflow:global_step/sec: 1.13548
2020-01-13 09:07:49,464 [INFO] tensorflow: global_step/sec: 1.13548
INFO:tensorflow:global_step/sec: 1.14765
2020-01-13 09:07:50,335 [INFO] tensorflow: global_step/sec: 1.14765
INFO:tensorflow:global_step/sec: 1.1434
2020-01-13 09:07:51,210 [INFO] tensorflow: global_step/sec: 1.1434
INFO:tensorflow:global_step/sec: 1.14241
2020-01-13 09:07:52,085 [INFO] tensorflow: global_step/sec: 1.14241
INFO:tensorflow:Saving checkpoints for step-320.
2020-01-13 09:07:52,087 [INFO] tensorflow: Saving checkpoints for step-320.
INFO:tensorflow:global_step/sec: 0.734195
2020-01-13 09:07:53,447 [INFO] tensorflow: global_step/sec: 0.734195
2020-01-13 09:07:53,448 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 40/120: loss: 0.00644 Time taken: 0:00:07.517500 ETA: 0:10:01.400000
INFO:tensorflow:epoch = 40.125, loss = 0.006629837, step = 321 (6.609 sec)
2020-01-13 09:07:54,313 [INFO] tensorflow: epoch = 40.125, loss = 0.006629837, step = 321 (6.609 sec)
INFO:tensorflow:global_step/sec: 1.15255
2020-01-13 09:07:54,315 [INFO] tensorflow: global_step/sec: 1.15255
INFO:tensorflow:global_step/sec: 1.14249
2020-01-13 09:07:55,190 [INFO] tensorflow: global_step/sec: 1.14249
INFO:tensorflow:global_step/sec: 1.13538
2020-01-13 09:07:56,071 [INFO] tensorflow: global_step/sec: 1.13538
INFO:tensorflow:global_step/sec: 1.13793
2020-01-13 09:07:56,950 [INFO] tensorflow: global_step/sec: 1.13793
2020-01-13 09:07:56,951 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 17.849
INFO:tensorflow:global_step/sec: 1.14276
2020-01-13 09:07:57,825 [INFO] tensorflow: global_step/sec: 1.14276
INFO:tensorflow:global_step/sec: 1.13823
2020-01-13 09:07:58,703 [INFO] tensorflow: global_step/sec: 1.13823
INFO:tensorflow:global_step/sec: 1.14011
2020-01-13 09:07:59,581 [INFO] tensorflow: global_step/sec: 1.14011
INFO:tensorflow:epoch = 41.0, loss = 0.007186246, step = 328 (6.146 sec)
2020-01-13 09:08:00,459 [INFO] tensorflow: epoch = 41.0, loss = 0.007186246, step = 328 (6.146 sec)
INFO:tensorflow:global_step/sec: 1.13612
2020-01-13 09:08:00,461 [INFO] tensorflow: global_step/sec: 1.13612
2020-01-13 09:08:00,462 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 41/120: loss: 0.00719 Time taken: 0:00:06.997418 ETA: 0:09:12.796022
INFO:tensorflow:global_step/sec: 1.14459
2020-01-13 09:08:01,334 [INFO] tensorflow: global_step/sec: 1.14459
INFO:tensorflow:global_step/sec: 1.14661
2020-01-13 09:08:02,207 [INFO] tensorflow: global_step/sec: 1.14661
INFO:tensorflow:global_step/sec: 1.14291
2020-01-13 09:08:03,082 [INFO] tensorflow: global_step/sec: 1.14291
INFO:tensorflow:global_step/sec: 1.13793
2020-01-13 09:08:03,960 [INFO] tensorflow: global_step/sec: 1.13793
INFO:tensorflow:global_step/sec: 1.14532
2020-01-13 09:08:04,833 [INFO] tensorflow: global_step/sec: 1.14532
INFO:tensorflow:global_step/sec: 1.13839
2020-01-13 09:08:05,712 [INFO] tensorflow: global_step/sec: 1.13839
INFO:tensorflow:epoch = 41.875, loss = 0.005918531, step = 335 (6.132 sec)
2020-01-13 09:08:06,591 [INFO] tensorflow: epoch = 41.875, loss = 0.005918531, step = 335 (6.132 sec)
INFO:tensorflow:global_step/sec: 1.13543
2020-01-13 09:08:06,593 [INFO] tensorflow: global_step/sec: 1.13543
INFO:tensorflow:global_step/sec: 1.14001
2020-01-13 09:08:07,470 [INFO] tensorflow: global_step/sec: 1.14001
2020-01-13 09:08:07,471 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 42/120: loss: 0.00602 Time taken: 0:00:07.005010 ETA: 0:09:06.390780
INFO:tensorflow:global_step/sec: 1.14369
2020-01-13 09:08:08,344 [INFO] tensorflow: global_step/sec: 1.14369
INFO:tensorflow:global_step/sec: 1.13675
2020-01-13 09:08:09,224 [INFO] tensorflow: global_step/sec: 1.13675
INFO:tensorflow:global_step/sec: 1.13864
2020-01-13 09:08:10,102 [INFO] tensorflow: global_step/sec: 1.13864
INFO:tensorflow:global_step/sec: 1.14223
2020-01-13 09:08:10,978 [INFO] tensorflow: global_step/sec: 1.14223
INFO:tensorflow:global_step/sec: 1.13427
2020-01-13 09:08:11,859 [INFO] tensorflow: global_step/sec: 1.13427
INFO:tensorflow:epoch = 42.75, loss = 0.0057198703, step = 342 (6.149 sec)
2020-01-13 09:08:12,740 [INFO] tensorflow: epoch = 42.75, loss = 0.0057198703, step = 342 (6.149 sec)
INFO:tensorflow:global_step/sec: 1.13338
2020-01-13 09:08:12,742 [INFO] tensorflow: global_step/sec: 1.13338
INFO:tensorflow:global_step/sec: 1.14358
2020-01-13 09:08:13,616 [INFO] tensorflow: global_step/sec: 1.14358
INFO:tensorflow:global_step/sec: 1.13482
2020-01-13 09:08:14,497 [INFO] tensorflow: global_step/sec: 1.13482
2020-01-13 09:08:14,499 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 43/120: loss: 0.00610 Time taken: 0:00:07.023872 ETA: 0:09:00.838144
INFO:tensorflow:global_step/sec: 1.14127
2020-01-13 09:08:15,373 [INFO] tensorflow: global_step/sec: 1.14127
INFO:tensorflow:global_step/sec: 1.13718
2020-01-13 09:08:16,253 [INFO] tensorflow: global_step/sec: 1.13718
INFO:tensorflow:global_step/sec: 1.14298
2020-01-13 09:08:17,128 [INFO] tensorflow: global_step/sec: 1.14298
INFO:tensorflow:global_step/sec: 1.14212
2020-01-13 09:08:18,003 [INFO] tensorflow: global_step/sec: 1.14212
INFO:tensorflow:epoch = 43.625, loss = 0.005818277, step = 349 (6.144 sec)
2020-01-13 09:08:18,884 [INFO] tensorflow: epoch = 43.625, loss = 0.005818277, step = 349 (6.144 sec)
INFO:tensorflow:global_step/sec: 1.13321
2020-01-13 09:08:18,886 [INFO] tensorflow: global_step/sec: 1.13321
2020-01-13 09:08:18,887 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 18.235
INFO:tensorflow:global_step/sec: 1.14762
2020-01-13 09:08:19,757 [INFO] tensorflow: global_step/sec: 1.14762
INFO:tensorflow:global_step/sec: 1.14771
2020-01-13 09:08:20,628 [INFO] tensorflow: global_step/sec: 1.14771
INFO:tensorflow:global_step/sec: 1.13819
2020-01-13 09:08:21,507 [INFO] tensorflow: global_step/sec: 1.13819
2020-01-13 09:08:21,508 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 44/120: loss: 0.00621 Time taken: 0:00:07.013538 ETA: 0:08:53.028888
INFO:tensorflow:global_step/sec: 1.14067
2020-01-13 09:08:22,384 [INFO] tensorflow: global_step/sec: 1.14067
INFO:tensorflow:global_step/sec: 1.14117
2020-01-13 09:08:23,260 [INFO] tensorflow: global_step/sec: 1.14117
INFO:tensorflow:global_step/sec: 1.14585
2020-01-13 09:08:24,133 [INFO] tensorflow: global_step/sec: 1.14585
INFO:tensorflow:epoch = 44.5, loss = 0.0053089373, step = 356 (6.126 sec)
2020-01-13 09:08:25,011 [INFO] tensorflow: epoch = 44.5, loss = 0.0053089373, step = 356 (6.126 sec)
INFO:tensorflow:global_step/sec: 1.13703
2020-01-13 09:08:25,012 [INFO] tensorflow: global_step/sec: 1.13703
INFO:tensorflow:global_step/sec: 1.14844
2020-01-13 09:08:25,883 [INFO] tensorflow: global_step/sec: 1.14844
INFO:tensorflow:global_step/sec: 1.14762
2020-01-13 09:08:26,754 [INFO] tensorflow: global_step/sec: 1.14762
INFO:tensorflow:global_step/sec: 1.14598
2020-01-13 09:08:27,627 [INFO] tensorflow: global_step/sec: 1.14598
INFO:tensorflow:Saving checkpoints for step-360.
2020-01-13 09:08:27,628 [INFO] tensorflow: Saving checkpoints for step-360.
2020-01-13 09:08:28,091 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 1, 0.00s/step
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/evaluation/metadata.py:38: UserWarning: One or more metadata field(s) are missing from ground_truth batch_data, and will be replaced with defaults: [‘frame/camera_location’]
Matching predictions to ground truth, class 1/2.: 100%|##################################| 1/1 [00:00<00:00, 464.69it/s]
Matching predictions to ground truth, class 2/2.: 100%|##############################| 78/78 [00:00<00:00, 76029.68it/s]
/usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/evaluation/compute_metrics.py:717: RuntimeWarning: invalid value encountered in true_divide
Epoch 45/120

Validation cost: 0.018751
Mean average_precision (in %): nan

class name average precision (in %)


person nan
vehicle 0

Median Inference Time: 0.374451
INFO:tensorflow:epoch = 45.0, loss = 0.0052545117, step = 360 (22.807 sec)
2020-01-13 09:08:47,817 [INFO] tensorflow: epoch = 45.0, loss = 0.0052545117, step = 360 (22.807 sec)
INFO:tensorflow:global_step/sec: 0.0495251
2020-01-13 09:08:47,819 [INFO] tensorflow: global_step/sec: 0.0495251
2020-01-13 09:08:47,820 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 45/120: loss: 0.00525 Time taken: 0:00:26.299269 ETA: 0:32:52.445175
INFO:tensorflow:global_step/sec: 1.17189
2020-01-13 09:08:48,672 [INFO] tensorflow: global_step/sec: 1.17189
INFO:tensorflow:global_step/sec: 1.16685
2020-01-13 09:08:49,529 [INFO] tensorflow: global_step/sec: 1.16685
INFO:tensorflow:global_step/sec: 1.17651
2020-01-13 09:08:50,379 [INFO] tensorflow: global_step/sec: 1.17651
INFO:tensorflow:global_step/sec: 1.16843
2020-01-13 09:08:51,235 [INFO] tensorflow: global_step/sec: 1.16843
INFO:tensorflow:global_step/sec: 1.16548
2020-01-13 09:08:52,093 [INFO] tensorflow: global_step/sec: 1.16548
INFO:tensorflow:global_step/sec: 1.17084
2020-01-13 09:08:52,947 [INFO] tensorflow: global_step/sec: 1.17084
INFO:tensorflow:epoch = 45.875, loss = 0.005457866, step = 367 (5.983 sec)
2020-01-13 09:08:53,801 [INFO] tensorflow: epoch = 45.875, loss = 0.005457866, step = 367 (5.983 sec)
INFO:tensorflow:global_step/sec: 1.16914
2020-01-13 09:08:53,802 [INFO] tensorflow: global_step/sec: 1.16914
INFO:tensorflow:global_step/sec: 1.15881
2020-01-13 09:08:54,665 [INFO] tensorflow: global_step/sec: 1.15881
2020-01-13 09:08:54,667 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 46/120: loss: 0.00542 Time taken: 0:00:06.869998 ETA: 0:08:28.379852
INFO:tensorflow:global_step/sec: 1.16422
2020-01-13 09:08:55,524 [INFO] tensorflow: global_step/sec: 1.16422
INFO:tensorflow:global_step/sec: 1.17172
2020-01-13 09:08:56,378 [INFO] tensorflow: global_step/sec: 1.17172
INFO:tensorflow:global_step/sec: 1.158
2020-01-13 09:08:57,241 [INFO] tensorflow: global_step/sec: 1.158
INFO:tensorflow:global_step/sec: 1.16647
2020-01-13 09:08:58,098 [INFO] tensorflow: global_step/sec: 1.16647
INFO:tensorflow:global_step/sec: 1.17048
2020-01-13 09:08:58,953 [INFO] tensorflow: global_step/sec: 1.17048
INFO:tensorflow:epoch = 46.75, loss = 0.0049010357, step = 374 (6.015 sec)
2020-01-13 09:08:59,815 [INFO] tensorflow: epoch = 46.75, loss = 0.0049010357, step = 374 (6.015 sec)
INFO:tensorflow:global_step/sec: 1.15818
2020-01-13 09:08:59,816 [INFO] tensorflow: global_step/sec: 1.15818
2020-01-13 09:08:59,817 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/sample_counter_hook.pyc: Samples / sec: 9.773
INFO:tensorflow:global_step/sec: 1.16719
2020-01-13 09:09:00,673 [INFO] tensorflow: global_step/sec: 1.16719
INFO:tensorflow:global_step/sec: 1.15324
2020-01-13 09:09:01,540 [INFO] tensorflow: global_step/sec: 1.15324
2020-01-13 09:09:01,542 [INFO] /usr/local/lib/python2.7/dist-packages/iva/detectnet_v2/tfhooks/task_progress_monitor_hook.pyc: Epoch 47/120: loss: 0.00551 Time taken: 0:00:06.870875 ETA: 0:08:21.573875
INFO:tensorflow:global_step/sec: 1.16139
2020-01-13 09:09:02,401 [INFO] tensorflow: global_step/sec: 1.16139
INFO:tensorflow:global_step/sec: 1.16233
2020-01-13 09:09:03,261 [INFO] tensorflow: global_step/sec: 1.16233
INFO:tensorflow:global_step/sec: 1.16739
2020-01-13 09:09:04,118 [INFO] tensorflow: global_step/sec: 1.16739
INFO:tensorflow:glob

Hello pangqingshuang, could you please attach your training specification file?

Hi pangqingshuang,
Could you please also attach the full log along with the training spec? Please attach here via a pdf or txt format file.

Sorry I closed the terminal after training. So I didnt have full log

Hi pangqingshuang,
Never mind, could you please paste your training config file? Thanks.

More, could you please trigger Jupyter notebook and follow detectnet_v2 sample to have a try run?
It will tell you how to generate tfrecords based on your own data, how to train with default spec, etc.

Hi pangqingshuang,

We haven’t heard back from you in a couple weeks, so marking this topic closed.
Please open a new forum issue when you are ready and we’ll pick it up there.